Parallel computing problem

[federico]

Hi everyone,
I tried to run an estimation using the parallel option using smets and wouters 2003 model.
I download PSTools and I should have add it into the window path.

When I run the code dynare give these warnings (Of course after loadings the data and computing the posterior mode)

Warning: File '\\localhost\$\\\slaveParallel_break.mat' not found.
> In dynareParallelDelete at 51
In masterParallel at 184
In random_walk_metropolis_hastings at 148
In dynare_estimation_1 at 931
In dynare_estimation at 70
In SW2003_parall at 375
In dynare at 120

C:\Users\federico\Desktop\dynare simulation\dynare simulation\SW2003>psexec -d -W C:\Users\federico\Desktop\dynare simulation\dynare simulation\SW2003 -a 0 -low C:\Program Files\MATLAB\R2011a\bin\matlab -nosplash -nodesktop -minimize -singleCompThread -r "addpath('C:\dynare\2013-06-17\matlab'), dynareroot = dynare_config(); fParallel(1,1,1,1,'random_walk_metropolis_hastings_core')"
"psexec" non Š riconosciuto come comando interno o esterno,
un programma eseguibile o un file batch.

Traslation of the last two lines ("psexec" is not recognize as an internal or external command, as an executable program or a batch file).

C:\Users\federico\Desktop\dynare simulation\dynare simulation\SW2003>psexec -d -W C:\Users\federico\Desktop\dynare simulation\dynare simulation\SW2003 -a 1 -low C:\Program Files\MATLAB\R2011a\bin\matlab -nosplash -nodesktop -minimize -singleCompThread -r "addpath('C:\dynare\2013-06-17\matlab'), dynareroot = dynare_config(); fParallel(2,2,2,1,'random_walk_metropolis_hastings_core')"
"psexec" non Š riconosciuto come comando interno o esterno,
un programma eseguibile o un file batch.

Besides , the waitbar of the mcmc pop up on the screen with the status of initialize but the computation of the mcmc does not start.

Any suggestion?

[federico]

I fix part of the problem.
Now I'm quite sure that PSTool is on the windows path but still the MCMC does not start

Launching the mod file I obtained this output


Warning: File '\\localhost\$\\\slaveParallel_break.mat' not found.
> In dynareParallelDelete at 51
In masterParallel at 184
In random_walk_metropolis_hastings at 148
In dynare_estimation_1 at 931
In dynare_estimation at 70
In SW2003_parall at 385
In dynare at 120

C:\Users\federico\Desktop\parallel>psexec -d -W C:\Users\federico\Desktop\parallel -a 1 -low -nosplash -nodesktop -minimize -r "addpath(''), dynareroot = dynare_config(); fParallel(1,1,1,1,'random_walk_metropolis_hastings_core')"

PsExec v1.98 - Execute processes remotely
Copyright (C) 2001-2010 Mark Russinovich
Sysinternals - www.sysinternals.com

PsExec executes a program on a remote system, where remotely executed console
applications execute interactively.

Usage: psexec [\\computer[,computer2[,...] | @file]][-u user [-p psswd][-n s][-l][-s|-e][-x][-i [session]][-c [-f|-v]][-w directory][-d][-<priority>][-a n,n,...] cmd [arguments]
-a Separate processors on which the application can run with
commas where 1 is the lowest numbered CPU. For example,
to run the application on CPU 2 and CPU 4, enter:
"-a 2,4"
-c Copy the specified program to the remote system for
execution. If you omit this option the application
must be in the system path on the remote system.
-d Don't wait for process to terminate (non-interactive).
-e Does not load the specified account's profile.
-f Copy the specified program even if the file already
exists on the remote system.
-i Run the program so that it interacts with the desktop of the
specified session on the remote system. If no session is
specified the process runs in the console session.
-h If the target system is Vista or higher, has the process
run with the account's elevated token, if available.
-l Run process as limited user (strips the Administrators group
and allows only privileges assigned to the Users group).
On Windows Vista the process runs with Low Integrity.
-n Specifies timeout in seconds connecting to remote computers.
-p Specifies optional password for user name. If you omit this
you will be prompted to enter a hidden password.
-s Run the remote process in the System account.
-u Specifies optional user name for login to remote
computer.
-v Copy the specified file only if it has a higher version number
or is newer on than the one on the remote system.
-w Set the working directory of the process (relative to
remote computer).
-x Display the UI on the Winlogon secure desktop (local system
only).
-priority Specifies -low, -belownormal, -abovenormal, -high or
-realtime to run the process at a different priority. Use
-background to run at low memory and I/O priority on Vista.
computer Direct PsExec to run the application on the remote
computer or computers specified. If you omit the computer
name PsExec runs the application on the local system,
and if you specify a wildcard (\\*), PsExec runs the
command on all computers in the current domain.
@file PsExec will execute the command on each of the computers listed
in the file.
program Name of application to execute.
arguments Arguments to pass (note that file paths must be
absolute paths on the target system).

You can enclose applications that have spaces in their name with
quotation marks e.g. psexec \\marklap "c:\long name app.exe".
Input is only passed to the remote system when you press the enter
key, and typing Ctrl-C terminates the remote process.

If you omit a user name the process will run in the context of your
account on the remote system, but will not have access to network
resources (because it is impersonating). Specify a valid user name
in the Domain\User syntax if the remote process requires access
to network resources or to run in a different account. Note that
the password is transmitted in clear text to the remote system.

Error codes returned by PsExec are specific to the applications you
execute, not PsExec.


As far I can understand PsExec is in the windows path and the program works but still no MCMC?
Suggestion?

Federico

[rattoma]

The first time you run psexec, there is a windows which pops up asking you to accept the license agreement [you have to accept it only once]. Perhaps this has never been done?

[federico]

Sorry, but in the meantime I was able to fix the problem by myself.
Now the parallel computation work fine at least on the local machine.
There was a mistake when I added PSTools to the windows path.

Thanx anyway for your answer.

Federico

[TaniaRA]

Hello Federico,

I am having the same problem with psexec. Could you please share how you solve the issue?

Please find below the error message I got.

Z:\Office Research\Economists Workplace\Tania\baseline2_est_140711\baseline2_code>psexec -d -W Z:\Office Research\Economists Workplace\Tania\baseline2_est_140711\baseline2_code -a 0 -low matlab -nosplash -nodesktop -minimize -singleCompThread -r "addpath('J:\MyDocuments\Dynare\4.4.2\matlab\'), dynareroot = dynare_config(); slaveParallel(1,1)"
'psexec' is not recognized as an internal or external command,
operable program or batch file.

Best,
Tania

[federico]

Hi Tania,
I have to be honest, right now i cannot remenber how i solved the problem one year ago.
I have to go check the last year work. I promise i will keep you update

[jduras]

I had the same problem - waiting bar would show up with 'Initialize ...' status but MCMC would not start. In my case the issue seemed to be that the name of one of the folders in the path where the mod file was located contained a space in its name. After I renamed it everything worked fine.

Jan

[rattoma]

Hi, it seems you need to add psexec in the path of the windows machine (NOT of matlab).
You need to edit the environment variable PATH of windows.
cheers
M

[wjgatt]

I have run into the same problem, MH dialog shows 'Initialize' and gets stuck there.

The PStools path is specified in the path of the windows machine, and I have manually opened all PStools .exe files to 'Accept' the terms and conditions.

There are also no spaces in the path containing the mod file or the data file.

Any idea what can be done please?

Many thanks
W

[rattoma]

apologies for having overlooked this thread. if you did not solve the issue yet, could you please provide some more info on the problem, and on which operating system are you getting the error.

[wjgatt]

I am working on a Core i5 quad core running Windows 10. The configuration file is in C:\dynare\4.4.3 (attached). When I run parallel_test I get the following:

Code: Select all


Testing computer -> localhost <- ...


Check on Local Variable ..... Ok!

Checking Hardware please wait ...
Hardware has 4 Cpu/Cores!
User requires 4 Cpu/Cores!
Check on CPUnbr Variable ..... Ok!



Test for Cluster computation, computer localhost ..... Passed!


AnalyseComputationalEnvironment returned with Error Code: 0


But when I try to have my code estimate my model (which works on a normal, sequential process) I get this:

Code: Select all

 Warning: File '\\localhost\$\slaveParallel_break.mat' not found.
> In dynareParallelDelete at 51
   [
    ...
         ]
  In dynare at 180
 
C:\Users\William\Desktop\Estimation>psexec -d -W C:\Users\William\Desktop\Estimation -a 0 -low  C:\Program Files\MATLAB\R2014b -nosplash -nodesktop -minimize -singleCompThread -r "addpath('C:\dynare\4.4.3\matlab'), dynareroot = dynare_config(); fParallel(1,1,1,1,'random_walk_metropolis_hastings_core')" 
 
PsExec v2.11 - Execute processes remotely
Copyright (C) 2001-2014 Mark Russinovich
Sysinternals - www.sysinternals.com
 
The system cannot find the file specified.

PsExec could not start C:\Program: 

I get the above errors 4 times (I suppose for each processor I am trying to access).

As I said in an earlier thread, I had opened the PSexec files to accept the user license agreement.

Any help would be really appreciated.

Thank you
William

[jpfeifer]

You can ignore the warning. The problem seems to be due to the PSTools not being in the Windows path. Please have a look at the manual to the current unstable version. It has a step by step guide for Windows parallel.

[wjgatt]

Thanks for your reply. I followed the steps, and when I type !psexec in Matlab at step 6, I get confirmation that the path is set:

Code: Select all

PsExec v2.11 - Execute processes remotely
Copyright (C) 2001-2014 Mark Russinovich
Sysinternals - www.sysinternals.com
 
PsExec executes a program on a remote system, where remotely executed console
applications execute interactively.
 
Usage: psexec [\\computer[,computer2[,...] | @file]][-u user [-p psswd][-n s][-r servicename][-h][-l][-s|-e][-x][-i [session]][-c [-f|-v]][-w directory][-d][-<priority>][-a n,n,...] cmd [arguments]
     -a         Separate processors on which the application can run with
                commas where 1 is the lowest numbered CPU. For example,
                to run the application on CPU 2 and CPU 4, enter:
                "-a 2,4"
etc...


but when I try to run the estimation I get the same problem,

Code: Select all

C:\Users\William\Desktop\Estimation>psexec -d -W C:\Users\William\Desktop\Estimation -a 0 -low  C:\Program Files\MATLAB\R2014b -nosplash -nodesktop -minimize -singleCompThread -r "addpath('C:\dynare\4.4.3\matlab'), dynareroot = dynare_config(); fParallel(1,1,1,1,'random_walk_metropolis_hastings_core')" 
 
PsExec v2.11 - Execute processes remotely
Copyright (C) 2001-2014 Mark Russinovich
Sysinternals - www.sysinternals.com
 
The system cannot find the file specified.

PsExec could not start C:\Program:


No idea what's going on! I've moved the conffile to the C:\, and I run

Code: Select all

dynare full_model_dec2015.mod parallel conffile='C:\dynare.ini'


And that's when I get the error, the MH dialog pops up, showing 4 bars and 'Initialize', and nothing else happens except for the warning.

[jpfeifer]

Are you sure that

Code: Select all

MatlabOctavePath = C:\Program Files\MATLAB\R2014b


is correct? Usually, it should be something like

Code: Select all

MatlabOctavePath = C:\Progra~1\MATLAB\R2014b\bin\matlab


or simply

Code: Select all

MatlabOctavePath = matlab


if Matlab is your path.

[wjgatt]

That solved my problem, thank you very much!