Differences between revisions 29 and 30
Revision 29 as of 2010-10-25 13:34:05
Size: 12584
Comment:
Revision 30 as of 2010-10-25 13:54:24
Size: 12659
Comment:
Deletions are marked like this. Additions are marked like this.
Line 162: Line 162:
  * {{{parallel}}}: trigger the parallel computation (using the first cluster specified in config file)   * {{{parallel}}}: trigger the parallel computation using the first cluster specified in config file. If no cluster is specified, use all slave nodes present in the config file.

This page documents the parallelization system developped by Marco Ratto for Dynare.

The idea is to provide a general framework in Dynare for parallelizing tasks which require very little inter-process communication.

The implementation is done by running several MATLAB or Octave processes, either on local or on remote machines. Communication between master and slave processes are done through SMB on Windows and SSH on UNIX. Input and output data, and also some short status messages, are exchanged through network filesystems.

Currently the system works only with homogenous grids: only Windows or only Unix machines.

Routines currently parallelized:

  • the Metropolis-Hastings algorithm (implemented in random_walk_metropolis_hastings.m)

  • the independent Metropolis-Hastings algorithm (implemented in independent_metropolis_hastings.m)

  • the Metropolis-Hastings diagnostics (implemented in McMCDiagnostics.m)

  • pm3.m (plotting routine)

  • Posterior_IRF.m

  • prior_posterior_statistics.m

1. Requirements

1.1. For a Windows grid

  1. a standard Windows network (SMB) must be in place
  2. PsTools must be installed in the path of the master Windows machine

1.2. For a UNIX grid

  1. MATLAB executable must be in the path of the slave machines
  2. SSH must be installed on the master and on the slave machines
  3. SSH keys must be installed so that the SSH connections from the slaves to the master can be done without passwords, or using an SSH agent (see SshKeysHowto)

2. Usage

The parallelization mechanism is triggered by the use of options_.parallel. By default, this option is equal to zero, no parallelization is used.

To trigger the parallelization, this option must be filled with a vector of structures. Each structure represents a slave machine (possibly using several CPU cores on the machine).

The fields are:

  • Local: equal to 0 or 1. Use 1 if this slave is the local machine, 0 if it is a remote machine
  • PcName: for a remote slave, name of the machine. Use the NETBIOS name under Windows, or the DNS name under Unix

  • NumCPU: a vector of integers representing the CPU cores to be used on that slave machine. The first core has number 0. So, on a quadcore, use [0:3] here to use the four cores
  • user: for a remote slave, username to be used. On Windows, the group needs also to be specified here, like DEPT\JohnSmith, i.e. user JohnSmith in windows group DEPT

  • passwd: for a remote slave, password associated to the username
  • RemoteDrive: for a remote Windows slave, letter of the remote drive (C, D, ...) where the computations will take place

  • RemoteFolder: for a remote slave, path of the directory on the remote drive where the computations will take place

There is currently no interface in the preprocessor to construct this option structure vector; this has to be done by hand by the user in the MOD file.

2.1. Example syntax for win and unix, for local parallel runs (assuming quad-core)

All empty fields, except Local and NumCPU

options_.parallel = struct('Local', 1, 'PcName','', 'NumCPU', [0:3], 'user','','passwd','',
'RemoteDrive', '', 'RemoteFolder','', 'MatlabOctavePath', '', 'DynarePath', '');

2.2. Example Windows syntax for remote runs

  • win passwd has to be typed explicitly!
  • RemoteDrive has to be typed explicitly!

  • for user, ALSO the group has to be specified, like DEPT\JohnSmith, i.e. user JohnSmith in windows group DEPT

  • PcName is the name of the computer in the windows network, i.e. the output of hostname, or the full IP adress

options_.parallel = struct('Local', 0, 'PcName','RemotePCName','NumCPU', [4:6], 'user',
'DEPT\JohnSmith','passwd','****', 'RemoteDrive', 'C', 'RemoteFolder','dynare_calcs\Remote');

2.2.1. Example to use several remote PC's to build a grid

A vector of parallel structures has to be built:

options_.parallel = struct('Local', 0, 'PcName','RemotePCName1','NumCPU', [0:3], 
'user', 'DEPT\JohnSmith', 'passwd','****', 'RemoteDrive', 'C', 'RemoteFolder','dynare_calcs\Remote');

options_.parallel(2) = struct('Local', 0, 'PcName','RemotePCName2','NumCPU', [0:3], 
'user', 'DEPT\JohnSmith','passwd','****', 'RemoteDrive', 'D', 'RemoteFolder','dynare_calcs\Remote');

options_.parallel(3) = struct('Local', 0, 'PcName','RemotePCName3','NumCPU', [0:1], 
'user','DEPT\JohnSmith','passwd','****', 'RemoteDrive', 'C', 'RemoteFolder','dynare_calcs\Remote');

options_.parallel(4) = struct('Local', 0, 'PcName','RemotePCName4','NumCPU', [0:3], 
'user','DEPT\JohnSmith','passwd','****', 'RemoteDrive', 'C', 'RemoteFolder','dynare_calcs\Remote');

2.2.2. Example of combining local and remote runs

options_.parallel=struct('Local', 1, 'PcName','','NumCPU', [0:3],
 'user','','passwd','','RemoteDrive', '', 'RemoteFolder','');

options_.parallel(2)=struct('Local', 0, 'PcName','RemotePCName','NumCPU', [0:1], 
'user','DEPT\JohnSmith','passwd','****', 'RemoteDrive', 'C', 'RemoteFolder','dynare_calcs\Remote');

2.3. Example Unix syntax for remote runs

  • no passwd and RemoteDrive needed!

  • PcName: full IP address or address

2.3.1. Example with only one remote slave

options_.parallel=struct('Local', 0, 'PcName','name.domain.org','NumCPU', [0:3], 
'user','JohnSmith','passwd','', 'RemoteDrive', '', 'RemoteFolder','/home/rattoma/Remote','MatlabOctavePath', 'matlab', 'DynarePath', '/home/rattoma/dynare/matlab');

2.3.2. Example of combining local and remote runs (on unix):

options_.parallel=struct('Local', 1, 'PcName','','NumCPU', [0:3], 
'user','','passwd','','RemoteDrive', '', 'RemoteFolder','','MatlabOctavePath', '', 'DynarePath', '');

options_.parallel(2)=struct('Local', 0, 'PcName','name.domain.org','NumCPU', [0:3], 'user','JohnSmith','passwd','', 'RemoteDrive', '', 'RemoteFolder','/home/rattoma/Remote','MatlabOctavePath', 'matlab', 'DynarePath', '/home/rattoma/dynare/matlab');

3. Informations for the Dynare developers

3.1. General architecture of the system

The generic parallelization system is organized around two routines: masterParallel and fParallel.

  • masterParallel is the entry point to the parallelization system. It is called from the master computer, at the point where the parallelization system should be activated. Its main arguments are the name of the function containing the task to be run on every slave computer, inputs to that function stored in two structures (one for local and the other for global variables), and the configuration of the cluster; this function exits when the task has finished on all computers of the cluster, and returns the output in a structure vector (one entry per slave)

  • fParallel is the top-level function to be run on every slave; its main arguments are the name of the function to be run (containing the computing task), and some information identifying the slave; the function will retrieve inputs on the filesystem, call the computing task, and transmit back the output to the master computer

A more complete developer documentation (but a bit outdated) is in parallel.pdf.

3.2. Improvements to be made (by decreasing order of importance)

  • Improve the way we deal with MATLAB's native multithreading, which was introduced in MATLAB 7.4, and enable by default on MATLAB 7.6 (see MatlabVersionsCompatibility). The default behavior of the parallel toolbox should be to disable that feature, which can be done using maxNumCompThreads or -singleCompThread depending on the MATLAB version. An option to the parallel code should exist for giving control of number of threads to MATLAB.

  • Rename internal options to reflect the names of options in the config file (see below)
  • Implement console mode for MATLAB (already done for Octave), by testing options_.console_mode

  • Allow for the possibility of specifying a weight for each slave in the cluster, for taking into account the heterogeneity of performances; slaves with a low weight would be allocated less blocks
  • Network performance: let the master download files from the slaves continuously, instead of waiting for the slaves to end their computations, in order to minimize transfer time

3.3. Interface

  • The general idea is to put all the configuration of the cluster in a config file different from the MOD file, and to trigger the parallel computation with option(s) on the “dynare” command line

  • Configuration file:
    • In a standard location ($HOME/.dynare under Unix, c:\Documents and Setting\<username>\Application Data\dynare.ini on Windows)

    • Should have provisions for other Dynare configuration parameters unrelated to parallel computation
    • Should allow to specify several clusters, each one associated with a nickname
    • For each cluster, specify a list of slaves with the following options for each slave [if not explicitly specified by the configuration file, the preprocessor sets the options to default]:

Node Options

type

default

Meaning

Req. Local Win

Req. Remote Win

Req. Local Unix

Req. Remote Unix

Name

string

empty

name of the node

*

*

*

*

CPUnbr

Integer

Linux: all available CPU's; Windows: needs to be set by the user

Number of CPU's to be used on that computer

*

*

*

*

ComputerName

string

localhost (implies options_.parallel(?).Local=1)

Computer name on the network or IP address

*

*

UserName

string

empty

required for remote login

*

*

Password

string

empty

required for remote login (only under Windows)

*

RemoteDrive

string

empty

Drive to be used on remote computer

*

RemoteDirectory

string

empty

Directory to be used on remote computer

*

*

DynarePath

string

empty

path to matlab directory within the Dynare installation directory

MatlabOctavePath

string

empty

path to MATLAB or Octave executable

SingleCompThread

boolean

true

disable MATLAB's native multithreading ?

Cluster Options

type

default

Meaning

Required

Name

string

empty

name of the node

*

Members

string

empty

list of members in this cluster

*

  • Command line options:
    • conffile=<path>: specify the location of the configuration file if it is not standard

    • parallel: trigger the parallel computation using the first cluster specified in config file. If no cluster is specified, use all slave nodes present in the config file.

    • parallel=<clustername>: trigger the parallel computation, using the given cluster

    • parallel_slave_open_mode: use the leaveSlaveOpen mode in the cluster

    • parallel_test: just test the cluster, don’t actually run the MOD file

  • options and commands triggered by the preprocessor:
    • options_.parallel: all the fields are filled in by the preprocessor: if some field is not explicitly defined in the configuration file, preprocessor sets them at their default value;
    • options_.parallel_info: the field options_.parallel_info.leaveSlaveOpen is set at 1 if parallel_slave_open_mode is set in configuraton file; the default value (=0), is set in global_initialization.m;
    • with parallel_test option, the preprocessor triggers simply:

InitializeComputationalEnviroment();
AnalizeComputationalEnvironment(options_.parallel);
  • after options_ are set, the processed matlab main file must contain two configuration commands, one before computations start, the second just before completing dynare computations:
    • InitializeComputationalEnviroment generates the temporary remote subfolder where remote jobs are run;

    • closeSlave closes all slave MATLAB instances and does a clean-up of working directories;

    • in the end, the processed matlab main script should be structured as follows:

preamble, options_ definitions, global initializations, etc...

InitializeComputationalEnviroment();

... all dynare computations ...

if options_.parallel_info.leaveSlaveOpen == 1,
    closeSlave(options_.parallel,options_.parallel_info.RemoteTmpFolder),
end

diary off

...

DynareWiki: ParallelDynare (last edited 2012-05-09 10:05:10 by HoutanBastani)