mpirun - Runs MPI programs
SYNOPSIS
mpirun [global_options] entry_object [:entry_object ...]
DESCRIPTION
The mpirun command is the primary job launcher for the
Message Passing Toolkit (MPT) implementations of MPI. The
mpirun command must be used when a user wants to run an
MPI application on IRIX or Linux systems. In addition,
Array Services software must be running to launch MPI
programs.
MPI implements the MPI 1.2 standard, as documented by the
MPI Forum in the spring 1997 release of MPI: A Message
Passing Interface Standard. In addition, certain MPI-2
functions are implemented. However, several MPI
implementations available today use a job launcher called
mpirun, and because this command is not part of the MPI
standard, each implementation's mpirun command differs in
both syntax and functionality.
You can run an application on the local host only (the
host from which you issued mpirun) or distribute it to run
on any number of hosts that you specify.
The mpirun command accepts the following operands:
The global_options operand applies to all MPI executable
files on all specified hosts. Global options must be
specified before local options specific to a host
(entry_object). The following global options are
supported:
Global Option Description
-a[rray] array_name Specifies the array to use when
launching an MPI application. By
default for multihost jobs, Array
Services uses the default array,
which is identified by the ainfo
dfltarray command. You can
obtain other valid values by
issuing the ainfo arrays command
or by viewing the
/usr/lib/array/arrayd.conf file.
[Note: This option is ignored
when using the -np option.]
-cpr (This option is supported on IRIX
systems only.) Allows users to
checkpoint or restart MPI jobs
that consist of a single
single system. Note that the
-cpr forces the use of the -np
option.
For example, the following
command is valid in ksh (OUTFILE
is the file to which stdout will
be redirected, which may also be
/dev/null):
mpirun -v -cpr -np 2 a.out > OUTFILE 2>&1 < /dev/null
The following commands are not
valid:
mpirun -cpr 2 ./a.out : 3 ./b.out
mpirun -cpr hosta -np 2 ./a.out>out 2>&1 </dev/null
The first one is not valid
because it consists of more than
one executable file (a.out and
b.out). The second one is not
valid because even if submitted
from hosta, it specifies a host
name.
For interactive users, the most
common methods of checkpointing
are by either ASH or HID. You
can use the array(1) command to
find the ASH of a job.
However, checkpoint/restart
currently uses the default array
from Array Services to checkpoint
jobs. To be able to checkpoint
by ASH you must also ensure that
the default array contains only
the localhost. If the default
array contains any remote hosts,
you must checkpoint by HID or one
of the other types supported by
cpr.
Interactive users should also
note that stdin, stdout, and
stderr should not be connected to
the terminal when this option is
being used.
Use of this option requires Array
The default behavior will allow
for jobs to be checkpointed if
the above rules for invoking have
been followed, but using the -cpr
option is recommended because it
provides specific error messages
instead of silently disabling.
-d[ir] path_name Specifies the working directory
for all hosts. In addition to
normal path names, the following
special values are recognized:
. Translates into the
absolute path name of
the user's current
working directory on
the local host. This is
the default.
~ Specifies the use of
the value of $HOME as
it is defined on each
machine. In general,
this value can be
different on each
machine.
-f[ile] file_name Specifies a text file that
contains mpirun arguments.
-h[elp] Displays a list of options
supported by the mpirun command.
-miser (This option is supported on IRIX
systems only.) Allows MPI jobs
that run on a single system to be
submitted to miser. The absence
of any host names in the mpirun
command indicates that a job is
running on a single system, and
thus can be submitted to miser.
For example, the following
command is valid:
miser_submit -q queue -f file mpirun -miser 2 ./a.out : 3 ./b.out
The following command is not
valid, even if submitted on
hosta:
Use of this option requires Array
Services 3.1 or later.
-p[refix] prefix_string Specifies a string to prepend to
each line of output from stderr
and stdout for each MPI process.
To delimit lines of text that
come from different hosts, output
to stdout must be terminated with
a new line character. If a
process's stdout or stderr
streams do not end with a new
line character, there will be no
prefix associated with the output
or error streams of that process
from the final new line to the
end of the stream.
If the MPI_UNBUFFERED_STDIO
environment variable is set, the
prefix string is ignored.
Some strings have special meaning
and are translated as follows:
* %g translates into the global
rank of the process producing
the output. This is
equivalent to the rank of the
process in MPI_COMM_WORLD
when not running in spawn
capable mode. In the latter
case, this translates to the
rank of the process within
the universe specified at job
launch.
* %G translates into the number
of processes in
MPI_COMM_WORLD, or, if
running in spawn capable
mode, the value of the
MPI_UNIVERSE_SIZE attribute.
* %h translates into the rank
of the host on which the
process is running, relative
to the mpirun command line.
This string is not relevant
for processes started via
MPI_Comm_spawn or
* %H translates into the total
number of hosts in the job.
This string is not relevant
for processes started via
MPI_Comm_spawn or
MPI_Comm_spawn_multiple.
* %l translates into the rank
of the process relative to
other processes running on
the same host.
* %L translates into the total
number of processes running
on the host.
* %w translates into the world
rank of the process, i.e. its
rank in a MPI_COMM_WORLD.
When not running in spawn
capable mode, this is
equivalent to %g.
* %W translates into the total
number of processes in
MPI_COMM_WORLD. When not
running in spawn capable
mode, this is equivalent to
%G.
* %@ translates into the name
of the host on which the
process is running.
For examples of the use of these
strings, first consider the
following code fragment:
main(int argc, char **argv)
{
MPI_Init(&argc, &argv);
printf("Hello world\n");
MPI_Finalize();
}
Depending on how this code is
run, the results of running the
mpirun command will be similar to
those in the following examples:
Hello world
Hello world
% mpirun -prefix ">" -np 2 a.out
>Hello world
>Hello world
% mpirun -prefix "%g" 2 a.out
0Hello world
1Hello world
% mpirun -prefix "[%g] " 2 a.out
[0] Hello world
[1] Hello world
% mpirun -prefix "<process %g out of %G> " 4 a.out
<process 1 out of 4> Hello world
<process 0 out of 4> Hello world
<process 3 out of 4> Hello world
<process 2 out of 4> Hello world
% mpirun -prefix "%@: " hosta,hostb 1 a.out
hosta: Hello world
hostb: Hello world
% mpirun -prefix "%@ (%l out of %L) %g: " hosta 2, hostb 3 a.out
hosta (0 out of 2) 0: Hello world
hosta (1 out of 2) 1: Hello world
hostb (0 out of 3) 2: Hello world
hostb (1 out of 3) 3: Hello world
hostb (2 out of 3) 4: Hello world
% mpirun -prefix "%@ (%h out of %H): " hosta,hostb,hostc 2 a.out
hosta (0 out of 3): Hello world
hostb (1 out of 3): Hello world
hostc (2 out of 3): Hello world
hosta (0 out of 3): Hello world
hostb (1 out of 3): Hello world
-stats Prints statistics about the
amount of data sent with MPI
calls during the MPI_Finalize
process. Data is sent to stderr.
Users can combine this option
with the -p option to prefix the
statistics messages with the MPI
rank. For more details, see the
MPI_SGI_stat_print(3) man page.
-up u_size Specifies the value of the
MPI_UNIVERSE_SIZE attribute to be
used in supporting MPI_Comm_spawn
and MPI_Comm_spawn_multiple.
This field must be set if either
of these functions are to be used
by the application being launched
by mpirun. Setting this field
implies the MPI job is being run
in spawn capable mode.
-v[erbose] Displays comments on what mpirun
is doing when launching the MPI
application.
Entry Objects
entry_object describes a host on which to run a program,
and the local options for that host. You can list any
number of entry_object entries on the mpirun command line.
In the common case (Single Program Multiple Data (SPMD)),
in which the same program runs with identical arguments on
each host, usually only one entry_object needs to be
specified.
Each entry_object has the following components:
* One or more host names (not needed if you run on the
local host)
* Number of processes to start on each host
* Name of an executable program
* Arguments to the executable program (optional)
entry_object has the following format:
host_list local_options program program_arguments
name) or a comma-separated list of hosts on which to run
an MPI program.
The local_options operand contains information that
applies to a specific host list. The following local
options are supported:
Local Option Description
-f[ile] file_name Specifies a text file that
contains mpirun arguments
(same as global_options.)
For more details, see the
subsection titled "Using a
File For mpirun Arguments"
on this man page.
-np num_proc Specifies the number of
processes on which to run.
This local option behaves
the same as -np.
-nt num_proc The -nt option is equivalent
to the -np option. It is
deprecated, and exists for
command line compatibility
with previously supported
architectures.
The program program_arguments operand specifies the name
of the program that you are running and its accompanying
options.
Using a File for mpirun Arguments
Because the full specification of a complex job can be
lengthy, you can enter mpirun arguments in a file and use
the -f option to specify the file on the mpirun command
line, as in the following example:
mpirun -f my_arguments
The arguments file is a text file that contains argument
segments. White space is ignored in the arguments file,
so you can include spaces and newline characters for
readability. An arguments file can also contain additional
-f options.
Launching Programs on the Local Host
For testing and debugging, it is often useful to run an
MPI program on the local host only without distributing it
to other systems. To run the application locally, enter
mpirun with the -np argument. Your entry must include the
executable file.
The following command starts three instances of the
application mtest, which is passed an arguments list
(arguments are optional):
mpirun -np 3 mtest 1000 "arg2"
You are not required to use a different host in each entry
that you specify on the mpirun command. You can launch a
job that has two executable files on the same host. In
the following example, both executable files use shared
memory.
mpirun host_a -np 6 a.out : host_a -np 4 b.out
Note that for IRIX hosts, both executable files must be compiled
as either 32-bit or 64-bit applications.
Launching a Distributed Program
You can use mpirun to launch a program that consists of
any number of executable files and processes and
distribute it to any number of hosts. A host is usually a
single machine, or, for IRIX systems, can be any
accessible computer running Array Services software. For
available nodes on systems running Array Services
software, see the /usr/lib/array/arrayd.conf file.
You can list multiple entries on the mpirun command line.
Each entry contains an MPI executable file and a
combination of hosts and process counts for running it.
This gives you the ability to start different executable
files on the same or different hosts as part of the same
MPI application.
The following examples show various ways to launch an
application that consists of multiple MPI executable files
on multiple hosts.
The following example runs ten instances of the a.out file
on host_a:
mpirun host_a -np 10 a.out
When specifying multiple hosts, the -np option can be
omitted with the number of processes listed directly. The
following example launches ten instances of fred on three
hosts. fred has two input arguments.
mpirun host_a, host_b, host_c 10 fred arg1 arg2
different hosts with different numbers of processes and
executable files, using an array called test:
mpirun -array test host_a 6 a.out : host_b 26 b.out
The following example launches an MPI application on
different hosts out of the same directory on both hosts:
mpirun -d /tmp/mydir host_a 6 a.out : host_b 26 b.out
Job Control
It is possible to terminate, suspend, and/or resume an
entire MPI application (potentially running across
multiple hosts) by using the same control characters that
work for serial programs. For example, sending a SIGINT
signal to mpirun terminates all processes in an MPI job.
Similarly, sending a SIGTSTP signal to mpirun suspends an
MPI job and sending a SIGCONT signal resumes a job.
Signal Propagation
It is possible to send some user signals to all processes
in an MPI application (potentially running across multiple
hosts). Presently, mpirun supports two user-defined
signals: SIGURG and SIGUSR1. To make use of this
feature, the MPI program needs to have a signal handler
that catches SIGURG or SIGUSR1. When the SIGURG or
SIGUSR1 signals are sent to the mpirun process ID, the
mpirun process will catch the signal and propagate it to
all MPI processes.
Troubleshooting
Problems you encounter when launching MPI jobs will
typically result in a could not run executable error
message from mpirun. There are many possible causes for
this message, including (but not limited to) the following
reasons:
* The . is missing from the user's search path. This
problem most commonly occurs when the -np syntax is
used.
* No permission has been granted to the local host to
launch processes on remote hosts. Because MPI
references the .rhosts file for authentication, this
can happen even if you are running your job on the
same machine. For example, if you specify mpirun
localhost 2 a.out, MPI will treat localhost as a
remote host. The usual solution to this problem is to
put the local host name in your ~/.rhosts file.
* The working directory is defaulting to $HOME instead
the -d option.
* localhost does not appear in the /etc/hosts.equiv file
(required for -np syntax).
* The Array Services daemon (arrayd) has been
incorrectly configured; use ascheck to test your
configuration.
* In general, if arshell fails, mpirun usually fails as
well.
Limitations
The following practices will break the mpirun parser:
* Using machine names that are numbers (for example, 3,
127, and so on)
* Using MPI applications whose names match mpirun
options (for example, -d, -f, and so on)
* Using MPI applications that use a colon (:) in their
command-lines.
NOTES
Running an MPI job in the background is supported only
when stdin is redirected.
The mpirun process is still connected to the tty when a
job is placed in the background. One of the things that
mpirun polls for is input from stdin. If it happens to be
polling for stdin when a user types in a window after
putting an MPI job in the background, and stdin has not
been redirected, the job will abort upon receiving a
SIGTTIN signal. This behavior is intermittent, depending
on whether mpirun happens to be looking for and sees any
stdin input.
The following examples show how to run an MPI job in the
background.
For a job that uses input_file as stdin:
mpirun -np 2 ./a.out < input_file > output &
For a job that does not use stdin:
mpirun -np 2 ./a.out < /dev/null > output &
RETURN VALUES
On exit, mpirun returns the appropriate error code to the
SEE ALSO
mpi(1)
termio(7)
Man(1) output converted with
man2html