mpirun - Runs MPI programs


SYNOPSIS

       mpirun [global_options] entry_object [:entry_object ...]


DESCRIPTION

       The mpirun command is the primary job launcher for the
       Message Passing Toolkit (MPT) implementations of MPI. The
       mpirun command must be used when a user wants to run an
       MPI application on IRIX or Linux systems.  In addition,
       Array Services software must be running to launch MPI
       programs.

       MPI implements the MPI 1.2 standard, as documented by the
       MPI Forum in the spring 1997 release of MPI: A Message
       Passing Interface Standard.  In addition, certain MPI-2
       functions are implemented.  However, several MPI
       implementations available today use a job launcher called
       mpirun, and because this command is not part of the MPI
       standard, each implementation's mpirun command differs in
       both syntax and functionality.

       You can run an application on the local host only (the
       host from which you issued mpirun) or distribute it to run
       on any number of hosts that you specify.

       The mpirun command accepts the following operands:

       The global_options operand applies to all MPI executable
       files on all specified hosts. Global options must be
       specified before local options specific to a host
       (entry_object).  The following global options are
       supported:

       Global Option            Description

       -a[rray] array_name      Specifies the array to use when
                                launching an MPI application.  By
                                default for multihost jobs, Array
                                Services uses the default array,
                                which is identified by the ainfo
                                dfltarray command.  You can
                                obtain other valid values by
                                issuing the ainfo arrays command
                                or by viewing the
                                /usr/lib/array/arrayd.conf file.
                                [Note: This option is ignored
                                when using the -np option.]

       -cpr                     (This option is supported on IRIX
                                systems only.)  Allows users to
                                checkpoint or restart MPI jobs
                                that consist of a single

                                single system.  Note that the
                                -cpr forces the use of the -np
                                option.

                                For example, the following
                                command is valid in ksh (OUTFILE
                                is the file to which stdout will
                                be redirected, which may also be
                                /dev/null):

                      mpirun -v -cpr -np 2 a.out > OUTFILE 2>&1 < /dev/null

                                The following commands are not
                                valid:

                      mpirun -cpr 2 ./a.out : 3 ./b.out
                      mpirun -cpr hosta -np 2 ./a.out>out 2>&1 </dev/null

                                The first one is not valid
                                because it consists of more than
                                one executable file (a.out and
                                b.out).  The second one is not
                                valid because even if submitted
                                from hosta, it specifies a host
                                name.

                                For interactive users, the most
                                common methods of checkpointing
                                are by either ASH or HID.  You
                                can use the array(1) command to
                                find the ASH of a job.

                                However, checkpoint/restart
                                currently uses the default array
                                from Array Services to checkpoint
                                jobs.  To be able to checkpoint
                                by ASH you must also ensure that
                                the default array contains only
                                the localhost.  If the default
                                array contains any remote hosts,
                                you must checkpoint by HID or one
                                of the other types supported by
                                cpr.

                                Interactive users should also
                                note that stdin, stdout, and
                                stderr should not be connected to
                                the terminal when this option is
                                being used.

                                Use of this option requires Array


                                The default behavior will allow
                                for jobs to be checkpointed if
                                the above rules for invoking have
                                been followed, but using the -cpr
                                option is recommended because it
                                provides specific error messages
                                instead of silently disabling.

       -d[ir] path_name         Specifies the working directory
                                for all hosts.  In addition to
                                normal path names, the following
                                special values are recognized:

                                     .    Translates into the
                                          absolute path name of
                                          the user's current
                                          working directory on
                                          the local host. This is
                                          the default.

                                     ~    Specifies the use of
                                          the value of $HOME as
                                          it is defined on each
                                          machine. In general,
                                          this value can be
                                          different on each
                                          machine.

       -f[ile] file_name        Specifies a text file that
                                contains mpirun arguments.

       -h[elp]                  Displays a list of options
                                supported by the mpirun command.

       -miser                   (This option is supported on IRIX
                                systems only.)  Allows MPI jobs
                                that run on a single system to be
                                submitted to miser. The absence
                                of any host names in the mpirun
                                command indicates that a job is
                                running on a single system, and
                                thus can be submitted to miser.
                                For example, the following
                                command is valid:

                      miser_submit -q queue -f file mpirun -miser 2 ./a.out : 3 ./b.out

                                The following command is not
                                valid, even if submitted on
                                hosta:


                                Use of this option requires Array
                                Services 3.1 or later.

       -p[refix] prefix_string  Specifies a string to prepend to
                                each line of output from stderr
                                and stdout for each MPI process.
                                To delimit lines of text that
                                come from different hosts, output
                                to stdout must be terminated with
                                a new line character.  If a
                                process's stdout or stderr
                                streams do not end with a new
                                line character, there will be no
                                prefix associated with the output
                                or error streams of that process
                                from the final new line to the
                                end of the stream.

                                If the MPI_UNBUFFERED_STDIO
                                environment variable is set, the
                                prefix string is ignored.

                                Some strings have special meaning
                                and are translated as follows:

                                *   %g translates into the global
                                    rank of the process producing
                                    the output. This is
                                    equivalent to the rank of the
                                    process in MPI_COMM_WORLD
                                    when not running in spawn
                                    capable mode.  In the latter
                                    case, this translates to the
                                    rank of the process within
                                    the universe specified at job
                                    launch.

                                *   %G translates into the number
                                    of processes in
                                    MPI_COMM_WORLD, or, if
                                    running in spawn capable
                                    mode, the value of the
                                    MPI_UNIVERSE_SIZE attribute.

                                *   %h translates into the rank
                                    of the host on which the
                                    process is running, relative
                                    to the mpirun command line.
                                    This string is not relevant
                                    for processes started via
                                    MPI_Comm_spawn or


                                *   %H translates into the total
                                    number of hosts in the job.
                                    This string is not relevant
                                    for processes started via
                                    MPI_Comm_spawn or
                                    MPI_Comm_spawn_multiple.

                                *   %l translates into the rank
                                    of the process relative to
                                    other processes running on
                                    the same host.

                                *   %L translates into the total
                                    number of processes running
                                    on the host.

                                *   %w translates into the world
                                    rank of the process, i.e. its
                                    rank in a MPI_COMM_WORLD.
                                    When not running in spawn
                                    capable mode, this is
                                    equivalent to %g.

                                *   %W translates into the total
                                    number of processes in
                                    MPI_COMM_WORLD.  When not
                                    running in spawn capable
                                    mode, this is equivalent to
                                    %G.

                                *   %@ translates into the name
                                    of the host on which the
                                    process is running.

                                For examples of the use of these
                                strings, first consider the
                                following code fragment:

                                     main(int argc, char **argv)
                                     {
                                        MPI_Init(&argc, &argv);

                                        printf("Hello world\n");

                                        MPI_Finalize();
                                     }

                                Depending on how this code is
                                run, the results of running the
                                mpirun command will be similar to
                                those in the following examples:

                                Hello world
                                Hello world

                                % mpirun -prefix ">" -np 2 a.out
                                >Hello world
                                >Hello world

                                % mpirun -prefix "%g" 2 a.out
                                0Hello world
                                1Hello world

                                % mpirun -prefix "[%g] " 2 a.out
                                [0] Hello world
                                [1] Hello world

                                % mpirun -prefix "<process %g out of %G> " 4 a.out
                                <process 1 out of 4> Hello world
                                <process 0 out of 4> Hello world
                                <process 3 out of 4> Hello world
                                <process 2 out of 4> Hello world

                                % mpirun -prefix "%@: " hosta,hostb 1 a.out
                                hosta: Hello world
                                hostb: Hello world

                      % mpirun -prefix "%@ (%l out of %L) %g: " hosta 2, hostb 3 a.out

                                hosta (0 out of 2) 0: Hello world
                                hosta (1 out of 2) 1: Hello world
                                hostb (0 out of 3) 2: Hello world
                                hostb (1 out of 3) 3: Hello world
                                hostb (2 out of 3) 4: Hello world

                      % mpirun -prefix "%@ (%h out of %H): " hosta,hostb,hostc 2 a.out

                                hosta (0 out of 3): Hello world
                                hostb (1 out of 3): Hello world
                                hostc (2 out of 3): Hello world
                                hosta (0 out of 3): Hello world

                                hostb (1 out of 3): Hello world

       -stats                   Prints statistics about the
                                amount of data sent with MPI
                                calls during the MPI_Finalize
                                process.  Data is sent to stderr.
                                Users can combine this option
                                with the -p option to prefix the
                                statistics messages with the MPI
                                rank.  For more details, see the
                                MPI_SGI_stat_print(3) man page.

       -up u_size               Specifies the value of the
                                MPI_UNIVERSE_SIZE attribute to be
                                used in supporting MPI_Comm_spawn
                                and MPI_Comm_spawn_multiple.
                                This field must be set if either
                                of these functions are to be used
                                by the application being launched
                                by mpirun.  Setting this field
                                implies the MPI job is being run
                                in spawn capable mode.

       -v[erbose]               Displays comments on what mpirun
                                is doing when launching the MPI
                                application.

   Entry Objects
       entry_object describes a host on which to run a program,
       and the local options for that host.  You can list any
       number of entry_object entries on the mpirun command line.

       In the common case (Single Program Multiple Data (SPMD)),
       in which the same program runs with identical arguments on
       each host, usually only one entry_object needs to be
       specified.

       Each entry_object has the following components:

       *   One or more host names (not needed if you run on the
           local host)

       *   Number of processes to start on each host

       *   Name of an executable program

       *   Arguments to the executable program (optional)

       entry_object has the following format:

            host_list local_options program program_arguments

       name) or a comma-separated list of hosts on which to run
       an MPI program.

       The local_options operand contains information that
       applies to a specific host list. The following local
       options are supported:

       Local Option                  Description

       -f[ile] file_name             Specifies a text file that
                                     contains mpirun arguments
                                     (same as global_options.)
                                     For more details, see the
                                     subsection titled "Using a
                                     File For mpirun Arguments"
                                     on this man page.

       -np num_proc                  Specifies the number of
                                     processes on which to run.
                                     This local option behaves
                                     the same as -np.

       -nt num_proc                  The -nt option is equivalent
                                     to the -np option. It is
                                     deprecated, and exists for
                                     command line compatibility
                                     with previously supported
                                     architectures.

       The program program_arguments operand specifies the name
       of the program that you are running and its accompanying
       options.

   Using a File for mpirun Arguments
       Because the full specification of a complex job can be
       lengthy, you can enter mpirun arguments in a file and use
       the -f option to specify the file on the mpirun command
       line, as in the following example:

            mpirun -f my_arguments

       The arguments file is a text file that contains argument
       segments.  White space is ignored in the arguments file,
       so you can include spaces and newline characters for
       readability. An arguments file can also contain additional
       -f options.

   Launching Programs on the Local Host
       For testing and debugging, it is often useful to run an
       MPI program on the local host only without distributing it
       to other systems. To run the application locally, enter
       mpirun with the -np argument. Your entry must include the

       executable file.

       The following command starts three instances of the
       application mtest, which is passed an arguments list
       (arguments are optional):

            mpirun -np 3 mtest 1000 "arg2"

       You are not required to use a different host in each entry
       that you specify on the mpirun command.  You can launch a
       job that has two executable files on the same host.  In
       the following example, both executable files use shared
       memory.

            mpirun host_a -np 6 a.out : host_a -np 4 b.out

       Note that for IRIX hosts, both executable files must be compiled
       as either 32-bit or 64-bit applications.

   Launching a Distributed Program
       You can use mpirun to launch a program that consists of
       any number of executable files and processes and
       distribute it to any number of hosts. A host is usually a
       single machine, or, for IRIX systems, can be any
       accessible computer running Array Services software.  For
       available nodes on systems running Array Services
       software, see the /usr/lib/array/arrayd.conf file.

       You can list multiple entries on the mpirun command line.
       Each entry contains an MPI executable file and a
       combination of hosts and process counts for running it.
       This gives you the ability to start different executable
       files on the same or different hosts as part of the same
       MPI application.

       The following examples show various ways to launch an
       application that consists of multiple MPI executable files
       on multiple hosts.

       The following example runs ten instances of the a.out file
       on host_a:

            mpirun host_a -np 10 a.out

       When specifying multiple hosts, the -np option can be
       omitted with the number of processes listed directly.  The
       following example launches ten instances of fred on three
       hosts.  fred has two input arguments.

            mpirun host_a, host_b, host_c 10 fred arg1 arg2

       different hosts with different numbers of processes and
       executable files, using an array called test:

            mpirun -array test host_a 6 a.out : host_b 26 b.out

       The following example launches an MPI application on
       different hosts out of the same directory on both hosts:

            mpirun -d /tmp/mydir host_a 6 a.out : host_b 26 b.out

   Job Control
       It is possible to terminate, suspend, and/or resume an
       entire MPI application (potentially running across
       multiple hosts) by using the same control characters that
       work for serial programs.  For example, sending a SIGINT
       signal to mpirun terminates all processes in an MPI job.
       Similarly, sending a SIGTSTP signal to mpirun suspends an
       MPI job and sending a SIGCONT signal resumes a job.

   Signal Propagation
       It is possible to send some user signals to all processes
       in an MPI application (potentially running across multiple
       hosts).  Presently, mpirun supports two user-defined
       signals:  SIGURG and SIGUSR1.  To make use of this
       feature, the MPI program needs to have a signal handler
       that catches SIGURG or SIGUSR1.  When the SIGURG or
       SIGUSR1 signals are sent to the mpirun process ID, the
       mpirun process will catch the signal and propagate it to
       all MPI processes.

   Troubleshooting
       Problems you encounter when launching MPI jobs will
       typically result in a could not run executable error
       message from mpirun. There are many possible causes for
       this message, including (but not limited to) the following
       reasons:

       *   The . is missing from the user's search path. This
           problem most commonly occurs when the -np syntax is
           used.

       *   No permission has been granted to the local host to
           launch processes on remote hosts.  Because MPI
           references the .rhosts file for authentication, this
           can happen even if you are running your job on the
           same machine. For example, if you specify mpirun
           localhost 2 a.out, MPI will treat localhost as a
           remote host.  The usual solution to this problem is to
           put the local host name in your ~/.rhosts file.

       *   The working directory is defaulting to $HOME instead

           the -d option.

       *   localhost does not appear in the /etc/hosts.equiv file
           (required for -np syntax).

       *   The Array Services daemon (arrayd) has been
           incorrectly configured; use ascheck to test your
           configuration.

       *   In general, if arshell fails, mpirun usually fails as
           well.

   Limitations
       The following practices will break the mpirun parser:

       *   Using machine names that are numbers (for example, 3,
           127, and so on)

       *   Using MPI applications whose names match mpirun
           options (for example, -d, -f, and so on)

       *   Using MPI applications that use a colon (:) in their
           command-lines.


NOTES

       Running an MPI job in the background is supported only
       when stdin is redirected.

       The mpirun process is still connected to the tty when a
       job is placed in the background.  One of the things that
       mpirun polls for is input from stdin.  If it happens to be
       polling for stdin when a user types in a window after
       putting an MPI job in the background, and stdin has not
       been redirected, the job will abort upon receiving a
       SIGTTIN signal.  This behavior is intermittent, depending
       on whether mpirun happens to be looking for and sees any
       stdin input.

       The following examples show how to run an MPI job in the
       background.

       For a job that uses input_file as stdin:

            mpirun -np 2 ./a.out < input_file > output &

       For a job that does not use stdin:

            mpirun -np 2 ./a.out < /dev/null  > output &


RETURN VALUES

       On exit, mpirun returns the appropriate error code to the



SEE ALSO

       mpi(1)

       termio(7)


Man(1) output converted with man2html