Message Passing Interface (MPI) is a standard application programming interface (API) for efficient interprocess communication. It is widely used in many existing research codes. It can be used to write parallel programs which span multiple computers or which reside on a single multiprocessor.
The default (and preferred) MPI implementation on our clusters is Open MPI. It's free, open source, production-quality MPI-2 implementation.
Note: Do not confuse the Open MPI library with the OpenMP interface.
- How it's spelled
- It's two words (not one) — Open MPI. It's also frequently abbreviated "OMPI" (pronounced "oom-pee").
- Default is 1.2.9 built with PGI. More recent versions also available.
- Benefits of Open MPI
- Supports all available types of interconnections (Ethernet, Myrinet, InfiniBand, on-host shared memory, etc.).
- The fastest available interconnection will be used by default.
- Better performance, especially on large shared-memory nodes.
- No need to recompile your application on every cluster.
- Better integration with Sun Grid Engine.
- Better integration with parallel debuggers (see TotalView).
- User friendly, easy to use.
You can use the following standard commands:
mpif90, mpif77, mpicc, mpiCC (or mpicxx or mpic++), mpirun
You can also browse man pages for the main Open MPI commands, for example:
$ man mpif90 $ man mpicc
Also you can get description and syntax for many MPI functions, for example:
$ man MPI_Init $ man MPI_Send
Running production jobs
Below is an example of a submission script for an MPI job. There is no need to specify the list of hosts and/or the number of processes for the
mpirun command because Open MPI will obtain this information directly from Grid Engine. You should use the
ompi* parallel environment for Open MPI jobs.
#$ -cwd #$ -pe ompi* 4 #$ -l h_rt=01:00:00 mpirun my_parallel_application
Running interactive jobs
- Main page: Interactive Jobs
Below is the short reference on how to run MPI job interactively. For a complete description, please read Interactive Jobs.
Interactive session through Grid Engine:
$ qrsh -cwd -l h_rt=00:10:00,test=true -pe "ompi*" 4 mpirun myapplication
If you prefer to get an interactive login shell and then run the job manually, you just need to specify the name of your favorite shell as a job name, like so:
$ qrsh -cwd -pe "ompi*" 8 -l h_rt=00:30:00 bash
Alternative builds of Open MPI
Some users may want to use alternative builds of Open MPI to compile their applications. We provide several versions of Open MPI built with PGI, Intel and GCC compilers. In order to use them, you should load a specific modulefile. All Open MPI modulefiles are mutually exclusive. In order to minimize user's confusion and prevent version conflict, we suggest users to load an Open MPI modulefile interactively before compiling an application and then again in the submission script to run it.