Open MPI test programs, by Douglas Eadline, 2/25/06
deadline@basement-supercomputing.com
Create/Edit file "machines" with machine list. Then load the module for the
appropriate compiler, for example:

  $ module load openmpi/1.8.4/gnu4

Build C version of pi program:

  $ mpicc -o cpi cpi.c

Run C pi program:

  $ mpiexec -np 8 -machinefile machines cpi

Build FORTRAN version of pi program:

  $ mpif77  -o fpi fpi.f

Run FORTRAN pi program:

  $ mpiexec -np 8 -machinefile machines fpi

If not using open-mx, you will see this error

"...Error in mx_init (error No device file)..."

It can be safely ignored as Open MPI will fall back
to tcp. If you start open mx (as root
#service open-mx start; #pdsh service open-mx start)
the message will go away and the open-mx layer
will be used. To force tcp, then to use

$ mpiexec -np 8 -machinefile machines  -mca btl tcp,self cpi

Finally, by default Open MPI maps processes by "socket," which 
means when launching jobs "a socket" (all cores in the socket) 
will be allocated first and then if more cores are needed
mpiexec will move to the next available socket on the node
and if there is none, move to the next node in the machinefile.
You can change this to "map by node" so it will place one
process on each node and move round-robin through the
machinefile until it assigns all processes.
For example:

  $ mpiexec -np 8 -machinefile machines -map-by node cpi

See the Open MPI mpiexec man page for more  options 
for the "map-by" argument.