14-Mar-2007

Introduction

The RTC has two network interconnect options between nodes (ethernet and Myrinet) which can be used for MPI applications. This document will describe how to compile, link, and run MPI code for use with both options as well as some issues to consider when compiling your code.

Compiling and Linking MPI Code

First, there is no default compilation environment. You must specify which compiler suite and MPI library you intend to use. You will do this using the module command.

The module command accomplishes several things. 1) It sets your PATH so that the appropriate compiler is invoked (either Intel or GNU), 2) it will set LD_LIBRARY_PATH to find the correct dynamic libraries at runtime, and 3) it will direct you to the appropriate man pages when you use man. Type module -h to see all of its options.

A typical usage, to compile using the Intel 9.0 compilers and link to the Myrinet MPI library, would be:

module load intel                        (load Intel compiler environment)
module load  mpich-gm/1.2.7-intel9       (load Myrinet MPI library environment)
mpif90 file.f90 -o executable.exe        (compile/link a Fortran 90 program)

After specifying a compiler suite, you can compile and link using the MPICH compile scripts: mpicc, mpicxx, mpif77, and mpif90. These will automatically link in the MPI library for you.

Considerations when Compiling Code

There are several things to consider when you compile your code:

1. Which Compiler Suite to Use?

We strongly recommend that you use the Intel compilers for both batch and interactive use. We prefer these compilers for three reasons: 1) the Intel compilers optimize for the Itanium far better than do the GNU compilers (producing perhaps 5X faster executables), 2) they include a f95 compiler (ifort); they all support OpenMP, and they have an autoparallelizer (for shared memory), and most importantly, 3) CITI and IT do not have the resources to fully support both Intel and GNU versions of all the MPI and parallel libraries. Therefore, if you want to use the GNU compilers, you may have to work harder. (Certainly your code will.)

2. Use Myrinet or Ethernet for MPI?

When compiling, you must link in one of two MPI libraries-the Myrinet or the ethernet version of MPICH. You must compile and link in the correct MPI library for the interconnect you choose. (This is done using the module utility which changes your PATH so that the compile script finds the correct version in the correct directory.)

Remember that you have no default compile configuration. To use the MPICH compile scripts, you must first specify a compiler and MPICH library using the module utility. Thereafter, you can compile using the MPICH scripts and the MPI library will be linked in automatically.

The Myrinet option is the best option because it provides access to the Myrinet network on the RTC, a dedicated high-speed low-latency network designed for parallel applications. However, it is only available on 81 nodes (as of 3/12/2007). Ethernet will be much slower and is subject to performance fluctuations due to non-MPI traffic that is on this network. It is available on every node.

Here are the module commands available for both Ethernet and Myrinet MPI libraries:

3. Use the MPICH compilation scripts

We strongly recommend using the MPICH compilation scripts (mpicc, mpicxx, mpif77, mpif90) since they're easier to use than compiling and linking by manually specifying all the include file paths and library locations.

Running MPI Code

To choose an MPI network in a PBS script, you must edit two lines-one to request the network via PBS, the other to begin execution of your MPI program on that network. To PBS, the default network is Gigabit Ethernet, so you must explicitly request Myrinet. To mpiexec, (the program that launches your MPI executable) the default network is Myrinet, so you must explicitly request Gigabit Ethernet.

Requesting Specific MPI Network via PBS
To specify the MPI network to PBS, specify one of the following lines in your PBS script.

To request Myrinet
To request Myrinet, you must mention it by name:
#PBS -l nodes=7:ppn=2:myrinet,walltime=1:00:00

To request Gigabit Ethernet
#PBS -l nodes=7:ppn=2,walltime=1:00:00

Launching Your Code with mpiexec
When launching your MPI executable using mpiexec, use one of the following lines.

To use Myrinet
mpiexec ./your-mpi-program
or
mpiexec -comm mpich-gm ./your-mpi-program

To use Gigabit Ethernet
To use Gigabit Ethernet, you must mention it by name:
mpiexec -comm mpich-p4 ./your-mpi-program