OpenMP

OpenMP (Open Multi-Processing) is a standard for building parallel applications on shared-memory computers (multiprocessors). It consists primarily of a set of compiler directives, with some library routines besides.

The maximum number of threads available for OpenMP jobs at ACEnet is either 4 or 16 depending on the execution host upon which the job is running. The number of threads is communicated to an OpenMP program at execution time with the environment variable OMP_NUM_THREADS. For example, if you want your program running on four cores use the following in bash

 export OMP_NUM_THREADS=4

or in csh:

 setenv OMP_NUM_THREADS 4

In order to force the compiler to interpret the OpenMP directives in the source code, you need to specify appropriate flags during the compilation, otherwise a serial code will be generated.

Compiler	Option
PGI compilers	-mp
GCC4	-fopenmp
Intel compilers	-openmp
Sun Studio 12	-openmp along with the third optimization level -O3

The good introduction to OpenMP can be found here.

Hello world program

Here is an example of the simple OpenMP program written in Fortran 90, along with the instructions on how to compile, run interactively and submit this program to the cluster.

program hello

implicit none
include 'omp_lib.h'
integer :: my_rank = 0, p = 1

!$omp parallel private( my_rank )

!$omp single
write(*,'("Hello parallel world!")')
!$ p = omp_get_num_threads()
write(*,'("Number of threads is ", i3)') p
!$omp end single

!$ my_rank = omp_get_thread_num()
write(*,'("Hello world from thread", i3)') my_rank

!$omp end parallel

write(*,'("Back to the sequential world.")')

end program hello

And this is how you compile the code and run it interactively on the head node in csh.

 $ pgf90 -mp -o hello hello.f90
 $ export OMP_NUM_THREADS=4
 $ ./hello
 Hello parallel world!
 Number of threads is   4
 Hello world from thread  3
 Hello world from thread  0
 Hello world from thread  1
 Hello world from thread  2
 Back to the sequential world.
 $

To submit the code hello to the scheduler, you need to create a small submission script. With this script you instruct the scheduler where to execute the code, where to write the output, and with how many threads you want you code to be run. Here is an example of such a script called submit_hello.sh.

 #$ -cwd
 #$ -j y
 #$ -l h_vmem=1G,h_rt=01:00:00
 #$ -pe openmp 4

 export OMP_NUM_THREADS=$NSLOTS
 ./hello

Finally, to submit the job, type in the command line

 $ qsub submit_hello.sh