| Legacy documentation
This page describes a service provided by a retired ACENET system. Most ACENET services are currently provided by national systems, for which please visit https://docs.computecanada.ca.
- CMAQ is an active open-source development project of the U.S. EPA Atmospheric Science Modeling Division that consists of a suite of programs for conducting air quality model simulations.
- CMAQ homepage
- User's experience working with CMAQ on our clusters might be different from that on a personal computer. The default CMAQ distribution appears to expect its users to have full access to the installation. The problem is that this approach is not compatible with how software gets installed on a shared facility to be accessible by all users, which is read-only. An advanced CMAQ user should be able to adjust to this setup rather quickly though.
- For an easy transition, we have modified the
run.*files (henceforth the "run scripts") found in the
scriptsdirectory so that
$BASEis set relative to the installation path, making the CMAQ run scripts independent of the current working directory. Moreover, the line that forcibly changes the current working directory to
- All the run-time relevant settings are provided via a modulefile file, and the CMAQ run scripts do not source
config.cmaq, allowing users to overwrite
- Users should be using C-shell in their submission scripts when utilizing CMAQ run scripts. Please ensure that there is an empty line at the end of your submissions script.
unlimitcommand has been commented in the run scripts, such as
run.cctm. Users should be setting memory requirements in their job submission scripts. CCTM likely requires an increased stack size, so try specifying
-l h_stack=100Mif the code fails shortly after starting (see Memory Management).
- The only component that could be run in parallel appears to be CCTM. In order to execute it in parallel, users are expected to make changes to a run script as described below.
Using standard run scripts
Here is an example how to reproduce the Run Instructions from the readme file. First, prepare the environment:
$ module purge $ module load intel openmpi/intel cmaq
Make a copy of the data directory (more than 6GB) in your space, either into
$ cd /home/$USER/scratch $ mkdir cmaq $ cd cmaq/ $ cp -a $M3HOME/data/* ./
Create a submission script (see below). The CCTM component will require a greater stack size as well as more than 2G of memory (see Memory Management), so specify them explicitly.
#$ -S /bin/csh #$ -cwd #$ -j y #$ -l h_rt=01:00:00 #$ -l h_vmem=4G,h_stack=100M module purge module load intel openmpi/intel cmaq setenv M3DATA /home/$USER/scratch/cmaq $M3HOME/scripts/icon/run.icon $M3HOME/scripts/bcon/run.bcon $M3HOME/scripts/cctm/run.cctm # Empty line. Keep it
Submit your script with the
qsub command to the scheduler.
Customized run scripts
If you need to make modifications to a CMAQ run script, you can simply copy it into your current directory, make modifications and then call it from your submissions script. Here is an example:
$ cp $M3HOME/scripts/bcon/run.bcon ./
run.cctm to your liking and then create a submission script:
#$ -S /bin/csh #$ -cwd #$ -j y #$ -l h_rt=01:00:00 #$ -l h_vmem=4G,h_stack=100M module purge module load intel openmpi/intel cmaq setenv M3DATA /home/$USER/scratch/cmaq ./run.bcon # Empty line. Keep it
Running CCTM in parallel
In order to run CCTM in parallel, one needs to make modifications to the CCTM run script. First, make a copy of the script into your current working directory:
$ cp $M3HOME/scripts/cctm/run.cctm ./
Then modify the script in a few places to allow it to run in parallel. Let's consider an example for a 4-process MPI job. Below are the lines that you would need to find in
run.cctm and change their values appropriately:
set PROC = mpi setenv NPCOL_NPROW "2 2"; set NPROCS = 4 # /usr/bin/time $BLD/$EXEC mpirun $BLD/$EXEC
Please note the commented line above. Also note that
mpirun does not require any options or parameters, because it will get the number of processes to run with from the scheduler. Below is an example of a submission script. The
h_vmem is per process, so it can be reduced compared to the serial run.
#$ -S /bin/csh #$ -cwd #$ -j y #$ -l h_rt=01:00:00 #$ -l h_vmem=2G,h_stack=100M #$ -pe ompi* 4 module purge module load intel openmpi/intel cmaq setenv M3DATA /home/$USER/scratch/cmaq ./run.cctm # Empty line. Keep it
Running MCIP on the sample data
If you have made a copy of the data directory as described above then you already have sample input data for MCIP. Here is a submissions script to run it:
#$ -S /bin/csh #$ -cwd #$ -j y #$ -l h_rt=01:00:00 #$ -l h_vmem=2G module purge module load intel openmpi/intel cmaq setenv M3DATA /home/$USER/scratch/cmaq/mcip $M3HOME/scripts/mcip/run.mcip # Empty line. Keep it
If you want to process any other data then you will need to modifying the run script for MCIP, and thus you will need to copy it into your working directory, make modifications and then call it from your submissions script. Here is how:
$ cp $M3HOME/scripts/mcip/run.mcip ./
run.mcip to your liking and then create a submission script like so:
#$ -S /bin/csh #$ -cwd #$ -j y #$ -l h_rt=01:00:00 #$ -l h_vmem=2G module purge module load intel openmpi/intel cmaq setenv M3DATA /home/$USER/scratch/cmaq/mcip ./run.mcip # Empty line. Keep it