Frequently Asked Questions
"Error: No suitable queues"
You will get the message "Unable to run job: error: no suitable queues" when you submit the job if Grid Engine finds it could never run. You may have requested more memory than can be provided, or mistyped the name of a queue or parallel environment. If you want some hints from Grid Engine about why it can't be scheduled, re-submit it with "-w v":
$ qsub -w v ...other options...
If you know the job can't be scheduled and want it to go in anyway, you can use "-w w" or "-w n" options to override the default "-w e".
No output or truncated output
If your job output is mysteriously truncated or you get no output at all, it might be because:
- the program exceeded a run-time limit (
h_rt). Look at the output you got and try to determine why the program is taking longer than you expected. If you think you know why, resubmit with a larger value for
- the program ran out of memory (
h_vmem). See Memory Management for a detailed discussion.
- the program crashed due to an internal error — check your code for bugs
- the disk quota for the storage the program is writing to was reached — See disk quotas for a list of quotas and commands to check disk usage.
If your job has been terminated unexpectedly (for example it has
exit_status 137 in the 'qacct' records) and it did not violate the run-time limit (
h_rt) then it may have violated a memory limit (
h_vmem). Try increasing
h_vmem or decreasing the size of some large arrays in your program, and resubmit. Use the
qacct command to get Grid Engine accounting data about your job: how much memory it consumed, how long it ran, etc.
Eqw: Job waiting in error state
If qstat or showq has "Eqw" in the STATE column for one of your jobs you can use
$ qstat -j jobid | grep error
to check the reason. If you understand the reason and can get it fixed, you can clear the error state with
$ qmod -cj jobid
Some common error messages are these:
can't chdir to directory: No such file or directory can't stat() "file" as stdout_path
These indicate that some directory or file (respectively) cannot be found. Verify that the file or directory in question exists, i.e., you haven't forgotten to create it and you can see it from the head node. If it appears to be okay, then the job may have suffered a transient condition such as a failed NFS automount, or an NFS server was temporarily down. Clear the error state as shown above.
My job is stuck in the 'dr' state
Sometimes users find their jobs being stuck indefinitely in the
dr state. The
d state indicates that a
qdel has been used to initiate job deletion. The reason a job gets stuck in this state is because Grid Engine loses communication with one of the compute nodes, and thus cannot cleanly remove the job from a queue - hence the dual state: (r)unning and being (d)eleted. Usually, this is due to a failed compute node, which was probably the reason why you tried to remove the job in the first place - because it was not making any progress. Please let us know about such jobs in the
dr state so that we can remove faulty nodes out of production as soon as possible. You can also try to force delete your job like so:
$ qdel -f job_id
If it does not help, let us know and we will delete it for you.
Job won't start
There are a lot of possible reasons why your job does not start right away.
- There may not be enough CPU slots free.
- There may not be enough slots in the time-limit queue (medium.q, long.q) your job qualifies for.
- There might be enough slots but not enough memory because some of the running jobs have reserved a lot of memory.
- A higher priority job may be reserving slots for a large parallel run.
- An outage may be scheduled to begin before your job would end. (See Cluster Status for planned outages.)
- Serial jobs are not scheduled on most 16-core shared memory hosts.
- An individual research group cannot occupy more than 80% of the slots on a given cluster.
- Some other requestable resource (e.g. Myrinet endpoints, Fluent licenses) may not be available.
You can query the Grid Engine for hints about why your job hasn't yet run this way:
$ qalter -w v job_id
However, even the output from qalter is sometimes obscure. If you want more assistance interpreting your jobs' situation, contact us.
Rr: Job re-started
A capital R in the job status in
showq signifies "rescheduled" or "rescheduling". If a host goes down while running a job, Grid Engine will put the job back in the waiting list to be run again. "Rr" means it has restarted and is running again; "Rq" means it is waiting to be restarted. We strongly recommend you verify that a restarted job is progressing as you would expect by checking the output file. Not all applications recover gracefully on a restart like this.
If you want your jobs not to be rescheduled when a host fails in the middle, set the "rerun" option to "no" in your job script like this:
#$ -r no
Why doesn't my job start right away?
This could be for a variety of reasons. When you submit a job to the N1 Grid Engine you are making a request for resources. There may be times when the cluster is busy and you will be required to wait for resources. If you use the qstat command, you may see
qw next to your job. This indicates that it is in the queue and waiting to be scheduled. If you see an
r next to your job then your job is running.
That said, it is often not clear what resources are missing that are preventing your job from being scheduled. Most often it is memory that is in short supply,
h_vmem. You may be able to increase your job's likelihood of being scheduled if it requires only few resources by reducing the job's memory requirements. For example:
$ qalter -l h_vmem=500M,h_rt=hh:mm:ss job_id
will reduce the virtual memory reserved for the job to 500 megabytes. You must re-supply the
h_rt and any other arguments to
-l when you use
qalter. The default values are listed here. Note that for parallel jobs, this
h_vmem request is per process. The scheduler will only start your job if it can find a host (or hosts) with enough memory unassigned to other jobs. You can determine the vmem available on various hosts with
$ qhost -F h_vmem
or you can see how many hosts have at least, say, 8 gigabytes free with
$ qhost -l h_vmem=8G
You can also try defining a short time limit for the job:
$ qalter -l h_rt=0:1:0,other args job_id
imposes a hard run-time limit of 0 hours, 1 minute, 0 seconds (0:1:0). In certain circumstances the scheduler will be able to schedule a job that it knows will finish quickly, where it cannot schedule a longer job.
qalter gotcha at Placentia: Following the simple example above can land your job in the subordinate queue where it is subject to being suspended. Now, if you want your job to start sooner this might be A Good Thing, but if you'd rather wait for a regular queue slot then here's how you avoid that:
-l resource request,
suspendable=true, is added to your job quietly by default.
qalter replaces the entire list of
-l arguments, so if you just want to change
h_rt you should first extract the resource list like this, and incorporate the modified but complete list into the
qalter command. For example:
$ qstat -j 1098765 | grep resource_list hard resource_list: test=false,suspendable=false,h_stack=10M,h_vmem=1G,h_rt=169200 $ qalter -l test=false,suspendable=false,h_stack=10M,h_vmem=800M,h_rt=12:0:0 1098765
As you can see,
suspendable is not the only resource request that gets quietly added to your job by default!
I need to change the order of my waiting jobs
You can shuffle the order of your own jobs with the "job share" option to
qsub. For example,
$ qalter -js 100 job_id
will boost the priority of the given job relative to the other jobs belonging to you. The change in priority may take 15 seconds or so to be reflected in qstat.
How to merge output files
Usually there are four output files generated by Grid Engine for a parallel job:
.*e files represent
stderr, and the
.*o files represent
stdout, while the
.p* files get generated when there is a parallel environment specified in a submission script.
Users may want to reduce that number of files to just one. Here is how. The
stdout streams can be merged with the following option in your submission script:
#$ -j y
This will yield two files
.po instead of four. You can merge these two further if you explicitly specify a name for the output file like so:
#$ -o output.log
The problem here is that if you submit several such jobs from the same directory, they will be writing to the same file, which is usually undesirable, so the advice would be to use a Grid Engine environment variable to set a unique file name like so:
#$ -o $JOB_ID.log
So, the recipe to get four files into one with a unique name is:
#$ -j y #$ -o $JOB_ID.log
Warning: No xauth data; using fake authentication data for X11 forwarding
~/.Xauthority file got corrupted. Please delete it manually, and next time you login with X11 forwarding enabled, it will get re-created.
I need to extend the run-time limit for a running job
You cannot alter parameters of a running job. If you are sure that your jobs will not be able to finish on time, then it might be better to terminate and then re-submit it again with a different run-time limit.
I need a lot of CPUs
You might find it necessary to turn on the job reservation option to qsub:
$ qsub -R y ...other parameters...
I need to run a quick test
You can run your job in the test queue only by asking for it explicitly like so:
$ qsub -l test=true,h_rt=0:0:10 ...other parameters...
Jobs in the test queue must also have run times of less than one hour (
I need to run jobs longer than a month
Check the conditions and caveats associated with Subordinate Queues to see if this solution might work for you. Otherwise, this requires special approval of the Resource Allocation Committee (RAC). Contact support first for advice on making an application to the RAC.