1. Documentation

Running Jobs in Torque

Status
Yes

To run jobs, you have to submit them to the queue. The queue controls allocation of the nodes so that you don't have to search for free nodes, etc.

Some simple commands to look at the queue:

  • qstat -a (will show you all jobs running in the queue)
  • showq (will show you what is in the queue, both running and waiting)

To run a job you need a simple script file which basically includes the number of nodes and cores needed and the name of your job. These are scripts written in C-shell or bash, and the directives for the queue are indicated by a line beginning with #PBS (so it's a comment, but the queue will read any line with a #PBS). A simple example is

#!/bin/bash
#PBS -l nodes=1:ppn=1
#  Make sure we're in right directory
cd $PBS_O_WORKDIR
sleep 10
/bin/hostname
sleep 20

which will ask for one node and one processing core (ppn is short for processor per node) It will sleep, print the hostname, sleep again and exit. This would be a simple serial job. You would run it with the command

qsub test.job

To run a parallel (MPI) job is slightly more complex

#!/bin/bash
############################################################################## 
# IMPORTANT:  the next line determines how many nodes to run on 
#  nodes is number of nodes, ppn= processors (cores) per node
#PBS -l nodes=2:ppn=4
# 
# Make sure that we are in the same subdirectory as where the qsub command 
# is issued. 
# 
cd $PBS_O_WORKDIR 
#
#  make a list of allocated nodes(cores)
#  Note that if multiple jobs run in same directory, use different names
#     for example, add on jobid nmber, e.g. nodes.$PBS_JOBID.
cat $PBS_NODEFILE > nodes

NODE_LIST=`cat $PBS_NODEFILE `
#
# Just for kicks, see which nodes we got.
echo $NODE_LIST
# 
# Run the executable. *DO NOT PUT* a '&' at the end!!
# $PBS_NP is how many cores your job allocated (i.e. nodes*ppn)
mpirun -np $PBS_NP -machinefile nodes ./pi3 >& log 
# 
#########################################

This script asks for 2 nodes with 4 cores/node.  There is a file (defined by $PBS_NODEFILE) that contains this list, and we need that list to give to the 'mpirun' command.  It also figures out how many total cores you've asked for (i.e. you don't have to remember to change the -np argument to 'mpirun' every time)

There are several other PBS directives that are helpful:

#PBS -l walltime=06:00:00 (asks for 6 hours of wall clock time). This is useful if you have shorter jobs because the queue can figure out how to run a shorter job sometimes by running them in between longer jobs.

#PBS -l medmem  (asks for medium memory nodes). The choices for RAM are

48 GB/node (is the default, i.e. if you don't specify any other mem)

64 GB/node (mem64)

#PBS -l nodes=1:ppn=12:mem64  (will allocate you an entire node with 64GB of RAM)

To run jobs that require even more memory, we have four nodes with more RAM - you can simply use

qsub -q largemem myfile.job

to select anyo f those nodes, or if you want more specific control, use the large mem queue and the following:

512GB/node (largemem) There are two nodes like this, with 32 cores/node

1 TBGB/node (xlargemem). There are two nodes like this, again with 32cores/node.

 

To run jobs that require GPUs:

 

qsub -q gpuq myfile.job