RCSG and CITI Deploying Enhanced Job Scheduler

Introduction

On January 31, 2007, the PBS scheduler on Ada will be replaced by Torque/Moab which will significantly enhance our capability to support user needs and manage QoS (Quality of Service). The new system includes a replacement for PBS called Torque for the resource manager, while the job scheduler will be replaced by a product called Moab (before July 2006 we had Maui). The Torque/Moab combination will replace and significantly enhance the functionality that are currently being offered with PBS. To ease any concerns you have regarding this change, PBS commands (such as qsub, qdel, qstat, and so on) will still work as usual. There are, however, a few notable exceptions, particularly for certain types of job submissions. Below is a list of the more notable changes.

showq, checkjob, and showstart Commands Available Once Again

The preferred way to check the status of jobs will be by using the showq and checkjob commands. These commands are provided by Moab and will reflect the position of a job in the queue (whether it is running, waiting to run, or blocked). Simply running the showq command will produce a detailed list of all jobs in the queue. Running the checkjob command followed by the job ID number will show a detailed status of the particular job. Running the showstart command followed by the job ID number will show the projected start time of a particular job. It is strongly recommended that you use these commands instead of qstat to check on the status of your jobs. Note that when a job is submitted, Torque will pass the job to Moab for scheduling. This process may take a few seconds. During these few seconds the new job will be visible with qstat but not with the showq, checkjob, and showstart commands. Please be patient until Moab assigns a position for your job in the queue.

Job Arrays No Longer Available

One feature of PBS that we know a few of you have used extensively is Job Arrays. The Moab/Torque installation does not offer Job Arrays at this time. Users who need to submit large numbers of jobs will need to rewrite their PBS batch scripts to submit jobs individually rather than as jobs within an array. Rather than having a single job submitted with an array of subjobs, users will now need to submit multiple single jobs. We plan on reintroducing Job Arrays during a future upgrade of Moab/Torque.

Node Request Method (select=X option is invalid)

With the PBS scheduler users were encouraged to use the select=X option in the PBS batch script to select the number of CPUs that a job needed. This allowed the user to scatter their jobs across the cluster rather than having to wait on entire nodes to be available. The scheduler would find the first X processors available regardless of which nodes they were on. This option will no longer work. The standard method will be nodes=X:ppn=Y. A user needing 20 processors, for example, would use either nodes=5:ppn=4 or nodes=10:ppn=2. The first choice would mean that the user would have exclusive access to all four processors on a node and all of the node's resources (memory, disk, etc.). The latter would mean that the user might share 2 processors on the node with someone else. So care should be taken when selecting processors per node (ppn).

Exclusive Access to Nodes

It is possible to request exclusive access to a node so that you will not share it with anyone else regardless of how many processors (i.e. independent of the value of ppn) you need on that node. This might be important for memory or disk intensive jobs where interference from other jobs would impact performance. There are two methods that can be used to gain exclusive access:

• Use the following lines in your PBS batch script:

#PBS -l nodes=1:ppn=1 #PBS -W x=NACCESSPOLICY:SINGLEJOB

This combination of options will give the user one node, and only one processor (ppn), but will have exclusive access to the entire node. The remaining 3 processors will be unused. Note that the showq command will show your job in the queue as having been assigned 4 processors, even though only one will be used.

• Request all four processors and then use mpiexec with the -n 1 option in your PBS batch script as follows:

#PBS -l nodes=1:ppn=4 mpiexec -n 1 $XD1LAUNCHER myprogram #run your program with mpiexec

This combination of options will give the user one node and all four processors (thus exclusive access) but will specify that mpiexec should use only one processor.

Improved Job Scheduling!

The primary motivation for changing to a new scheduling system is to enhance the performance and capabilities of the scheduler, ultimately providing better support for our users. The new system will permit us to reinstate the queue policy and should (everything else being constant) result in shorter wait times for large parallel jobs. In fact, the new system will (as per policy) assign a higher priority to jobs that request large numbers of processors, therefore favoring parallel jobs over single-CPU jobs. Additionally, job priorities will work as expected with users running large numbers of jobs receiving a lower priority over time (fairshare policy). Furthermore, the new system supports a mechanism that can be used to backfill short and small jobs such that these jobs can use empty nodes so long as they do not prevent jobs of higher priority from running when it is their turn (backfill policy). So it is possible, in fact very likely, that jobs will be running out of order. It is also anticipated that the improved scheduling system will eliminate the need for the select=X option and Job Arrays, as both were being used to bypass deficiencies in the old scheduler.

Getting Help

As with any major software change there are certain to be unforseen issues and questions. Some tuning of the system will likely be required once a full load of jobs is introduced. If you discover that something does not appear to be working as you expected, please check our FAQ for topics related to your questions. If the FAQs do not help resolve your problem, please notify the Help Desk. Please be sure to provide us with the job ID number and location of your PBS batch script so we can help debug the problem.