Skip to main content
U.S. flag

An official website of the United States government

Partitions or Queues

Compute jobs are run on functional groups of nodes called partitions or queues. Each different partition has different capabilities (e.g. regular memory versus high memory nodes) and resource restrictions (e.g. time limits on jobs). Nodes may appear in several partitions.

Some of the Ceres compute nodes have been purchased by individual researchers or research groups. These nodes are available to the owners in the priority* partitions but can also be used by anyone on the cluster through *-low and scavenger* partitions. These partitions have been introduced to increase usage of the priority nodes while still allowing node owners to have guaranteed fast access to priority nodes. All *-low partitions have 2-hour time limit. Scavenger* partitions have 3-weeks time limit, but jobs in this partition will be killed when resources are requested for the jobs in priority* partitions. Since jobs in the scavenger* partitions can be killed at any moment, running in those partitions does not affect job priorities in the community partitions.

The following table lists partitions. Number of nodes in a specific partition can be adjusted from time to time and be different from the one published in this document.

Community partitions

Name Nodes Logical Cores per Node Maximum Simulation Time Default Memory per Core Function
short 41 72,96 48 hours 3000 MB short simulation queue (default)
medium 32 72,96 7 days 3000 MB medium length simulation queue
long 11 72,96 21 days 3000 MB long simulation queue
long60 2 72 60 days 3000 MB extra long simulation queue
mem 4 80 7 days 16000 MB large memory queue
longmem 1 80 1000 hours 16000 MB long simulation large memory queue
mem768 1 80 7 days 7900 MB new node with 768GB of memory
debug 2 72,96 1 hour 3000 MB for testing scripts and runs before submitting them

Partitions that allow all users access to priority nodes

Name Nodes Logical Cores per Node Maximum Simulation Time Default Memory per Core Function
mem768-low 3 80 2 hours 7900 MB priority nodes with 768GB of memory
mem-low 16 80,96 2 hours 16000 MB priority nodes with 1.5TB of memory
gpu-low 1 72 2 hours 3000 MB priority GPU node
brief-low 92 72,96 2 hours 3000 MB all new nodes with 384GB of memory
scavenger 49 72,80 21 days 3000 MB non-GPU priority nodes; scavenger jobs can be killed at any moment
scavenger-gpu 1 72 21 days 3000 MB GPU priority node; jobs can be killed at any moment

Priority partitions available only to those users who purchased nodes

Name Nodes Maximum Simulation Time Default Memory per Core Function
priority 49 2 weeks 3000 MB priority nodes with 384GB memory
priority-mem 16 2 weeks 16000 MB priority nodes with 1.5TB memory
priority-mem768 3 2 weeks 7900 MB priority nodes with 768 GB memory
priority-gpu 1 2 weeks 3000 MB priority GPU node

At most 1440 cores and 5760 GB of memory can be used by all simultaneously running jobs per user across all community and *-low partitions. In addition, up to 800 cores and 2100 GB of memory can be used by jobs in scavenger* partitions. Any additional jobs will be queued but won’t start. At times these limits can be lowered to prevent a small group of users overtaking the whole cluster.

Users that have access to priority partitions are limited by the amount of resources purchased by the group. For example, if a group has purchased one 768GB node, then group members cannot use more than an equivalent of one 768GB node across all jobs simulteniously running in priority-mem768 partition even when there are idle nodes in the partition. However all users on the system can use these idle nodes through *-low and scavenger* partitions. Each group that has purchased nodes on Ceres, has a special QOS created for it. To list QOSes for your account, issue “sacctmgr -Pns show user format=qos”. The group’s QOS needs to be specified when submitting a job to a priority partition via the “-q” salloc/sbatch/srun option. When users submit a job to a priority partition, any node in the partition can be assigned to the job.

To get current details on all partitions use the following scontrol command:

$ scontrol show partitions