Cedar cluster

Cedar (GP2) is a heterogeneous HPC cluster located at Simon Fraser University and operated by WestGrid. It is one of several new Compute Canada national systems.

• Cedar - CC Doc

Nodes

There are seven kinds of CPU nodes:

The first four types are in the base partitions, and the last three are in the large partitions. The memory amounts are nominal, with the actual usable space potentially being a bit less.

There are also 338 GPU nodes with all sorts of different specs, having P100 or V100 cards.

All of the processors in the cluster are Xeons, clocked anywhere from 2.1 GHz to 2.4 GHz, and spanning several architectures.

Partitions

There are five special CPU partitions: cpubase_interac, cpularge_interac, cpubackfill, c12hbackfill, and cpupreempt. The first two are for interactive jobs (maximum time: 3 hours). The next two are automatically added to the list of partitions for jobs not exceeding 24 or 12 hours, respectively, and are used for backfilling lower priority short jobs into otherwise wasted slots. The cpubackfill partition includes some of every kind of CPU node, so even jobs requiring huge amounts of memory can get backfilled, and c12hbackfill even includes contributed nodes. The last of the special partitions has a time limit of 122 days (4 months), but must be requested manually and has PreemptMode=REQUEUE, so jobs from other partitions allocated to the same hardware will cause jobs on this partition to be restarted.

The other 24 CPU partitions are of the form cpuX_Y_bZ, where X is either base or large, Y is either bycore or bynode, and Z is a digit between 1 and 6; for example, cpularge_bycore_b4. The difference between bycore and bynode is whether the entire node is allocated to the job; this happens when the job is submitted as exclusive, or otherwise depending on the core and memory requirements (gory details in /etc/slurm/job_submit.lua).

The base memory bucket is used by default (88,992 cores). If the memory requirements call for it, the large bucket is used instead (1,664 cores).

The time buckets are much more straightforward:

Every job goes into the smallest time bucket that is large enough to contain it.

The total number of cores in each partition is as follows:

Here's a shell one-liner to see the number of free cores in each partition:

1

for x in base large; do for y in core node; do for z in {1..6}; do partition="cpu${x}_by${y}_b${z}"; echo -n "$partition "; sinfo --noheader -o '%C' -p "$partition" | tr / ' ' | awk '{print $2, $4}'; done; done; done | column -t

It's important not to run this too frequently to avoid further stressing the scheduler, which is (seemingly permanently) under heavy load.

Examples

Some concrete examples of where jobs end up based on their parameters:

See also

• Graham cluster