Graham cluster

Graham (GP3) is a heterogeneous HPC cluster located at the University of Waterloo and operated by SHARCNET. It is one of several new Compute Canada national systems.

• Graham - CC Doc
• SHARCNET: Cluster graham.sharcnet.ca

Nodes

There are five kinds of CPU nodes:

The first three types are in the base partitions, and the last two are in the large partitions. The memory amounts are nominal, with the actual usable space potentially being a bit less.

There are also 203 GPU nodes with all sorts of different specs, having P100, V100, or T4 cards.

All of the processors in the cluster are Xeons and most (except in a handful of GPU nodes) are clocked at 2.1 GHz, but they have different architectures.

Partitions

There are three special CPU partitions: cpubase_interac, cpularge_interac, and cpubackfill. The first two are for interactive jobs (maximum time: 3 hours). The last one is automatically added to the list of partitions for jobs not exceeding 24 hours and is used for backfilling lower priority short jobs into otherwise wasted slots. The cpubackfill partition includes some of every kind of CPU node, so even jobs requiring huge amounts of memory can get backfilled.

The other 24 CPU partitions are of the form cpuX_Y_bZ, where X is either base or large, Y is either bycore or bynode, and Z is a digit between 1 and 6; for example, cpularge_bycore_b4. The difference between bycore and bynode is whether the entire node is allocated to the job; this happens when the job is submitted as exclusive, or otherwise depending on the core and memory requirements (gory details in /etc/slurm/job_submit.lua).

The base memory bucket is used by default (33,856 cores). If the memory requirements call for it, the large bucket is used instead (960 cores).

The time buckets are much more straightforward:

Every job goes into the smallest time bucket that is large enough to contain it. No partition allows jobs longer than 4 weeks.

The total number of cores in each partition is as follows:

Here's a shell one-liner to see the number of free cores in each partition:

1

for x in base large; do for y in core node; do for z in {1..6}; do partition="cpu${x}_by${y}_b${z}"; echo -n "$partition "; sinfo --noheader -o '%C' -p "$partition" | tr / ' ' | awk '{print $2, $4}'; done; done; done | column -t

It's important not to run this too frequently to avoid further stressing the scheduler, which is (seemingly permanently) under heavy load.

Examples

Some concrete examples of where jobs end up based on their parameters:

See also

• Cedar cluster