HPC Ground Rules

Oscar / CCV — shared compute etiquette for the group

Quick links: Cheat sheet · sbatch templates — see References below.

CCV's own monitoring staff already emails you directly if a job does something that threatens the cluster itself (overloading a login node, filling storage, etc.) — that backstop already exists and isn't ours to duplicate. These rules are the layer above that: about not quietly draining the group's shared allocation, and not surprising a labmate with a queue that's suddenly full because of a job they didn't know about. Rule of thumb: keep it small and you don't owe anyone a heads-up. Go bigger, and post in the CCV Slack channel first.

✅ Debugging with a small job? Nothing to report. Short walltime and a couple of GPUs is exactly what the cluster is for. Don't feel like you need to announce every test run — see Rule 1.

⚠️ Bigger job? Say so before you submit it. Past a certain size, your job can visibly delay or block someone else's — that's the one thing worth a 30-second Slack message. See Rule 2.

The rules

0. Never build or run on the login node

No make, no cmake --build, no test binaries, no scripts that actually do work. Configuring with cmake (no compilation) is fine — anything that spends real CPU is not. Compile and run only inside a SLURM allocation (sbatch/srun).

Why

The login node is shared by the entire cluster, not just the group. A heavy build or run there degrades it for everyone logged in at that moment, and it's the fastest way to get an account flagged by CCV directly — this is the one rule with essentially zero tolerance.

1. Small debug jobs are always fine — no report needed

Walltime under 20 min and ≤ 2 GPUs (or ≤ 16 CPUs)? Just run it. This is normal, expected cluster use — you are never "in trouble" for debugging at this scale, and there's no channel post to make.

Why

Most wasted-hour incidents don't come from small debug runs — they come from someone scaling up before they've confirmed the small case works. Explicitly carving out "small is always OK" means people don't hesitate to iterate, and it keeps the reporting rule below meaningful instead of noise.

2. Report before you submit anything bigger

Post in the CCV Slack channel before submitting if either is true:

• Walltime > 1 hour — always report, regardless of size.
• Walltime between 20 min and 1 hour and either > 2 GPUs or > 16 CPUs.

Why

A job at this scale can sit in the queue long enough, or hold enough of the shared allocation, to visibly delay someone else's work. A quick heads-up costs you nothing and means nobody is confused later about why the queue is backed up or the group's hours dropped — it turns a mystery into an expected event.

3. Estimate before you request

Know roughly how long the job should take, then set --time to about 2x that estimate — not the partition's max wall-clock "just in case."

Why

An inflated --time reserves resources you probably won't use, which can push you into "needs to be reported" territory for no reason, and makes scheduling harder for everyone else waiting on the same partition.

Recommendations

Good habits, not hard rules — nobody's reporting you to CCV for skipping these, but they're how you avoid becoming Rule 0 material.

A. Scope every job to what you actually need

Checking one thing? Run one thing. Don't launch a full sweep or benchmark suite "just to be safe" when you only need a single data point — pass the specific target as an argument. Don't request more GPUs/nodes than the job will use.

Why

Every hour spent on jobs you didn't need to run is an hour someone else in the group doesn't have. It's the single most direct way ordinary carelessness turns into hundreds of wasted hours.

B. Monitor your own jobs — don't fire-and-forget

Check back 1-2 minutes after submitting — most failed jobs fail within that window, so a quick squeue/sacct or log peek right away catches most problems for the price of a glance. Beyond that, check in periodically; if a job is running well past your estimate with no sign of progress, kill it (scancel) instead of letting it run out the clock.

Why

A silently hung or immediately-crashing job is the exact shape of the "400 hours for nothing" failure — the person just never checked back in. Most failures announce themselves almost immediately, so the highest-value check is the cheapest one: look again a minute or two after you hit submit.

C. Capture full logs

Redirect full output to the job's log (sbatch -o, or 2>&1 | tee job.log) — don't trim with head/tail/narrow grep while the job runs.

Why

If something goes wrong, you want the full record the first time. Re-running an expensive job just to recover a detail you threw away is wasted allocation for a problem you already solved once.

D. When in doubt, ask first

New to HPC and not sure if a command or job size is safe to run? Ask a labmate or the PI before submitting.

Why

A 30-second question is always cheaper than losing a weekend of shared allocation — and it's exactly what these rules exist to make the easy, default choice.

References

• Cheat sheet — the SLURM, log-watching, and Unix commands you'll actually use day to day, plus copy-paste one-liners for the "check back in 1-2 minutes" habit (Recommendation B).
• sbatch templates — ready-to-adapt job scripts, each with a DEBUG (Rule 1) and PRODUCTION (Rule 2) toggle built in, so the walltime/size thresholds above are already wired into the #SBATCH directives:
  • serial_cpu.sbatch — single task, single core, no MPI, no GPU.
  • serial_gpu.sbatch — single task, single GPU, no MPI.
  • parallel_cpu.sbatch — MPI across multiple CPU tasks, no GPU.
  • parallel_gpu.sbatch — MPI across multiple ranks, one GPU per rank.

---

Numbers here reflect current group agreement — update in place as thresholds change.