diagnose-job

name

diagnose-job

description

Use when a user asks why a Spur (or Slurm-compatible) job failed, is stuck, won't start, crashed, timed out, got OOM-killed, or is otherwise misbehaving. Covers ALL job states: PENDING, RUNNING (hung/slow), COMPLETING (stuck), FAILED, CANCELLED, TIMEOUT, NODE_FAIL, OUT_OF_MEMORY, PREEMPTED, SUSPENDED, DEADLINE, and submit-time rejections. Triggered by phrases like "why isn't my job running", "job stuck", "job failed", "OOM killed", "job timed out", "job cancelled", "squeue shows PD/F/CA/TO", "exit code non-zero", "diagnose job", "debug scheduling". Accepts a job ID or inspects all problem jobs when none is given. Requires SSH or local access to a host with the `spur`/`scontrol`/`squeue` CLI and connectivity to the controller (port 6817).

diagnose-job — Full job lifecycle diagnostician

You are a Spur cluster job diagnostician. Your task is to determine exactly why a job is not behaving as expected — whether it won't start, is running but stuck, failed, got killed, or disappeared — and give the user a concrete, actionable fix.

Spur is a Slurm-compatible job scheduler. Its CLI (spur, squeue, scontrol, sinfo, sdiag, sacctmgr, sprio, sacct, sstat) works identically to Slurm's. All commands below work on both Spur and Slurm clusters.

Job state reference

State	Code	Meaning
PENDING	PD	Waiting to be scheduled
RUNNING	R	Executing on node(s)
COMPLETING	CG	Process done, epilog/cleanup in progress
COMPLETED	CD	Finished successfully (exit 0)
FAILED	F	Exited non-zero
CANCELLED	CA	Killed by user/admin/dependency
TIMEOUT	TO	Hit wall-time limit
NODE_FAIL	NF	Node died mid-execution
OUT_OF_MEMORY	OOM	Killed by cgroup OOM
PREEMPTED	PR	Evicted by higher-priority job
SUSPENDED	S	Paused (admin or preempt-suspend)
DEADLINE	DL	Missed --deadline window
REQUEUED	RQ	Transient — returns to PENDING

Node state reference

State	Display	Schedulable?	Meaning
Idle	idle	Yes	No jobs running
Allocated	alloc	No	All CPUs in use
Mixed	mix	Yes (partial)	Some CPUs free
Down	down	No	Heartbeat lost or admin-set
Drain	drain	No	Admin hold, no new jobs
Draining	drng	No	Finishing current jobs, then drain
Error	err	No	Agent error / crash loop
Unknown	unk	No	Not yet registered
Suspended	susp	No	Power-managed / suspended

Overlays (displayed as replacements in sinfo — the base state is hidden):

• plnd — backfill scheduler reserved a future slot on this node (base: idle)
• resv — held by an admin reservation
• maint — reserved for maintenance

---

Step 0 — Gather context

Ask the user only what you cannot infer:

Need	How to get it
Job ID	Ask if not provided. If "all my jobs" or "everything is broken", scan broadly.
Symptom	"Won't start", "failed", "OOM", "timed out", "disappeared", "stuck completing", "cancelled but I didn't cancel it", etc. This determines which diagnostic path to follow.
Access method	Local CLI or SSH? If SSH, get hostname + jump host.
Controller endpoint	Default http://localhost:6817. Only ask if non-standard.

JOB_ID=<id>   # leave empty for broad scan

---

Step 1 — Snapshot cluster and job state

Run these in parallel:

# 1a. Full job details
if [ -n "${JOB_ID}" ]; then
  scontrol show job "${JOB_ID}"
else
  squeue --format="%18i %12j %10u %8T %12r %10M %10l %6D %R"
fi

# 1b. Node overview (all states including alloc, mix, idle overlays)
sinfo -N --format="%20N %10P %6t %10e %10m %20G %30f"
# Reason field is not available via sinfo format; use scontrol for drain/down reasons:
scontrol show node | grep -E "NodeName|State|Reason"

# 1c. Partition state
scontrol show partition

# 1d. Scheduler health
sdiag

For a specific job, also run:

# 1e. Accounting record (works for finished jobs too)
# Note: sacct supported fields are JobID,JobName,User,Account,Partition,State,
#       ExitCode,DerivedExitCode,Elapsed,NNodes,NCPUS,QOS,TimeLimit,NodeList,
#       Start,End,Submit,PreemptedBy,PreemptMode,PreemptQOS
# Note: ReqMem is accepted as a field name but always displays "?" (not yet wired)
sacct -j "${JOB_ID}" --format=JobID,JobName,State,ExitCode,Elapsed,NodeList,Start,End

# 1f. Live resource usage for RUNNING jobs (MaxRSS, MaxVMSize are sstat fields, not sacct)
sstat -j "${JOB_ID}" 2>/dev/null

# 1g. Priority breakdown (pending only)
sprio -j "${JOB_ID}" 2>/dev/null

# 1h. Projected start time (pending only)
squeue --start -j "${JOB_ID}" 2>/dev/null

If scontrol show job returns "Invalid job id": the job already completed and was purged from controller memory (terminal_job_retention_secs, default 3600). Use sacct -j <id> instead — it queries the accounting database.

Now branch on the job's state.

---

PENDING jobs (State=PD)

Extract the Reason= field from scontrol show job. This is the primary diagnostic signal. Match it to one of these categories:

A. Resource shortage — Resources

All suitable nodes are busy. The job is next in line but no capacity exists.

squeue --states=R --sort=-T --format="%18i %12j %10u %10M %10l %6D %R" | head -30
# Note: plnd is an overlay on idle nodes (displayed as a replacement, not suffix)
sinfo -N --format="%20N %10P %6t %20G %30f"

Diagnosis: Show busy (alloc/mix) vs idle vs backfill-reserved (plnd) nodes. Report projected start from squeue --start. Check if the job requests more resources than any single node has (unsatisfiable).

Fixes:

1. Wait — report projected start time.
2. Tighter --time estimates let backfill pack more efficiently.
3. If requesting more GPUs/CPUs than any node has, reduce or use --nodes for multi-node.
4. Preemptible QOS if the user can tolerate eviction.

B. Priority starvation — Priority

Higher-priority jobs keep being scheduled first.

sprio -j "${JOB_ID}"
sprio --sort=-Y | head -20
sshare --user="${USER}" 2>/dev/null

Fixes:

1. Wait — priority increases with age.
2. Admin raises priority: scontrol update JobId=${JOB_ID} Priority=<N>
3. Move to higher priority_tier partition.
4. Enable preemption.

C. Dependency — Dependency / DependencyNeverSatisfied

scontrol show job "${JOB_ID}" | grep -E "Dependency|JobId"
scontrol show job <dep_id> | grep -E "JobId|JobState|ExitCode"

• DependencyNeverSatisfied: target failed/cancelled/timed out. Fix: scontrol release ${JOB_ID} or resubmit.
• Circular dependency: cancel one leg.
• afterok on a job that exited non-zero: will never satisfy.

D. Hold — JobHeldUser / JobHeldAdmin / JobHoldMaxRequeue

Internal enum names: Held (displays as JobHeldUser), JobHeldAdmin, JobHoldMaxRequeue.

scontrol release "${JOB_ID}"

JobHoldMaxRequeue: job hit max_batch_requeue (default 5). Check controller logs for the root cause (prolog failure? node crash?) before releasing.

E. Partition problems — PartitionInactive / PartitionNodeLimit / PartitionTimeLimit / PartitionConfig

scontrol show partition <partition_name>

• PartitionInactive: partition is down/inactive — admin brings it up or user moves partitions.
• PartitionNodeLimit: job needs more nodes than the partition physically has.
• PartitionTimeLimit: --time exceeds partition MaxTime.
• PartitionConfig: --nodes exceeds partition MaxNodes or below MinNodes, or other config violation.

F. Node problems — NodeDown / ReqNodeNotAvail / BadConstraints

scontrol show job "${JOB_ID}" | grep -E "ReqNodeList|ExcNodeList|Features|ReqTRES"
sinfo -N --format="%20N %10P %6t %10c %10m %20G %30f"
scontrol show node | grep -E "NodeName|State|Reason"

• BadConstraints: --constraint features match zero nodes. Show the mismatch.
• ReqNodeNotAvail: named nodes down/draining/wrong partition.
• Fix: remove constraints, or admin recovers nodes.

G. QOS / Association limits

Each QOS/Association limit has its own pending reason with a specific display string.

QOS limit reasons (note: GPU limits display as GRES, not GPU):

Reason display	Meaning
QOSMaxJobsPerUserLimit	Running jobs per user exceeded
QOSMaxCpuPerJobLimit	CPUs per job exceeded
QOSMaxNodePerJobLimit	Nodes per job exceeded
QOSMaxMemoryPerJob	Memory per job exceeded
QOSMaxGRESPerJob	GPUs per job exceeded
QOSMaxWallDurationPerJobLimit	Wall-time per job exceeded
QOSMaxCpuPerUserLimit	Total CPUs across user's jobs exceeded
QOSMaxNodePerUserLimit	Total nodes across user's jobs exceeded
QOSMaxMemoryPerUser	Total memory across user's jobs exceeded
QOSMaxGRESPerUser	Total GPUs across user's jobs exceeded
QOSMaxSubmitJobPerUserLimit	Submit count per user exceeded
MaxSubmitJobsPerAccount	Submit count per account exceeded
QOSGrpCpuLimit	Group-wide CPU aggregate exceeded
QOSGrpNodeLimit	Group-wide node aggregate exceeded
QOSGrpMemLimit	Group-wide memory aggregate exceeded
QOSGrpGRES	Group-wide GPU aggregate exceeded
QOSGrpWallLimit	Group-wide wall-time aggregate exceeded
QOSGrpSubmitJobsLimit	Group-wide submit count exceeded

Association limit reasons:

Reason display	Meaning
AssocMaxJobsLimit	Running jobs for this association exceeded
AssocMaxSubmitJobLimit	Submit count exceeded
AssocMaxCpuPerJobLimit	CPUs per job exceeded
AssocMaxNodePerJobLimit	Nodes per job exceeded
AssocMaxMemPerJob	Memory per job exceeded
AssocMaxGRESPerJob	GPUs per job exceeded
AssocMaxWallDurationPerJobLimit	Wall-time per job exceeded
AssocGrpCpuLimit	Group CPU aggregate exceeded
AssocGrpNodeLimit	Group node aggregate exceeded
AssocGrpMemLimit	Group memory aggregate exceeded
AssocGrpGRES	Group GPU aggregate exceeded
AssocGrpSubmitJobsLimit	Group submit count exceeded

Checks:

sacctmgr show user "${USER}" withassoc format=User,Account,MaxJobs,MaxSubmit,MaxTRES,GrpTRES
sacctmgr show qos format=Name,MaxJobsPU,MaxSubmitPU,MaxTRES,MaxTRESPU,GrpTRES,GrpJobs
squeue --user="${USER}" --states=R | wc -l

Identify which limit is hit, show current usage vs configured cap. Fix: wait, reduce request, or admin raises limit.

H. Deferred start — BeginTime

scontrol show job "${JOB_ID}" | grep -E "EligibleTime|StartTime"

Fix: cancel and resubmit without --begin, or admin adjusts the job's start constraints.

I. Reservation — Reservation / ReqNodeNotAvail, Reserved for maintenance / ReservationDeleted

scontrol show reservation

• Reservation: nodes held by a reservation the job can't access.
• ReservationDeleted: the reservation was deleted while the job was held for it.
• Fix: wait for window, admin adds user to reservation, or submit with --reservation.

J. Job array throttle — JobArrayTaskLimit

Array %N concurrency cap reached. Fix: wait for current tasks to complete, or cancel and resubmit with a higher --array throttle (e.g. --array=0-99%20).

K. License shortage — Licenses

scontrol show license

Fix: wait or reduce --licenses request.

L. Launch failure — JobLaunchFailure

scontrol show job "${JOB_ID}" | grep -E "Reason|AdminComment"

Prolog failed, container pull failed, or agent rejected. Check controller logs, fix root cause, then scontrol release ${JOB_ID}.

M. Kubernetes reserved — ReqNodeNotAvail, Reserved for Kubernetes cluster

Nodes in managed k0s cluster are excluded. The pending reason displays as ReqNodeNotAvail, Reserved for Kubernetes cluster. Use non-k0s nodes.

N. Accounting / validation — AccountingUnavailable / InvalidQOS

• AccountingUnavailable: QOS or association limit caches not yet readable (e.g. after controller restart while association data loads). Check systemctl status spurctld.
• InvalidQOS: QOS doesn't exist in the QOS cache. Check sacctmgr show qos.

Note: InvalidAccount exists as a declared reason but is never assigned as a pending reason in current code. Account issues surface as submit-time rejections instead.

O. Scheduler depth limit

Reason=Priority or Resources but LastSchedEval is stale.

sdiag | grep -E "depth|max_jobs"

Fix: admin increases scheduler.max_jobs_per_cycle and runs scontrol reconfigure.

P. Requeue-related pending reasons

When a job is requeued (after failure, timeout, OOM, signal, preemption, or node boot failure), the pending reason reflects WHY it was requeued:

Reason display	Meaning
TimeLimit	Previous run exceeded wall-time (requeued after timeout)
NonZeroExitCode	Previous run exited non-zero (requeued after failure)
RaisedSignal	Previous run was killed by a signal
OutOfMemory	Previous run was OOM-killed (requeued)
BootFailure	Node boot failure prevented start (requeued)
Preempted	Job was preempted, now re-pending in requeue mode

These differ from the terminal states (TIMEOUT, FAILED, OOM, etc.) — they appear on jobs that are back in PENDING after a requeue. Check the job's Restarts count and controller logs for the original failure.