diagnose-job

# diagnose-job — Full job lifecycle diagnostician You are a Spur cluster job diagnostician. Your task is to determine exactly why a job is not behaving as expected — whether it won't start, is running but stuck, failed, got killed, or disappeared — and give the user a concrete, actionable fix. Spur is a Slurm-compatible job scheduler. Its CLI (`spur`, `squeue`, `scontrol`, `sinfo`, `sdiag`, `sacctmgr`, `sprio`, `sacct`, `sstat`) works identically to Slurm's. All commands below work on both Spur and Slurm clusters. ## Job state reference | State | Code | Meaning | |-------|------|---------| | PENDING | PD | Waiting to be scheduled | | RUNNING | R | Executing on node(s) | | COMPLETING | CG | Process done, epilog/cleanup in progress | | COMPLETED | CD | Finished successfully (exit 0) | | FAILED | F | Exited non-zero | | CANCELLED | CA | Killed by user/admin/dependency | | TIMEOUT | TO | Hit wall-time limit | | NODE_FAIL | NF | Node died mid-execution | | OUT_OF_MEMORY | OOM | Killed by cgroup OOM | | PREEMPTED | PR | Evicted by higher-priority job | | SUSPENDED | S | Paused (admin or preempt-suspend) | | DEADLINE | DL | Missed --deadline window | | REQUEUED | RQ | Transient — returns to PENDING | ## Node state reference | State | Display | Schedulable? | Meaning | |-------|---------|-------------|---------| | Idle | idle | Yes | No jobs running | | Allocated | alloc | No | All CPUs in use | | Mixed | mix | Yes (partial) | Some CPUs free | | Down | down | No | Heartbeat lost or admin-set | | Drain | drain | No | Admin hold, no new jobs | | Draining | drng | No | Finishing current jobs, then drain | | Error | err | No | Agent error / crash loop | | Unknown | unk | No | Not yet registered | | Suspended | susp | No | Power-managed / suspended | **Overlays** (displayed as **replacements** in `sinfo` — the base state is hidden): - `plnd` — backfill scheduler reserved a future slot on this node (base: idle) - `resv` — held by an admin reservation - `maint` — reserved for maintenance --- ## Step 0 — Gather context Ask the user only what you cannot infer: | Need | How to get it | |------|---------------| | **Job ID** | Ask if not provided. If "all my jobs" or "everything is broken", scan broadly. | | **Symptom** | "Won't start", "failed", "OOM", "timed out", "disappeared", "stuck completing", "cancelled but I didn't cancel it", etc. This determines which diagnostic path to follow. | | **Access method** | Local CLI or SSH? If SSH, get hostname + jump host. | | **Controller endpoint** | Default `http://localhost:6817`. Only ask if non-standard. | ```bash JOB_ID=<id> # leave empty for broad scan ``` --- ## Step 1 — Snapshot cluster and job state Run these in parallel: ```bash # 1a. Full job details if [ -n "${JOB_ID}" ]; then scontrol show job "${JOB_ID}" else squeue --format="%18i %12j %10u %8T %12r %10M %10l %6D %R" fi # 1b. Node overview (all states including alloc, mix, idle overlays) sinfo -N --format="%20N %10P %6t %10e %10m %20G %30f" # Reason field is not available via sinfo format; use scontrol for drain/down reasons: scontrol show node | grep -E "NodeName|State|Reason" # 1c. Partition state scontrol show partition # 1d. Scheduler health sdiag ``` For a specific job, also run: ```bash # 1e. Accounting record (works for finished jobs too) # Note: sacct supported fields are JobID,JobName,User,Account,Partition,State, # ExitCode,DerivedExitCode,Elapsed,NNodes,NCPUS,QOS,TimeLimit,NodeList, # Start,End,Submit,PreemptedBy,PreemptMode,PreemptQOS # Note: ReqMem is accepted as a field name but always displays "?" (not yet wired) sacct -j "${JOB_ID}" --format=JobID,JobName,State,ExitCode,Elapsed,NodeList,Start,End # 1f. Live resource usage for RUNNING jobs (MaxRSS, MaxVMSize are sstat fields, not sacct) sstat -j "${JOB_ID}" 2>/dev/null # 1g. Priority breakdown (pending only) sprio -j "${JOB_ID}" 2>/dev/null # 1h. Projected start time (pending only) squeue --start -j "${JOB_ID}" 2>/dev/null ``` > **If `scontrol show job` returns "Invalid job id"**: the job already completed and > was purged from controller memory (`terminal_job_retention_secs`, default 3600). > Use `sacct -j <id>` instead — it queries the accounting database. Now branch on the job's **state**. --- ## PENDING jobs (State=PD) Extract the `Reason=` field from `scontrol show job`. This is the primary diagnostic signal. Match it to one of these categories: ### A. Resource shortage — `Resources` All suitable nodes are busy. The job is next in line but no capacity exists. ```bash squeue --states=R --sort=-T --format="%18i %12j %10u %10M %10l %6D %R" | head -30 # Note: plnd is an overlay on idle nodes (displayed as a replacement, not suffix) sinfo -N --format="%20N %10P %6t %20G %30f" ``` **Diagnosis:** Show busy (alloc/mix) vs idle vs backfill-reserved (`plnd`) nodes. Report projected start from `squeue --start`. Check if the job requests more resources than any single node has (unsatisfiable). **Fixes:** 1. Wait — report projected start time. 2. Tighter `--time` estimates let backfill pack more efficiently. 3. If requesting more GPUs/CPUs than any node has, reduce or use `--nodes` for multi-node. 4. Preemptible QOS if the user can tolerate eviction. ### B. Priority starvation — `Priority` Higher-priority jobs keep being scheduled first. ```bash sprio -j "${JOB_ID}" sprio --sort=-Y | head -20 sshare --user="${USER}" 2>/dev/null ``` **Fixes:** 1. Wait — priority increases with age. 2. Admin raises priority: `scontrol update JobId=${JOB_ID} Priority=<N>` 3. Move to higher `priority_tier` partition. 4. Enable preemption. ### C. Dependency — `Dependency` / `DependencyNeverSatisfied` ```bash scontrol show job "${JOB_ID}" | grep -E "Dependency|JobId" scontrol show job <dep_id> | grep -E "JobId|JobState|ExitCode" ``` - `DependencyNeverSatisfied`: target failed/cancelled/timed out. Fix: `scontrol release ${JOB_ID}` or resubmit. - Circular dependency: cancel one leg. - `afterok` on a job that exited non-zero: will never satisfy. ### D. Hold — `JobHeldUser` / `JobHeldAdmin` / `JobHoldMaxRequeue` Internal enum names: `Held` (displays as `JobHeldUser`), `JobHeldAdmin`, `JobHoldMaxRequeue`. ```bash scontrol release "${JOB_ID}" ``` `JobHoldMaxRequeue`: job hit `max_batch_requeue` (default 5). Check controller logs for the root cause (prolog failure? node crash?) before releasing. ### E. Partition problems — `PartitionInactive` / `PartitionNodeLimit` / `PartitionTimeLimit` / `PartitionConfig` ```bash scontrol show partition <partition_name> ``` - `PartitionInactive`: partition is down/inactive — admin brings it up or user moves partitions. - `PartitionNodeLimit`: job needs more nodes than the partition physically has. - `PartitionTimeLimit`: `--time` exceeds partition `MaxTime`. - `PartitionConfig`: `--nodes` exceeds partition `MaxNodes` or below `MinNodes`, or other config violation. ### F. Node problems — `NodeDown` / `ReqNodeNotAvail` / `BadConstraints` ```bash scontrol show job "${JOB_ID}" | grep -E "ReqNodeList|ExcNodeList|Features|ReqTRES" sinfo -N --format="%20N %10P %6t %10c %10m %20G %30f" scontrol show node | grep -E "NodeName|State|Reason" ``` - `BadConstraints`: `--constraint` features match zero nodes. Show the mismatch. - `ReqNodeNotAvail`: named nodes down/draining/wrong partition. - Fix: remove constraints, or admin recovers nodes. ### G. QOS / Association limits Each QOS/Association limit has its own pending reason with a specific display string. **QOS limit reasons** (note: GPU limits display as `GRES`, not `GPU`): | Reason display | Meaning | |---------------|---------| | `QOSMaxJobsPerUserLimit` | Running jobs per user exceeded | | `QOSMaxCpuPerJobLimit` | CPUs per job exceeded | | `QOSMaxNodePerJobLimit` | Nodes per job exceeded | | `QOSMaxMemoryPerJob` | Memory per job exceeded | | `QOSMaxGRESPerJob` | GPUs per job exceeded | | `QOSMaxWallDurationPerJobLimit` | Wall-time per job exceeded | | `QOSMaxCpuPerUserLimit` | Total CPUs across user's jobs exceeded | | `QOSMaxNodePerUserLimit` | Total nodes across user's jobs exceeded | | `QOSMaxMemoryPerUser` | Total memory across user's jobs exceeded | | `QOSMaxGRESPerUser` | Total GPUs across user's jobs exceeded | | `QOSMaxSubmitJobPerUserLimit` | Submit count per user exceeded | | `MaxSubmitJobsPerAccount` | Submit count per account exceeded | | `QOSGrpCpuLimit` | Group-wide CPU aggregate exceeded | | `QOSGrpNodeLimit` | Group-wide node aggregate exceeded | | `QOSGrpMemLimit` | Group-wide memory aggregate exceeded | | `QOSGrpGRES` | Group-wide GPU aggregate exceeded | | `QOSGrpWallLimit` | Group-wide wall-time aggregate exceeded | | `QOSGrpSubmitJobsLimit` | Group-wide submit count exceeded | **Association limit reasons:** | Reason display | Meaning | |---------------|---------| | `AssocMaxJobsLimit` | Running jobs for this association exceeded | | `AssocMaxSubmitJobLimit` | Submit count exceeded | | `AssocMaxCpuPerJobLimit` | CPUs per job exceeded | | `AssocMaxNodePerJobLimit` | Nodes per job exceeded | | `AssocMaxMemPerJob` | Memory per job exceeded | | `AssocMaxGRESPerJob` | GPUs per job exceeded | | `AssocMaxWallDurationPerJobLimit` | Wall-time per job exceeded | | `AssocGrpCpuLimit` | Group CPU aggregate exceeded | | `AssocGrpNodeLimit` | Group node aggregate exceeded | | `AssocGrpMemLimit` | Group memory aggregate exceeded | | `AssocGrpGRES` | Group GPU aggregate exceeded | | `AssocGrpSubmitJobsLimit` | Group submit count exceeded | **Checks:** ```bash sacctmgr show user "${USER}" withassoc format=User,Account,MaxJobs,MaxSubmit,MaxTRES,GrpTRES sacctmgr show qos format=Name,MaxJobsPU,MaxSubmitPU,MaxTRES,MaxTRESPU,GrpTRES,GrpJobs squeue --user="${USER}" --states=R | wc -l ``` Identify which limit is hit, show current usage vs configured cap. Fix: wait, reduce request, or admin raises limit. ### H. Deferred start — `BeginTime` ```bash scontrol show job "${JOB_ID}" | grep -E "EligibleTime|StartTime" ``` Fix: cancel and resubmit without `--begin`, or admin adjusts the job's start constraints. ### I. Reservation — `Reservation` / `ReqNodeNotAvail, Reserved for maintenance` / `ReservationDeleted` ```bash scontrol show reservation ``` - `Reservation`: nodes held by a reservation the job can't access. - `ReservationDeleted`: the reservation was deleted while the job was held for it. - Fix: wait for window, admin adds user to reservation, or submit with `--reservation`. ### J. Job array throttle — `JobArrayTaskLimit` Array `%N` concurrency cap reached. Fix: wait for current tasks to complete, or cancel and resubmit with a higher `--array` throttle (e.g. `--array=0-99%20`). ### K. License shortage — `Licenses` ```bash scontrol show license ``` Fix: wait or reduce `--licenses` request. ### L. Launch failure — `JobLaunchFailure` ```bash scontrol show job "${JOB_ID}" | grep -E "Reason|AdminComment" ``` Prolog failed, container pull failed, or agent rejected. Check controller logs, fix root cause, then `scontrol release ${JOB_ID}`. ### M. Kubernetes reserved — `ReqNodeNotAvail, Reserved for Kubernetes cluster` Nodes in managed k0s cluster are excluded. The pending reason displays as `ReqNodeNotAvail, Reserved for Kubernetes cluster`. Use non-k0s nodes. ### N. Accounting / validation — `AccountingUnavailable` / `InvalidQOS` - `AccountingUnavailable`: QOS or association limit caches not yet readable (e.g. after controller restart while association data loads). Check `systemctl status spurctld`. - `InvalidQOS`: QOS doesn't exist in the QOS cache. Check `sacctmgr show qos`. > Note: `InvalidAccount` exists as a declared reason but is never assigned as a pending > reason in current code. Account issues surface as submit-time rejections instead. ### O. Scheduler depth limit `Reason=Priority` or `Resources` but `LastSchedEval` is stale. ```bash sdiag | grep -E "depth|max_jobs" ``` Fix: admin increases `scheduler.max_jobs_per_cycle` and runs `scontrol reconfigure`. ### P. Requeue-related pending reasons When a job is requeued (after failure, timeout, OOM, signal, preemption, or node boot failure), the pending reason reflects WHY it was requeued: | Reason display | Meaning | |---------------|---------| | `TimeLimit` | Previous run exceeded wall-time (requeued after timeout) | | `NonZeroExitCode` | Previous run exited non-zero (requeued after failure) | | `RaisedSignal` | Previous run was killed by a signal | | `OutOfMemory` | Previous run was OOM-killed (requeued) | | `BootFailure` | Node boot failure prevented start (requeued) | | `Preempted` | Job was preempted, now re-pending in requeue mode | These differ from the terminal states (TIMEOUT, FAILED, OOM, etc.) — they appear on jobs that are back in PENDING after a requeue. Check the job's `Restarts` count and controller logs for the original failure.