job-monitor

name

job-monitor

description

Monitor SLURM/PBS/LSF jobs, tail logs live, detect errors immediately as they appear in log output, cancel the job on fatal error so the user can iterate fast, and report status. Invoke when the user asks to "check my jobs", "monitor", "what's running", "did it finish", "job status", "check logs", "is it done", or after any sbatch/qsub submission. Also invoke proactively after every job submission — the default behavior is to immediately start watching logs for errors.

Job Monitor

Track compute jobs from submission through completion with a focus on fast failure detection. The goal is minimal round-trips: catch errors as soon as they hit the log, kill the job, and report so the code can be fixed.

Workflow

1. Immediate post-submission check

After every sbatch / qsub, immediately:

squeue -u $USER --format="%i %j %T %M %l" 2>/dev/null

Wait ~5 seconds for the scheduler to create the log file, then tail the .err and .out for the first signs of trouble:

# Wait for log files to appear
for i in $(seq 1 6); do
    if [ -f "logs/<jobname>_$JOBID.err" ]; then break; fi
    sleep 2
done

2. Live error scan (primary workflow)

While the job is RUNNING, repeatedly scan the error log for Python tracebacks shell/fatal errors. The moment one appears:

# Check .err log for tracebacks or fatal errors
grep -c "Traceback\|Error:\|error:\|FATAL\|ModuleNotFoundError\|ImportError\|AttributeError\|NameError\|SyntaxError\|FileNotFoundError" logs/<jobname>_$JOBID.err 2>/dev/null

If any match found:

1. Immediately cancel the job: scancel $JOBID
2. Read the full traceback from the .err file
3. Report the exact error to the user so they can fix and resubmit
4. Show the last few lines of .out for context

Do NOT wait for the scheduler to mark the job as FAILED. The scheduler may not notice for minutes. A Python AttributeError kills the script instantly — cancel and report right away.

3. When to poll vs when to tail

• First 30 seconds after submission: Poll every 3 seconds — most fatal errors (imports, syntax, path issues) happen immediately
• After 30 seconds with no error: Switch to checking every 10 seconds while showing progress from .out
• Once processing visible output appears: Poll every 15 seconds, show the last processed subject/file count

4. Health marks

Look for positive signals in stdout while running:

"Processing sub-"  → extraction started
"OK"               → subject completed successfully  
"FAIL"             → subject failed but pipeline continued
"DONE"             → all subjects finished

Count successes/failures mid-stream:

grep -c "OK " logs/*.out   # how many succeeded
grep -c "FAIL " logs/*.out  # how many failed (pipeline tolerated)

5. Determine job state

State	Meaning	Action
PENDING/PD	Waiting for resources	Report queue position, retry in 10s
RUNNING/R + no stderr errors	Healthy	Show progress counts from stdout
RUNNING/R + traceback in stderr	Fatal error	Cancel immediately, report error
COMPLETED/CD	Finished	Validate outputs exist
FAILED/F/TIMEOUT	Scheduler-detected failure	Read logs, diagnose

6. Reporting format

Job 5 (fc_extract): CANCELLED after 12s - FATAL ERROR
  Error: AttributeError: 'NiftiLabelsMasker' object has no attribute 'clone'
  File: analysis_04_fc_extraction.py, line 275

Or for a running healthy job:
Job 6 (fc_extract): RUNNING 45s [3/10 OK, 0 FAIL] -- no errors yet

7. When to stop monitoring

Stop when one of these happens:

• The job fully completes (DONE marker or scheduler COMPLETED)
• A hard error is detected and we cancel
• The user interrupts

Constraints

• Never assume a specific scheduler — detect and adapt
• Cancel jobs with fatal Python errors immediately — don't let them waste queue time
• Always show the exact error line number and message
• For running jobs, always provide a running count of good vs failed subjects
• Default to aggressive error scanning for the first minute