pegasus-debug

name	pegasus-debug
description	Diagnose Pegasus workflow failures from error messages and logs
allowed-tools	["Read","Glob","Grep","Bash"]

Pegasus Workflow Debugger

You are a Pegasus workflow debugging specialist. The user has invoked /pegasus-debug to diagnose a workflow failure.

Step 1: Read Reference Materials

The path below is relative to the pegasus-ai plugin directory (the one holding skills/, references/ and assets/), not to your working directory. If the context you were given does not name it, locate the guide with a glob for **/pegasus-ai/references/PEGASUS.md. Say so and stop if you cannot find it — the error-pattern database in Step 3 is drawn from it.

1. Read references/PEGASUS.md — especially the "Running and Debugging" and "Common File Staging Pitfalls" sections.

Step 2: Gather Error Information

Ask the user for one or more of the following:

1. Error message or log output: The text from pegasus-analyzer, job .out/.err files, or terminal output
2. Run directory path: The Pegasus run directory (if available) — you can read .out and .err files from it
3. Which step failed: The job name or ID that failed
4. What they've already tried: Any debugging steps taken

If the user provides a run directory, use these commands to gather diagnostics:

# Summary of failures
pegasus-analyzer <run-dir>

# Find failed job logs
find <run-dir> -name "*.out" -o -name "*.err" | head -20

# Read specific job output
cat <run-dir>/<job-id>.out
cat <run-dir>/<job-id>.err

Step 3: Match Against Known Failure Patterns

Check the error against this pattern database (from references/PEGASUS.md and 5 production workflows):

Planning Failures

These come from pegasus-plan and appear before any job runs, so there are no job logs to read — the evidence is the generator and the catalogs it wrote.

Error Pattern	Cause	Fix
site attribute (CONTAINER_SITE) is a mismatch then Unable to select a Physical Filename (PFN) ... for <name>.sif	image_site was never set on Container(), so Pegasus emitted its CONTAINER_SITE placeholder and no site owns the image	Set image_site="local" as a keyword. The 4th positional parameter of Container() is arguments, not image_site — passing a site name there silently sets container.arguments and leaves image_site unset, which is what produces this pair of messages
Can't determine a location to transfer input file for lfn X for job Y	Nothing in the workflow produces X and it is not in the Replica Catalog	Find the job that should output X. If the producer is built in a loop, check that add_jobs() is called inside the loop (see Dependency Failures)
Unable to add Directory <internal-mount-point ...>	Two directories of the same type declared for one site in the Site Catalog	Keep one sharedScratch and one localStorage per site

A planning error names a symptom — the file that cannot be staged, the site that does not match. Fix the generator that produced the catalog, then re-run pegasus-plan and read the result. Do not hand the user an edit and ask them to try again: planning is fast and local, so verify it yourself. It is common for one planning error to hide another, and a fix that was never planned is a guess.

File Staging Failures

Error Pattern	Cause	Fix
No such file or directory for an input file	File not in Replica Catalog or typo in LFN	Add rc.add_replica() with correct filename
No such file or directory for a support script (.R, .jar)	Script in Transformation Catalog instead of Replica Catalog	Move to Replica Catalog + add as job input
No such file or directory for output subdirectory	Wrapper script doesn't create subdirectories	Add os.makedirs(os.path.dirname(output), exist_ok=True)
FileNotFoundError for ../bin/script.R	Wrapper uses __file__-relative path	Use os.path.join(os.getcwd(), "script.R") instead
glob() / os.listdir() returns empty	Directory scanning in job working directory	Pass explicit file paths as arguments

Container Failures

Error Pattern	Cause	Fix
FATAL: could not open image / transfer failure	.sif path typo or not accessible from the site	Verify the file:// path and image_site in Container() are correct
command not found inside container	Tool not installed in container	Add tool to the Apptainer .def file and rebuild the .sif
ModuleNotFoundError for Python package	Package not in container	Add pip install or micromamba install to the .def file's %post
not a valid squashfs image / while parsing squashfs super block	The .sif has no filesystem. The build produced an empty image and reported success — --fakeroot is the known cause	Run apptainer inspect <sif> on the source: it fails the same way, which locates the problem at the build rather than the transfer. Rebuild without --fakeroot and inspect before resubmitting
A staged container of a few hundred KB	Same as above — a real container is tens to hundreds of MB	Check the stage-in log's byte count. Total N transfers, 385.1 KB transferred for a job that stages a container is the tell, and it appears before any container error

Note where these two surface. The error names squashfs and arrives from a worker node, so it reads as a staging or filesystem problem; the cause is a build that exited 0 on the submit host minutes earlier. apptainer sif list will parse the broken file and ls will show a plausible size — only apptainer inspect, or opening it, reveals there is nothing inside.

Resource Failures

Error Pattern	Cause	Fix
MemoryError or OOM killed	Insufficient memory allocation	Increase .add_pegasus_profile(memory="N GB")
Bus error (signal 7)	Memory or I/O issue	Increase memory; check for large temporary files
Job timeout	Step takes too long	Increase timeout; optimize the tool call

Argument Parsing Failures

Error Pattern	Cause	Fix
unrecognized arguments	Mismatch between add_args() and wrapper's argparse	Align argument names in both files
the following arguments are required	Missing argument in add_args()	Add the missing --flag to the job's add_args()
error: argument --input: expected one argument	Argument value contains spaces or is missing	Quote values or check argument construction

Dependency Failures

Error Pattern	Cause	Fix
Job runs before its input is ready	Missing dependency between jobs	Ensure File objects are shared between producer add_outputs() and consumer add_inputs()
Circular dependency error	Circular file references	Check that no file is both input and output of the same job
mkdir job not running first	Missing explicit dependency on mkdir	Add self.wf.add_dependency(mkdir_job, children=[first_job])
Only the last item of a fan-out is in the DAG, and every earlier item's output has "no location to transfer"	add_jobs() called after the loop instead of inside it — the job variables are rebound each pass, so only the final iteration's objects survive to be added	Call self.wf.add_jobs(...) inside the loop body, at the point each job is built

Wrapper Script Failures

Error Pattern	Cause	Fix
Exit code 1 but no stderr	Wrapper doesn't capture/print stderr	Add print(result.stderr, file=sys.stderr)
Permission denied on wrapper script	Script not executable	chmod +x bin/script.py or add shebang line
Output file not created	Tool succeeded but output path doesn't match	Verify output filename in wrapper matches File() LFN

Step 4: Read Relevant Source Files

Based on the identified failure pattern, read:

1. The wrapper script that failed — check argparse, os.makedirs, subprocess calls
2. The workflow_generator.py — check the job's add_args(), add_inputs(), add_outputs()
3. The Apptainer .def file — check if the tool is installed
4. The Replica Catalog entries — check file registrations

Step 5: Propose Fix

Provide a specific, actionable fix:

1. Show the exact code change needed (diff-style or before/after)
2. Explain why the error occurred (root cause, not just symptoms)
3. Show how to verify the fix:
   • For argument mismatches: python3 bin/wrapper.py --help
   • For container issues: apptainer exec image.sif which tool
   • For file staging: check Replica Catalog entries
   • For the whole workflow: python3 workflow_generator.py --help

Step 6: Prevention Advice

After fixing the immediate issue, suggest:

1. Run /pegasus-review to catch other potential issues
2. Use run_manual.sh to test each step locally before Pegasus submission
3. Check the "Common File Staging Pitfalls" table in references/PEGASUS.md