workflow-errors

name	workflow_errors
description	Diagnose and fix CatGo workflow engine errors including HPC connection failures, missing POTCARs, stuck tasks, and REMOTE_ERROR states.

CatGo Workflow Engine Error Troubleshooting Skill

When to Use

Use this skill when the problem is NOT with the DFT code itself, but with CatGo's workflow engine, HPC connectivity, or job management. Route here when:

• Tasks are stuck in READY or SUBMITTED state
• HPC connection fails (SSH, SFTP)
• POTCAR or pseudopotential files not found
• REMOTE_ERROR status on a task
• Workflow does not start after submission
• Job disappears from queue unexpectedly

Diagnostic Steps

Step 1: Check system status

catgo_system(action: "status")

This reports:

• Backend health (FastAPI server running)
• HPC connection status (SSH connected, SFTP available)
• Engine status (scanner running, poller active)

Step 2: Check workflow and task status

catgo_workflow_engine(action: "status", params: { workflow_id: "<wf_id>" })

Step 3: Get recent error logs

catgo_system(action: "errors")

---

Error Reference

No HPC Connection

Symptoms:

• Tasks stay in READY state indefinitely
• catgo_system(action: "status") shows no active HPC connections
• Error: "No HPC connection available"

Diagnosis:

catgo_system(action: "status")

Fixes:

1. Verify HPC credentials are configured in the CatGo settings
2. Check that the HPC host is reachable (not under maintenance)
3. If using SSH key + OTP, ensure the OTP token is current
4. Reconnect via the CatGo HPC settings panel in the frontend

Note: CatGo uses asyncssh with keyboard-interactive authentication. Some HPC systems have specific prompt formats for OTP that may need special handling. See server/CLAUDE.md for the KbdintSSHClient OTP issue.

POTCAR Not Found

Symptoms:

• VASP task fails immediately after submission
• Error message contains "POTCAR" or "pseudopotential"
• Error: "Could not find POTCAR for element X"

Diagnosis:

catgo_workflow_engine(action: "get_result", params: {
  workflow_id: "<wf_id>",
  task_id: "<task_id>"
})

Fixes:

1. Check that VASP_PP_PATH is set correctly on the HPC:

   • Should point to the directory containing potpaw_PBE/, potpaw_LDA/
   • Each element dir must contain a POTCAR file

2. For CP2K, check that the basis set and pseudopotential files are in the CP2K data directory (usually $CP2K_DATA_DIR).

3. If the element was recently added to the structure (e.g., a dopant), the POTCAR generation may not include it. Verify the structure has correct elements:

catgo_view(action: "get_state")

Task Stuck in READY

Symptoms:

• Task shows state: "READY" for a long time
• No HPC job is submitted
• No error message

Diagnosis:

catgo_workflow_engine(action: "status", params: { workflow_id: "<wf_id>" })

Possible causes and fixes:

1. Upstream task not complete: The task depends on another task that has not finished. Check the DAG:

catgo_workflow_engine(action: "get_dag", params: { workflow_id: "<wf_id>" })

2. Engine scanner not running: The workflow engine scanner picks up READY tasks periodically. Check system status:

catgo_system(action: "status")

3. Workflow not submitted: The workflow was created but never submitted:

catgo_workflow_engine(action: "submit", params: { workflow_id: "<wf_id>" })

4. Manual reset: Force the task back to READY and re-trigger:

catgo_workflow_engine(action: "reset", params: {
  workflow_id: "<wf_id>",
  task_id: "<task_id>"
})

REMOTE_ERROR

Symptoms:

• Task status shows state: "REMOTE_ERROR"
• The job was submitted to HPC but something failed during execution

Diagnosis:

catgo_workflow_engine(action: "get_result", params: {
  workflow_id: "<wf_id>",
  task_id: "<task_id>"
})

Common causes:

1. Job killed by scheduler: Exceeded walltime or memory limit. Increase resources in run_config.
2. File system error: HPC scratch space full or quota exceeded.
3. Module not loaded: Required software module not available. Check the HPC submission script.
4. SFTP failure during file transfer: CatGo could not upload input files or download results. The SFTP fallback (exec-based transfer) should handle this automatically, but some HPC systems restrict both.

Fix: After addressing the root cause:

catgo_workflow_engine(action: "retry", params: {
  workflow_id: "<wf_id>",
  task_id: "<task_id>"
})

Task Stuck in SUBMITTED / RUNNING

Symptoms:

• Task shows SUBMITTED or RUNNING but the HPC job has already finished
• The poller is not picking up the completed job

Diagnosis:

catgo_system(action: "status")

Check if the engine poller is active. If not, the backend may need a restart.

Fix: Reset the task to re-trigger polling:

catgo_workflow_engine(action: "reset", params: {
  workflow_id: "<wf_id>",
  task_id: "<task_id>"
})

Workflow Paused Unexpectedly

An error handler may pause a workflow after task failure to prevent cascading failures. Fix the failed task first, then resume:

catgo_workflow_engine(action: "resume", params: { workflow_id: "<wf_id>" })

Recovery Actions Summary

• Retry: catgo_workflow_engine(action: "retry", params: {workflow_id, task_id})
• Reset: catgo_workflow_engine(action: "reset", params: {workflow_id, task_id})
• Modify + retry: modify_params then retry
• Check DAG: catgo_workflow_engine(action: "get_dag", params: {workflow_id})

When to Escalate

• Backend down: user needs to restart (pnpm desktop:serve)
• Repeated HPC connection failures: re-enter credentials or check maintenance
• Same REMOTE_ERROR after multiple retries: likely environmental (HPC config)