troubleshooting-workflow-failures

name	troubleshooting-workflow-failures
description	Systematically investigate failed Syntropic137 workflow executions; find the root cause before retrying, diagnose phase failures, blocked tools, budget exhaustion, and workspace errors

Troubleshooting Workflow Failures: Syntropic137

When a workflow execution fails, the instinct is to re-run it and hope it works. That instinct is wrong.

NEVER re-run a failed execution without first identifying which phase failed and why. Re-running without diagnosis wastes budget, masks the real problem, and produces the same failure, possibly 3 more times before you hit the max_attempts limit.

Three re-runs with the same failure = a configuration problem, not bad luck.

When to Use This Skill

Use this when a workflow execution has status: failed or status: interrupted and you need to understand why before deciding what to do next.

Not needed for executions that are RUNNING but seem slow; use the execution-control skill's monitoring section for that. Not needed for choosing a fix; this skill diagnoses, and the workflow-management skill redesigns.

The Four Investigation Phases

Phase 1: Get the Execution Detail

syn control status <execution-id>

Or via API: curl -sf http://localhost:8137/api/v1/executions/<id> | python3 -m json.tool

Look at:

• Which phase has status: failed? That's where the failure happened.
• What is error_message? This is often the complete answer.
• What is that phase's session_id? You'll need it in Phase 2.

If error_message is clear (e.g., "budget exceeded", "workspace provision failed"), skip to the fix table below. If it's vague or empty, continue to Phase 2.

Phase 2: Inspect the Session Operations Log

syn sessions show <session-id>

The operations log shows every message and tool call in sequence. Read the last 10-20 operations; the failure signature is almost always visible here. Look for:

• The last tool call before the session ended
• An ERROR operation with a message
• An abrupt end with no SESSION_COMPLETED event (workspace crash or timeout)

Phase 3: Trace the Tool Timeline

syn observe tools <session-id>

This gives you precise timing and success/failure for every tool call. Look for:

• TOOL_BLOCKED entries: a SAFETY VALIDATOR blocked the tool call. This does NOT come from the phase's allowed_tools, which is not enforced at runtime (syntropic137#803) - widening that list will not clear a TOOL_BLOCKED. Read the block reason recorded on the event.
• Long duration_ms on a Bash tool: a hanging command that eventually caused a timeout.
• Failed Read or Write calls: permission issues or paths that don't exist in the workspace.
• Repeated calls to the same file: the agent is looping, usually because a previous step produced unexpected output.

Phase 4: Check Token and Budget Metrics

syn observe tokens <session-id>

If the session ended abruptly without a clear tool error, budget or context exhaustion is likely:

• High total_input_tokens (>100K) with the session cutting off: context window limit hit
• Compare total_cost_usd to the phase's budget_per_trigger_usd or execution's max_budget_usd; if equal, budget was the kill switch

Failure Patterns and Fixes

Symptom	Root Cause	Fix
Phase fails immediately	TOOL_BLOCKED: a safety validator blocked the call	Read the block reason on the event; allowed_tools is not the cause and editing it changes nothing
Phase fails on Bash	Command hangs or returns error	Check the bash command in the prompt; add timeout or restrict the command
Phase times out	timeout_seconds too low for the task	Increase timeout_seconds in the phase definition
Budget exhausted	Cost hit max_budget_usd	Increase budget or reduce phase scope; use haiku for cheaper phases
Workspace provision fails	Docker image missing or corrupt	just workspace-build to rebuild; check Docker disk space
Agent loops on same calls	Previous phase output was unexpected format	Inspect the artifact from the previous phase; revise its prompt to produce cleaner output
Context window hit	Too much input in a single turn	Restructure prompts to load targeted files rather than scanning broadly

Red Flags: Stop and Reassess

If you see any of these, don't keep iterating on the same execution:

• Three runs, same phase failing with the same error: the workflow design is wrong, not the execution
• TOOL_BLOCKED on a tool that's obviously needed: a safety validator refused it. Read the reason on the event; running again won't fix it, and neither will editing allowed_tools
• Workspace provision failing repeatedly: infrastructure problem; run just health-check and just workspace-build before any more executions

After Diagnosing

Once you've identified the root cause:

• Tool blocked: read the validator's block reason on the TOOL_BLOCKED event. Editing the phase's allowed_tools does not affect it
• Budget too low: re-run with --max-budget-usd <higher> or edit the phase config
• Workspace issue: fix infrastructure first (platform-ops skill), then re-run
• Prompt producing bad output: revise the phase prompt_template and validate the YAML before re-registering

Integration

This skill follows execution-control (which runs and monitors executions) and feeds back into workflow-management (which redesigns templates). Use /syn-insights for quick session and cost queries during investigation.