debugging-workflows

# Debugging Workflows Skill Use this skill when you need to debug GitHub Actions workflows, download workflow logs or summaries, or understand how agentic workflows and the AWF firewall work together. ## Quick Start ### Download Workflow Logs Use the `download-workflow-logs.ts` script to download logs from a workflow run: ```bash # Download logs from the latest workflow run npx tsx .github/skills/debugging-workflows/download-workflow-logs.ts # Download logs from a specific run ID npx tsx .github/skills/debugging-workflows/download-workflow-logs.ts --run-id 1234567890 # Download logs from a specific workflow npx tsx .github/skills/debugging-workflows/download-workflow-logs.ts --workflow test-integration.yml # Save logs to a specific directory npx tsx .github/skills/debugging-workflows/download-workflow-logs.ts --output ./my-logs ``` ### Download Workflow Summary Use the `download-workflow-summary.ts` script to get a summary of workflow runs: ```bash # Get summary of latest workflow runs npx tsx .github/skills/debugging-workflows/download-workflow-summary.ts # Get summary for a specific workflow run npx tsx .github/skills/debugging-workflows/download-workflow-summary.ts --run-id 1234567890 # Get summary for a specific workflow file npx tsx .github/skills/debugging-workflows/download-workflow-summary.ts --workflow test-integration.yml # Get summary as JSON npx tsx .github/skills/debugging-workflows/download-workflow-summary.ts --format json ``` ## GitHub CLI Commands The `gh` CLI is essential for debugging workflows. Here are the most useful commands: ### List Workflow Runs ```bash # List recent workflow runs gh run list --limit 10 # List runs for a specific workflow gh run list --workflow test-integration.yml --limit 10 # List only failed runs gh run list --status failure --limit 10 # List runs in JSON format for parsing gh run list --json databaseId,name,status,conclusion,createdAt --limit 10 ``` ### View Workflow Run Details ```bash # View a specific run gh run view <run-id> # View run with job details gh run view <run-id> --verbose # View run as JSON gh run view <run-id> --json jobs,conclusion,status ``` ### Download Run Logs ```bash # Download all logs for a run gh run download <run-id> # Download specific artifact gh run download <run-id> --name <artifact-name> # Download to specific directory gh run download <run-id> --dir ./logs ``` ### Watch a Running Workflow ```bash # Watch a workflow run in real-time gh run watch <run-id> # Watch with exit code (useful for CI) gh run watch <run-id> --exit-status ``` ### Re-run Failed Jobs ```bash # Re-run failed jobs only gh run rerun <run-id> --failed # Re-run all jobs gh run rerun <run-id> ``` ## Understanding Agentic Workflows ### What are Agentic Workflows? Agentic workflows are GitHub Actions workflows that use AI agents (like GitHub Copilot or Claude) to perform tasks. They are defined using **markdown + YAML frontmatter** format in `.github/workflows/*.md` files and compiled to GitHub Actions YAML (`.lock.yml` files). ### Key Components 1. **Workflow File Format**: `.github/workflows/<name>.md` - YAML frontmatter for configuration - Markdown body for AI instructions - Compiles to `.github/workflows/<name>.lock.yml` 2. **Triggers** (`on:` field): - Standard GitHub events: `issues`, `pull_request`, `push`, `schedule` - Command triggers: `/mention` in issues/comments - `workflow_dispatch` for manual triggers 3. **Safe Outputs**: Controlled way for AI to create GitHub entities - `create-issue:` - Create GitHub issues - `create-pull-request:` - Create PRs with git patches - `add-comment:` - Add comments to issues/PRs - `add-labels:` - Add labels to issues/PRs - `create-discussion:` - Create GitHub discussions 4. **Tools Configuration** (`tools:` field): - `github:` - GitHub API tools - `agentic-workflows:` - Workflow introspection tools - `edit:` - File editing tools - `web-fetch:` / `web-search:` - Web access tools - `bash:` - Shell command tools ### Compiling Workflows ```bash # Compile all workflows gh aw compile # Compile a specific workflow gh aw compile <workflow-name> # Compile with strict security checks gh aw compile --strict ``` ### Debugging Agentic Workflows ```bash # View status of all agentic workflows gh aw status # Download and analyze logs from previous runs gh aw logs <workflow-name> --json # Audit a specific run for issues gh aw audit <run-id> --json ``` ### Common Issues 1. **Missing Tool Calls**: Check `missing_tools` in audit output 2. **Safe Output Failures**: Review `safe_outputs.jsonl` artifact 3. **Permission Issues**: Verify `permissions:` block in frontmatter 4. **Network Blocked**: Check `network:` configuration for allowed domains ## Understanding the AWF Firewall ### What is AWF? AWF (Agent Workflow Firewall) is a tool that provides L7 (HTTP/HTTPS) egress control for GitHub Copilot CLI and other agents. It restricts network access to a whitelist of approved domains using Squid proxy and Docker containers. ### Architecture Overview ``` ┌─────────────────────────────────────────┐ │ Host (GitHub Actions Runner / Local) │ │ │ │ ┌────────────────────────────────────┐ │ │ │ Firewall CLI (awf) │ │ │ │ - Parse arguments │ │ │ │ - Generate Squid config │ │ │ │ - Start Docker Compose │ │ │ └────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌──────────────────────────────────┐ │ │ │ Docker Compose │ │ │ │ ┌────────────────────────────┐ │ │ │ │ │ Squid Proxy Container │ │ │ │ │ │ - Domain ACL filtering │ │ │ │ │ │ - HTTP/HTTPS proxy │ │ │ │ │ └────────────────────────────┘ │ │ │ │ ▲ │ │ │ │ ┌────────┼───────────────────┐ │ │ │ │ │ Agent Container │ │ │ │ │ │ - Full filesystem access │ │ │ │ │ │ - iptables redirect │ │ │ │ │ │ - All traffic → Squid │ │ │ │ │ └────────────────────────────┘ │ │ │ └──────────────────────────────────┘ │ └─────────────────────────────────────────┘ ``` ### Key Containers - **`awf-squid`** - Squid proxy container (IP: 172.30.0.10) - Filters HTTP/HTTPS traffic based on domain allowlist - Logs all traffic decisions - **`awf-agent`** - Agent execution container (IP: 172.30.0.20) - Runs the actual command/agent - Has iptables rules to redirect traffic to Squid - Full filesystem access via `/host` mount ### Traffic Flow 1. Command runs in agent container 2. All HTTP/HTTPS traffic → iptables DNAT → Squid proxy 3. Squid checks domain against allowlist 4. Allowed → forward to destination 5. Blocked → return 403 Forbidden ### Squid Log Analysis ```bash # View Squid access log (shows traffic decisions) docker exec awf-squid cat /var/log/squid/access.log # Find blocked domains docker exec awf-squid grep "TCP_DENIED" /var/log/squid/access.log | awk '{print $3}' | sort -u # Count blocked by domain docker exec awf-squid grep "TCP_DENIED" /var/log/squid/access.log | awk '{print $3}' | sort | uniq -c | sort -rn # Real-time blocked traffic docker exec awf-squid tail -f /var/log/squid/access.log | grep --line-buffered TCP_DENIED ``` ### Squid Decision Codes - `TCP_TUNNEL:HIER_DIRECT` = **ALLOWED** (HTTPS) - `TCP_MISS:HIER_DIRECT` = **ALLOWED** (HTTP) - `TCP_DENIED:HIER_NONE` = **BLOCKED** ### Running Commands Through Firewall ```bash # Basic usage sudo awf --allow-domains github.com 'curl https://api.github.com' # With debug logging sudo awf --allow-domains github.com --log-level debug 'your-command' # Keep containers for inspection sudo awf --allow-domains github.com --keep-containers 'your-command' ``` ### Preserved Logs Locations **With `--keep-containers`:** - Squid: `/tmp/awf-<timestamp>/squid-logs/access.log` - Agent: `/tmp/awf-<timestamp>/agent-logs/` **Normal execution (after cleanup):** - Squid: `/tmp/squid-logs-<timestamp>/access.log` - Agent: `/tmp/awf-agent-logs-<timestamp>/` ```bash # Find preserved logs ls -ldt /tmp/awf-* /tmp/squid-logs-* 2>/dev/null | head -5 # View preserved Squid logs sudo cat $(ls -t /tmp/squid-logs-*/access.log 2>/dev/null | head -1) ``` ## Debugging Workflow Failures ### Step-by-Step Process 1. **Identify the failing workflow run** ```bash gh run list --status failure --limit 5 ``` 2. **Get run details** ```bash gh run view <run-id> --verbose ``` 3. **Download logs** ```bash gh run download <run-id> --dir ./logs # Or use the script: npx tsx .github/skills/debugging-workflows/download-workflow-logs.ts --run-id <run-id> ``` 4. **Analyze the failure** - Check job logs for error messages - Look for timeout issues - Check for permission errors - Review network-related errors 5. **For agentic workflows, audit the run** ```bash gh aw audit <run-id> --json ``` 6. **If firewall-related, check Squid logs** ```bash # If containers are still running docker exec awf-squid cat /var/log/squid/access.log # Or check preserved logs sudo cat /tmp/squid-logs-*/access.log ``` ### Common Failure Patterns #### Permission Denied ``` Error: Resource not accessible by integration ``` **Fix:** Check `permissions:` in workflow frontmatter #### Domain Blocked ``` curl: (56) Recv failure: Connection reset by peer ``` **Fix:** Add domain to `--allow-domains` or `network:` configuration #### Timeout ``` Error: The operation was canceled. ``` **Fix:** Increase `timeout-minutes` in workflow configuration #### Missing Tool ``` Tool 'xyz' not found ``` **Fix:** Add tool to `tools:` configuration in workflow frontmatter ## Related Documentation - [Architecture](../../../docs/architecture.md) - System architecture details - [Troubleshooting](../../../docs/troubleshooting.md) - Common issues and solutions - [GitHub Actions Integration](../../../docs/github_actions.md) - CI/CD setup - [Logging Documentation](../../../LOGGING.md) - Comprehensive logging guide - [Debug Firewall Skill](../debug-firewall/SKILL.md) - Firewall-specific debugging ## Start here for diagnosis If you are diagnosing a failure rather than exploring, enter through the [`diagnose-awf`](../diagnose-awf/SKILL.md) skill and the canonical diagnosis registry in [`docs/diagnostics/README.md`](../../../docs/diagnostics/README.md). This skill is one of the specialist references it routes to.