samba-check-workflows

# Check Workflows Check GitHub Actions workflow runs from the past day for severe or consistent failures and file a GitHub issue if actionable problems are found. ## Arguments - `$ARGUMENTS`: (Optional) Number of hours to look back (default: 24) ## Instructions ### 1. Gather recent workflow runs Fetch all workflow runs from the past N hours (default 24): ``` gh run list --limit 100 --json workflowName,status,conclusion,event,headBranch,createdAt,databaseId,url,name ``` Filter to only runs created within the lookback window. Group runs by workflow name. ### 2. Classify each failure For each failed run, determine if it is **expected** or **actionable** by checking these rules: #### Expected failures (IGNORE these): 1. **Nightly Runner Cleanup**: This workflow intentionally reboots self-hosted macOS runners, which kills the runner process mid-job. It will almost always show as "failed" even when working correctly. **Always skip this workflow entirely.** 2. **Cascading failures from CI**: When the main CI workflow fails, these downstream workflows will also fail because they depend on CI artifacts (e.g. `html-report`, blob reports). This is noise, not an independent problem: - Playwright Report Comment (fails with "artifact not found") - Upload to Flakiness.io (fails when no flakiness reports exist) - Merge PR when ready (skipped/fails when CI hasn't passed) 3. **CLA Assistant**: Failures just mean a contributor hasn't signed the CLA yet. This resolves on its own. 4. **Cancelled runs**: Runs cancelled due to concurrency groups (newer push cancels older run) are normal. 5. **`action_required` / `neutral` conclusions**: Standard GitHub behavior for fork PRs or first-time contributors needing manual approval. 6. **CI failures on non-main branches**: Individual PR CI failures are expected — contributors may have formatting issues, lockfile mismatches, test failures, etc. These are the contributor's responsibility. 7. **Claude Deflake E2E**: This workflow is expected to sometimes have long runs or partial failures as it investigates flaky tests. #### Actionable failures (FLAG these): 1. **Permission errors**: Workflow can't access secrets, missing `GITHUB_TOKEN`, 403/401 errors on API calls that should be authenticated, `Resource not accessible by integration` errors. 2. **Consistent CI failures on main branch**: If the CI workflow fails on 2+ consecutive pushes to main, something is likely broken. Check if different commits are failing for the same reason. 3. **Infrastructure failures**: Self-hosted runners not coming back online (check if Nightly Runner Cleanup's verify steps are failing), runners consistently unavailable, disk space issues. 4. **Repeated rate limiting**: If GitHub API rate limiting is causing the same workflow to fail across multiple runs (not just a one-off). 5. **Action version issues**: Deprecated or broken GitHub Action versions causing failures. 6. **Workflow configuration errors**: YAML syntax errors, invalid inputs, missing required secrets (distinct from permission issues). 7. **Scheduled workflow failures**: If a scheduled/cron workflow (other than Nightly Runner Cleanup) fails consistently, it likely indicates a systemic issue. ### 3. Investigate actionable failures For each potentially actionable failure, get more details: ``` gh run view <run_id> --log-failed 2>/dev/null | head -100 ``` Look for: - The specific error message - Whether the failure is in a setup step (infrastructure) vs. a test/build step (code) - Whether the same failure appears across multiple runs ### 4. Determine severity After investigation, categorize actionable failures: - **SEVERE**: Permission errors, infrastructure down, main branch consistently broken, workflow configuration errors - **MODERATE**: Repeated rate limiting, deprecated action warnings, intermittent infrastructure issues - **LOW**: One-off transient failures that resolved on retry Only proceed to file an issue if there are SEVERE or MODERATE findings. ### 5. Check for existing issues Before creating a new issue, check if there's already an open issue about workflow problems: ``` gh issue list --label "workflow-health" --state open --json number,title,body ``` If an existing issue covers the same problems, do not create a duplicate. Instead, add a comment to the existing issue with the latest findings. ### 6. File a GitHub issue If there are actionable findings (SEVERE or MODERATE), create a GitHub issue: ``` gh issue create --title "Workflow issues: <X>, <Y>, and <Z>" --label "workflow-health" --body "$(cat <<'EOF' ## Workflow Health Report **Period:** <start_time> to <end_time> **Total runs checked:** <N> **Failures found:** <N actionable> actionable, <N expected> expected (ignored) ## Issues Found ### <Issue 1 Title> - **Workflow:** <workflow name> - **Severity:** SEVERE / MODERATE - **Failed runs:** - [Run #<id>](<url>) — <date> - [Run #<id>](<url>) — <date> - **Error:** <brief error description> - **Suggested fix:** <how to resolve> ### <Issue 2 Title> ... ## Expected Failures (Ignored) <Brief summary of expected failures that were skipped and why> --- *This issue was automatically created by the daily workflow health check.* EOF )" ``` The issue title should list the specific problems found (e.g., "Workflow issues: CI permissions error, flakiness upload rate-limited"). Keep it concise but descriptive. ### 7. Report results Summarize: - How many workflow runs were checked - How many were expected failures (and which categories) - How many were actionable (and what was found) - Whether an issue was filed (with link) or if everything looks healthy - If no actionable issues were found, report "All workflows healthy" and do not create an issue