analyze-ci

name	analyze-ci
description	Analyze CI failures by fetching logs and producing a structured fix plan. Use before attempting automated CI fixes.

CI Failure Analysis Skill

You are given a list of failed CI checks with their log URLs. Fetch the actual logs, understand the root cause of each failure, and produce a structured fix plan for a developer agent to follow.

Workflow

Step 0 — Check for prior attempts (ALWAYS do this first)

Before downloading any logs, check whether a previous fix attempt already ran:

1. If .forge/fix-plan.md exists, read it in full. Identify:

   • Which tests were in Fixable Failures and what fix was applied
   • Which tests were in Skipped Failures and why

2. Run git log --oneline -10 to see what changes were committed by previous fix attempts. This tells you what code was already modified.

3. For each test that is still failing in the current run:

   • Was it in the previous Fixable Failures list? If yes, the previous fix did not work. Do not repeat it.
   • Was it in the previous Skipped Failures list? Re-evaluate — a test that was skipped but continues to fail across multiple CI runs may have been incorrectly classified.

4. For each test that is no longer failing: it was fixed. Do not include it in this plan.

If this is a retry (attempt > 1) and a test is still failing despite a prior fix:

• The prior approach was wrong. You MUST propose a different fix category or direction.
• If the prior fix was a timeout increase and the test still fails: the timeout is not the root cause. The operation is stuck, not slow. Read the source code — look for early-return paths that skip scheduling a follow-up requeue or reconcile.
• If the prior fix was a code change and the test still fails: re-read the code change in git diff HEAD~3..HEAD and reconsider whether it actually addresses the failure mode.
• Never write a plan that repeats an approach already tried on a test that is still failing.

1. Read the failures file at the path provided in the prompt using read_file

2. Create .forge/logs/ in the workspace if it doesn't exist: mkdir -p .forge/logs

3. For each failed check that has a log URL, download to .forge/logs/ first — one download, then analyze locally:

   Single log file (GitHub Actions, plain Prow log URL):

   • GitHub Actions: gh api repos/{owner}/{repo}/actions/jobs/{job-id}/logs > .forge/logs/{check-name}.txt
   • Plain URL: curl -sL "{url}" -o .forge/logs/{check-name}.txt

   Log bundle / archive (Prow often uploads a .tar.gz bundle of all logs):

   • Download: curl -sL "{url}" -o .forge/logs/{check-name}.tar.gz
   • Extract: tar -xzf .forge/logs/{check-name}.tar.gz -C .forge/logs/{check-name}/
   • The bundle typically contains build-log.txt, controller logs, and test output

   GitHub Actions artifacts (uploaded on failure):

   • gh run download {run-id} -n {artifact-name} -D .forge/logs/{check-name}/

4. Analyze the downloaded files using local tools — use your judgment:

   • grep -i "error\|fail\|panic\|FAIL" .forge/logs/{check-name}.txt | tail -50
   • tail -100 .forge/logs/{check-name}/build-log.txt
   • Search through extracted bundle files for the root cause

5. Categorize each failure (see categories below)

6. For fixable failures, determine exactly what needs to change

7. Write the fix plan to .forge/fix-plan.md in the workspace (see output format)

Important: Do not print large log content to the conversation. Analyse locally and write only the structured fix plan to the output file.

Failure Categories

Fixable by code change

• codegen-outdated: Generated files (zz_generated.*, CRD YAML, mocks) are out of sync with source. Fix: run the project's codegen command.
• format: Code formatting violations (gofmt, ruff, etc.). Fix: run the formatter on the affected files.
• lint: Lint rule violations with clear error messages. Fix: address the specific violations.
• compile: Compilation errors. Fix: correct the specific syntax/type errors.
• unit-test: Test assertion failures caused by a code bug. Fix: correct the implementation or update the test.
• e2e-code-bug: An e2e/integration test fails with a consistent, reproducible assertion error pointing to a code logic defect — not an infrastructure problem. The failure message is the same across runs and environments. Fix: correct the implementation. Examples: wrong condition status, missing requeue, incorrect state machine transition.

Not fixable by code change — skip

• e2e-infra: End-to-end tests that fail due to infrastructure problems: DevStack/cluster startup failures, OpenStack API timeouts, networking errors, resource quota issues, or missing environment resources. The failure is in the test harness or cloud environment, not the code under test.
• infra: Timeouts, network errors, missing cluster resources, quota issues outside of e2e test context.
• flaky: Genuinely non-deterministic failures — fails with different errors across runs, or fails <30% of the time with no consistent root cause.

Distinguishing e2e-code-bug from e2e-infra and flaky

Before marking any e2e failure as skipped, apply these checks:

1. Failure rate: Does the same test fail in ≥70% of runs across multiple environments? High, consistent failure rates indicate a code bug, not a flaky environment. A test that fails 3 out of 4 runs with the same error is almost certainly a code bug.

2. Error consistency: Is the same assertion failing with the same error message across runs and environments? Consistent errors point to code; varied errors (different steps, timeouts vs. assertion mismatches) point to infrastructure or true flakiness.

3. Error type: Does the failure message mention assertion mismatches on computed values (condition status, timestamps, counts) vs. infrastructure errors (connection refused, resource not found in OpenStack, DevStack not ready)?

4. Failure isolation: Does the failure happen at a specific, named test step that exercises business logic, rather than during setup/teardown or cluster bootstrapping?

If any of these checks points to a code defect, classify as e2e-code-bug and investigate the implementation, not the test harness. Read the relevant source code to confirm the root cause before writing the fix plan.

Timing and scheduling failures — special handling

When a test measures time-based behavior (resyncs, requeues, reconcile periods, backoff timers), apply these additional checks before concluding it is environment load:

Stuck vs slow: A slow operation makes progress but takes longer than the timeout. A stuck operation never makes progress at all. Check the logs:

• If the measured value (e.g. lastSyncTime, status.id) changes at least once during the timeout window → the system is slow, a timeout increase may be appropriate.
• If the measured value is completely static for the entire timeout window — unchanged for 60–120s when it should update every 10s — the operation is stuck. A stuck state is a code bug, not a timing issue. A longer timeout will not fix it.

To confirm stuck vs slow: search the logs for the field name and check whether its value ever changes between test start and assertion failure. If it does not change at all, the controller never scheduled a follow-up reconcile.

Probabilistic code bugs: Not all code bugs fail 100% of the time. A bug that triggers with ~50% probability per resource will fail:

• ~50% of runs when 1 resource is involved (looks like a flake)
• ~87.5% of runs when 3 resources are involved (looks like a bug)

When related tests that exercise the same code path show inconsistent failure rates, compute the per-resource failure probability. If it is consistent across tests (e.g. all explained by ~50% per resource), that is a single probabilistic code bug, not independent flakes. Treat it as an e2e-code-bug and read the source code.

Read the source code for timing failures: For any failure involving timing, scheduling, or state machine transitions, read the relevant controller/handler code before drawing conclusions. Look for: early-return paths that skip scheduling a follow-up requeue, race conditions between jitter and period checks, or conditions that prevent the next reconcile from being registered. A timeout increase is only valid if you have read the code and confirmed the operation can complete correctly. If any code path prevents the operation from ever completing, that is a code bug.

Never propose a timeout increase as the sole fix without first verifying via source code that the operation is capable of completing. Timeout increases mask stuck states and leave the underlying bug unresolved.

Output Format

Write the fix plan to .forge/fix-plan.md in this exact structure so the fix agent can follow it mechanically:

# CI Fix Plan

## Summary
[1-2 sentences: what failed and what the fix involves]

## Fixable Failures

### [check-name]
**Category**: [codegen-outdated | format | lint | compile | unit-test]
**Root Cause**: [exact error message or description]
**Affected Files**: [list of files to change]
**Fix**:
1. [exact command or edit to apply]
2. [verification command to confirm fix]

### [next check-name]
...

## Skipped Failures

### [check-name]
**Reason**: [e2e-infra | infra | flaky] — [brief explanation]

### [next check-name]
...

Important

• Fetch the actual log content — do not guess based on the check name alone
• Be specific: include exact file paths, line numbers, and error messages
• For codegen failures, identify the exact go:generate directive or script to run
• If a failure log is unavailable, mark it as skipped with reason "log unavailable"

Documentation ripple — include stale references in the fix plan

For every fixable failure that changes a constant, threshold, algorithm, or behavior, search the repository for stale documentation before writing the fix plan:

grep -r "<old value>" . --include="*.go" --include="*.md" -l

Include any files with stale references in Affected Files alongside the implementation files. The fix agent will update them as part of the same commit.

Examples of what to search for:

• Old numerical values or percentages mentioned in comments or docs
• Old behavior descriptions in enhancement documents or user guides
• Old flag names, condition names, or error message strings in any documentation

Do not skip this step just because the stale references are in documentation rather than code — documentation that contradicts the implementation is a bug.