analyze-failures

name	analyze-failures
description	Analyze a context file of test failures and produce a fix report
user-invocable	true

Analyze Failures

Read a previously gathered context file and produce a detailed failure analysis report with root cause grouping and suggested fixes.

Usage

/analyze-failures <context-file>

• <context-file> (required) — path to a context file generated by /gather-failures or houston analyze

Examples:

• /analyze-failures resources/analysis/proton_ipv4_sat-stream-rhel9-Insights-Advisor/35/context.md

Instructions

Step 1: Read the context file

Read the entire context file provided in the arguments. It may be large — use offset/limit parameters to read in chunks. The file contains:

• Build metadata (job name, build number, result, URL)
• For each failure: test ID, error type, exception, stacktrace, stdout/stderr
• Resolved test source code, fixtures, and imported entities
• Product code references (controllers, models) from foreman, katello, or foreman_rh_cloud

Step 2: Check for known fixes in the conversation context

If /search-fixes was run before this step (as part of /analyze-build), there may be known fixes from the knowledge base already in the conversation. Use these as reference when analyzing failures:

• If a known fix matches a current failure, use the past fix's root cause and diff to inform your recommendation
• Reference the past fix in the report (e.g., "Similar to fix_abc123 which resolved the same error by updating the airgun locator")
• If the past fix's approach applies directly, recommend the same pattern rather than guessing

Step 3: Read screenshots

If the context file includes a Screenshots section for a failure, read each screenshot file using the Read tool. These are PNG images captured at the moment of failure during UI tests. Use them to:

• Verify what the UI actually showed when the test failed (e.g., missing elements, wrong page, error dialogs)
• Confirm or refute root cause hypotheses from the stacktrace alone
• Identify product-side UI changes (renamed tabs, moved elements, layout changes)

Not all failures will have screenshots — only UI tests that failed will have them.

Step 4: Read the actual test and product source files

For deeper analysis, read the relevant source files directly from the local repo clones:

• Test code: resources/cache/repos/robottelo/ — read the full test files referenced in the context (e.g. tests/foreman/ui/test_rhcloud_inventory.py)
• Product code: When product code refs appear in the context, read the full files for additional context:
  • resources/cache/repos/foreman/ (theforeman/foreman)
  • resources/cache/repos/katello/ (Katello/katello)
  • resources/cache/repos/foreman_rh_cloud/ (theforeman/foreman_rh_cloud)

Cross-reference test code and product code together to determine whether each failure is a test bug or a product bug.

Step 5: Produce the report

Analyze all failures and write a report to resources/analysis/<job>/<build>/report.md (save it alongside the context file in the same build directory).

The report MUST include for each failure:

1. Test name and file path
2. Brief description of what the test does (1-2 sentences)
3. Where it failed — the specific line/assertion and why
4. Root cause — whether this is a test bug or a product bug, using product code as context to make that determination
5. Suggested fix — actual code changes in unified diff format, targeting only test repositories (robottelo, airgun, nailgun). Never suggest changes to product code (foreman, katello, foreman_rh_cloud). If the root cause is a product bug, the test fix should adapt the test to the current product behavior (e.g., update locators, adjust assertions, add skips with a Jira reference).

Group failures by root cause when multiple parameterized tests fail for the same reason.

End the report with a summary table showing:

• Root cause groups
• Number of failures per group
• Whether the root cause is a test bug or product bug
• All fixes target test repos (robottelo/airgun/nailgun)

Step 6: Tell the user

Print the path to the saved report and show a summary:

• Total failures
• Number of root cause groups
• How many are test fixes vs product bugs

Report Format

Use this structure:

# Test Failure Report: <job> #<build>

**Build**: <result> | **<N> failures** | **<M> root causes** | <date>

**Product component**: `<CaseComponent>` (<repo>)

---

## Group 1: <root cause description> (<N> failures)

**Tests**: list of affected test names
**File**: file path

**What they do**: 1-2 sentence description
**Where they fail**: specific line and assertion

**Product code context**: what the product code reveals about this failure

**Fix**:
\```diff
--- a/path/to/file.py
+++ b/path/to/file.py
@@ ...
\```

---

## Summary

| Root Cause | Count | Type | Fix Target | Investigation Needed |
|-----------|-------|------|------------|---------------|
| ... | ... | test bug / product bug | robottelo / airgun / nailgun | yes / no |