compare-builds

name	compare-builds
description	Compare two builds and report new vs recurring failures
user-invocable	true

Compare Builds

Compare two Jenkins builds of the same job and produce a report highlighting new failures, recurring failures, and resolved failures between them.

Usage

/compare-builds <job-path> --build <number1> --build <number2>

• <job-path> (required) — full Jenkins job path: <team>/<ipv4|ipv6>/<component> (e.g. proton/ipv4/sat-stream-rhel9-Insights-Advisor)
• --build <number1> --build <number2> (required) — the two build numbers to compare. The lower number is treated as the older build.

Examples:

• /compare-builds proton/ipv4/sat-stream-rhel9-Insights-Advisor --build 34 --build 35
• /compare-builds proton/ipv4/sat-stream-rhel9-RHCloud --build 1018 --build 1019

Instructions

Step 1: Gather failures for both builds

Run houston analyze for each build to generate context files:

houston analyze "<job-path>" --build <older-build> --verbose 2>&1

houston analyze "<job-path>" --build <newer-build> --verbose 2>&1

Note the context file paths from each run.

Step 2: Read both context files

Read both context files to extract the failure data from each build:

• Test IDs (the full file::test_name[params] string)
• Error types and exception classes
• Error messages and stacktraces

Step 3: Categorize failures

Compare the two builds by test ID to categorize each failure:

• New failures — tests that failed in the newer build but NOT in the older build. These are the most important to investigate.
• Recurring failures — tests that failed in BOTH builds. These are known issues that persist.
• Resolved failures — tests that failed in the older build but NOT in the newer build. These have been fixed or are no longer triggered.

When matching test IDs, strip parameterized suffixes (e.g., [rhel9-ipv4-local]) to group parameterized variants together. If all variants of a parameterized test fail in both builds with the same error, treat them as one recurring failure group.

Step 4: Write the comparison report

Save the report to resources/analysis/<job>/<older-build>_vs_<newer-build>_comparison.md.

The report MUST follow this structure:

# Build Comparison: <job> #<older> vs #<newer>

**Older build**: #<older> — <result> | <N> failures | <date>
**Newer build**: #<newer> — <result> | <N> failures | <date>

**New failures**: <count> | **Recurring**: <count> | **Resolved**: <count>

---

## New Failures (<count>)

These failures appeared in #<newer> but were NOT present in #<older>.

### <test_name> (<count> variants)

**Test ID**: `<full test ID>`
**Error**: `<exception class>`

**Traceback**:
\```
<full stacktrace from the context file>
\```

**Error message**:
\```
<error message>
\```

---

## Recurring Failures (<count>)

These failures are present in both builds.

### <test_name> (<count> variants)

**Test ID**: `<full test ID>`
**Error**: `<exception class>`
**Same error in both builds**: yes/no

**Traceback** (from #<newer>):
\```
<full stacktrace>
\```

---

## Resolved Failures (<count>)

These failures were present in #<older> but are NOT in #<newer>.

### <test_name>

**Test ID**: `<full test ID>`
**Error was**: `<exception class>`

---

## Summary

| Category | Count |
|----------|-------|
| New failures | ... |
| Recurring failures | ... |
| Resolved failures | ... |
| Total failures (older) | ... |
| Total failures (newer) | ... |

Step 5: Tell the user

Print a summary:

• How many new, recurring, and resolved failures
• List the new failure test names (these need attention first)
• Path to the comparison report