| name | integration-test-triage |
| description | Triage a red any-llm integration test suite. Use when integration tests are failing, when deciding whether a failure is a real regression, or before skipping or "fixing" an integration test. |
Integration test triage
Integration tests run on main post-merge, and on a PR only via the run-integration-tests
label (auto-removed after each run). They are not a merge gate, so failures accumulate.
A red suite is usually a backlog of unrelated causes, not one regression. Build a test's
history before assuming a recent break: list the recent failed runs
(gh run list --workflow "Integration Tests" --status failure), then read the failing steps
of the ones you need (gh run view <id> --log-failed). Root-cause each failure as one of:
- An any-llm bug: fix it and add a test.
- A deprecated or wrong test model: update
tests/conftest.py and cite the provider's docs.
- Missing or invalid credentials: a 401/403 from one provider means its key is absent or
expired. Name the secret; never turn it into a code change.
- Missing CI infrastructure: skip it, naming the missing piece in the skip reason.
- Rate limiting or quota exhaustion: 429s and quota errors. The suite runs under
-n auto,
so re-run serially to tell contention apart from an exhausted account.
- A provider outage: failures correlated on one provider, possibly narrowed to a single
model, endpoint, region, or operation rather than all of its tests. Treat the correlation
plus that provider's public status page as the evidence before calling it transient.
Never record a cause as "flaky" or "does not reliably support". Every skip and fix needs a
concrete reason (HTTP status, deprecated model, missing CI infra).
