integration-test-triage

name	integration-test-triage
description	Triage a red any-llm integration test suite. Use when integration tests are failing, when deciding whether a failure is a real regression, or before skipping or "fixing" an integration test.

Integration test triage

Integration tests run on main post-merge, and on a PR only via the run-integration-tests label (auto-removed after each run). They are not a merge gate, so failures accumulate.

A red suite is usually a backlog of unrelated causes, not one regression. Build a test's history before assuming a recent break: list the recent failed runs (gh run list --workflow "Integration Tests" --status failure), then read the failing steps of the ones you need (gh run view <id> --log-failed). Root-cause each failure as one of:

• An any-llm bug: fix it and add a test.
• A deprecated or wrong test model: update tests/conftest.py and cite the provider's docs.
• Missing or invalid credentials: a 401/403 from one provider means its key is absent or expired. Name the secret; never turn it into a code change.
• Missing CI infrastructure: skip it, naming the missing piece in the skip reason.
• Rate limiting or quota exhaustion: 429s and quota errors. The suite runs under -n auto, so re-run serially to tell contention apart from an exhausted account.
• A provider outage: failures correlated on one provider, possibly narrowed to a single model, endpoint, region, or operation rather than all of its tests. Treat the correlation plus that provider's public status page as the evidence before calling it transient.

Never record a cause as "flaky" or "does not reliably support". Every skip and fix needs a concrete reason (HTTP status, deprecated model, missing CI infra).