| name | detect-flaky-tests |
| description | Detects flaky Go tests by analyzing GitHub Actions workflow runs across the last 7 days and all PRs — covering both the run-tests job (unit/integration) and the e2e-test job (gVisor and microVM lanes). For each newly-detected flaky test or infra issue, opens a GitHub issue with full evidence and a draft fix PR. Does not touch BigQuery, dashboards, or any external storage — those are cron-job concerns layered on top.
|
Detect Flaky Tests
A test is flaky when it produces both PASS and FAIL outcomes across multiple independent
CI runs in the last 7 days, with no code change to that test's package explaining the
inconsistency. Cross-PR analysis provides the strongest signal: if the same test fails on
PR-A but passes on PR-B, that inconsistency is almost certainly non-determinism, not a
legitimate regression.
Flakiness threshold (keeps false-positive rate low)
A test is flagged only when all three conditions hold in the 7-day window:
| Condition | Rationale |
|---|
fail_count >= 2 | One failure could be infra noise |
pass_count >= 2 | One pass could be a pre-fix lucky run |
0.05 < fail_rate < 0.95 | Outside this band it is either reliably broken or reliably passing |
Step 1 — Collect workflow run IDs (last 7 days)
SINCE=$(date -u -v-7d +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ)
gh api --paginate \
"repos/agent-substrate/substrate/actions/workflows/pr-workflow.yaml/runs?status=completed&per_page=100&created=>=$SINCE" \
--jq '.workflow_runs[] | {id: .id, conclusion: .conclusion, head_sha: .head_sha, created_at: .created_at}'
--paginate is required: a typical week has several hundred completed runs (600+ as of
August 2026), far more than one page of 100.
Collect all run IDs. Process both successful and failed runs — both contain test output.
Step 2 — Download and parse logs: two jobs, three lanes per run
For each run, you need logs from two jobs:
| Job name | Coverage |
|---|
run-tests | Unit + integration tests (go test -race -v ./...) |
e2e-test | E2E suite, both sandbox classes — two sequential steps in the one job |
gh api "repos/agent-substrate/substrate/actions/runs/<RUN_ID>/jobs" \
--jq '.jobs[] | {id: .id, name: .name, conclusion: .conclusion}'
gh api "repos/agent-substrate/substrate/actions/jobs/<JOB_ID>/logs" > /tmp/job_<JOB_ID>.log
The two e2e lanes are NOT a matrix — e2e-test is a single job that runs the step
"Run E2E tests (gVisor)" followed by "Run E2E tests (micro-VM)" (same
hack/run-e2e-kind.sh command, the second with E2E_SANDBOX_CLASS: microvm). Split the
one e2e-test job log at the "Run E2E tests (micro-VM)" step boundary: PASS/FAIL lines
before it belong to the gVisor lane, lines after it to the microVM lane. Because the
steps are sequential, a gVisor-lane failure means the micro-VM step never ran — record
no microVM results for that run rather than counting them as failures.
Parse go test -v output from each log:
grep -E '^--- (PASS|FAIL): ' /tmp/job_<JOB_ID>.log \
| awk '{print $2, $3}' | sed 's/://'
Track results per (test_name, job_type) where job_type is one of:
unit, e2e-gvisor, e2e-microvm.
Step 3 — Infra issue triage (do this BEFORE aggregating flakiness)
An infra failure is when the job itself breaks before or during setup — not when a test
produces FAIL output. Infra failures must be identified and reported separately; they do
NOT count toward a test's fail_count.
How to detect an infra failure
A job log is an infra failure (not a test failure) when it contains ANY of:
| Signal | Example log pattern |
|---|
| Go module proxy error | INTERNAL_ERROR, proxy.golang.org: dial, go mod download: ...500 |
| kind cluster creation failure | ERROR: failed to create cluster, node(s) not ready, timed out waiting for the condition |
| Image pull failure | failed to pull image, ErrImagePull, ImagePullBackOff |
| Docker/containerd failure | failed to start containerd, Error response from daemon |
| OOM / out of disk | OOMKilled, No space left on device |
| Network/DNS failure in setup | dial tcp: lookup, connection refused during setup steps (not inside a test) |
| No test output at all | Log ends before any --- PASS or --- FAIL line appears |
| Setup step non-zero exit | A step before the go test or hack/run-e2e-kind.sh command fails |
Critical rule: If a job log contains --- FAIL: TestFoo AND infra error patterns, you
must determine which came first chronologically. If the infra error appears before the first
test ran, treat the whole job as an infra failure (zero test results). If tests started
running and then an infra error interrupted them mid-run, count only the completed test
results and note the truncation.
Triple-check protocol
For any run where you suspect an infra issue, verify it three ways:
- Pattern match: does the log contain one of the signals above?
- Timeline check: does the error appear before
=== RUN Test lines, or between test completions?
- Cross-run consistency: did the same infra error appear in ≥2 other runs around the same time? If yes, it is definitely infrastructure. If it appeared only once, it may be a transient fluke — still classify as infra but flag lower confidence.
Infra issue aggregation
Collect infra failures separately:
| infra_pattern | job_type | fail_count | example_run_ids |
|---|
proxy.golang.org INTERNAL_ERROR | unit | 4 | [run_1, run_2, ...] |
kind cluster: node(s) not ready | e2e-gvisor | 2 | [run_5, ...] |
An infra pattern that appears in ≥2 runs warrants a GitHub issue (Step 5b).
Step 4 — Aggregate test flakiness across runs
Using only the runs NOT classified as infra failures, build a per-test table:
| test_name | job_type | fail_count | pass_count | total_runs |
|---|
Treat each lane independently. A test that is flaky only in e2e-gvisor is still
flagged — it does not need to be flaky in e2e-microvm too.
Apply the threshold: fail_count >= 2 AND pass_count >= 2 AND 0.05 < fail_rate < 0.95.
Step 5a — Create issues for flaky tests
Before creating, check for an existing open issue:
gh issue list \
--repo agent-substrate/substrate \
--state open \
--label "kind/bug,area/tests" \
--search "flaky: <TEST_NAME>" \
--json number,title
If none exists, create:
gh issue create \
--repo agent-substrate/substrate \
--title "flaky: <TEST_NAME>" \
--label "kind/bug,area/tests" \
--body "$(cat <<'BODY'
## Flaky test detected
**Test:** `<TEST_NAME>`
**Job:** `<e2e-gvisor | e2e-microvm | unit>`
### Evidence (last 7 days)
| Metric | Value |
|---|---|
| Runs analysed | <TOTAL_RUNS> |
| Failures | <FAIL_COUNT> |
| Passes | <PASS_COUNT> |
| Flake rate | <FLAKE_RATE>% |
| Infra-failure runs excluded | <INFRA_EXCLUDED> |
### Failing run examples
<links to 2-3 failing runs>
### Passing run examples
<links to 1-2 passing runs>
### Infra triage
Infra failures were excluded before computing this flake rate. The remaining
failures cannot be explained by cluster setup, image pull, or proxy errors.
A draft fix PR will be opened by the detect-flaky-tests agent.
BODY
)"
Step 5b — Create issues for recurring infra failures
For each infra pattern appearing in ≥2 runs:
gh issue list \
--repo agent-substrate/substrate \
--state open \
--search "infra: <PATTERN_SUMMARY>" \
--json number,title
If none exists, create:
gh issue create \
--repo agent-substrate/substrate \
--title "infra: <PATTERN_SUMMARY>" \
--label "kind/bug,area/dev-infra" \
--body "$(cat <<'BODY'
## Recurring infrastructure failure in CI
**Pattern:** `<infra error pattern>`
**Job type:** `<unit | e2e-gvisor | e2e-microvm>`
### Evidence
| Metric | Value |
|---|---|
| Occurrences in last 7 days | <COUNT> |
| Example runs | <links> |
### Impact
This failure causes entire CI jobs to abort before tests run. It inflates
apparent failure rates and masks real test flakiness. Fixing it will improve
flakiness signal quality.
### Log excerpt
\`\`\`
<paste 3-5 lines of the actual error from the log>
\`\`\`
BODY
)"
Step 6 — Open a draft fix PR for each new flaky test
Read the test source file. Diagnose the likely cause using the patterns below (unit tests
and e2e tests share most root causes, but e2e has additional patterns):
Common Go flakiness patterns and fixes
| Pattern | Symptoms | Fix |
|---|
| Timing / sleep | time.Sleep before an assertion | Replace with require.Eventually or testutil.WaitFor |
| Shared global state | Package-level var mutated without cleanup | Move to test-local; t.Cleanup to restore |
| Port conflicts | Hardcoded port or race on ephemeral port | Use ln.Addr() from the actual listener |
| Goroutine leak | Goroutines from one test race the next | t.Cleanup(cancel) + wait for goroutines to exit |
| File system races | Shared temp path across parallel tests | Use t.TempDir() |
| Context not cancelled | Long operation outlives test | t.Context() (Go 1.21+) or t.Cleanup(cancel) |
| Order dependency | Test relies on prior test's side effects | Make each test self-contained |
| E2E: actor/pod not ready | Test proceeds before actor reaches Running state | Poll with require.Eventually on status, increase timeout with justification |
| E2E: resource cleanup race | Prior test's namespace/actor not fully deleted before next test | Add explicit WaitForDeletion in t.Cleanup |
| E2E: network policy timing | Policy applied but not yet enforced at assertion time | Retry the connectivity check, not just the policy application |
Steps for the fix PR:
- Create a branch:
fix/flaky-<test-name-kebab> from main
- Apply the fix
- Commit:
fix(tests): resolve flakiness in <TestName>\n\nFixes #<issue_number>
- Open a draft PR:
gh pr create \
--repo agent-substrate/substrate \
--title "fix(tests): resolve flakiness in <TestName>" \
--draft \
--body "$(cat <<'BODY'
## Summary
Fixes the flaky test `<TestName>` in `<package>` (`<job_type>` lane).
**Root cause:** <one sentence>
**Fix:** <one sentence>
Closes #<issue_number>
## Evidence
Flake rate over last 7 days: <FLAKE_RATE>% (<FAIL_COUNT> fail / <PASS_COUNT> pass)
Infra-failure runs excluded from count: <INFRA_EXCLUDED>
Failing runs: <links>
Passing runs: <links>
## Test plan
- [ ] Run `go test -race -count=10 ./path/to/package/...` locally for unit tests
- [ ] For e2e: re-run the affected suite 3+ times against a kind cluster
BODY
)"
Do not open a fix PR for infra issues — those require infra investigation, not test code changes.
Step 7 — Report
Output two tables:
Flaky tests
| Test | Job | Flake rate | Fail/Pass | Infra excluded | Issue | Fix PR | Action |
|---|
| TestFoo | e2e-gvisor | 40% | 4/6 | 2 runs | #NNN | #MMM | created |
Infra issues
| Pattern | Job | Occurrences | Issue | Action |
|---|
proxy.golang.org INTERNAL_ERROR | unit | 4 | #OOO | created |
If nothing found in either category: No new flaky tests or infra issues detected.
Notes
- This skill does NOT write to BigQuery, update a dashboard, or perform any storage
operations. Those are handled by the cron job that invokes this skill.
- If log download fails for a run (e.g. logs expired after 90 days), skip that run and
note it in the report.
- The
--add-label flag may fail if a label does not exist on the repo; fall back to
omitting labels and add a comment instead.
- E2E flakiness in one lane (gVisor or microVM) is flagged independently — a test does
not need to be flaky in both lanes to warrant an issue.
