summarize-test-suite

name	summarize-test-suite
description	Summarize a GitHub Actions test suite run for the mongodb/terraform-provider-mongodbatlas repo — fetches job logs, classifies failures, and produces a Slack summary led by an explicit code-regression verdict. Use when the user shares a workflow run URL or run_id, or when triggered by the test-suite webhook to decide whether on-call should investigate.

Summarize Test Suite Execution

Inputs

The only input is a run_id (GitHub Actions run identifier). The repo is always mongodb/terraform-provider-mongodbatlas, and the run URL is https://github.com/mongodb/terraform-provider-mongodbatlas/actions/runs/<run_id> — derive both from run_id.

run_id can arrive as a workflow run or job URL the user pastes (extract the numeric ID from …/actions/runs/<run_id> or …/actions/runs/<run_id>/job/<job_id>), as a number passed by the test-suite webhook, or as a value the caller spells out directly.

Output

The output is always Slack-flavoured markdown (mrkdwn) — *bold* (never **bold**), backticks for code, • for bullets, <url|label> for links. The caller is responsible for posting it to Slack; the skill only produces the text.

The skill operates only on the run it is given — do not look at prior runs or other workflows.

Goal

The first character of the output is one of three emojis that telegraph urgency to on-call at a glance:

• :red_circle: — at least one code regression. Investigate now.
• :yellow_circle: — only infrastructure noise (capacity / timeout / cleanup / API contract). No on-call action.
• :green_circle: — all tests passed. No on-call action; this is the daily heartbeat.

Confidence

Decide a confidence level for your verdict and embed it on the second line of the summary (templates below). Always show it — high confidence is useful information for on-call, not just low.

• high — the failure pattern maps cleanly onto the rules in Step 4, the categorisation matches the logs you read, and you are sure about the verdict.
• medium — the verdict is clear, but some category counts are estimated (you didn't enumerate every package's logs) or one failure type didn't fit any category cleanly.
• low — the failure pattern is novel, the logs were incomplete or truncated, you had to make a judgement call between two categories for ≥1 failure, or the rules in this skill don't cover something you saw.

Confidence modulates urgency on the second line: lower confidence on a yellow or green verdict raises the action from "no immediate action" to "please review", because the agent is signalling it might have missed something. Red is always "investigate immediately" regardless of confidence — but the confidence prefix still appears so the on-call knows whether to fully trust the regression call.

When confidence is medium or low, the summary must end with two extra lines:

• *Why <confidence> confidence*: <one or two sentences naming the specific ambiguity> — e.g., "saw an INVALID_REGION API error that doesn't match the category 3a indicators", "a flex_cluster package failed with no --- RUN/--- PASS and an unfamiliar unable to start dispatcher line, ambiguous between build error and cleanup", "two failing tests share a deferred-test-helper panic that doesn't match any category 1 example", or "the generic job's logs returned HTTP 404, so its failures could not be enumerated". Be concrete; vague phrases like "complex output" aren't useful.
• If this ambiguity recurs, consider updating the skill rules. — verbatim, immediately after the Why line. This is a standing nudge that medium / low confidence often reflects a missing rule rather than a one-off run; the maintainer can use it to decide when to revisit the skill.

Both lines are omitted on high confidence.

Provider repo layout

When you cite "code regressions in ", use the actual Go package path, not the test-job display name. The job name maps to a Go package by dropping underscores, lowercasing, and prefixing with internal/service/ — e.g., advanced_cluster → internal/service/advancedcluster, global_cluster_config → internal/service/globalclusterconfig. Apply this rule for any job not covered by the special cases below.

Special cases (do not apply the simple rule):

• network is a super-job covering several packages (networkcontainer, networkpeering, privatelinkendpoint*, privateendpointregionalmode). Cite the specific package the failing test lives in, not "network".
• autogen_fast / autogen_slow run the autogenerated resources under internal/serviceapi/.
• config_sa_mig runs the same packages as the config group, but only the migration tests, in SA-auth mode.
• clean-before / clean-after are cleanup utilities under internal/testutil/clean/, not provider code. Failures in these never count as code regressions.

Workflow

Step 1: Run context and failed jobs

First, fetch the run metadata so the summary can cite which environment was tested:

gh api repos/mongodb/terraform-provider-mongodbatlas/actions/runs/{run_id} \
  --jq '{head_sha, run_number}'

Extract commit: first 7 chars of head_sha.

Then list jobs:

gh api repos/mongodb/terraform-provider-mongodbatlas/actions/runs/{run_id}/jobs --paginate

Derive env and auth from job names (the test-suite run's display_title is always just "Test Suite" and does not embed them):

• auth: any test-job name has the matrix prefix <terraform_version>-<provider_version>-<auth> (e.g. 1.14.x-latest-pak / ... or 1.14.x-latest-sa / ...). Extract the third dash-separated token of the prefix (pak or sa).
• env: nested called-workflow jobs are named tests-<terraform_version>-<provider_version>-<env> / ... (e.g. tests-1.14.x-latest-dev or tests-1.14.x-latest-qa). Extract the suffix after the last dash of that segment (dev or qa).

If no test jobs exist (only cleanup ran), default env/auth to "n/a".

Split jobs with conclusion: "failure" into two buckets:

• Cleanup jobs: any job whose name starts with clean-before or clean-after. Treat all failures inside these collectively as category 5 (cleanup) — surface as a single line ("Cleanup: N stuck projects from prior runs"), do not enumerate the TestCleanProjectAndClusters/<id> subtests in the Failing tests list. Cleanup-job failures are leftovers from previous runs, never signals of a code regression in the current run.
• Test jobs: everything else. These go through Steps 2 to 4.

If both buckets are empty, emit the green template (:green_circle:) immediately and return.

Step 2: Fetch failure verdicts strictly

For each test job, fetch the per-test verdicts using a strict filter. Do not grep for Error or generic FAIL — [ERROR] sdk.proto: lines and Error in create/update/delete are emitted by many tests that ultimately PASS:

gh api repos/mongodb/terraform-provider-mongodbatlas/actions/jobs/{job_id}/logs 2>&1 \
  | grep -E "(--- FAIL|^FAIL\t|panic:)"

The --- FAIL: TestName line is the source of truth for an individual test failure. Never classify a test as failed without one.

Package-level FAIL\t<package> without any --- FAIL: TestName for that package is not a test failure. Sub-classify it before counting:

• If the job log shows tests started for that package (any --- RUN or --- PASS lines for it), the failure is post-test teardown (deferred TestMain, cleanup) — category 5 (cleanup).
• If no tests started for that package (no --- RUN or --- PASS lines, just compile errors like # <package> followed by line:column errors, or cannot find module, command not found, undefined:, dependency-resolution failures, or panic: during go test startup, dependency download, or binary installation), it is a build / tooling error that prevents the runner from executing any tests in that package — category 1 (code regression). Examples: a recent provider commit broke a transitive import, a dependency yanked a version, a test-infra change broke binary discovery, a third-party installer panicked verifying a signature or checksum. These warrant immediate investigation regardless of whether the root cause is internal or upstream — the runner can't verify provider behaviour at all.

In neither case should the package name go in the Failing tests list — instead, surface a build error as its own bullet under code regressions ("Build error in <package>: ") and a cleanup-only package failure as part of the cleanup count.

When gh api .../jobs/{job_id}/logs returns an HTTP error (404, 403, 500) or empty output: do not guess the job's category. The job's tests cannot be verified. Surface it as a single bullet in the summary body — • Logs unavailable: \` (failed after min — logs inaccessible)— and do **not** count its tests toward any category and do **not** list them in the *Failing tests* line. Logs-unavailable jobs do not, by themselves, escalate the verdict to red, but they are still failed jobs: when a logs-unavailable job exists, the floor for the leading emoji is:yellow_circle:(never:green_circle:), even if no category 1–5 indicators were found in any other job. Set confidence to **medium** with this exact text used as the Why medium confidence:line at the end of the summary:logs unavailable for `` (); all other jobs cleanly classified, where isHTTP (e.g.HTTP 404) when the API returned a non-2xx status, or empty log response` when the call succeeded but the body was empty. Rationale: a job that ran tens of minutes and then failed is most likely infra noise, but without logs you cannot confirm; medium confidence asks on-call to glance without triggering a false red alert.

Subtest aggregation: a --- FAIL: Parent/<subname> line is one subtest of Parent. If multiple subtests of the same parent fail in the same job, list Parent once with the count (Parent (×N subtests)), not N separate entries.

Step 3: Get error context for each failed test

For each unique parent test (after subtest aggregation), fetch the surrounding context:

gh api repos/mongodb/terraform-provider-mongodbatlas/actions/jobs/{job_id}/logs 2>&1 \
  | grep -B10 "FAIL: TestName"

Step 4: Classify failures

Assign each failing test (or aggregated subtest group) to exactly one category, first match wins:

Priority	Category	Indicators
1	Code regression (high-signal)	Error: Provider produced inconsistent result after apply / provider produced an unexpected new value / This is a bug in the provider / any panic: line (in test execution, in TestMain, during dependency install, or during binary-installer signature/checksum verification) EXCEPT the Go test framework's deadline panic (see exception below) / Plugin did not respond / rpc error: / non-empty plan after a successful apply / assertion failures on attribute values / unexpected plan diffs / build or tooling errors that prevented any tests from running in a package (per the Step 2 sub-classification) / INVALID_ATTRIBUTE from the Atlas API when a test fails (the provider sent a field or value the API rejects; root cause may be provider code, test config, or a new Atlas API validation, but all cases require investigation and a fix)
2	Cloud capacity	OUT_OF_CAPACITY, NO_CAPACITY, No Capacity
3	API errors	Generic HTTP 4xx/5xx with API error codes (only when not covered by category 1; raw API errors that are not provider bugs go here); polled-state failures of the form unexpected state '<terminal-fail-state>', wanted target '<success-state>' (emitted by terraform-plugin-sdk's retry helper when an Atlas-side resource enters a terminal failure state during create / update / delete polling — e.g. ). The test got a definitive negative answer from the Atlas backend, not no answer