test-intelligence

name	test-intelligence
description	Pull and analyze test results from a Semaphore pipeline — failure detail, flaky-test detection, JUnit parsing. Use when a job failed and you need to see WHICH test broke without changing the pipeline config or re-running; when a junit-only reporter hides failure detail in the log; when you're about to add a stdout reporter just to debug — pull the published JUnit artifact instead via `sem-ai test summary` / `sem-ai test report` / `sem-ai artifact get`. Supports Go, pytest, rspec, minitest, jest, ExUnit.
user-invocable	false

Test Intelligence

Quick summary

sem-ai test summary --pipeline <id>

Returns: verdict, total/passed/failed/skipped, failures with test name + file:line + message.

Detailed per-job report

sem-ai test report --pipeline <id>

Each job shows: framework detected, source (artifact or log), counts, failed test details.

Strategy: tries artifact JUnit JSON first (richer data), falls back to log parsing.

Frameworks: Go/gotestsum, pytest, rspec, minitest, jest, ExUnit, JUnit JSON.

Flaky detection

sem-ai test flaky --project my-app --branch main --count 10

Analyzes last N workflows. Flaky = test that sometimes passes, sometimes fails.

Common workflows

"Did tests pass?"

sem-ai test summary --pipeline <id>   # check "verdict" field

"Which test broke?"

sem-ai test summary --pipeline <id>   # check "failures" array

"Is this flaky?"

sem-ai test flaky --project <p> --count 10
# Cross-reference failing test name with flaky_tests list

"Get raw test artifact"

sem-ai artifact get --scope jobs --id <job-id> --path test-results/junit.json --output results.json

"Show raw test output"

sem-ai job log <job-id> --format table