test-quality-assurance

name	test-quality-assurance
version	0.1.27
description	Use when assessing whether a test suite follows TDD discipline, validates behavior vs implementation, has a sound unit/integration/white-box balance, or needs prioritized quality actions beyond raw coverage numbers. Produces heuristic, evidence-tiered 0-24 rubric scores (script_baseline / mixed-tier provisional only; a human reviewer is required for an expert-confirmed final grade), delta reports, and action plans. Supports coverage.json for Numba, Cython, and standard Python projects.

Test Quality Assurance

Overview

Use this skill to evaluate test-suite quality architecture. It focuses on TDD discipline, contract coverage, behavior-vs-implementation coupling, and actionable improvement plans.

Autonomous/unattended runs can only reach script_baseline or mixed-tier provisional status. Reaching expert-confirmed final requires a human reviewer to confirm every dimension and classify all limitations — an agent must stop at provisional and hand off.

This skill does NOT make keep/merge/delete redundancy decisions. It assesses quality and produces evidence-linked recommendations.

Boundaries

• Keep/merge/delete redundancy decisions are out of scope. If test-redundancy-triage is available and the user asks for deletion or merge decisions, hand off there; otherwise list this as an external follow-up
• Measured profiling and benchmark comparisons are out of scope. If perf-benchmark is available and the user asks for measurement, hand off there; otherwise limit this skill to QA evidence about benchmark intent, gates, and coverage
• Report duplicate/overlap signals only as QA architecture risks; do not make keep/merge/delete decisions
• This skill audits quality architecture and produces evidence-linked findings, open questions, and a prioritized improvement plan

Use This Skill When

• A team asks whether tests follow TDD best practices
• You need to assess if tests validate behavior or implementation details
• You need a unit/integration/white-box/black-box balance review
• You need quality gaps and prioritized next actions, not just coverage numbers

Workflow

0. Audit Scope and Provenance

Record the audit charter before collecting evidence. This step pins the review target so that every score, finding, and limitation is traceable to a specific revision and environment. Skip this step only when the audit is a draft heuristic run with script_baseline status.

Required charter fields:

Field	Description
Target repository	Path or URL of the repository under audit
Revision	Git commit SHA, tag, or branch head at audit time
Environment	OS, Python version, and key dependency versions
CI-equivalent test command	Full pytest command (or equivalent) that represents the CI test matrix
Included / excluded suites	Explicit list of test directories, markers, or files in and out of scope
Audit depth	heuristic-only (script run only), runtime-partial (some execution evidence), or full (all evidence gates collected)
Risk focus	Known high-risk areas, if the audit is scoped to specific modules or contract families
Tool / skill / script versions	audit_test_quality.py version, skill version (from SKILL.md frontmatter), and any external tool versions (coverage.py, mutmut, etc.)
Generated artifact paths and hashes	Paths and SHA-256 hashes of generated JSON/MD/coverage artifacts
Known constraints	Time limits, unavailable environments, policy restrictions, or missing tooling declared up front

Provenance example:

Target: /path/to/project @ a1b2c3d
Env: Linux, Python 3.12.3, pytest 8.3.4, coverage 7.8.0
CI test cmd: python3 -m pytest -q --durations=10
Scope: tests/ (all), --ignore=tests/integration/slow (excluded)
Depth: runtime-partial (single run + coverage; no mutation or repeat runs)
Risk focus: algorithm module src/algo/, public API src/api/
Skill version: 0.1.25
Audit script: audit_test_quality.py v0.1.25
Artifacts:
  test_quality.json  sha256:a1b2...
  test_quality.md    sha256:c3d4...
  coverage.json      sha256:e5f6...
Constraints: pytest-xdist unavailable; mutation tooling not installed

Reports that omit the audit scope block must note it as a limitation classified blocker-for-final — they cannot reach expert-confirmed final without provenance metadata.

1. Runtime and Tooling Baseline

Target-Repository Safety

When the target repository is under git version control, always verify a clean tree before collecting execution evidence. Commands that run tests, generate coverage, or produce mutation data may create side-effect artifacts (.coverage, __pycache__, mutation reports, temporary test files).

Pre-execution safety check:

# Verify clean target tree (run from the target repository root)
git -C /path/to/target/repo status --short

If the tree is dirty, record the dirty files as a limitation and assess whether the dirt affects audit validity. Do not proceed with execution evidence collection that could contaminate the target repository — use a dedicated artifact directory outside the target tree.

Planned artifact directory: Before collecting any execution evidence, create a dedicated, unique artifact directory outside the target repository. Each audit run must allocate its own empty directory so that coverage data, reports, logs, hashes, and provenance records cannot be overwritten by another same-day run. All generated files (.coverage, coverage.json, test output logs) must be written to this directory, never to the target repository.

artifact_dir=$(mktemp -d "${TMPDIR:-/tmp}/tqa-audit.XXXXXXXXXX")

Post-execution verification:

# Confirm no side effects leaked (run from the target repository root)
git -C /path/to/target/repo status --short
# Expected output: empty (same as pre-execution check)

If new files appear, list them as a limitation and assess impact.

Side-effecting suites: Some test suites create or modify files during execution (temporary databases, cache files, generated test data). Before running such suites, review the test commands for known side effects and either run them in a sandbox environment (container, dedicated worktree) or record the expected side effects as a limitation. Never run a suite with known data-mutating side effects against a production-like environment without explicit approval.

Baseline Commands

Confirm the interpreter, dependencies, and test collection before analysis:

export PYTHONDONTWRITEBYTECODE=1
python3 -V
python3 -m pip check
python3 -m pytest --collect-only -q -o cache_dir="$artifact_dir/.pytest_cache"

Collect bounded execution evidence for determinism assessment:

# Recommended: single quiet run for pass/fail baseline
python3 -m pytest -q -o cache_dir="$artifact_dir/.pytest_cache"

# For flake-sensitive suites: repeat N times (default 3)
python3 -m pytest -q --count=3 -o cache_dir="$artifact_dir/.pytest_cache"      # if pytest-repeat is available
# or:
for i in 1 2 3; do python3 -m pytest -q -o cache_dir="$artifact_dir/.pytest_cache"; done

# Optional: parallel stability (only when pytest-xdist is available)
python3 -m pytest -q -n auto -o cache_dir="$artifact_dir/.pytest_cache"

When test execution is not feasible (missing environment, time constraints, or external dependencies), record this as a limitation. Skipped execution evidence must be listed explicitly and keeps the Determinism and Isolation data_evidence_tier: static-provisional. Whether the overall audit can reach expert-confirmed final depends separately on the reviewer's review_status and limitation classification: if the skipped execution is accepted as nonblocking, the dimension may still be review_status: reviewed when the reviewer endorses the capped/static score and records the limitation as accepted/capped/nonblocking or limitation-only. If skipped execution is a blocker for a given audit, classify it explicitly as blocker-for-final, not as a side effect of the data evidence tier.

See references/methodology-completeness.md for per-dimension evidence-tier upgrade rules: repeated/parallel runs upgrade Determinism and Isolation; coverage JSON upgrades only the coverage sub-signal; mutation tooling or manual review upgrades the mutation sub-signal; benchmark artifacts upgrade Non-Functional Contract Coverage.

2. Contract Map

Build a concise map of public API contracts, algorithm invariants, and non-functional contracts (performance/memory/stability). This is the truth source for behavior-first assessment.

Required Fields. Each contract entry must include:

1. Contract ID: unique short identifier (e.g. CT-01, ALG-sort, NF-perf-latency)
2. Source / Test Basis: public function, invariant rule, or non-functional threshold
3. Risk / Criticality: high, medium, or low based on failure impact
4. Owner / Source Evidence: team, module, or document owning the contract
5. Linked Tests: test file paths or function names exercising the contract
6. Evidence Status: source of the contract evidence (source-only, linked-test, runtime-artifact, or reviewed) plus any relevant artifact IDs. Contract-map evidence status is not the same as the rubric score table's data_evidence_tier / review_status fields.
7. Gap Status: covered, partial, or missing

This schema is compact and standards-style; it does not claim formal ISO/ISTQB compliance. Reference contract IDs in findings and scoring rationale. Contract risk/criticality levels feed into scoring guardrails (see references/rubric.md Risk-Based Scoring Guardrails).

3. Process and Gate Inventory

Inventory process controls and quality gates before scoring. Record each item as reviewed (source cited) or unavailable (listed as a limitation).

Required inventory items:

1. Test strategy documentation: existing test policy docs or README sections
2. CI gate configuration: CI pipeline test stages, coverage thresholds, must-pass suites
3. Threshold / ratchet policies: any enforced coverage or mutation score gates
4. Must-pass contract suite: test markers or directories treated as gating
5. Ownership map: CODEOWNERS, git-blame patterns, or team assignments for test areas
6. Test review checklist: documented review criteria for new or modified tests
7. Regression / defect lineage: evidence of tests written after bug fixes (regression tests), red-green-refactor commit/PR patterns, or documented regression test ownership. Required when high-confidence TDD or Contract Coverage claims are made.

This step captures process evidence without expanding into full QMS compliance. Mark unavailable items as limitations in the final report, and classify each unavailable control by its score impact: limitation only, finding, score cap, or blocker for expert-confirmed final status. See references/methodology-completeness.md for the process-control-to-rubric mapping table.

4. Coverage and Mutation Evidence Collection

When coverage or mutation data is available, collect it with deterministic commands before running the static inventory. When unavailable, declare it explicitly as a limitation.

Artifact lifecycle and isolation: All generated artifacts (.coverage, coverage.json, mutation reports) must be written to a dedicated artifact directory outside the target repository (see Step 1 Target-Repository Safety). This prevents side effects and keeps audit artifacts isolated.

Coverage JSON

# Prevent bytecode side effects and route cache to artifact directory
export PYTHONDONTWRITEBYTECODE=1
export COVERAGE_FILE="$artifact_dir/.coverage"

# Preferred: coverage.py with branch coverage
python3 -m coverage run --branch -m pytest -q -o cache_dir="$artifact_dir/.pytest_cache"
python3 -m coverage json -o "$artifact_dir/coverage.json"

# Alternative: pytest-cov with branch coverage
python3 -m pytest --cov=. --cov-branch --cov-report=json:"$artifact_dir/coverage.json" -q -o cache_dir="$artifact_dir/.pytest_cache"

Artifact hashing: After generating each artifact, record its SHA-256 hash for inclusion in the Audit Scope and Provenance section:

sha256sum "$artifact_dir/coverage.json"
sha256sum "$artifact_dir/.coverage"

When a .coveragerc or pyproject.toml already enables branch = True, the --branch and --cov-branch flags are optional but recommended for visibility. If branch coverage is not collected and the existing config does not activate it, record branch coverage evidence unavailable as a limitation.

If the suite cannot execute or neither tool is available, skip and record "coverage collection unavailable" as a limitation. Without coverage data the Coverage and Mutation Signal dimension starts at static-provisional.

Mutation Evidence

Mutation evidence is required for score 3 on the Coverage and Mutation Signal dimension. Accepted sources, in order of preference:

1. Tool-generated: mutmut, cosmic-ray, mutpy, or similar mutation testing tool output (JSON or report)
2. CI gate: documented mutation threshold enforced in CI with historical runs
3. Manual expert review: documented review of mutation kills in critical algorithm modules, with kill/survive counts per mutant

When no mutation evidence is available, declare it as a limitation: the dimension is capped at the coverage-only ceiling (score 2). Expert review of the capped score still qualifies as expert-confirmed (coverage) / unavailable (mutation) for finalization purposes.

5. Static Inventory

Run scripts/audit_test_quality.py for a fast heuristic baseline. Inspect --help and generated outputs before reading script internals unless you are implementing or debugging it.

Installed-skill invocation (from any target repository):

# Resolve the script path relative to the loaded SKILL.md directory.
# Common default (assumes install dest ~/.agents/skills):
python3 $HOME/.agents/skills/test-quality-assurance/scripts/audit_test_quality.py \
  --root /path/to/target/repo \
  --json-out "$artifact_dir/test_quality.json" \
  --md-out "$artifact_dir/test_quality.md"

# Robust fallback when SKILL_DIR or CODEX_HOME is set, otherwise use ~/.agents:
SKILL_DIR="${SKILL_DIR:-${CODEX_HOME:+$CODEX_HOME/skills/test-quality-assurance}}"
SKILL_DIR="${SKILL_DIR:-$HOME/.agents/skills/test-quality-assurance}"
python3 "$SKILL_DIR/scripts/audit_test_quality.py" --root /path/to/target/repo ...

Local checkout invocation (from within this skill's repository):

python3 scripts/audit_test_quality.py \
  --root /path/to/repo \
  --public-hint "compute(" --public-hint "run_pipeline(" \
  --cov-json "$artifact_dir/coverage.json" \
  --json-out "$artifact_dir/test_quality.json" \
  --md-out "$artifact_dir/test_quality.md"

Portability knobs:

• --tests-dir / --test-glob when tests are not in a standard tests/ layout
• --internal-import-pattern for repo-specific internal import regex
• --public-hint for public API call substrings (auto-inferred from __init__.py exports by default)
• --no-auto-public-hints to disable inference
• --cov-json for a pre-generated coverage.json (prefer the artifact generated at $artifact_dir/coverage.json; any external coverage input must have its provenance/hash documented and an accepted limitation recorded before finalizing).
• --baseline-json for delta reporting (see Step 5a below)

5a. Delta Reporting (Optional)

Delta reporting compares two audit runs using a promoted expert-final baseline to track score trends and evidence-tier changes. The single key rule: --baseline-json accepts only a promoted expert-final baseline JSON produced by --final-baseline-json; raw script output is never an accepted baseline. See references/delta-reporting.md for the full procedure, schemas, tables, and interpretation rules.

6. Score With the Rubric

Evaluate against references/rubric.md (8 dimensions, 0-3 each). The static script emits heuristic scores with its raw evidence_tier field; the final audit report maps that raw script evidence into script_data_evidence_tier and script_review_status: unreviewed, then records final data_evidence_tier and review_status after expert review. Every final report score carries both axes per the model defined in references/methodology-completeness.md.

Apply risk-based scoring guardrails from references/rubric.md: high-criticality contract gaps cap dimension scores and raise finding severity. For example, a high-criticality contract with no linked test caps Contract Coverage at ≤1.

7. Classify Tests

Label tests as contract, white-box sentinel, or change indicator. The script provides candidate counts; the auditor completes missing role labels (unit/integration/property/e2e).

8. Findings and Questions

Produce severity-ordered, evidence-linked findings. Use references/question-bank.md to expand open questions.

9. Improvement Plan

Produce 3-8 concrete actions with expected impact. Each action must follow the lifecycle schema defined in references/rubric.md Action Plan Lifecycle Schema: action ID, owner/role, acceptance criteria, verification command/artifact, due date or trigger event, expected impact (dimension and estimated signal gain), and status (open/in_progress/closed/wont_fix). See references/sample-report.md for an example table.

Artifact Interpretation

• Script JSON/markdown output is a static heuristic inventory — not a final grade
• Automated scores are raw evidence_tier values in script JSON/markdown. Map raw script output to script_data_evidence_tier with script_review_status: unreviewed. Evidence-tier upgrades follow the per-dimension rules in references/methodology-completeness.md.
• Without mutation evidence the Coverage and Mutation Signal dimension is capped at score 2 (coverage-only ceiling). Score 3 requires mutation evidence and expert confirmation.
• Final findings must be evidence-linked and may override script scores
• Every final report rubric score carries a data_evidence_tier and review_status. See references/methodology-completeness.md for the two-axis evidence model, per-dimension upgrade rules, and limitation taxonomy.

Outputs

Score Finalization Policy

Every total score must carry an explicit total status that gates how the total may be interpreted:

Total Status	Definition	Can total be read as final?
script_baseline	All dimensions have data_evidence_tier: static-provisional and review_status: unreviewed; no expert review applied.	No — use only as a heuristic starting point.
mixed-tier provisional	Any report that is not script_baseline and not expert-confirmed final. This includes when at least one dimension has received expert review (i.e. review_status: reviewed) but not all dimensions are confirmed, or at least one blocker-for-final limitation is documented. Accepted/capped/nonblocking and limitation-only entries alone do not force mixed-tier provisional — the report can reach expert-confirmed final with these documented if all dimensions are confirmed.	No — the total is provisional until all dimensions are expert-confirmed and no blockers remain.
expert-confirmed final	Every dimension's review_status is reviewed OR, for the Coverage and Mutation Signal dimension, available sub-signals are reviewed and unavailable sub-signals are explicitly classified as accepted limitations (e.g. reviewed (coverage) / unavailable (mutation)). No blocker-for-final limitations remain. The reviewer has confirmed each score and classified all limitations.	Yes — the total is the final grade.

Limitations are classified per the taxonomy defined in references/methodology-completeness.md Limitation Classification Taxonomy: accepted (capped, nonblocking), limitation-only, or blocker-for-final. Only blocker-for-final limitations prevent a report from reaching expert-confirmed final status.

A report whose total line omits the total status is incomplete. Rubric interpretation bands apply conditionally based on the total status (see references/rubric.md).

Required Sections

Final reports must include:

0. Audit scope and provenance: charter fields as defined in Step 0 (target repository, revision, environment, CI-equivalent test command, included/excluded suites, audit depth, risk focus, tool/skill/script versions, generated artifact paths and hashes, and known constraints). Reports without scope metadata cannot reach expert-confirmed final status.
1. Evidence-tiered score table: all 8 rubric dimensions with script score (0-3), final score (0-3), script data evidence tier and review status, final data evidence tier and review status, confirm/override status, rationale, reviewer/role, review date, evidence IDs, max (3), limitations or unavailable evidence. See references/rubric.md Score Provenance and Finalization for the full provenance schema.
2. Best-practice rules
3. Evidence map with source references: source/test claims require file:line citations; generated evidence (script output, coverage data, tool reports) may use command, artifact path, JSON key/pointer, and a short output excerpt/summary
4. Contract traceability table (see Contract Map required fields)
5. Process evidence section (see Process and Gate Inventory)
6. Execution evidence section with commands run, results, and unavailable evidence
7. Severity-ordered findings
8. Limitations and unavailable evidence: a standalone section listing every limitation that affects score confidence, including unavailable mutation data, skipped execution evidence, missing process controls, and any other evidence gaps. Each limitation must carry its classification per the taxonomy defined in references/methodology-completeness.md: accepted (capped, nonblocking), limitation-only, or blocker-for-final.
9. Open questions / question-bank prompts
10. Prioritized action plan with lifecycle metadata per references/rubric.md Action Plan Lifecycle Schema: action ID, owner/role, acceptance criteria, verification command/artifact, due date or trigger, expected impact, and status

Framework-Specific Caveats

• Numba: Set NUMBA_DISABLE_JIT=1 before coverage collection
• Cython: Collect coverage via tracing:
  1. Compile with linetrace=True and define CYTHON_TRACE=1 as needed
  2. Enable [run] plugins = Cython.Coverage in .coveragerc
  3. Collect coverage JSON/XML through coverage.py as normal
  4. cython --annotate is optional visualization, not collection
  5. Mark Cython coverage unavailable/limited when this setup is absent
• ctypes/CFFI: C extensions not instrumented by coverage.py

References

• references/rubric.md: scoring dimensions
• references/question-bank.md: guided questions
• references/sample-report.md: format example
• references/methodology-completeness.md: QA practice matrix
• references/delta-reporting.md: delta reporting workflow
• scripts/audit_test_quality.py: static inventory