| name | test-effectiveness-evaluation |
| description | Use to interpret test, JIT test, or mutation results and classify weak tests, effective tests, validation gaps, spec gaps, implementation issues, and recommended improvements. |
Test Effectiveness Evaluation
Purpose
Interpret validation results to decide whether tests are effective, weak, incomplete, unstable, or blocked by spec ambiguity.
Script-First Execution
Before classifying effectiveness, extract deterministic evidence from available test or mutation result artifacts:
python agent-skills/test-effectiveness-evaluation/scripts/collect_validation_result_evidence.py --json <result-file> [<result-file> ...]
The helper reads text result files, detects unittest run counts and pass/fail status, and extracts mutation term counts such as killed, survived, equivalent, skipped, and blocked. It does not decide whether a survived mutation is a test gap, spec gap, implementation issue, or equivalent mutation; use LLM judgement for that classification after evidence is collected.
Run the helper from Windows PowerShell or a POSIX shell (Linux/macOS) with the same command shape; use python3 when python is unavailable.
Prompt Contract
Stable prefix:
- Skill:
test-effectiveness-evaluation
- Execution Profile:
hybrid
- Reusable Rules: passing tests alone do not prove effectiveness; survived mutation is a gap until classified; do not promote generated tests without evidence; distinguish test weakness from spec ambiguity.
- Scope / Governance Defaults: do not edit tests or code unless explicitly asked; do not decide human governance questions; do not hide flaky or inconclusive results.
- Output Contract: see the Test Effectiveness Report template under
## Output.
Dynamic run packet:
- User Request:
- Deterministic Evidence:
- Relevant Files Or Artifacts:
- Current Assumptions Or Gaps:
- Requested Judgement Or Transformation:
Workflow
- Read deterministic validation-result evidence, baseline test results, JIT test results, mutation results, spec refs, risk items, and impact analysis.
- Identify which tests passed baseline and which killed meaningful mutations.
- Classify survived mutations as test gap, spec gap, implementation issue, equivalent mutation, accepted risk, or unclear.
- Identify flaky, brittle, redundant, or coverage-only tests.
- Recommend refine tests, update spec, fix implementation, discard candidate tests, or request human decision.
- Provide promotion recommendation for generated tests when appropriate.
- Hand decisions to
decision-proposal, test-promotion, or spec-test-evolution.
Mandatory Rules
- Passing tests alone do not prove effectiveness.
- Survived mutation is a gap until classified.
- Do not promote generated tests without evidence.
- Distinguish test weakness from spec ambiguity.
- Report tooling limitations clearly.
Boundaries
- Do not edit tests or code in this skill unless explicitly asked.
- Do not decide human governance questions.
- Do not hide flaky or inconclusive results.
Validation
Check:
- Baseline result is considered.
- Mutation or manual mutation result is considered.
- Gap classification is explicit.
- Recommended next action is clear.
- Human decision needs are identified.
- If the helper script changed, run its unit tests and a CLI smoke check.
Output
Use this report template:
### Test Effectiveness Report
- Effective Tests:
- Weak Tests:
- Candidate Tests:
- Survived Mutation Classification:
- Validation Gaps:
- Recommended Improvements:
- Promotion Recommendation:
- Human Decision Needed:
