test-challenger

name

test-challenger

description

Challenge AI-generated (or any) unit tests to find false positives — tests that pass against wrong behavior. Use this skill whenever the user asks to review, audit, validate, or challenge test quality, especially tests generated by AI to increase code coverage. Triggers on phrases like 'review these tests', 'are these tests actually correct?', 'challenge my tests', 'test quality audit', 'false positive tests', 'these tests were auto-generated', 'AI wrote these tests', 'coverage went up but I don't trust it', 'are these tests testing the right thing?', 'validate test correctness', or any situation where tests exist alongside source code and someone wants to know whether the tests are genuinely verifying correct behavior vs. merely mirroring implementation bugs. Also trigger when the user mentions increasing code coverage with AI, copilot-generated tests, or test generation tools — these are high-risk contexts for false positive tests.

Test Challenger

A test is only as valuable as its independence from the code it tests. When expected behavior is derived by observing code rather than understanding the spec, the test cannot catch bugs — only regressions. This skill identifies tests likely validating what the code does rather than what the code should do.

---

The Oracle Problem

Every meaningful test has:

1. Stimulus — calling the code with inputs
2. Oracle — expected result, from an independent source of truth

Human-written: oracle from spec, domain knowledge, requirements. AI-generated by observing code: oracle from the code itself. Circular: the test validates the code, but the code defined the test. Tautology — never fails for the right reasons.

This skill applies challenge patterns to estimate the probability that a test's oracle was derived from the code rather than from genuine domain understanding.

---

How to Execute

STEP 1 — Gather Context

Need:

1. Test file(s) — tests being challenged
2. Source file(s) — code being tested

If only one provided, ask for the other.

Optional (ask if available):

• Specification, requirements, or documentation — the independent oracle
• Git/PR context — AI-generated origin increases prior suspicion
• Language/framework — some patterns are framework-specific

STEP 2 — Classify Each Test

Work through the False Positive Pattern Catalog. A test may exhibit multiple patterns. Assign a Challenge Verdict:

Verdict	Meaning
Solid	Independent oracle; meaningful assertions; would catch real bugs
Suspicious	1-2 weak patterns; warrants human review
Likely False Positive	Strong indicators oracle came from code, not domain knowledge
Vacuous	Zero verification value — passes regardless of correctness

STEP 3 — Apply the Catalog

Patterns ordered most obvious to most subtle. Skip irrelevant ones.

---

False Positive Pattern Catalog

Pattern 1: Tautological Test (Likely FP)

What: Test reimplements the function's logic in the assertion. Structurally impossible to fail.

Detect:

• Assertion mirrors implementation step-by-step
• Expected value computed using same algorithm as source
• Essentially "does f(x) equal f(x)?"

# Source
def calculate_tax(amount, rate):
    return amount * rate * 0.01

# Tautological — reimplements the formula
def test_calculate_tax():
    result = calculate_tax(100, 20)
    assert result == 100 * 20 * 0.01  # just repeating the code

Challenge: "If implementation had a bug (rate * 0.1 instead of rate * 0.01), would this catch it?" If no because you'd change the expected value too — tautological.

---

Pattern 2: Hardcoded Oracle From Execution (Likely FP)

What: Expected value obtained by running code and copying output, not from domain knowledge.

Detect:

• Magic numbers without comment
• Suspiciously specific values (many decimals, odd structures)
• Value only knowable by running code first
• No spec/formula justifies it

def test_compute_score():
    result = compute_score(users=[...], weights=[...])
    assert result == 73.4821  # where did this number come from?

Challenge: "Could a developer who has never seen the implementation derive this from the requirements?" If not, oracle came from execution.

---

Pattern 3: Snapshot/Approval Test Without Specification (Suspicious)

What: Captures snapshot of current behavior. Legitimate for regression detection; zero value for correctness validation.

Detect:

• assertSnapshot, toMatchSnapshot, toMatchInlineSnapshot, similar
• Large expected outputs that look copy-pasted
• Tests named should_match_expected_output without specifying "expected"

Challenge: "Was this snapshot reviewed against a spec before commit, or auto-accepted?" Auto-accepted = locking in current behavior, bugs included.

---

Pattern 4: Assertion-Free or Weak-Assertion (Vacuous)

What: Executes the code but asserts nothing meaningful — or trivially true.

Detect:

• No assertions ("smoke test")
• Only non-null / non-empty / no-exception
• Asserts type but not value (assertIsInstance(result, dict))
• Asserts length but not content

def test_process_data():
    result = process_data(input_data)
    assert result is not None  # tells us almost nothing
    assert len(result) > 0     # nor does this

Challenge: "What specific wrong behavior would this catch?" Can't name one → vacuous.

---

Pattern 5: Happy-Path-Only Coverage (Suspicious)

What: Only exercises paths that already work. Never probes boundaries, edges, errors — where bugs live.

Detect:

• All inputs are "nice" (small positives, short strings, non-empty lists)
• No tests for: null/empty, boundaries, max sizes, negatives, Unicode, concurrency, errors
• Covers lines, not decision boundaries

Challenge: "What if I fed this malformed, empty, enormous, or adversarial input? Test for that?" If not, coverage is cosmetic.

---

Pattern 6: Over-Mocked Test (Suspicious)

What: So many mocks the test only verifies internal wiring. Passes regardless of whether real dependencies work.

Detect:

• More mock setup than test logic
• Mocks return hardcoded values matching what code expects
• Would pass if real dependency was completely broken
• Verifies method was called, not that result is correct

def test_send_notification():
    mock_email = Mock()
    mock_email.send.return_value = True
    service = NotificationService(email=mock_email)
    result = service.notify(user, message)
    mock_email.send.assert_called_once()  # only checks wiring
    assert result is True                  # always true — mock returns True

Challenge: "If I replaced the mock with a real (broken) implementation, would this still pass?" If yes, testing the mocks.

---

Pattern 7: Implementation-Coupled Assertion (Likely FP)

What: Asserts on implementation details (internal state, call order, private methods) rather than observable behavior. Breaks on refactor; misses behavioral bugs.

Detect:

• Assertions on private/internal method calls
• Assertions on execution order when order doesn't matter
• Assertions on internal data structures rather than output
• Breaks on refactor without behavior change

Challenge: "Could I refactor the implementation (no behavior change) and would this break?" Yes = coupled to implementation.

---

Pattern 8: Coincidental Pass (Likely FP)

What: Assertion passes by coincidence — expected and actual match for wrong reasons.

Detect:

• Generic values many incorrect implementations would produce (0, 1, [], true)
• Single input tested, happens to be a fixed point
• Trivially wrong implementation passes (return 0 passes a test expecting 0)

def test_multiply():
    assert multiply(0, 5) == 0    # passes even if multiply() returns 0
    assert multiply(1, 7) == 7    # passes even if multiply() returns the second arg

Challenge: "Simplest wrong implementation that still passes?" Trivially wrong passes → low discrimination.

---

Pattern 9: Missing Negative Test (Suspicious)

What: Verifies correct inputs → correct outputs, but never that bad inputs are rejected.

Detect:

• No tests for invalid input (wrong types, out-of-range, malformed)
• No tests for error conditions the code claims to handle
• No tests verifying exceptions raised when they should be
• Error-handling branches with zero coverage

Challenge: "Does this code have error handling? Tests proving it works?"

---

Pattern 10: Specification Blindness (Meta-Pattern, Likely FP)

What: Across the entire suite, no evidence any test was informed by a spec, requirement, or domain rule. All derived from code observation.

Detect:

• Test names describe implementation (test_returns_map_with_keys) not behavior (test_discounted_price_applies_to_premium_members)
• No domain language in test descriptions
• Expected values lack explanatory comments connecting to business rules
• Reads like a mechanical trace of code, not an independent contract

Challenge: "If I deleted the source and only had these tests + a spec, could I rewrite the code correctly?" If not, tests are mirrors, not contracts.

---

STEP 4 — Produce the Challenge Report

Per test (or logical group):

### [Test Name / Description]
**Verdict**: Solid / Suspicious / Likely FP / Vacuous
**Patterns detected**: [list of pattern numbers and names]
**Reasoning**: [1-3 sentences why suspicious]
**What would make it solid**: [specific fix]

Then:

## Challenge Summary
- Total tests analyzed: N
- Solid: N (XX%)
- Suspicious: N (XX%)
- Likely False Positive: N (XX%)
- Vacuous: N (XX%)

## Highest-Risk Findings
[Top 3 with brief reasoning]

## Recommendations
[Prioritized actions]

---

STEP 5 — Suggest Remediation

Per suspicious or false-positive test:

1. Rewrite with independent oracle — derive expected from spec, not execution
2. Add boundary/edge variants — keep existing test, add tests where bugs hide
3. Replace with property-based test — check invariants, not specific outputs
4. Delete and replace — vacuous or tautological? Removing beats false confidence.
5. Add specification comment — if test IS correct, comment why the expected value is correct, link to requirement

---

Calibration Notes

Not every AI-generated test is a false positive. Some are fine when:

• Behavior is simple and obviously correct
• Test checks well-known invariant (sorting produces sorted output)
• Generated from a clear docstring/type signature encoding the spec
• Exercises error-handling paths independently understandable

Goal: flag tests where oracle is suspect and human domain knowledge is needed. Not reject all generated tests.

---

Edge Cases and Judgment Calls

"All my tests were generated and coverage went 30% → 80%!" Coverage measures lines executed, not assertion meaningfulness. No-assertion tests achieve coverage. Wrong-assertion tests achieve coverage. Challenge each independently.

"The test name says 'should calculate correct tax' — doesn't that mean it's correct?" Name reflects intent, not reality. If 20.0 was derived by running calculate_tax(100, 20) rather than knowing 20% of $100 is $20, the name is aspirational.

"Aren't regression tests meant to mirror current behavior?" Yes — but explicitly label as regression/snapshot, not correctness. Generated to lock current behavior is only valuable if someone first verified current behavior is correct. Otherwise, locking in bugs.