stelow-product-testing-ai-code

name	stelow-product-testing-ai-code
description	[stelow] AI-aware testing strategy skill for software products. Generates AI-aware testing plans, security gates, and risk-based coverage targets. Activated automatically when product_type is "software" or "hybrid"; usable standalone. Based on empirical research: AgentAssay (2026), MSR 2026, Veracode 2025, CodeRabbit 2025, LLM4TDD (2023), TDD-with-AI-Agents (2026).
metadata	{"frequency":"monthly","category":"code","context-cost":"medium","author":"calionauta","author-url":"https://github.com/calionauta"}

AI-Aware Testing Strategy

Based on empirical research:

• AgentAssay (2026): Non-deterministic agent testing framework
• MSR 2026: Over-mocking anti-patterns in AI-generated tests
• Veracode 2025: 45% of AI code contains vulnerabilities
• CodeRabbit 2025: AI code has 1.7x more bugs than human code
• CoderEval (2023): 43.1% of AI code is less robust

Standalone awareness: when inside stelow, triggered automatically by product_type in spec-product.md frontmatter. When standalone, invoke directly with a spec-product.md path. Appetite defaults to Core if not found — documented in output. All test-breadth tables and quality baselines work identically in both modes.

Activation

• Trigger: product_type: software or product_type: hybrid in spec-product.md frontmatter
• Phase: Phase 11 (Tech Planning)
• Prerequisite: approved spec-product.md with scope defined

Step 2: Read Appetite and Product Context

Read appetite from spec-product.md before generating test scopes. When running standalone, appetite defaults to Core if not found in frontmatter — the skill documents this assumption in the output.

Appetite controls test breadth, not quality baseline.

Appetite	Test breadth
Lean	Behavior/E2E (1 happy path) + smoke tests + critical-path unit tests. Add integration only when an external seam is in scope.
Core	Behavior/E2E (happy path + variations) + unit tests for main logic + integration tests for DB/API/external seams.
Complete	Behavior/E2E (full coverage + edge cases) + unit + integration + security tests/scans.

Quality baseline applies to every appetite: build/test/lint/typecheck always run when available, and a11y checks run whenever UI files exist. Appetite changes exploration breadth, not whether quality gates exist.

Then determine the product context:

Before generating testing strategy, determine the product context:

Context	Description	Testing Approach
Greenfield	New product, no existing code	TDD-first, appetite-specific coverage targets, clean slate
Brownfield	Existing product with features	TDD for critical paths, test-after for existing code, regression focus
Hybrid	Adding features to existing product	Separate new from existing, protect invariants

Based on context from setup or spec-product.md:

• greenfield: TDD recommendation, appetite-specific coverage targets
• brownfield: TDD for critical paths only, test-after + regression for existing code
• hybrid: Add test-regression scopes for existing functionality

Input Detection (Standalone Mode)

When called outside the workflow with no pre-existing spec-product.md:

Input:
  ├── User provided a spec-product*.md path?
  │   └→ Read product_type, appetite, scope from frontmatter
  ├── User provided a description of the project?
  │   └→ Extract: language, product type, risk level
  └── No structured input?
      └→ Default: product_type=software, appetite=Core, context=brownfield

Graceful degradation: The skill works standalone. If product_type or appetite cannot be determined from frontmatter, sensible defaults are used (product_type=software, appetite=Core, context=brownfield). Every step documents its assumptions and flags them in the output.

Input

From Tech Planning context (or standalone):

• spec-product.md (frontmatter with product_type, appetite)
• spec-tech.md (scopes to add test-* types, if available)
• Tech stack detection from project files

Output

Artifact	Path
testing-strategy.md	.stelow/{date}/{_dir}/plans/

---

Process

Step 1: Detect Tech Stack

Auto-detect testing frameworks from project files:

Language	Detection File	Unit Framework	Coverage Tool	Security Tool
Python	requirements.txt, pyproject.toml	pytest	pytest-cov	Bandit
JavaScript/TypeScript	package.json	Vitest, Jest	c8 / V8 coverage	ESLint + SAST
Go	go.mod	testing	go test -cover	Gosec
Rust	Cargo.toml	cargo test	cargo-tarpaulin	cargo-audit
Java	pom.xml, build.gradle	JUnit	JaCoCo	SpotBugs

Step 3: Classify Scope Risk

Based on spec-tech.md scopes:

Risk Level	Examples	Testing Depth
Critical	Payment, auth, data persistence, security	High coverage + negative cases + security gates
Standard	CRUD, UI, API endpoints	Core paths + obvious edge cases
Experimental	Prototypes, new features	Smoke tests + regression hooks

Step 4: Generate Appetite-Specific Test Targets

From research: coverage alone is insufficient. A test suite can execute every line but still miss behavioral bugs, security gaps, and non-deterministic agent failures. Appetite selects breadth; risk selects which paths inside that breadth are mandatory.

Appetite	Path Type	Testing Depth
Lean	Critical path	E2E/behavior (1 happy path) + smoke + unit for the one negative case
Lean	Standard/experimental	E2E/behavior (1 happy path); no broad integration suite
Core	Critical path	E2E/behavior + unit tests + negative cases + integration seams
Core	Standard features	E2E/behavior (key variations) + unit tests + integration for external seams
Complete	Critical path	E2E/behavior (full) + unit + integration + security gates
Complete	Complex flows	E2E/behavior for multi-step UI or agent workflows

Step 5: Define Test Scope Types

For each IN scope in spec-product.md, add corresponding test scopes. Appetite controls breadth:

Lean:

Code Type	Test Type	When to Use	TDD?
User-facing flows	test-behavior	1 E2E test for happy path	No — browser/e2e
Critical business logic	test-unit	Happy path + one negative case	Yes for deterministic logic
External APIs	test-integration	Only if external seam is in Lean IN scope	No — test-after
Security-sensitive	test-security	Only if auth/payment/data is in Lean IN scope	Automated SAST

Core:

Code Type	Test Type	When to Use	TDD?
User-facing flows	test-behavior	E2E for happy path + key variations	No — browser/e2e
Agent workflows	test-behavior	Multi-step agents	Multi-run validation
Business logic	test-unit	Main flows + obvious edge cases	Yes — critical paths
External APIs	test-integration	DB, APIs, queues	No — test-after
Security-sensitive	test-security	Auth, payment, data	Automated SAST

Complete:

Code Type	Test Type	When to Use	TDD?
User-facing flows	test-behavior	Full E2E coverage + edge cases	No — browser/e2e
Complex UI/user flows	test-behavior	Forms, modals, multi-step flows	Browser/e2e
Agent workflows	test-behavior	Multi-step agents	Multi-run validation
Business logic	test-unit	Full edge mapping	Yes — critical paths
External APIs	test-integration	All external seams	No — test-after with contract checks
Security-sensitive	test-security	Auth, payment, data, permissions	Automated SAST + targeted tests

Brownfield/Hybrid (existing code):

Code Type	Test Type	When to Use	Context
Protect existing	test-regression	Detect regressions	Brownfield/Hybrid
Document behavior	test-characterization	Golden tests	Brownfield
Replay past tasks	test-simulation	Verify consistency	Brownfield
Impact analysis	test-impact	TDAD-style	Brownfield

Based on MSR 2026 research (agents use mocks 36% vs 26% humans):

Anti-patterns to flag:

• ❌ Mock count > 3 per test
• ❌ Mocks for simple objects (use real objects instead)
• ❌ 100% coverage target (coverage ≠ test quality)
• ❌ Snapshot tests for non-UI components
• ❌ Single-run validation (agents are non-deterministic)
• ❌ Same AI for both code AND test generation (circular validation)

Step 6: Evaluate Mutation Testing Fit

Mutation testing evaluates whether a test suite would notice a regression — not just whether it executed a line. Research shows LLM-generated tests cluster around the same blind spots as the code they test (Test Homogenization Trap, AgentPatterns 2026). Mutation forces tests to prove they'd catch a defect.

Evaluate whether mutation testing adds value for this project:

# Heuristic: mutation testing fit score (0-10)
SCORE=0

# +1 if language is compiled (Go, Rust, Java, TypeScript with strict config)
+2 if language is Go, Rust, or Java

# +1 if there are critical scopes (auth, payment, data persistence)
+2 if @critical scopes exist

# +2 if project has >5 scopes or appetite=Complete
+3 if appetite=Complete or scope count > 5

# -1 if appetite=Lean (prototype/validation)
-2 if appetite=Lean

# -1 if pure CRUD with no critical paths (no payment, auth, data persistence)
-2 if pure CRUD and no @critical scopes

# Score >= 5: recommend mutation testing
# Score 2-4: recommend only for specific critical modules
# Score < 2: skip (document why)

Context	Mutation Testing Recommendation
Production app with critical paths (auth, payment, data)	✅ Recommend: [TYPE] test-mutation with target 50%, scoped to critical modules, run nightly
CRUD web app, REST API without sensitive paths	🟡 Optional: [TYPE] test-mutation with target 40%, only if existing test suite is mature
Prototype, validation, appetite=Lean	❌ Skip — cost (CI minutes, equivalent-mutant triage) > regression risk
Service/no-code product	❌ Skip — no code to test
Skill/CLI/workflow package (e.g., stelow itself)	❌ Skip — integration + smoke tests sufficient for low regression risk

If recommending, add to testing strategy:

mutation_testing:
  recommended: true
  rationale: "Project has {critical_count} critical scopes in {language}; mutation testing
    validates test suite can detect regressions in business logic."
  scope: "critical modules only, not full suite"
  target: "50% starter"
  tool: "Stryker (JS/TS), mutmut (Python), PIT (Java), go-mutate (Go)"
  schedule: "nightly, not per-PR — mutation runs are too slow for gate gating without
    selective mutation (Zenseact 2023, Google 2021)"

Validated loop (AgentPatterns 2026, MUTGEN 2025):

1. Generate tests (AI)
2. Run mutation tool
3. Feed surviving mutants into prompt
4. Generate additional tests targeting survivors
5. Repeat (plateau ~4 iterations)

Anti-patterns for mutation testing with AI code:

• ❌ Same model that wrote code generates tests AND judges equivalent mutants (circular bias)
• ❌ Full-suite mutation as per-PR gate (hours of CI, Zenseact 2023)
• ❌ High targets on first pass (start at 50%, not 70%)
• ❌ No equivalent-mutant filter (>50% survival = noise, Facebook 2020)

Step 7: Create CI/CD Gates

GATES:
  critical_path_coverage:
    condition: "missing required tests"
    action: BLOCK
    rationale: "Critical paths need executable regression checks"
  
  security_findings:
    condition: "> 0 on critical paths"
    action: BLOCK
    rationale: "45% of AI code contains vulnerabilities (Veracode 2025)"
  
  flaky_rate:
    condition: "> 5%"
    action: WARN
    rationale: "Agents generate non-deterministic tests"

---

TDD Guidance (Research-Based)

Empirical findings on TDD with AI agents:

Research	Key Finding
LLM4TDD (2023)	Including test cases alongside problem statements enhances code generation and increases success rate on benchmarks like MBPP and HumanEval
TDD-with-AI-Agents (2026)	TDD reduces bugs by 40-80% compared to test-after for AI workflows
AgentPatterns.ai	Tests as specifications constrain AI behavior — "the test is a contract the agent cannot fake"
QASkills (2026)	Without TDD, AI agents write tests that validate their own broken logic

When to Use TDD (Based on Research)

Code Type	TDD Recommended?	Rationale
Critical business logic	✅ Yes — recommended	TDD provides design feedback, validates understanding, and constrains AI output
Security-sensitive	⚠️ TDD + automated gates	Write tests first, then run SAST continuously (45% vulnerability rate)
External APIs	❌ No — test-after	Over-mocking is anti-pattern; use real dependencies
Agent workflows	❌ No — behavioral testing	Non-deterministic — needs multi-run validation
Standard features	⚠️ Optional	Use TDD for clarity; risk-based tests for standard paths

Brownfield Testing (Existing Products)

When evolving an existing product (research-based):

Aspect	Strategy	Rationale
Existing tests	Adapt, don't replace	High coverage = regression focus; Low coverage = characterization tests
New features	TDD for critical, test-after for standard	Protect existing, innovate safely
Existing invariants	Regression + simulation/replay testing	AI agents can break invariants without detection
Technical debt	Risk-aware testing targets	Higher depth for risky areas

---

test-regression: Protect Existing Functionality

Purpose: Run existing test suite to detect regressions when AI modifies code.

When to use: Any scope that touches existing code in brownfield/hybrid context.

Steps:

# 1. Identify affected tests before changes
find . -path ./node_modules -prune -o \n  -name "*.test.*" -print -o \n  -name "*.spec.*" -print | xargs rg -l "module_name" > affected_tests.txt

# 2. Run baseline (before changes)
npm test -- --testPathPattern="$(cat affected_tests.txt | tr '\n' '|')" > baseline_results.json

# 3. After scope changes, rerun same tests
npm test -- --testPathPattern="$(cat affected_tests.txt | tr '\n' '|')" > post_change_results.json

# 4. Compare: any new failures = regression
diff baseline_results.json post_change_results.json

CI/CD Gate:

regression:
  condition: "baseline_failures != post_change_failures"
  action: BLOCK
  rationale: "6.08% regression rate in vanilla agent runs (TDAD paper)"

---

test-characterization: Document Existing Behavior

Purpose: Create golden tests that capture current behavior before AI changes.

When to use: Before modifying complex existing modules (no or few tests).

Steps:

# 1. Identify target module
TARGET_MODULE="src/auth/session.ts"

# 2. Generate characterization tests (capture existing behavior)
# Use AI to generate tests that pass with current implementation
npx vitest create test --filter "$TARGET_MODULE" --type characterization

# 3. Run and confirm all pass (baseline)
npm test -- --grep "$TARGET_MODULE"

# 4. Only then proceed with changes
# These tests become the regression guard

Output: *.characterization.test.ts files that document current behavior.

Key principle: Tests should PASS initially — they document what the code currently does, not what it should do.

---

test-simulation: Replay Past Tasks

Purpose: Replay successful agent tasks from history to verify consistent behavior.

When to use: After AI completes similar tasks — verify it behaves the same way.

Steps:

# 1. Record task execution (from git history or logs)
task_id="fix-login-2026-05-15"
echo "Task: $task_id" > simulation_input.txt
cat commit_message.txt >> simulation_input.txt
cat changed_files.txt >> simulation_input.txt

# 2. Replay with different agent configuration
# Compare output to original successful run
agent --config "$AGENT_CONFIG" --replay simulation_input.txt > replay_output.txt

# 3. Diff against expected (original successful output)
diff expected_output.txt replay_output.txt

# 4. If diff > threshold → behavioral regression

Tool integration: AgentPatterns.ai recommends replay testing for agent verification.

CI/CD Gate:

simulation:
  condition: "diff > tolerance_threshold"
  action: WARN
  rationale: "Behavioral drift from baseline"

---

test-impact: TDAD-Style Impact Analysis

Purpose: Graph-based analysis to identify which tests to run before AI commits.

When to use: Before ANY scope execution in brownfield/hybrid context.

Steps:

# 1. Build code-to-test dependency graph
# Python example with pytest:
pytest --co -q | awk '{print $1}' | while read test; do
  deps=$(rg -o "import.*from.*['\"]\([^'\"]*\)['\"]\|require\(['\"]\([^'\"]*\)['\"]\)" "$test" | cut -d: -f2 | sort -u)
  echo "$test: $deps"
done > code_test_graph.json

# 2. For proposed change, query affected tests
TARGET_FILES="src/auth/ src/payment/"
cat code_test_graph.json | jq -r '.[] | select(.code_files | split(",") | inside($target)) | .test_file' \
  --arg target "$TARGET_FILES"

# 3. Run impact subset before changes (baseline)
npm test -- --testPathPattern="affected_tests" > impact_baseline.txt

# 4. After scope changes, run same tests
# Any new failures → scope not complete until fixed

Alternative (simpler) using madge:

# Generate dependency graph
npx madge --image dependencies.svg --format dot .

# Find tests affecting modified modules
npx madge --inverse src/payment/

CI/CD Gate:

impact:
  condition: "new_failures_in_impact_set > 0"
  action: BLOCK
  rationale: "TDAD reduced regressions by 70%"

---

Additional test scopes for Brownfield:

Greenfield Testing (New Products)

When building a new product:

Aspect	Strategy	Rationale
TDD adoption	Full recommended	No legacy constraints, clean architecture
Coverage/risk targets	Appetite-specific: Lean critical path only; Core main logic + external seams; Complete full edge mapping + security	Establish quality baseline from day one
Coverage	Define target upfront	80% baseline, higher for critical
Technical debt	None yet	Focus on clean patterns, not remediation

Hybrid (Feature Addition)

When adding features to existing product:

Aspect	Strategy	Rationale
New code	TDD for critical, test-after for standard	Same as greenfield
Existing code	Regression + protection	Same as brownfield
Integration points	Extra verification	Ensure new doesn't break old
Agent behavior	Behavioral + regression	Non-deterministic risk

Anti-patterns for Brownfield/Hybrid:

• ❌ AI refactoring without test protection
• ❌ AI modifying existing code without characterization tests
• ❌ Over-mocking existing integrations
• ❌ Ignoring technical debt in scope planning

TDD Cycle for AI Agents

1. RED: Write failing test (human or AI with explicit constraints)
2. GREEN: AI implements only enough to pass test
3. REFACTOR: Clean up with tests still passing

Key difference from human TDD:
- AI must see failing test BEFORE implementation
- Tests must be written independently of implementation
- Human validates test quality via critical-path coverage and negative cases

---

Risk-Based Test Feedback Loop

Use this loop when tests miss important behavior or regressions appear after AI changes:

1. Identify the missed behavior or regression
2. Add a focused test that captures the invariant
3. Rerun the affected test suite
4. Feed the invariant back into future test scopes

Test quality signals:

• Critical-path coverage
• Negative-case coverage
• Security gate results
• Flaky-rate monitoring

---

Output Format

testing-strategy.md

---
version: 1
product_type: software
generated_by: stelow-product-testing-ai-code
generated_at: {YYYY-MM-DD}
---

# Testing Strategy for {product_name}

## Tech Stack
- Language: {language}
- Unit: {framework}
- Coverage: {tool}
- Security: {tool}

## Coverage and Risk Targets
| Path Type | Required Evidence |
|-----------|-------------------|
| Critical | Branch coverage, negative cases, security gates |
| Standard | Core paths and obvious edge cases |
| Experimental | Smoke tests and regression hooks |

## Test Scopes
| Scope | Type | Required Evidence |
|-------|------|----------------|
| {feature-name} | test-unit | Critical path + negative cases |
| {feature-name} | test-integration | External seams covered |
| {feature-name} | test-security | SAST clean on critical paths |

## CI/CD Gates
- BLOCK: missing required critical-path tests
- BLOCK: security_findings > 0 on critical paths
- WARN: flaky_rate > 5%

## Anti-Patterns
- ❌ > 3 mocks per test
- ❌ Mocks for simple objects
- ❌ 100% coverage target
- ❌ Same AI for code AND tests

---

Success Criteria

• testing-strategy.md generated
• test-* scopes added to spec-tech.md
• Coverage and risk targets calibrated per risk level
• CI/CD gates documented
• Anti-patterns checklist included

---

Fallback

If tech stack detection fails:

• Python → pytest + pytest-cov + Bandit
• JavaScript/TypeScript → Vitest + c8/V8 coverage + ESLint
• Go → testing + go test -cover + Gosec
• Rust → cargo test + cargo-tarpaulin + cargo-audit

---

Related Skills

• stelow-product-tech-planning: Produces scopes to test
• stelow-product-scope-executor: Executes test scopes
• stelow-product-plan-critique: Can review testing strategy