testing-patterns

name	testing-patterns
description	Use when writing tests, analyzing test results, or improving test coverage. Provides conventions for workflow testing, comment syntax, and improvement patterns.
title	Testing Patterns
stratum	2
branches	["agentic"]

Domain expertise for testing AI agent workflows. Use this when writing tests, analyzing results, or planning test improvements.

---

Quick Reference

Concept	Pattern
Test categories	Adoption, Modularity, Portability, Intuitiveness, Docs, Deterministic
Comment syntax	%%PASS%%, %%FAIL%%, %%NOTE%%, %%TODO%%, %%QUESTION%%
Results format	RESULTS.md with category tables
Improvement loop	Test → Comment → Report → Improve → Repeat

---

Test Category Framework

Category A: Adoption Tests

Core question: Can someone new get productive quickly?

Good adoption tests measure:

• Time to first useful interaction
• Clarity of entry points (README, Quick Start)
• Error recovery when setup goes wrong
• Minimal viable setup path

Category B: Modularity Tests

Core question: Can you use pieces independently?

Good modularity tests validate:

• Single skill works without scaffold
• Single agent works without other agents
• No hidden dependencies between components
• Partial installation is viable

Category C: Portability Tests

Core question: Does it work across tools and projects?

Good portability tests verify:

• Same skill works in Claude Code AND OpenCode
• Drop into existing project without conflicts
• Global vs local priority works correctly
• No tool-specific assumptions baked in

Category D: Intuitiveness Tests

Core question: Do things work as expected?

Good intuitiveness tests check:

• Natural language triggers right behavior
• Users can discover what's available
• Same input produces consistent output
• No surprising side effects

Category E: Documentation Tests

Core question: Are docs accurate and helpful?

Good documentation tests verify:

• Links work and point to real content
• Examples actually execute
• No stale or outdated information
• Cross-references are valid

Category F: Deterministic Checks

Core question: Can we validate programmatically?

Good deterministic tests are:

• Scriptable (bash/grep/find)
• Repeatable (same result every time)
• Fast (run on every change)
• Clear (pass/fail, no ambiguity)

---

Test Structure Template

Every test should have:

### [ID]: [Test Name]

**Goal:** [One sentence - what success looks like]

**Setup:**
1. [Prerequisite step]
2. [Prerequisite step]

**Steps:**
1. [Action to take]
2. [Action to take]
3. [Action to take]

**Pass Criteria:**
- [ ] [Checkable criterion]
- [ ] [Checkable criterion]
- [ ] [Checkable criterion]

**Results:**
| Tool | Pass? | Notes |
|------|-------|-------|
| Claude Code | | |
| OpenCode | | |

**Comments:** %%Add observations here%%

---

Comment Syntax

Use Obsidian-compatible comments for inline test feedback:

Comment	When to Use	Example
%%PASS: message%%	Test passed as expected	%%PASS: Skill loaded on first try%%
%%FAIL: message%%	Test failed	%%FAIL: Agent didn't trigger, manual invoke needed%%
%%NOTE: message%%	Interesting observation	%%NOTE: OpenCode behaves differently here%%
%%TODO: message%%	Action needed	%%TODO: Add edge case for empty input%%
%%QUESTION: message%%	Needs clarification	%%QUESTION: Is this expected behavior?%%

Collecting comments:

grep -r "%%" test-workspace/ --include="*.md"

Comment best practices:

• Be specific - include what you expected vs what happened
• Include context - tool, scenario, input
• One observation per comment
• Keep comments near the relevant test section

---

RESULTS.md Format

Track results in a structured table by category:

## Category A: Adoption

| ID | Test | Claude Code | OpenCode | Last Run |
|----|------|-------------|----------|----------|
| A01 | Fresh Start | PASS | PASS | 2025-12-24 |
| A02 | Minimal Setup | PASS | FAIL | 2025-12-24 |

### A02 Notes
- OpenCode: [specific issue observed]
- Action: [what needs to change]

Status values:

• PASS - All criteria met
• FAIL - One or more criteria failed
• PARTIAL - Some criteria met, some unclear
• SKIP - Not applicable or blocked
• TODO - Not yet tested

---

Improvement Loop

1. Run tests
   └── Add %%comments%% as you go

2. Generate report
   └── Run /test-report to aggregate findings

3. Analyze patterns
   └── What's failing? What's unclear?

4. Prioritize fixes
   └── Critical failures > Edge cases > Polish

5. Implement changes
   └── Fix the actual issue, not just the symptom

6. Re-run affected tests
   └── Verify the fix works

7. Repeat
   └── Until all priority tests pass

---

Writing Good Pass Criteria

Bad criteria (vague):

• It works
• The skill loads
• User can do the thing

Good criteria (specific, checkable):

• Skill permission prompt appears within 3 seconds
• Skill content visible in response (quote specific text)
• No error messages in output
• Total time < 5 minutes (timed)

Criteria guidelines:

1. Observable - Can be seen or measured
2. Binary - Pass or fail, no "mostly works"
3. Independent - Each criterion stands alone
4. Specific - Exact values, not ranges

---

Common Testing Anti-Patterns

Testing What You Built, Not What Users Need

Problem: Tests verify internal mechanics, not user value Fix: Start with user goal, work backward to what to test

Assuming Success

Problem: Only testing happy path Fix: Include error cases, edge cases, recovery scenarios

Vague Criteria

Problem: "It should work" is not testable Fix: Specify exact expected behavior

Testing in Isolation Only

Problem: Components work alone but fail together Fix: Include integration tests across components

Not Recording Findings

Problem: Run tests, forget results Fix: Always update RESULTS.md and add %%comments%%

---

Deterministic Check Patterns

File Structure Validation

# Skills in correct format
find .claude/skills -name "SKILL.md" -type f | wc -l

# No loose skill files
find .claude/skills -maxdepth 1 -name "*.md" -type f

YAML Field Validation

# Check for required fields in agent
grep -l "^name:" .claude/agents/**/*.md
grep -l "^description:" .claude/agents/**/*.md
grep -l "^tools:" .claude/agents/**/*.md

Link Validation

# Find all wikilinks
grep -oh '\[\[[^]]*\]\]' *.md | sort | uniq

# Check if targets exist (manual or scripted)

Naming Consistency

# Directory name should match YAML name field
for dir in .claude/skills/*/; do
  name=$(basename "$dir")
  grep "^name: $name" "$dir/SKILL.md" || echo "Mismatch: $dir"
done

---

Related

• TESTING.md - Active test scenarios
• RESULTS.md - Test result tracking
• /validate command - Run deterministic checks
• /test command - Interactive test runner
• /test-report command - Aggregate findings
• test-improver agent - Proactive improvement suggestions