testing-patterns

name	testing-patterns
description	Use when writing tests, analyzing test results, or improving test coverage. Provides conventions for workflow testing, comment syntax, and improvement patterns.
title	Testing Patterns
stratum	2
branches	["agentic"]

Domain expertise for testing AI agent workflows. Use this when writing tests, analyzing results, or planning test improvements.

---

Quick Reference

Concept	Pattern
Test categories	Adoption, Modularity, Portability, Intuitiveness, Docs, Deterministic
Comment syntax	%%PASS%%, %%FAIL%%, %%NOTE%%, %%TODO%%, %%QUESTION%%
Results format	RESULTS.md with category tables
Improvement loop	Test → Comment → Report → Improve → Repeat

---

Test Category Framework

Category A: Adoption Tests

Core question: Can someone new get productive quickly?

Good adoption tests measure:

• Time to first useful interaction
• Clarity of entry points (README, Quick Start)
• Error recovery when setup goes wrong
• Minimal viable setup path

Category B: Modularity Tests

Core question: Can you use pieces independently?

Good modularity tests validate:

• Single skill works without scaffold
• Single agent works without other agents
• No hidden dependencies between components
• Partial installation is viable

Category C: Portability Tests

Core question: Does it work across tools and projects?

Good portability tests verify:

• Same skill works in Claude Code AND OpenCode
• Drop into existing project without conflicts
• Global vs local priority works correctly
• No tool-specific assumptions baked in

Category D: Intuitiveness Tests

Core question: Do things work as expected?

Good intuitiveness tests check:

• Natural language triggers right behavior
• Users can discover what's available
• Same input produces consistent output
• No surprising side effects

Category E: Documentation Tests

Core question: Are docs accurate and helpful?

Good documentation tests verify:

• Links work and point to real content
• Examples actually execute
• No stale or outdated information
• Cross-references are valid

Category F: Deterministic Checks

Core question: Can we validate programmatically?

Good deterministic tests are:

• Scriptable (bash/grep/find)
• Repeatable (same result every time)
• Fast (run on every change)
• Clear (pass/fail, no ambiguity)

---

Test Structure Template

Every test should have:

### [ID]: [Test Name]

**Goal:** [One sentence - what success looks like]

**Setup:**
1. [Prerequisite step]
2. [Prerequisite step]

**Steps:**
1. [Action to take]
2. [Action to take]
3. [Action to take]

**Pass Criteria:**
- [ ] [Checkable criterion]
- [ ] [Checkable criterion]
- [ ] [Checkable criterion]

**Results:**
| Tool | Pass? | Notes |
|------|-------|-------|
| Claude Code | | |
| OpenCode | | |

**Comments:** %%Add observations here%%

---

Comment Syntax

Use Obsidian-compatible comments for inline test feedback:

Comment	When to Use	Example
%%PASS: message%%	Test passed as expected	%%PASS: Skill loaded on first try%%
%%FAIL: message%%	Test failed	%%FAIL: Agent didn't trigger, manual invoke needed%%
%%NOTE: message%%	Interesting observation	%%NOTE: OpenCode behaves differently here%%
%%TODO: message%%	Action needed	%%TODO: Add edge case for empty input%%
%%QUESTION: message%%	Needs clarification	%%QUESTION: Is this expected behavior?%%

Collecting comments:

grep -r "%%" test-workspace/ --include="*.md"

Comment best practices:

• Be specific - include what you expected vs what happened
• Include context - tool, scenario, input
• One observation per comment
• Keep comments near the relevant test section

---

RESULTS.md Format

Track results in a structured table by category:

## Category A: Adoption

| ID | Test | Claude Code | OpenCode | Last Run |
|----|------|-------------|----------|----------|
| A01 | Fresh Start | PASS | PASS | 2025-12-24 |
| A02 | Minimal Setup | PASS | FAIL | 2025-12-24 |

### A02 Notes
- OpenCode: [specific issue observed]
- Action: [what needs to change]

Status values:

• PASS - All criteria met
• FAIL - One or more criteria failed
• PARTIAL - Some criteria met, some unclear
• SKIP - Not applicable or blocked
• TODO - Not yet tested

---

Improvement Loop

1. Run tests
   └── Add %%comments%% as you go

2. Generate report
   └── Run /test-report to aggregate findings

3. Analyze patterns
   └── What's failing? What's unclear?

4. Prioritize fixes
   └── Critical failures > Edge cases > Polish

5. Implement changes
   └── Fix the actual issue, not just the symptom

6. Re-run affected tests
   └── Verify the fix works

7. Repeat
   └── Until all priority tests pass

---

Writing Good Pass Criteria

Bad criteria (vague):

• It works
• The skill loads
• User can do the thing

Good criteria (specific, checkable):

• Skill permission prompt appears within 3 seconds
• Skill content visible in response (quote specific text)
• No error messages in output
• Total time < 5 minutes (timed)

Criteria guidelines:

1. Observable - Can be seen or measured
2. Binary - Pass or fail, no "mostly works"
3. Independent - Each criterion stands alone
4. Specific - Exact values, not ranges

---

Common Testing Anti-Patterns

Testing What You Built, Not What Users Need

Problem: Tests verify internal mechanics, not user value Fix: Start with user goal, work backward to what to test

Assuming Success

Problem: Only testing happy path Fix: Include error cases, edge cases, recovery scenarios

Vague Criteria

Problem: "It should work" is not testable Fix: Specify exact expected behavior

Testing in Isolation Only

Problem: Components work alone but fail together Fix: Include integration tests across components

Not Recording Findings

Problem: Run tests, forget results Fix: Always update RESULTS.md and add %%comments%%

Declaring a Test Surface "Env-Fragile" Without Diagnostic Effort

Problem: When one of the project's configured test surfaces (E2E, integration, headless-browser, etc.) appears to fail, declaring it "env-fragile, deferred" without diagnosing why. Why it matters: Multi-surface test setups exist precisely because each surface catches different failure modes (unit catches logic; E2E catches integration; visual-regression catches CSS rot). Skipping a surface drops coverage in that mode silently — the regressions ship and surface later as "I thought we tested this." Common causes mistaken for "env-fragile":

• First-run binary/asset downloads — e.g., wdio-obsidian-service downloads ~150 MB Obsidian binary on first run; subsequent runs use cache. Playwright's npx playwright install is similar. Headless-Chrome services often need a one-time fetch.
• Missing display server in WSL/CI — usually fixable with xvfb-run, WSLg, or a --headless flag.
• Cache not yet populated — the first run writes to cache; subsequent runs are fast.
• Plugin / fixture not built — the test harness may expect dist/ or test-vault/.obsidian/plugins/<x>/main.js to exist.
• Wrong-tool-on-PATH collision — multiple tools with the same binary name (e.g., obsidian-cli resolving to a test-runs uploader instead of the headless vault tool).

Fix — the diagnostic checklist (run BEFORE concluding "fragile"):

1. Read the project's testing-patterns skill / TESTING.md / E2E config comments in full — they typically note first-run cost, display-server requirements, and known gotchas.
2. Inspect cache state — was anything actually downloaded? E.g. ls -la .obsidian-cache/ or the equivalent for the harness.
3. Try a single specific test (smoke) with a longer timeout — e.g. 240s instead of 5min. Does it pass when given more time?
4. Check the failed-test output for the actual error vs. the timeout error — timeouts hide the underlying issue. Tail the harness's own log file.
5. Verify the binary on PATH is the one the harness expects — name collisions are real.

When a phase ships without one of the configured surfaces: log the reason explicitly in the phase's commit message + zz-log entry — "deferred to follow-up because [specific observed failure + what was tried + what was missing]" — so future agents can reproduce and fix rather than assume it's permanently fragile. The project's existing per-milestone-E2E-requirement table (if present) should be honored, not treated as aspirational.

Source case: Crosswalker v0.1.6 Phase 3 — see agent-context/zz-log/2026-05-10.md for the concrete failure + correction.

---

Deterministic Check Patterns

File Structure Validation

# Skills in correct format
find .claude/skills -name "SKILL.md" -type f | wc -l

# No loose skill files
find .claude/skills -maxdepth 1 -name "*.md" -type f

YAML Field Validation

# Check for required fields in agent
grep -l "^name:" .claude/agents/**/*.md
grep -l "^description:" .claude/agents/**/*.md
grep -l "^tools:" .claude/agents/**/*.md

Link Validation

# Find all wikilinks
grep -oh '\[\[[^]]*\]\]' *.md | sort | uniq

# Check if targets exist (manual or scripted)

Naming Consistency

# Directory name should match YAML name field
for dir in .claude/skills/*/; do
  name=$(basename "$dir")
  grep "^name: $name" "$dir/SKILL.md" || echo "Mismatch: $dir"
done

---

Related

• TESTING.md - Active test scenarios
• RESULTS.md - Test result tracking
• /validate command - Run deterministic checks
• /test command - Interactive test runner
• /test-report command - Aggregate findings
• workflow-expert agent - Proactive improvement suggestions