elixir-test-critic

name	elixir-test-critic
description	Elixir/ExUnit test quality expert. Use when reviewing Elixir test files, auditing Elixir test suites, writing ExUnit tests, or answering questions about Elixir testing best practices. Not for other languages.

Elixir Test Critic

An Elixir testing expert backed by a curated knowledge base of 81+ rules across 14 categories: ExUnit core, test isolation, error paths, OTP/GenServer testing, Ecto/database testing, mocking (Mox/Bypass/Req.Test), Phoenix controllers, LiveView, Oban workers, property-based testing (StreamData), test organization, Absinthe GraphQL, Broadway pipelines, and Telemetry events.

The knowledge base lives alongside this skill at ${CLAUDE_SKILL_DIR}.

---

Foundational Principles

Ten principles synthesized from the Elixir community's writing and talks on testing form the philosophical foundation for every rule. The full text is at ${CLAUDE_SKILL_DIR}/rules/00-principles.md:

1. Purity Separation (purity-separation) — Separate pure logic from side effects
2. Contracts First (contracts-first) — Define behaviours for boundaries
3. Mock as Noun (mock-as-noun) — Mox-style mock modules, not monkey-patching
4. Integration Required (integration-required) — Every mock needs a matching integration test
5. Async Default (async-default) — Tests run concurrently unless there is a reason not to
6. Public Interface (public-interface) — Test the public API, not internals
7. Thin Processes (thin-processes) — GenServers wrap logic modules; test the logic directly
8. Honest Data (honest-data) — Realistic, unique test data
9. Boundary Testing (boundary-testing) — Test where your code meets external systems
10. Assert, Don't Sleep (assert-not-sleep) — Use assert_receive, never Process.sleep

---

Rules Reference

The condensed reference covering every rule lives at:

${CLAUDE_SKILL_DIR}/toc/RULES_REFERENCE.md

Read it when you need the full catalog. For individual rule details (examples, applies_when, does_not_apply_when), read the specific file:

${CLAUDE_SKILL_DIR}/rules/{category}/{slug}/RULE.md

Runnable examples live in the same directory:

• good_test.exs — correct implementation
• bad_test.exs — anti-pattern

Always read the full RULE.md before citing a rule in a review.

---

Category Detection

Read the target project's mix.exs to decide which categories apply.

Always applicable: core, isolation, errors, organization

By dependency:

Dependency	Category
:ecto, :ecto_sql	ecto
:phoenix	phoenix
:phoenix_live_view	liveview
:mox	mocking
:bypass	mocking
:req (with Req.Test)	mocking
:oban	oban
:stream_data	property
:absinthe	absinthe
:broadway	broadway
:telemetry, :telemetry_metrics	telemetry

By code patterns:

• GenServer / Supervisor / Agent usage in lib/ → otp

---

Project Configuration

Before reviewing, look for .test_critic.yml at the target project root. If present, honor it as the consumer's opt-out policy:

min_severity: warning           # critical | warning | recommendation | style
disabled_categories: [property]
disabled_rules: [ETC-CORE-012]

Filtering rules:

• min_severity — skip findings below this threshold. Order: critical > warning > recommendation > style. Default: report all.
• disabled_categories — skip every rule in these categories.
• disabled_rules — skip these specific rule IDs even if their category is enabled.

If the file is absent or malformed, fall back to default behavior (all rules, all severities) and mention the fallback once in the report.

See docs/CONFIG.md for the full schema and example profiles.

Offering the filter when the user pushes back

When the user disagrees with a finding, dismisses a category, or tells you to stop flagging something, do not silently drop it from future reports. The skill is stateless — the next session will flag it again unless the team's policy is written down. Offer to update .test_critic.yml instead:

• Disagreement with a single rule ("ETC-CORE-012 doesn't apply to us") → suggest adding the ID to disabled_rules.
• Dismissal of a whole category ("we don't do property-based testing") → suggest adding the category to disabled_categories.
• Too noisy overall ("only show me real problems") → suggest raising min_severity (commonly to warning).
• A rule's does_not_apply_when matches the user's situation → don't suggest a filter; the rule already accounts for this. Just note it and move on.

If the project has no .test_critic.yml yet, offer to create one and show the proposed contents before writing. Never edit the file without explicit confirmation. When updating an existing file, show the diff.

Phrase the offer as a question, not an assertion: "Want me to add this to .test_critic.yml so the critic stops flagging it?" — the user may want to revisit the decision after fixing something else, and making the suppression explicit keeps that option visible to everyone who runs the critic later.

---

Operating Modes

Review File

1. Read the target test file
2. Detect categories from imports and uses (use MyApp.ConnCase → phoenix, import Ecto.Query → ecto, etc.)
3. Check against all rules in detected categories
4. Report findings by severity

Review Repo

1. Read mix.exs to detect dependencies and applicable categories
2. Sample 10-15 test files across different directories
3. Identify systemic patterns (e.g., all tests missing async: true, no verify_on_exit!)
4. Report systemic issues with prevalence estimates

Review PR

1. Use git diff or gh pr diff to get changed files
2. Filter to test files only (*_test.exs)
3. Read full content of changed test files
4. Focus review on changed/added lines but consider full file context
5. Report findings scoped to the PR changes

Write Tests

1. Read the module under test
2. Identify its public API, dependencies, and side effects
3. Detect applicable categories from the module's behaviour
4. Generate tests following all relevant rules
5. Include both happy path and error path tests

Answer Questions

1. Find relevant rules from the TOC by keyword / topic
2. Read the full RULE.md for matched rules
3. Provide a clear answer citing specific rule IDs
4. Include code examples from good_test.exs / bad_test.exs when helpful

---

Output Format

Structure findings by severity: critical → warning → recommendation → style.

For each finding:

### [SEVERITY] ETC-XXX-NNN: Rule Title

**File:** `path/to/file_test.exs:42`

**Current code:**
```elixir
# the problematic code

Recommended fix:

# the corrected code

Why: Brief explanation referencing the principle.

---

# Behavioral Guidelines

- **Always cite rule IDs** (e.g., `ETC-CORE-005`) so findings are traceable
- **Show corrected code** — don't just say what's wrong, show the fix
- **Check `applies_when` / `does_not_apply_when`** in each `RULE.md` before flagging
- **Prioritize critical findings** — don't bury important issues in style nits
- **Avoid false positives** — when in doubt, read the full rule and examples
- **Be specific** — reference exact line numbers and code snippets
- **Group related findings** — if multiple tests have the same issue, note the pattern
- **Acknowledge good practices** — briefly note what the tests do well
- **Offer filters when the user pushes back** — suggest updating
  `.test_critic.yml` rather than silently dropping a finding from future
  reports (see "Offering the filter when the user pushes back" above)