effective-software-testing

name

effective-software-testing

description

Systematically derive test cases — and judge how thorough a test suite is — in the style of Maurício Aniche's "Effective Software Testing." This is the "how do I come up with the right inputs" complement to behavior-focused unit testing. Use this skill whenever the user asks what test cases to write, how to cover a function or method, how to find missing edge cases, what to test at a boundary, how to test a complex boolean condition, whether a suite is thorough enough, how to use code coverage well, whether to reach for property-based testing, or how to make a class testable — even if they never mention the book or Aniche. Also for questions like "what inputs should I test here", "is 100% coverage enough", "how do I test all these combinations", or "why is this class so hard to test". Enforces: derive cases from the specification first (partitions + boundaries); use coverage as a guide to find gaps, never as a target; and design for controllability and observability.

Effective Software Testing (Aniche)

This skill makes you derive test cases systematically instead of guessing, the way Maurício Aniche teaches in Effective Software Testing. It is the natural complement to khorikov-unit-testing and art-of-unit-testing: those tell you what makes a test good (behavior, not implementation; the right doubles); this tells you which tests to write — how to engineer a small set of inputs that finds the bugs.

The core attitude: testing is an engineering activity, not an afterthought. Given a piece of code, you don't sprinkle a few happy-path asserts and move on — you analyze the specification, partition the input space, probe the boundaries, and only then write tests. Then you use the code structure (coverage) to reveal what your spec-based pass missed. The result is far more bugs caught per test than ad-hoc testing, with no wasted redundant cases.

Examples are language-agnostic pseudocode; adapt to the user's stack (JUnit, pytest, Jest, xUnit…) and match their conventions.

---

The order of operations (don't skip to coverage)

1. Specification-based testing first. Derive cases from what the code is supposed to do. This is primary because it can catch missing logic — a bug where required behavior was never implemented. Coverage tools can't see code that isn't there.
2. Structural testing second, as a guide. Run your spec-based tests under coverage to find parts of the existing code they didn't exercise, then ask why — usually it reveals a partition you missed. Add tests for the behavior, not for the line.
3. Assess the suite with mutation testing and a smell check.

Keep that order. Coverage is a flashlight for finding gaps, never the goal (see "Coverage" below).

---

Workflow — Specification-based testing in 7 steps

This is the heart of the skill. Apply it to any function/method with non-trivial input→output logic. Full worked detail in references/specification-based-testing.md.

1. Understand the requirements, inputs, and outputs. What should this do? What are the inputs, their types and ranges, and the expected outputs/effects?
2. Explore what the program does for a few example inputs — enough to ground your understanding (read the code, run it in your head). Don't write final tests yet.
3. Identify the partitions (equivalence classes). Split each input's domain into classes the program treats the same way (e.g. "empty", "valid", "too long"; negative / zero / positive). One representative input per class exercises that class. This is what collapses an infinite input space to a handful of cases.
4. Analyze the boundaries. Bugs cluster where behavior changes. For each boundary, test the on point (the value the condition names) and the off point (the nearest value on the other side). Optionally add an in point (clearly inside) and out point (clearly outside). On + off is the minimum that pins a boundary.
5. Devise the test cases. Combine the partitions/boundaries into concrete cases. When inputs multiply (3 inputs × 4 classes = 64 combinations), don't take the Cartesian product — apply constraints to drop impossible combinations, then use pairwise (all-pairs) selection to cover every pair of values with far fewer cases. (Detail in references/specification-based-testing.md.)
6. Automate the test cases. Write them as parameterized tests — one case per partition/boundary, named for the behavior it covers (house style: behavior sentences; see khorikov-unit-testing). No logic in the test.
7. Augment with creativity and experience. Add cases your systematic pass won't produce: known nasty inputs (null, empty, huge, Unicode, duplicates), domain intuition, and past bugs. The procedure is a floor, not a ceiling.

---

Structural testing & coverage as a guide

After the spec-based pass, measure structural coverage to find untested parts of the code, and choose a criterion deliberately. The criteria, weakest to strongest:

• Line (statement) — each line executed. Weakest; a line can run without its logic being meaningfully checked.
• Branch — each branch of each decision (every if taken both ways) executed.
• Condition — each individual boolean sub-condition evaluated both true and false.
• Condition + branch — both of the above together (the practical default for decisions with compound conditions).
• Path — every path through the code. Strongest, but explodes with loops; usually infeasible in full.
• MC/DC (Modified Condition/Decision Coverage) — for complex boolean expressions, test the combinations that show each condition independently affects the outcome. It targets the important combinations instead of all 2ⁿ — the right tool when a decision has many &&/|| operands (and what safety-critical standards require).

Subsumption (X subsumes Y = achieving X guarantees Y): path ⊇ MC/DC ⊇ condition+branch ⊇ {branch, condition} ⊇ line. A stronger criterion finds more but costs more. Full hierarchy and the one exception (basic condition coverage does not necessarily subsume line) in references/structural-and-coverage.md.

Coverage is a guide, not a target — the repo-wide house line. A high percentage doesn't prove the tests assert anything real (you can cover a line with no meaningful assertion), and chasing the number produces tests that hit lines instead of catching bugs (Goodhart's law). Use coverage's negative signal — an uncovered branch is a genuine gap worth a test — and ignore its positive signal as proof of quality. This is exactly the stance in khorikov-unit-testing/references/four-pillars-and-styles.md §4.

---

Property-based testing

When a function has a property that holds for all valid inputs — an invariant, a round-trip (decode(encode(x)) == x), an algebraic law, a relation to a simpler reference — prefer a property-based test over hand-picked examples. You state the property and a generator for inputs; the framework throws hundreds of random inputs at it and, on failure, shrinks to a minimal counterexample. It explores far more of the input space than examples and often surfaces edge cases you'd never enumerate. Reach for it for parsers, serializers, data structures, numeric/algorithmic code; keep example-based tests for specific known scenarios. Detail and the "finding good properties" guidance in references/property-based-and-testability.md.

---

Designing for testability

Hard-to-test code is a design problem, not a testing problem. Two properties make code testable, and you should design for both:

• Controllability — how easily a test can drive the code into the state it wants. Achieved mainly through dependency injection: a class receives its collaborators (constructor/parameters) so a test can substitute a stub/fake/mock, rather than instantiating databases and gateways itself.
• Observability — how easily a test can inspect the outcome. If a result is buried in private state, provide a simple, honest way to observe it (a query method, an isValid()), or have the unit return the decision instead of hiding it.

At the architecture scale this is separating domain from infrastructure (Hexagonal Architecture): pure domain logic you test directly, thin infrastructure adapters you mock or integration-test. This overlaps the Humble Object / functional-core pattern in khorikov-unit-testing/references/what-to-test.md — same idea from the testability angle.

House-style note: Aniche is pragmatic about adding a getter or isValid() purely to observe state. Khorikov is stricter — he'd rather restructure (extract a pure unit that returns the value) than widen visibility just for a test. Prefer restructuring when it's cheap; accept a small, honest observability hook when it isn't. Don't expose mutable internals or test-only branches either way.

---

Assessing the suite: mutation testing & smells

Coverage tells you what ran; it doesn't tell you whether your assertions would catch a bug. Mutation testing does: a tool injects small bugs (mutants) into the production code and reruns your suite. A mutant your tests catch is killed; one they miss survives. Surviving mutants point at weak or missing assertions — genuinely actionable, unlike a raw coverage number. Use it to harden the suite on critical code. (Detail and the test-smell checklist in references/mutation-and-test-quality.md; the smell catalogue overlaps Meszaros — see xunit-test-patterns/references/test-smells.md.)

---

Reference files

• references/specification-based-testing.md — the 7 steps in full, equivalence partitioning, boundary on/off/in/out points, and taming combinations with pairwise.
• references/structural-and-coverage.md — every coverage criterion, the subsumption hierarchy, MC/DC worked through, and using coverage as a guide.
• references/property-based-and-testability.md — properties & generators, when to prefer property-based testing, controllability/observability, and hexagonal design.
• references/mutation-and-test-quality.md — mutation testing, the competent- programmer & coupling-effect assumptions, and a test-code quality checklist.

../../EXAMPLES.md (repo root) has worked before/after pairs under "Effective Software Testing."