| name | code-testing |
| description | Use when the user asks to add tests, write a regression test for a bug, set up CLI execution tests or coverage in CI, or pick the kind of test (unit / property / blackbox / acceptance) to write. Triggers on phrases like "add a test for this", "write a regression test", "add test coverage", "test this CLI", "property-based test", "fuzzing", "what kind of test should I write", "edge-case test". |
Code testing
Serokell's testing philosophy is built around two qualities of the
system being developed:
- External Quality — how well the system meets its functional
requirements and external constraints (what the customer sees).
- Internal Quality — how well the system is designed, how easily
it can be changed (what the developers feel).
Both matter. Tests serve both. Different kinds of tests serve them in
different ratios.
Mandatory rules
Defect-driven testing
When you fix a bug, you must add a test that fails before the fix
and passes after. Commit them together (or the test in the same PR).
This:
- Documents the bug — the test explains the failure mode.
- Increases coverage on a previously uncovered code path.
The PR template's "Tests" checkbox under "Related changes" enforces
this for bugs.
Edge-case safety
While writing code, identify edge cases (empty inputs, boundary
values, error paths, etc.) and add a test for each one — even if it
passes trivially. The point is not the assertion but the marker: the
test will be there when someone later modifies the code, and it will
make them pay attention to the case.
Execution tests for CLI tools
If the product ships any CLI tool (a binary the user invokes, not a
long-running daemon), the test suite shall execute that tool
through a standard shell-like invocation and validate every subcommand
and option as thoroughly as possible. This is the single most
important kind of test for shipping a high-quality CLI product.
Test coverage in CI
- CI should produce a test coverage report.
- CI should fail merges that reduce coverage.
- CI jobs should exercise every executable file in the
repository — including small scripts — to catch breakage in
rarely-touched code.
Test classification (vocabulary)
These terms are used consistently across Serokell — use them when
discussing or naming tests.
Level
- Acceptance tests (highest level) — verify the system as a whole.
Cover all components and their interactions.
- Integration tests — verify that specific modules interact
correctly.
- Unit tests (lowest level) — verify the smallest independently
testable component in isolation.
Higher levels → more feedback on External Quality, less on Internal.
Lower levels → opposite.
Interface use
- Edge-to-edge (aka end-to-end) — the test feeds inputs through
the public interface (UI, API) and reads outputs through the public
interface, just like a real user.
- Blackbox — an edge-to-edge test that does not peek at internal
state at all. Strictly measures External Quality.
- Greybox — between blackbox and unit; uses private interfaces to
customise initialisation or observe internal state, but tests
higher-level behaviour. Balances External / Internal feedback.
Data generation
- Specific test — given a known input, expects a known output.
- Invariant test (aka Property / Generative test) — random
data generated, output checked against a predicate.
Purpose
- Customer test — derived from customer-facing specs; mostly
measures External Quality.
- Developer test — written for developers' own benefit; mostly
measures Internal Quality.
Preferred testing strategies
Pure unit testing
Lowest-level developer tests of pure components. Greatest impact on
Internal Quality.
- Prefer probabilistic invariant (property-based) testing as the
primary tactic.
- Fall back to specific tests for edge cases that property-based
generators don't reliably hit.
The mere existence of a unit test forces the surrounding code to be
decoupled enough to be tested — that's a benefit independent of the
assertions.
Functional blackbox testing
Highest-level customer blackbox tests for functional requirements.
Most important for External Quality. Non-intrusive: they don't need
support from the system under test, and the framework is independent
of the technology stack.
Durability / load / stress / fuzzing
Highest-level blackbox tests for non-functional constraints
(reliability, resilience, security). Deploy the system in a typical
configuration, then:
- Durability — leave it under normal load for a month.
- Load — leave it under very high load for a couple of hours.
- Stress — slowly increase load over ~10 minutes until it fails.
- Fuzzing — throw random or semi-random data at the public
interface.
These are valuable when the project has infrastructure for them, but
require no input from the customer or developer beyond instrumentation.
Product-level expectations
If you're authoring or maintaining a Serokell product (library or
application):
- Fuzzing generators: the product should ship an executable or API
that produces random valid inputs, with a knob for "meaningfulness"
of generated data. Useful both for fuzzing the product itself and
for fuzzing things that consume the product.
- Benchmarks: every product, including libraries, should ship
executables for benchmarking. CI should run them, report results, and
collect profiling data (language-specific tools plus
perf where
applicable). This catches performance regressions, e.g. after
updating dependencies.
Where this skill does NOT prescribe
- Impure unit testing (mocking IO-bound components) — Serokell
knows it matters but doesn't have a settled methodology. Use your
judgement.
- Functional greybox testing — same status: valuable in principle,
no settled approach. Don't avoid it, but don't expect a template.
- TDD — supported in principle but not mandated.
Local Haskell specifics
The haskell-style skill notes that tasty is the assumed primary
test runner in this repo and that tests should be built with
-threaded -with-rtsopts=-N. The setup-ci skill covers Haskell-CI
test jobs (multiple GHC versions, multiple stack snapshots, hlint).
