Property-based and metamorphic testing expert for choosing a library, finding good properties (invariants, round-trips, idempotence, oracle/differential, metamorphic relations), and wiring generative tests into the existing test runner.
Instrucciones de origen · Vista previa de solo lectura
name
property-based-testing
version
1.0.0
description
Property-based and metamorphic testing expert for choosing a library, finding good properties (invariants, round-trips, idempotence, oracle/differential, metamorphic relations), and wiring generative tests into the existing test runner.
auto_activates
["property-based testing","property based test","metamorphic testing","generative testing","fuzz property","proptest","quickcheck","hypothesis testing library","fast-check","fscheck","jqwik","shrinking"]
Expert guidance for property-based and metamorphic testing: instead of
asserting one hand-picked example, you state a rule that must hold for every
input, and the library generates hundreds of inputs to try to break it.
Where This Fits — the third leg of the formal-methods triad
amplihack ships two other formal-methods skills. Property-based testing is the
practical middle ground between them:
Approach
Skill
What it checks
Cost
Example tests
(unit tests)
Behavior on inputs you thought of
Low
Property-based / metamorphic
property-based-testing
Invariants over generated inputs, at runtime, against real code
Medium
BDD acceptance
gherkin-expert
Human-readable behavior scenarios
Low–medium
Model checking / proof
tla-plus-expert
Exhaustive state-space or temporal properties of a model
You can state a rule the code must obey but can't enumerate every input.
You have a reference implementation (an old version, a slow-but-correct
oracle, or a sibling implementation in another language) to compare against.
Prefer TLA+ instead when the property is about concurrency, ordering, or
temporal/liveness behavior of a design you can model. Prefer Gherkin when the
value is a shared, human-readable acceptance spec. These are complements, not
substitutes — a mature component often has all three.
Pick the library for your stack
Stack
Library
Runner it plugs into
Rust
proptest (preferred) or quickcheck
cargo test
Python
Hypothesis
pytest /
unittest
.NET
FsCheck (F#-friendly) or CsCheck (C#-friendly)
xunit / nunit
JS/TS
fast-check
jest / vitest / mocha
Java
jqwik
JUnit 5 platform
Do not build a separate harness. Every library above runs inside your
existing test runner as ordinary test cases. A property test is just a test
that loops over generated inputs.
Finding good properties
A "property" is a rule that holds for all valid inputs. The reliable families:
Invariant — something always true of the output (e.g. output length ≤
input length; a sorted list is still a permutation of the input).
Round-trip — decode(encode(x)) == x. Great for serializers, parsers,
compressors, config load/save.
Commutativity / associativity — order of operations doesn't change the
result: merge(a, b) == merge(b, a).
Oracle / differential — compare against a trusted reference: a naive
slow implementation, the previous release, or a sibling implementation in
another language (cross-language conformance).
Metamorphic relation — you don't know the exact output, but you know how
the output must change when the input changes. E.g. adding an item to a set
can never shrink its size; re-scaling all weights by k scales the total by k;
searching a superset returns at least as many hits.
When you can't state an exact expected value, a metamorphic relation is usually
still available. That is the whole point of metamorphic testing.
Shrinking, seeding, reproducibility
Shrinking: when a property fails, the library automatically reduces the
failing input to the smallest counterexample (e.g. from a 400-char string
to "a"). Never disable shrinking — the minimal case is the bug report.
Seeding: every run prints/records a seed. On failure, re-run with the
same seed to reproduce deterministically. Commit the seed (or the reduced
counterexample as a regression example test) so the bug stays fixed.
Budget: start with the default case count (usually 100–1000). Raise it in
CI for critical invariants; keep it low locally for fast feedback.
Worked examples (concrete property families)
The snippets below use amplihack's own invariants as worked examples.
using FsCheck;
using FsCheck.Xunit;
publicclassCoverageProperties
{
// bundle >= in_scope AND scanned + excluded == in_scope
[Property]
public Property CoverageTallyHolds(PositiveInt inScope, PositiveInt extra, NonNegativeInt excluded)
{
int in_scope = inScope.Get;
int bundle = in_scope + extra.Get; // bundle is a supersetint exc = Math.Min(excluded.Get, in_scope);
int scanned = in_scope - exc;
bool bundleSuperset = bundle >= in_scope;
bool tallyBalances = scanned + exc == in_scope;
return (bundleSuperset && tallyBalances).ToProperty();
}
}
JS/TS — fast-check (telemetry totality / safety)
import fc from"fast-check";
import { recordTelemetry } from"../src/telemetry";
// totality: recordTelemetry never throws and always returns a well-formed eventtest("telemetry is total and safe", () => {
fc.assert(
fc.property(fc.anything(), fc.string(), (payload, name) => {
const event = recordTelemetry(name, payload); // must not throwexpect(event.name).toBe(name);
expect(typeof event.timestamp).toBe("number");
expect(typeof event.dropped).toBe("boolean"); // always a boolean, never undefined
}),
);
});
For amplihack's multi-runtime components, run the same generated inputs
through each language implementation and assert identical outputs. Generate the
corpus once (with a fixed seed), feed it to every implementation, and diff the
results. Any divergence is a conformance bug — shrink it to the minimal input.
Quick quiz
Check your understanding (answers below).
A function normalize_path is claimed to be safe to call more than once.
Which property family proves that, and how do you write it?
You are testing a JSON serializer in TypeScript and want to catch encoding
bugs. Which library and which property family fit best?
You can't predict the exact ranking a search function returns, but you know
adding a matching document must never reduce the number of results. What
kind of property is this?
Which stack does jqwik target, and which runner does it plug into?
A redaction function must satisfy two rules. Name them and classify each.
Answers
Idempotence: assert normalize_path(normalize_path(p)) == normalize_path(p)
over generated paths.
fast-check, using a round-trip property: decode(encode(x)) == x.
A metamorphic relation — you constrain how the output must change, not
its exact value.
Java, plugged into the JUnit 5 platform.
Idempotence (redact(redact(x)) == redact(x)) and a no-secret-leak
invariant (the output never matches the secret). The first is an
idempotence property; the second is a safety invariant.
Related Skills
tla-plus-expert — exhaustive model checking of designs (the high-cost leg).
