| name | wads-test-coverage |
| description | Diagnose and close test-coverage gaps in a wads-managed Python repo (pytest + doctests + uv CI). Measures coverage exactly as CI does (doctests included), adds branch coverage and per-test contexts, buckets every public function/class as untested / doctest-only / test-covered, then writes the missing tests. Use for "what's untested", "check test coverage", "coverage gaps", "which functions have no tests", "is this only covered by doctests", "improve coverage", "raise coverage", "write tests for the uncovered parts", "add tests for this module". Also trigger when CI coverage numbers look wrong or a doctest-heavy package looks untested. Does NOT analyze failing-test output (pipe pytest output to the wads-test-analyze CLI), measure docstring presence (wads-docs-coverage), or check CI workflow health (wads-ci-health). |
| metadata | {"audience":"users"} |
Wads Test Coverage
Diagnose which parts of a package are genuinely untested — distinguishing
"covered only by doctests" from "covered by dedicated tests" — and then write
the tests that close the real gaps. The golden rule is observe, don't
invent: every gap claim comes from an actual coverage measurement, and every
new test is verified to pass (and to move the numbers) before you report it.
This ecosystem is doctest-first: doctests are simultaneously docs and tests,
and CI always runs them. A measurement that skips doctests is wrong here.
Step 0 — Resolve the target repo and package dir
Operate on the repo path the user gives, else the current working directory.
Then establish ground truth about what to measure and how CI measures it:
cd /path/to/repo
python - <<'EOF'
import pathlib
try:
import tomllib
except ImportError:
import tomli as tomllib
t = tomllib.load(open('pyproject.toml', 'rb'))
name = t['project']['name']
pkg = name.replace('-', '_')
print(f"project name : {name}")
print(f"package dir : {pkg} (exists: {pathlib.Path(pkg).is_dir()})")
tp = t.get('tool', {}).get('pytest', {}).get('ini_options', {}).get('testpaths', [])
print(f"testpaths : {tp} (exist: {[pathlib.Path(p).is_dir() for p in tp]})")
c = t.get('tool', {}).get('wads', {}).get('ci', {}).get('testing', {})
print(f"pytest_args : {c.get('pytest_args', ['-v', '--tb=short'])}")
print(f"exclude_paths: {c.get('exclude_paths', ['examples', 'scrap'])}")
print(f"coverage : enabled={c.get('coverage_enabled', True)}, "
f"threshold={c.get('coverage_threshold', 0)} (threshold is NOT enforced by CI)")
EOF
PKG below means the package directory printed above. ⚠️ Pitfall:
--cov must name the package dir, not [project].name. A hyphenated
project name (my-pkg) with an underscored dir (my_pkg) — or a dir that
differs from the name entirely — makes --cov=<project-name> collect nothing
and report misleading or empty coverage. Verify the dir exists first.
⚠️ Pitfall: stale testpaths. Many repos declare testpaths = ["tests"]
with no top-level tests/ dir (tests live in PKG/tests/ instead). When the
declared dir is missing, pytest falls back to rootdir-wide collection — so CI
quietly collects more (or different) things than the config suggests. Note it
as a finding; the measurement below sidesteps it by passing paths explicitly.
Diagnose
1. Measure — mirror CI, plus what CI doesn't give you
Run from the target repo root (capture output to a file — see pitfall below):
python -m pytest --doctest-modules \
-o doctest_optionflags='ELLIPSIS IGNORE_EXCEPTION_DETAIL' \
--cov=PKG --cov-branch --cov-context=test \
--cov-report=term-missing --cov-report=json:coverage.json \
PKG > /tmp/coverage_run.log 2>&1
rc=$?
tail -40 /tmp/coverage_run.log; echo "pytest exit: $rc"
Why each piece, relative to what the uv CI (run-tests-uv) does:
| Flag | In CI? | Why it's here |
|---|
--doctest-modules -o doctest_optionflags=... | always | Non-optional. CI unconditionally runs doctests with exactly these flags. Omit it and every doctest-primary package looks untested — the single most misleading mistake a coverage diagnosis can make here. |
--cov=PKG --cov-report=term-missing | yes (when coverage_enabled) | Same line-coverage CI reports. |
--cov-branch | no | CI is line-only. Branch coverage exposes untested if/except arms — exactly where bugs hide in "covered" code. |
--cov-context=test | no | Records which test or doctest executed each line — the raw data for doctest-vs-test attribution. |
--cov-report=json:coverage.json | no | Machine-readable artifact (CI produces none) with per-function/class regions (coverage >= 7.5). |
For full CI fidelity, append the repo's [tool.wads.ci.testing] values from
Step 0: each pytest_args entry as-is, and each exclude_paths entry as
--ignore=<path>. Example with the defaults:
python -m pytest --doctest-modules \
-o doctest_optionflags='ELLIPSIS IGNORE_EXCEPTION_DETAIL' \
--cov=PKG --cov-branch --cov-context=test \
--cov-report=term-missing --cov-report=json:coverage.json \
--ignore=examples --ignore=scrap -v --tb=short \
PKG > /tmp/coverage_run.log 2>&1
If the repo keeps its test suite outside the package (a real top-level
tests/), append that dir too (... PKG tests) — otherwise dedicated tests
aren't collected and everything mislabels as doctest-only or untested.
⚠️ Pitfall: never read an exit code through a pipe. pytest ... | tail
makes $? report tail's status, not pytest's (a verified false-green source).
Always redirect to a file, capture $? immediately, then inspect the file.
⚠️ Pitfall: the suite must be green before you measure. A collection
error or failing test truncates coverage data. If the run fails, fix or
report that first (for failure triage, pipe the log to the wads-test-analyze
CLI — it parses pytest text output, not junit XML).
2. Per-module gap table
python -m coverage report --format=markdown --sort=cover --skip-covered
Worst modules first, fully-covered files skipped — paste-ready for an issue
or report. (--format=markdown landed in coverage 7.0; the same data is in
coverage.json under files.*.summary if the flag is unavailable.)
3. Bucket the public API: untested / doctest-only / test-covered
Export contexts, then run the bundled analyzer:
python -m coverage json --show-contexts -o coverage_ctx.json
python scripts/coverage_gaps.py /path/to/repo
python scripts/coverage_gaps.py /path/to/repo --json
(scripts/ is relative to this skill; the repo path defaults to cwd.)
The script reads coverage_ctx.json and buckets every public function/class
region by its execution contexts:
- doctest contexts look like
pkg/mod.py::pkg.mod.func|run — the nodeid
file is the module itself, the item a dotted object name.
- test contexts look like
tests/test_x.py::test_name|run.
- No contexts at all (or only import-time execution) → untested.
"Public" = no component of the dotted name starts with _. Test files are
excluded from the buckets. The script needs per-region data from
coverage >= 7.5 (files.*.functions / files.*.classes) and errors
clearly if it's absent — upgrade coverage rather than approximating with AST.
⚠️ Pitfall: coverage.json (from --cov-report=json:) has regions but NO
contexts. Attribution needs coverage_ctx.json from coverage json --show-contexts. The script tells you exactly this if you feed it the wrong
file.
Doctrine: doctest-only is a category, not a verdict
In this ecosystem doctest-only coverage is legitimate — doctests are the
documentation and the tests, and CI runs them on every push. Do not report
the doctest-only bucket as a gap per se. Do flag, within it:
- error paths and edge cases covered only by a happy-path doctest
(the
--cov-branch missing-branches column points right at these);
- functions whose doctest asserts trivia (e.g. just constructs an object)
while the meaningful behavior goes unexercised.
The untested bucket is the real gap list. Prioritize it.
Complete — write the missing tests
Priority order
- Untested public API — the
untested bucket from coverage_gaps.py,
worst modules first (the script already orders them).
- Branch gaps in modules you're already touching — missing branches from
term-missing / coverage.json (files.*.summary.missing_branches),
especially error paths behind if/raise/except.
- Doctest-only regions whose edge cases deserve a real test (see doctrine).
Style rules
- Doctests for example-shaped behavior — the happy path a reader should
see. They double as documentation, which is the point here. Two admission
criteria for a doctest:
-
(a) Low setup. A line or two of setup is fine; if the docstring fills
up with hard-to-read scaffolding that exists only to create test
conditions, it stops being documentation — write a pytest test instead.
-
(b) Robust assertions. These run on different platforms and Python
versions. Anything whose repr is unordered or environment-dependent
(sets, dict views, object reprs, float noise, paths) must not be
asserted by displayed output — use an explicit comparison instead:
>>> result = f('x')
>>> assert result == {'a', 'b'}
>>> sorted(other_result)
['a', 'b']
Never show memory addresses, ids, or timestamps; use ELLIPSIS (...)
where partial output is unavoidable, and # doctest: +SKIP only as a
last resort (skipped examples don't count as coverage).
- Pytest tests (in the repo's existing test dir) for edge cases, error
paths (
pytest.raises), parametrized matrices, and anything needing
fixtures/tmp dirs — things that would clutter a docstring.
- A broken doctest is a broken test — when the measurement run (or the
rollout gate) fails on an existing doctest, fix it like any failing test:
decide whether the code or the documented example is wrong, never delete
the example just to go green.
- Match the existing test style — read a neighboring test file first;
mirror its naming, imports, and assertion idioms.
- No new dependencies. Stdlib + pytest + what the repo already declares.
- ⚠️ Pitfall: a package utility named
test_* gets collected as a test
under this measurement (and in CI). If you write or meet a non-test helper
named test_* in package code, set __test__ = False on it (or rename).
- ⚠️ Pitfall: doctests on
functools.partial objects are NEVER collected.
Doctest's finder only inspects functions/classes/lambdas, so an example in a
some_partial.__doc__ = "...>>> ..." assignment is a dormant test — it
never runs, and may be silently wrong (verified in the wild: one documented
behavior the function didn't have). When you meet the pattern, check the
example by hand and prefer converting the partial to a small def so the
doctest goes live. Grep for the pattern: rg -n '__doc__\s*=' PKG.
- ⚠️ Pitfall: import-unclean modules break
--doctest-modules. It imports
every module; one that reads env vars or hits the network at import time
fails collection. Mitigation: declare the var in
[tool.wads.ci.env].test_envvars (CI) and export it locally; longer term,
make imports lazy — that fix is wads-import-time's job. Don't "fix" it
by dropping --doctest-modules.
Verify
- Full suite green before AND after — run the Step 1 command both times.
Never break an existing test; if a new test exposes a real bug, report the
bug rather than bending the test to pass.
- Re-run the measurement and report the delta: totals
(
python -c "import json; print(json.load(open('coverage.json'))['totals']['percent_covered'])"),
the per-module table, and the before/after bucket counts from
coverage_gaps.py --json.
CI enforcement (optional, propose first)
[tool.wads.ci.testing].coverage_threshold is dead config today — CI
reads it but never enforces it (no --cov-fail-under anywhere in the uv
actions). To actually gate, override the test command:
[tool.wads.ci.commands]
test = "pytest --doctest-modules -o doctest_optionflags='ELLIPSIS IGNORE_EXCEPTION_DETAIL' --cov=PKG --cov-report=term-missing --cov-fail-under=80"
This is a CI-affecting change: propose it to the user, set the bar at (or
slightly below) the measured level so it ratchets rather than blocks, and
verify on a non-default branch before merging.
Scope guardrails
- Diagnosis is read-only. The measurement writes scratch artifacts into
the repo root (
.coverage, coverage.json, coverage_ctx.json) — never
commit them; delete them when done unless .gitignore already covers them.
- Never game the metric: don't delete, skip, or weaken existing tests;
don't add
# pragma: no cover or shrink exclude_paths/test scope to make
numbers look better. Coverage config changes need explicit user agreement.
- Don't touch CI without proposing — threshold gating and
[tool.wads.ci.*] edits are verified on a non-default branch first.
- Stay in your lane: this skill measures and writes tests. Failing-CI
forensics, docstring quality, and publishing are other skills' territory
(below).
Related skills
- wads-repo-doctor — the orchestrator; runs this skill as the
test-coverage step of a full repo health pass.
- wads-docs-coverage — docstring presence and README alignment.
Docstring presence ≠ doctest coverage: a function can have a beautiful
docstring and still sit in the
untested bucket (and vice versa).
- wads-ci-health — is CI green, current stub, secrets/publish sane.
Coverage findings about CI wiring (dead threshold, line-only) land there.
wads-test-analyze CLI — failure triage: pipe pytest text output to
it (pytest ... | wads-test-analyze); it does not parse junit XML.
Closing checklist
