quality-report-setup

name

quality-report-setup

description

One-time bootstrap that stands up a weekly quality-report dashboard in a repo that has none. Inspects the codebase to discover its real stack (languages, test/coverage tooling, security scanner, e2e matrix, a11y lint, default branch, local-vs-container exec), decides which of the seven metrics the repo can actually support, then BUILDS the per-repo analyzers and installs the shipped, stack-invariant report-generator template — it ships the generator (identical everywhere) but builds the analyzers (which vary by stack). Wires unit-coverage (blended line+branch), e2e coverage, maintainability (cognitive complexity), accessibility, and security into a rotating dashboard with whole-repo and new-code (Clean-as-You-Code) lenses, grades, RAG, and trends. Verifies by generating the report once. Run ONCE per repo. Use when the user says 'set up the quality report', 'add a quality dashboard', 'bootstrap quality metrics', 'reverse-engineer the quality report into this repo', or before the first weekly report run on a repo that has none.

Quality Report Setup — one-time bootstrap

Stands up a weekly quality dashboard + its analyzers in a repo that has none. Run once per repo. The job is to look at this codebase and build the smallest set of analyzers that surface the metrics for this stack — not to drop in pre-made ones.

Read reference/metrics.md first. It is the source of truth for what each metric means, how its data is surfaced, the grade/RAG thresholds, and the report layout. For the one metric that needs real analysis code — maintainability — also read reference/complexity-example.md, a worked example (algorithm + reference implementation + hand-checked validation set) to port from. This skill is the process; those files are the methodology and example. Everything below assumes you've internalised them.

The seam that makes this portable: every metric reduces to (produce one intermediate artifact) → (collect into a normalized metrics.json) → (the generator renders it). Your job per metric is to produce that artifact however this repo's tooling allows. Change the stack → change how an artifact is produced, never the generator. Only maintainability needs a genuinely custom analyzer; the rest are thin readers over tools the repo already runs.

What ships vs what you build: the report generator is shipped — template/generate.py is stack-invariant (grades, RAG, trends, This/Last-week rotation, History, backing numbers) and is copied in, not rewritten (exactly as e2e-setup ships harness/). You build only the per-repo analyzers and the small collect step that feeds the generator its metrics.json. See template/README.md for the input contract.

Conventions (apply to every step below)

These four rules prevent the silent-failure modes that otherwise produce plausible-but-wrong dashboards. They matter more than any single metric.

• Run every tool through one wrapper. Decide once (Step 1) whether commands run directly or inside a container / compose service, and route all coverage/analyzer/scanner commands through that same path. When containerised, make sure each artifact lands somewhere readable from where the report generator runs (a mount, docker cp, or a shared work dir) — an artifact the generator can't open reads as "metric missing".
• Normalize every path to repo-relative, the same way, everywhere. Coverage reports, the complexity analyzer, a11y/security output, and git diff must all express paths in one form: repo-relative, forward slashes, no absolute or container prefix (strip a leading /workspaces/<repo>/, the absolute repo root, etc.). The new-code lens intersects diff paths with each artifact's paths — if they don't match, the intersection is silently empty and new-code coverage/maintainability/a11y all read 0%. The trap is that each format anchors paths differently:
  • Cobertura XML (coverage.py, coverlet, gocover-cobertura) anchors filename to the <sources><source> element, not the repo root — so <source> is /abs/repo/src/requests and a class is filename="adapters.py", which is really src/requests/adapters.py. You must read <source>, compute its repo-relative prefix (src/requests), and prepend it to every filename — otherwise the diff's src/requests/adapters.py never matches the bare adapters.py and new-code coverage reads 0%.
  • Istanbul JSON (jest/vitest) keys by absolute path → strip the repo root.
  • lcov SF: records are usually repo-relative already, but can be absolute or ./-prefixed → normalize. Cross-check before trusting it: take one file you know changed this week and confirm its path string is byte-identical in the diff and in each artifact. (When a week's diff touches no product code, do this check against any source file present in both the diff history and the coverage artifact, so you've still proven the prefixing is right.)
• Label any degraded metric; never fake one. If a metric can only be computed partially, put the caveat in its reported value and list it as a limitation — don't pass a partial measure off as the full one (see the line-only-coverage and skipped-new-code cases below). A loud caveat beats a silent wrong number.
• Measure the product, not the scaffolding. Exclude the quality/ pipeline you add — the analyzers, the collect step, and especially the copied generate.py — from every analyzer and scanner input. Otherwise the dashboard measures its own tooling, and the maintainability analyzer double-counts the copied generator against its template source. Likewise scope each code metric to product code: exclude tests, fixtures, generated, and vendored code from maintainability (a long, branch-free test body otherwise surfaces as a false "problem function"; a deliberately-pathological fixture skews the percentage). Decide the measured surface explicitly and record it as a scope note in the report — "what we measured" is itself a number a reader needs to trust the rest.

Step 0 — Detect: already set up?

Look for an existing dashboard (e.g. QUALITY_REPORT.md) and a report builder.

• Found → setup has already run. Stop. Day-to-day regeneration is the quality-report-update skill, not this one.
• Not found → continue.

Step 1 — Confirm the target (one round, then proceed)

Ask only what you can't determine from the repo:

1. Which repo / working tree is the report for (default: the current repo root).
2. Where the dashboard should live (default: QUALITY_REPORT.md at repo root) and whether it should be gitignored (it's often a personal/working artifact) or committed.
3. Default branch for the weekly diff (default: auto-detect origin/HEAD, fall back to the current branch).
4. Execution — do tests/tools run directly, or inside a dev container / compose service? If containerised, get the service/compose details so analyzer commands can be wrapped.

Don't block on anything inferable. Confirm the repo actually builds / its tests run before wiring coverage — a metric whose tool can't run is a metric you can't surface.

Bootstrap a runnable toolchain — don't assume one exists. A fresh clone often has no usable environment: no virtualenv, dependencies uninstalled, and the default interpreter on PATH the wrong version. Before wiring any metric:

• Check the runtime version against what the repo declares (requires-python in pyproject.toml, engines in package.json, the go directive in go.mod, etc.). The system default is frequently too old — e.g. system python3 is 3.9 while the repo needs ≥3.10. Pick an interpreter that satisfies the declared floor (python3.13, the right node, …).
• Create a dedicated, isolated environment and install the project + its test/dev deps into it — a private venv (python3.13 -m venv .venv-quality), a clean npm ci, etc. Keep it separate from any env the developer uses so the pipeline is reproducible and doesn't pollute the repo. Install from the repo's own dev/test extras (pip install -e '.[test]', the test dependency-group, requirements-dev.txt, npm ci). npm ci hard-fails on a stale lockfile — an abandoned repo's package-lock.json is often out of sync with package.json, and npm ci refuses outright ("can only install when … in sync"). Don't treat that as unbuildable: fall back to npm install --no-audit --no-fund --package-lock=false, which populates node_modules without mutating the tracked lockfile (the "don't edit the repo's tracked files" convention still holds). Verify it worked by checking node_modules actually populated — and grep the install log for errors rather than trusting a chained command's exit code, which can mask the installer's own failure.
• The package manager binary itself may be absent — not just the deps. A fresh clone can pin a PM the machine doesn't have on PATH at all (packageManager: "yarn@1.22.22" in package.json, a pinned pnpm, a Corepack-managed version). Don't assume yarn/pnpm exist — bring up the declared version first (corepack prepare yarn@1.22.22 --activate, corepack enable) before installing, and invoke that PM consistently thereafter. This is the JS counterpart of the version-correct-interpreter check above.
• Invoke repo-local binaries by path — never bare npx <tool>. npx react-scripts (or any npx <tool>) silently downloads a tool when the local one is missing — and often a different version than the repo pins (e.g. react-scripts@5.0.1 when the repo declares 5.0.0), then runs a broken environment that fails on missing deps and writes misleading zeros. So once deps are installed, run the pinned binary directly (./node_modules/.bin/react-scripts, ./node_modules/.bin/eslint) — same "pin the bootstrapped toolchain" rule as Step 5, and it also surfaces a failed install instead of papering over it.
• Then run the suite once to confirm it's green before wiring coverage. If you cannot get the tests running at all, coverage (and anything that needs the suite) becomes a labelled limitation — never a fabricated number.
• Confirm the coverage artifact is actually written — some runners suppress it on a failing suite. A green suite isn't guaranteed: a fresh clone often has a couple of environmental failures (a font/snapshot test under jsdom, a timing-sensitive test). The trap is that several runners then write no coverage report at all — most importantly vitest, whose coverage.reportOnFailure defaults to false, so one failing test silently produces an empty coverage/ dir and the whole coverage metric vanishes. So: after the run, check the coverage file exists; if a few env-dependent tests fail, force coverage to emit anyway at invocation time (vitest: --coverage.reportOnFailure=true; the override-don't-edit rule from Step 4 applies) and record the failing tests as a labelled limitation rather than treating them as a blocker. Only a suite you genuinely cannot run at all makes coverage n/a.
• The regenerate command (Step 5) must pin this bootstrapped environment — its interpreter and tool binaries by path (.venv-quality/bin/python, not bare python/pytest on PATH), so the weekly run doesn't silently fall back to the wrong/global toolchain. This is the same "one wrapper, decided once" rule as the container case in the Conventions.
• Watch repo-injected test options — but verify before overriding, don't blanket-wipe. A repo's pytest addopts, default jest config, or similar can break or distort a coverage run — a -n auto parallel flag that confuses coverage measurement, a test-selection default that silently narrows the suite, or --doctest-modules collecting failing source doctests. But many such options are harmless: --doctest-modules only collects doctests from the paths being collected, so with testpaths=["tests"] it never touches src/ and changes neither pass counts nor coverage. So: first run the suite once with the repo's real options and confirm it's green and the counts are sane. Only override the specific offending option if it actually fails or distorts the metric, and re-confirm the pass/skip counts match the un-overridden run. Don't reach for a blanket pytest -o addopts="" — it can silently drop config the coverage run needs (markers, --strict-config, required plugins). If you do override anything, note it.
• Record the bootstrap as a scope note if it diverges from the repo's documented setup (a pinned interpreter the repo doesn't ship, branch coverage the repo's own config doesn't enable — see the coverage note in Step 4).

Step 2 — Inventory the repo

Survey the codebase and write down, for each metric, whether it's achievable and how:

• Languages & layout, and the tier set. Which languages, and is it a monorepo with multiple apps/projects? Decide the coverage/maintainability tiers here — a tier is a meaningful reporting group, not "one per language by reflex". Prefer grouping by deployable / app (e.g. web, api), keep the count small (roughly ≤4 rows per metric — a 12-row table helps nobody), and propose the grouping to the user in Step 3 rather than inventing it silently. A single-language repo is just one tier.
• Test & coverage tooling — and whether it produces branch data. What runner is configured (jest/vitest, coverlet, pytest-cov, go test -cover, SimpleCov…) and what format it emits (Istanbul json, Cobertura xml, lcov…) — you read that format, not impose one. Critically, check whether branch coverage is actually produced, and distinguish two reasons it might be absent — they have different fixes:
  • The repo's config simply doesn't ask for it (the common case). .coveragerc / pyproject has no branch = True, make coverage omits --cov-branch, the jest config doesn't set coverageProvider/branch reporting. The tool can produce branches; the committed setup just doesn't. This is fixable — you override it (Step 4), it is not a limitation.
  • The tool genuinely cannot emit branches (Go -cover is statement-only; some lcov setups). No override helps → it's a real line-only limitation. Don't run the repo's own coverage command/config blindly and assume the blend has branches to blend — inspect the config and confirm branch data is in the artifact.
• Security scanner — is one available (semgrep, trivy, or another scanner the repo uses)? Note which, because their severity scales differ and must be normalized (Step 4). If none is present, default to Semgrep — free, open-source SAST, broad language coverage — and propose installing it in the Step 3 gate rather than treating security as unsupported. Only fall back to "security is a limitation" if the user declines the install or Semgrep can't run in the repo's environment — never fabricate a zero.
• Accessibility lint — is there a web frontend with an a11y linter (e.g. jsx-a11y for React, eslint-plugin-vuejs-accessibility for Vue, axe)? Pick the one matching the framework. If not a web UI, a11y is n/a.
• E2E matrix — is there a coverage-matrix file (e.g. from the e2e skills)? If not, e2e is n/a.
• Default branch & git history. Confirm the default branch (auto-detect origin/HEAD; if absent — e.g. no remote — fall back to the current branch and say so). Detect a shallow clone (git rev-parse --is-shallow-repository) and whether there's ≥7 days of history: both affect whether BASE..HEAD can resolve for the new-code slice (Step 4 fallback).

Step 3 — Decide the metric set (gated)

Present the achievable metric set, the proposed tiers, and the limitations before building anything:

GATE — confirm with the user:

• "I propose these tiers: … (e.g. web, api). OK, or group differently?"
• "I can surface these metrics: …. I cannot surface these and here's why: … (e.g. PHP/Go have no validated complexity analyzer yet; coverage is line-only here; no e2e matrix). Proceed with the achievable set?"
• If the repo has no security scanner: "No scanner is configured — I can install Semgrep (free, open-source SAST) to power the security metric, or leave security as a labelled limitation. Install Semgrep?"

Two honesty rules, applied later in the report:

• Defer freely, label loudly. A language whose complexity analyzer you can't validate, or a metric whose tool isn't available, is a deferred limitation — shown as n/a with a reason, never silently dropped and never faked.
• A smaller honest dashboard beats a complete dishonest one.

Step 4 — Build the analyzers (the actual work)

For each agreed metric, build the smallest thing that produces its intermediate artifact in the shape reference/metrics.md describes. Put the analyzers/scripts somewhere sensible in the repo (e.g. a quality/ or scripts/quality/ dir) and have each write its artifact to a known work location.

• Unit coverage (per tier) — run the repo's coverage tool, then a thin reader that emits the blended line+branch numbers. Multiple projects in a tier → union per line, never sum. Verify the denominator matches the product surface — it can be wrong in both directions. After the run, compare the artifact's file set against the product source files on disk (git ls-files of the measured surface):
  • Too small (narrowed): many tools count only files some test imports, so a source file with zero tests vanishes entirely (it doesn't read as 0% — it's simply absent), inflating the headline by exactly the files you most want flagged. Re-run with the include-everything option (vitest coverage.all — on by default but a repo can disable it or narrow coverage.include; coverage.py source-package config) so untested files count as uncovered, or label it "tested-files-only".
  • Too big (polluted): some defaults sweep in tests/scaffolding — e.g. Create React App's default collectCoverageFrom is all of src/, which pulls test files and fixtures into the denominator. Scope it at invocation to the product surface (--collectCoverageFrom='src/components/**' --collectCoverageFrom='src/examples/**', the right jest globs, coverage.py --source). Either way the override goes at invocation time, not by editing the repo's config. (This is distinct from the path cross-check: that catches mismatched paths, this catches a wrong denominator.) Confirm the reporter you intend to read is actually enabled. Reading a format the run didn't emit reads as "metric missing". A repo's default reporters often omit what the collect step needs — e.g. CRA/jest defaults to json/lcov/clover/text with no json-summary — so add it at invocation (--coverageReporters=json-summary --coverageReporters=lcov, --cov-report=xml, etc.) rather than assuming it's there. Same shape as the branch-data gotcha below. Branch-data fallback (two cases — see Step 2):
  • Config-off — override the invocation, don't edit the repo. When the tool supports branches but the committed config doesn't enable them, turn them on at invocation time via a flag, which takes precedence over the config file — pytest --cov-branch …, coverage run --branch, the jest/vitest branch options, coverlet /p:CoverletOutputFormat+branch. Do not edit the repo's .coveragerc / pyproject / jest config — that mutates tracked files and changes the repo's own coverage runs. And don't just call the repo's make coverage-style target if it bakes in line-only settings; run your own branch-enabled invocation instead. Because this diverges from the repo's documented coverage command, record it as a scope note (e.g. "branch coverage enabled by the quality pipeline via --cov-branch; the repo's own make coverage is line-only").
  • Tool can't emit branches — label it. If branches genuinely aren't available (Go -cover, branch-less lcov) and no override helps, report line coverage only and label it — e.g. value "61.0% (lines only — no branch data)" and list it as a limitation. Never blend against zero branches and present it as a line+branch figure.
• Maintainability — the one place you write real analysis code. Build a per-language cognitive-complexity AST analyzer emitting {file, line, name, cog, lines, ncloc} per function. Work from reference/complexity-example.md — it carries the full algorithm, a worked reference implementation to port from, per-language gotchas (Python/Go/PHP/C#), and a validation set of hand-checked functions. Port the approach to each target language, then prove your analyzer against the validation set (run it over reference/fixtures/ example-cases.js for a JS port, or author an equivalent hand-derived fixture for the target language) and confirm the cog/ncloc match before wiring the metric in. A language whose port you cannot make reproduce a hand-checked set → unsupported-for-maintainability (a limitation), never unvalidated numbers.
• Accessibility — run the a11y linter in JSON mode; reader computes violations per kLOC. Count only the a11y plugin's own rules. A dedicated lint pass is not clean of other rules: even with eslint --no-eslintrc and only the a11y plugin loaded, ESLint can still surface rules pulled in transitively (a shared parser/base config re-enabling react-hooks/*, react/*, …). Counting every message inflates the density. Filter to the plugin's namespace (ruleId starts with jsx-a11y/, vuejs-accessibility/, etc.) — never len(messages).
• Security — run the scanner over the tree; reader buckets findings into critical / high / medium / low. If the Step 3 gate approved installing Semgrep, install it (e.g. pip install semgrep or brew install semgrep) and run semgrep --config auto --json; map its ERROR/WARNING/INFO levels per the normalization below. Normalize severities first: the grade thresholds assume CVSS-like bands, but scanners differ — map each scanner's scale to those bands explicitly (e.g. semgrep ERROR→high, WARNING→medium, INFO→low; trivy CRITICAL/HIGH/MEDIUM/LOW direct; a 0–100 scale: ≥90 crit, ≥70 high, ≥40 medium, else low) and record the mapping you used. Without it, the crit/high counts aren't comparable to the thresholds.
• E2E — parse the coverage-matrix totals row.
• New-code slice — derive added lines from the BASE..HEAD diff (language-agnostic) and intersect with each metric's data (paths normalized per the Conventions). BASE-resolution fallback: BASE = the first-parent default-branch commit just before the 7-day window. If that can't resolve — shallow clone, <7 days of history, or a branch that doesn't descend from the default — first try git fetch --unshallow / deepen; if it still can't, skip the new-code lens and show it as n/a (new-code window unavailable: <reason>) rather than diffing against the wrong base. Squash/rebase workflows: confirm the chosen BASE actually predates the window's merges before trusting the diff. A freshly-created repo is the common case, not an edge case: when the whole repo is younger than the 7-day window there is no BASE, so the new-code lens is n/a for every metric on the first run(s) and self-heals once ≥7 days of history exist — say so plainly rather than treating it as a failure. The mirror case — a dormant repo with no commits in the window — also yields an empty slice: here BASE resolves fine but to HEAD itself (the last commit predates the window), so BASE..HEAD is empty. Detect BASE == HEAD (or an empty diff) and report the same n/a (no commits in the 7-day window) rather than emitting a confusing BASE==HEAD window — it likewise self-heals on the next commit.

Step 5 — Install the generator template + write the collect step

Do not write the grading/rotation/trend/History logic yourself — it ships. Copy the template generator into the repo (e.g. cp -R <this-skill>/template <repo>/quality/) and read template/README.md for the metrics.json contract. Then:

• Write the collect step (the per-repo glue): a small script that reads the Step 4 artifacts, computes each metric's value / display / backing, and writes metrics.json in the shape template/README.md documents. Use a stable key per metric/tier (for trends), the right kind (selects the grade scale), and null for any new-code lens that's unavailable.
• Adapt template/static-sections.md — fill in the regenerate command and record the repo's limitations (line-only coverage, deferred languages, missing scanner) in the "Metrics tracked" notes.
• Wire one regenerate command that runs analyzers → collect → generate.py, invoking the generator through the bootstrapped interpreter (Step 1), not a bare python3 on PATH: .venv-quality/bin/python quality/generate.py --input metrics.json --out QUALITY_REPORT.md --static quality/static-sections.md.
• Gitignore the intermediate artifacts and the generator's *.state.json sidecar (and the dashboard itself if the user chose so).

The generator is stack-invariant and already validated — your only correctness surface here is the collect step producing the contract correctly (and the analyzers behind it).

Step 6 — Verify

Generate the report once, end to end. Sanity-check every wired metric: numbers are non-empty and in range, grades/RAG line up with the thresholds, the backing numbers reconcile with the headline values, and limitations are shown for anything not wired. Fix wiring until the dashboard is internally consistent. A clean first generation is the success gate.

Step 7 — Hand off

• Report what was wired, what's a limitation and why, where the dashboard lives, and the single command to regenerate it weekly.
• Note that this skill is run-once; weekly regeneration is the quality-report-update skill (which drives that command, validates the result, and narrates week-over-week changes).

Done-when checklist

• Runnable toolchain bootstrapped (Step 1): a dedicated environment with a version-correct interpreter and the project's test/dev deps installed; the suite runs green (or coverage is a labelled limitation), and the regenerate command pins that environment.
• Repo inventoried; tiers and achievable metric set agreed with the user (Step 3 gate).
• An analyzer/reader exists per agreed metric, each emitting its artifact in the documented shape; the maintainability analyzer reproduces the validation set (not just a spot check) for every language wired.
• Path cross-check passed: a known-changed file's path is byte-identical across the diff and every artifact (else the new-code lens is silently empty).
• Degraded metrics labelled: line-only coverage, skipped new-code lens, and the security severity mapping are each captioned in the report and listed as limitations.
• Generator template installed (not rewritten) and a collect step produces a valid metrics.json; the generated dashboard shows whole-repo + new-code lenses with grades/RAG/ trends, rotation, History, and backing numbers.
• Limitations for any un-wired metric are shown in the report (n/a + reason), not hidden.
• One regenerate command exists and is documented.
• Report generates cleanly and is internally consistent (Step 6).
• Work artifacts gitignored; dashboard placed/gitignored per the user's choice.