| name | corpus-maintenance |
| description | Use for authored-source correspondence corpus maintenance: verify the vendored CIVIL/EVIL snapshot for drift against upstream, re-harvest or re-pin it, run the offline benchmark, and decide when to advance the pin. |
Corpus maintenance
Use this skill when the user asks to maintain, refresh, drift-check, re-pin, or
benchmark the authored-source correspondence corpus — the vendored source
oracle behind the decompiler's offline benchmark. Produce a short status report:
whether the pinned snapshot still corresponds to upstream, whether it needs a
re-harvest or a pin bump, and what the benchmark says about the current
decompiler.
Run from the repository root. The corpus tooling is the DecompilerHarness
(tools/DecompilerHarness), a dev tool, not the shipped dotnet-inspect
product; run it with dotnet run --project tools/DecompilerHarness -c Release --.
What the corpus is
The corpus is a vendored JSONL where each row is a real method identity plus a
checksum-verified authored member body captured through SourceLink at harvest
time. It lets the benchmark grade source correspondence fully offline, so
SourceUnavailable becomes a drift signal rather than an expected network
outcome. It has two halves:
- CIVIL (Curated Index of Varied IL) —
civil/corpus.jsonl, harvested from
the 14 pinned assemblies in eng/prepare-decompiler-corpus.sh. This is the
stable regression target that grows in lock step with the fixed real-world
corpus.
- EVIL (Edge-case Verification of IL Legibility) —
evil/corpus.jsonl, an
adversarial stress set of the most difficult real methods drawn from a much
broader pool (eng/prepare-evil-corpus.sh), difficulty-ranked off the shared
IL substrate.
Both live on the vendor/authored-source-corpus orphan branch so the harvested
third-party source snapshots never enter main's history. Restore them into a
git worktree first:
bash eng/restore-authored-source-corpus.sh
That materializes external/authored-source-corpus. Because it is a worktree on
the orphan branch, edits made there (a re-harvest) commit directly to
vendor/authored-source-corpus.
The pin, and why it is stable
Every corpus assembly is a pinned published package version, so the only
thing that varies across a decompiler change is the tool, never the input. The
dotnet-inspect self-assemblies come from the published dotnet-inspect.any
package (SELF_VERSION in eng/prepare-decompiler-corpus.sh), not the local
artifacts/bin build — that removes corpus drift (#1404) and breaks the
circularity where a decompiler change would rebuild both the tool and its own
corpus at once.
Two pin axes matter:
- Package version —
SELF_VERSION (and the third-party versions in
prepare-decompiler-corpus.sh).
- Source commit — each row's SourceLink commit, which is where its authored
body was captured.
Pinning buys attribution and reproducibility; "update to latest" is a
deliberate, reviewed re-pin (a re-harvest), because pins rot as upstream moves.
Drift verification is the bridge that turns "should we re-pin?" into an
evidence-driven decision.
Verify drift (start here)
Drift verification re-acquires each vendored row's authored source today and
compares it byte-for-byte (newline-normalized) against the stored snapshot. It
does not run the decompiler; it audits the corpus, not the tool. Prepare the
pinned assemblies, then verify CIVIL, resolving dotnet-inspect's own rows from
local git via --repo:
bash eng/prepare-decompiler-corpus.sh /tmp/corpus-assemblies.txt
dotnet run --project tools/DecompilerHarness -c Release -- \
--verify-authored-corpus external/authored-source-corpus/civil/corpus.jsonl \
--repo "$(git rev-parse --show-toplevel)" \
$(cat /tmp/corpus-assemblies.txt)
Each row lands in one of three states:
- Verified — the re-acquired body matches the vendored snapshot.
- Drifted — the body differs; the upstream source, the harvest slice, or the
stored row itself has changed. The report names the row with a short
line-count/first-diff summary.
- Unavailable — the source could not be re-acquired or sliced (offline with
no
--repo, commit gone, checksum mismatch, or an extraction regression). The
row is surfaced, never silently dropped.
It is report-only by default (exit 0). The run's own integrity still governs the
exit code with a named blocker: any corpus row whose assembly was not supplied
(unmatchedRows > 0) or a run that evaluated no rows fails, so an empty or
partially-unmatched run is never a success. Add --fail-on-drift to make it a
fail-closed gate — the run then exits nonzero unless every evaluated row is
Verified (any Drifted or Unavailable row fails):
dotnet run --project tools/DecompilerHarness -c Release -- \
--verify-authored-corpus external/authored-source-corpus/civil/corpus.jsonl \
--repo "$(git rev-parse --show-toplevel)" --fail-on-drift \
$(cat /tmp/corpus-assemblies.txt)
Always supply the same pinned assemblies the corpus was harvested from; a
missing assembly is a blocker, not a pass.
Interpret and decide
- All Verified — the pinned snapshot still corresponds to upstream. No
action; the corpus is a faithful oracle.
- Drifted rows — upstream (or the slice) moved off the pinned commit. This is
expected as repositories evolve and is not a bug — the fix is a deliberate
re-harvest / re-pin (below), not a code change. Confirm the drift is upstream
churn and not a harvester regression before advancing.
- Unavailable rows — usually an outage, a missing PDB, or a bad/incomplete
--repo checkout, not corpus rot. Re-run with a complete --repo and network
before concluding the source is gone. Persistent Unavailable at a known-good
commit points at an extraction regression worth an issue.
Distinguish "the corpus drifted from upstream" (re-pin) from "the harness can no
longer resolve known-good source" (investigate the harness).
Re-harvest and re-pin
Advancing the snapshot is a deliberate, reviewed act. To re-harvest CIVIL from
the current pins straight into the vendored worktree:
bash eng/prepare-decompiler-corpus.sh /tmp/corpus-assemblies.txt
dotnet run --project tools/DecompilerHarness -c Release -- \
--harvest-authored-corpus external/authored-source-corpus/civil/corpus.jsonl \
--harvest-target 12000 \
--repo "$(git rev-parse --show-toplevel)" \
$(cat /tmp/corpus-assemblies.txt)
--repo (repeatable) reads each target's authored source from a local git clone,
arbitrated by the PDB checksum, falling back to the network on any mismatch or
miss. Pointing it at this checkout resolves the dotnet-inspect self-corpus rows
with no GitHub round-trip; it also unlocks private, large, or offline source
repositories. Third-party rows are checksum-arbitrated and fall back to the
network.
To update to latest (advance the pin, not just re-capture the same commit),
bump SELF_VERSION in eng/prepare-decompiler-corpus.sh (and any third-party
version), re-emit the real-world baseline, then re-harvest as above. Re-harvest
EVIL from its broad pool the same way:
bash eng/prepare-evil-corpus.sh /tmp/evil-pool
dotnet run --project tools/DecompilerHarness -c Release -- \
--harvest-evil-corpus external/authored-source-corpus/evil/corpus.jsonl \
--harvest-target 12000 \
$(cat /tmp/evil-pool/assemblies.txt)
Commit the regenerated corpus.jsonl in the external/authored-source-corpus
worktree (it commits to the orphan branch), and land the pin bump on main
through a normal reviewed PR. Keep the two in step so the benchmark and the
fixed real-world corpus stay aligned.
Benchmark
The benchmark feeds the vendored authored bodies into the same
source-correspondence oracle the on-demand census uses and emits a taste card. It
validates the corpus and the current decompiler together; run it after a
re-harvest or when asked for a decompiler correspondence read:
dotnet run --project tools/DecompilerHarness -c Release -- \
--benchmark-authored-corpus external/authored-source-corpus/civil/corpus.jsonl \
$(cat /tmp/corpus-assemblies.txt)
Add --json for structured output. Each target is Correct, one of four
valid-but-different taste buckets, Invalid, or a diagnostic bucket. The run
exits nonzero on any Invalid, Drift, or Unsupported, and on a dishonest run
(unmatched rows or zero targets). A Drift bucket here means the corpus identity
no longer resolves in the pinned assembly — treat it like a Drifted verify row.
Guardrails
- Keep the benchmark hermetic: it must stay fully offline over the vendored
bodies. Do not add live network resolution to the benchmark path.
- Do not delete the
vendor/authored-source-corpus orphan branch or migrate its
bodies onto main. The vendored snapshot is the reproducible oracle.
- Build the harness with the solution graph:
dotnet build dotnet-inspect.slnx -c Release.
- Verify and benchmark over the pinned assemblies, never a fresh local build of
the tool (that reintroduces #1404 circularity).
Report shape
Keep the report short:
Corpus maintenance: <verified | drifted | re-pinned | benchmarked>
- Corpus: CIVIL <verified/drifted/unavailable counts>; EVIL <if checked>.
- Drift cause: <none | upstream churn (re-pin) | outage/--repo (retry) | harness regression (issue)>.
- Action: <none | re-harvest | bump SELF_VERSION + re-harvest | file issue>.
- Benchmark (if run): <correct / taste-bucket / invalid / drift counts>, exit <code>.
