| name | vmtests |
| description | Run the ethereum/legacytests VMTests conformance suite against the evaluator (stepF/run), interpret the pass/fail/incon/crash results, and refresh the committed CI baseline. Use when asked to run vmtests, check conformance, investigate a failing or crashing test, or update the baseline after an evaluator fix. |
vmtests
Run the legacy ethereum VMTests suite against the executable evaluator. This
is the conformance suite matching the project scope: single-frame EVM, no
inter-contract calls, no transaction processing. Read AGENTS.md for project
scope and VMTESTS.md for the full harness writeup and known gaps.
Get the corpus (one-time)
Pin to the exact CORPUS_REV that CI uses (in .github/workflows/ci.yml) —
the committed .github/vmtests-baseline.txt was generated against that revision,
so a different corpus HEAD will make your counts and any refreshed baseline
disagree with CI:
REV=$(grep -m1 'CORPUS_REV:' .github/workflows/ci.yml | awk '{print $2}')
git clone https://github.com/ethereum/legacytests
git -C legacytests checkout "$REV"
The suite lives in legacytests/Constantinople/VMTests. (CI does a sparse
blobless fetch of just that dir at the same rev.)
Run
lake build vmtests
./.lake/build/bin/vmtests <path>/legacytests/Constantinople/VMTests
./.lake/build/bin/vmtests --file <path>/.../add0.json
The full suite runs tests as in-process Lean Tasks across jobs workers
(-j N / VMTESTS_JOBS, default 8) — there is no subprocess isolation (a
deliberate ~7× speedup over the old subprocess-per-file design). A worker Task
that throws is recorded as a crash, but a hard panic aborts the whole run.
Use --file to run a single test in its own process when you need to isolate
one that panics.
Reading the summary
Line looks like: pass=… fail=… incon=… crash=…
- pass — every test runs with its real
exec.gas budget; on a test
with a post block, the remaining gas is compared too.
- fail — storage / return-data / balance / nonce / gas mismatch
against the test's
post block.
- incon — out-of-gas / fuel-exhausted cases the harness can't
reproduce (the loop legitimately exceeds the 2M-step fuel cap, or the
test relies on enforcement we don't yet model — e.g. the 1024-stack
cap).
- crash — a worker
Task threw and the parent recorded it (a hard
panic instead aborts the whole run). Should be 0; a new crash is a
real regression — isolate it with --file on that test.
Refresh the baseline (after an evaluator fix turns fails into passes)
CI's VMTests job is non-gating but compares against
.github/vmtests-baseline.txt. When you improve results, move the floor:
./.lake/build/bin/vmtests <path>/legacytests/Constantinople/VMTests > raw.txt
.github/scripts/vmtests_summary.sh raw.txt > .github/vmtests-baseline.txt
If you regenerated against a newer corpus, bump CORPUS_REV in
.github/workflows/ci.yml in the same commit — the baseline and pinned corpus
revision must always move together. Never bump one without the other.
Reporting
Restate the summary counts in text (the extractor can't see tool output). Call
out any regression vs baseline (a previously-passing test now FAIL/CRASH) — that
is the thing the CI check exists to catch.
