| name | bench-setup |
| metadata | {"internal":true} |
| description | Set up the nu-validator benchmark environment — initialise the validator submodule, ensure Docker is running, populate the gitignored raw snapshots via yarn bench:update. Also covers Docker / nu-runner troubleshooting. Use when bench commands fail with "no snapshots found", "Docker daemon not reachable", port 28888 conflicts, or healthcheck timeouts; or when refreshing the bench after a long break. Trigger keywords: bench setup, bench Docker, validator submodule, no snapshots found, Docker daemon not reachable, healthcheck times out, port 28888, yarn bench:update first time, bench install, fresh clone bench.
|
nu-validator Bench Setup Skill
Get the bench runnable on this machine, or recover when it isn't.
Once this skill's checks pass, drop into bench-triage for actual
coverage work.
Prerequisites
Fresh-clone bootstrap
Raw snapshot trees are gitignored. Populate them once:
git submodule update --init tests/external/validator
yarn bench:update
This pulls the nu-validator Docker image
(ghcr.io/validator/validator), runs both tools across the 5442
fixtures, and writes snapshots/{nu-validator,markuplint}/**,
snapshots/diff/*, snapshots/meta.json, and the generated spec.
Takes ~2 minutes at default concurrency.
Routine refresh after dev sync
git submodule update --remote tests/external/validator
yarn bench:update
Review snapshots/diff/summary.md and snapshots/diff/*.json in
the PR. The raw snapshot trees stay local-only.
Markuplint rule change (skip the Docker leg)
yarn bench:update:ml
Only re-runs the markuplint side; reuses existing nu snapshots.
Commands reference
| Command | What it does |
|---|
yarn bench:update | Full sweep: Docker → nu + ml snapshots → compare → generate-spec → report. |
yarn bench:update:ml | Same chain, skips Docker / nu. Use after markuplint rule edits. |
yarn bench:compare | Re-run compare against existing snapshots. |
yarn bench:generate-spec | Rewrite spec/nu-validator.spec.ts from coverage.json. |
yarn bench:report | Rewrite snapshots/diff/summary.md. |
yarn bench:verify | vitest run --config vitest.nu-validator.config.ts. |
yarn bench:update flags:
| Flag | Effect |
|---|
--target nu|markuplint|all | Restrict which snapshot tree is rewritten. |
--filter '<glob>' | Subset of validator/tests paths to visit. |
--concurrency <n> | Default os.availableParallelism() - 1 (capped at 8 for nu). Use 1 for deterministic nu output. |
--image-tag <tag|digest> | Override the nu-validator image. |
--dry-run | List files and exit. |
--skip-refresh | Stop after the snapshot phase. |
Troubleshooting
| Symptom | Likely cause and fix |
|---|
Docker daemon not reachable | Start Docker Desktop / the daemon. The bench does not install or start Docker. |
docker pull failed | docker login ghcr.io if behind authenticated pulls, otherwise check connectivity. |
| Healthcheck times out | nu-validator takes 10–30 s to come up cold. If > 60 s, docker logs ml-nu-validator. |
Port 28888 already allocated | Stop the conflicting container, or edit DEFAULT_PORT in bench/docker.ts. |
yarn bench:verify / yarn bench:compare "no snapshots found" | Raw trees are gitignored on a fresh clone. Run yarn bench:update. |
| One or two snapshot files flip between runs | Expected nu parallelism flicker. Rerun with --concurrency 1 when reproducibility matters. |
| Unexpected diffs after a submodule bump | Check meta.json for image-digest / markuplint-version changes. |
