| name | sync-pi-evals |
| description | Vendor and refresh the unpublished pi-evals harness from earendil-works/pi into tests/evals/vendor, and keep pi-* devDependencies compatible with it. Use when asked to set up, refresh, or update the evals harness or vendored pi-evals code in pi-processes. |
Syncing vendored pi-evals
@earendil-works/pi-evals (packages/evals in earendil-works/pi) is "private": true and not published to npm.
Until it ships publicly, the harness source is copied into tests/evals/vendor/pi-evals/ (gitignored) and patched.
Why not a pnpm git dependency: installing github:earendil-works/pi#<ref>&path:packages/evals does fetch the code,
but the harness must be patched, and pnpm patch on that dependency fails with
ERR_PNPM_GIT_DEP_PREPARE_NOT_ALLOWED — it tries to prepare the whole pi-monorepo root and demands an
allowBuilds entry for it. Revisit if pi publishes the package or upstreams extension injection.
Do not run the eval suite as part of syncing: evals make real, paid model calls. pnpm typecheck:evals is the free
check that a vendor drop is sound.
Run the script
./.agents/skills/sync-pi-evals/scripts/sync-pi-evals.sh
./.agents/skills/sync-pi-evals/scripts/sync-pi-evals.sh --ref v0.85.0
It clones the pi repo at the ref, copies pi-harness.ts and vitest-evals/, records the upstream package.json
as upstream-package.json, applies every patch in tests/evals/patches/, and writes tests/evals/vendor/SYNC.json
with the ref, resolved SHA, and date. Re-running is safe: rsync --delete restores pristine files before patching.
The script only mechanises the copy. The two decisions below are yours.
Decision 1: which ref
Read tests/evals/vendor/SYNC.json first to see what is currently vendored.
Default rule: use the git tag matching the @earendil-works/pi-coding-agent version pinned in this repo's
package.json devDependencies (devDependency 0.84.1 → tag v0.84.1). The harness calls
AgentSession / createAgentSessionServices APIs, so it must match the coding-agent version we build against.
Exception: if the harness change you need landed on main after the latest tag, pin the exact SHA rather than a
branch, and check whether that commit's packages/coding-agent API is compatible with the version installed here.
If it is not, stop and ask the user before syncing — see decision 2.
Never sync from a floating branch. The script accepts one, but the result is not reproducible.
Decision 2: dependency reconciliation — always ask before bumping
After syncing, compare tests/evals/vendor/pi-evals/upstream-package.json devDependencies against this repo's
package.json:
@earendil-works/pi-ai@earendil-works/pi-coding-agentvitestvitest-evals (already a devDependency here, pinned exact)
Then run pnpm typecheck:evals. It compiles the vendored and patched harness against this repo's installed
versions and is the real compatibility signal.
If it passes and versions match: done.
If versions differ or the typecheck fails:
- Report the mismatch explicitly: which package, the version pinned here, the version the vendored code expects,
and whether that version is published (
npm view @earendil-works/pi-coding-agent version).
- If it is published: propose bumping
@earendil-works/pi-ai, @earendil-works/pi-coding-agent,
@earendil-works/pi-tui, and vitest (plus matching peerDependencies ranges), and ask for confirmation
before running pnpm add -D <pkg>@<version>. This repo uses pnpm. Never bump silently.
- If it is not published: say the vendored harness is ahead of what is installable, and ask whether to re-sync
from the last tag (losing the new harness feature) or wait for the next pi release.
The patches
tests/evals/patches/0001-harness-additional-resource-paths.patch makes four changes, all required:
- adds
additionalExtensionPaths / additionalSkillPaths options,
- always forwards
resourceLoaderOptions (upstream only does when transformSystemPrompt is set),
- relaxes the zero-extension guard to expect exactly the injected extensions,
- widens
resolveModelSelection's environment parameter to Record<string, string | undefined>, because
@types/node 25 here makes ProcessEnv incompatible with upstream's inline literal type.
Upstream's harness deliberately refuses to load extensions, so without 1–3 there is no way to get the processes
extension into an eval session.
If the script reports a failed patch, upstream moved. Re-derive the change by hand against the newly vendored
pi-harness.ts, regenerate with diff -u, and commit the updated patch file. Do not skip it.
After syncing
pnpm typecheck:evals must pass. pnpm typecheck excludes tests/evals on purpose, so it stays green without a
vendor drop.- Verify with
git status that nothing under tests/evals/vendor/ is staged; it is gitignored.
- CI runs this same script in the gated
evals job, so a working sync locally means a working sync there.
For writing, running, and verifying the evals themselves, see the writing-pi-evals skill and
tests/evals/README.md.
Re-sync when
@earendil-works/pi-coding-agent is bumped here.- An eval needs a harness capability that only exists in a newer pi commit.
SYNC.json predates a pi release this repo has adopted.
