| name | check-benchmark-projects-for-info-leaks |
| description | Scan benchmark premade projects and manifest.yaml's description-for-refactoring-agent (and other files shown as-is to evaluated agents) for comments, metadata, or descriptions that leak this repo's benchmark/evaluation intent, including explicit spoilers of the upcoming update-prompt. Use when asked to review premades, "bad"/"good" premade variants, or check for info leaks, either after adding/editing a premade or as a periodic sweep. Read $ARGUMENTS for an optional scope (a benchmark name or path); default to all benchmarks. |
check-benchmark-projects-for-info-leaks
Why this matters
This benchmark (see repo-root README.md) scores an agent by having other agents
extend or refactor its code. Anything an evaluated agent actually reads must look like a
real codebase — if it tips off the agent that it's inside a benchmark, or that a design
was deliberately engineered to be good/bad/hard-to-extend, the agent could game the
scoring instead of reacting the way it would to a real codebase.
In scope: premades/{good,bad}/<variant> (plus any premade-prerefactor dirs that are
still git-tracked — check git ls-files <dir> | wc -l first, most are stale untracked
leftovers), and manifest.yaml's checkpoints.initial.description-for-refactoring-agent
(harness/lib/benchmark-manifest.ts's getDescriptionForRefactoringAgent) — this one
manifest field is passed to the refactoring agent as real input. Everything else in
manifest.yaml (summary/what/why/similar-examples) is harness-internal, never
shown to an agent, and fine to be explicit — don't flag it.
What counts as a leak
- Naming the benchmark/harness/eval apparatus itself: "benchmark", "harness", "premade",
"checkpoint", "feature agent", "refactoring agent", "update prompt".
- Saying a design is deliberately good/bad for testing purposes, or referencing sibling
variants by name or good/bad category (e.g. "the bad designs tend to...", "kept as a
separate site to trip up refactoring").
- Narrating the exact upcoming change a bad variant resists (e.g. a comment inside code
that hardcodes sort direction saying "adding a
reverse field would require touching
every call site here") — previews the test itself.
- In
description-for-refactoring-agent: naming a capability/entity/verb that appears in
update-prompt.txt but not initial-prompt.txt. Compare all three files. A generic,
initial-scope-only description is fine even if broad ("...among other things...");
naming the update's actual feature (e.g. "supports blocking users" when blocking is
the update) is the violation.
Ordinary comments about domain logic or this file's own design rationale are NOT leaks
("deliberately thin controller", "evaluator is ignorant of storage") — don't over-flag
plausible real-codebase commentary.
Where to look
Per variant: comments in every source file (.ts/.tsx/.js/.jsx/.mjs/.html) including
scripts/ and public/, not just src/ (build scripts and static HTML get missed);
package.json name/description/keywords (matching the variant's own dir name is
fine; referencing "bad"/"benchmark"/"pre-refactor" is not); any README.md; .gitignore
comments; stale gitignored build output like dist/ (rebuild via npm run build if it
lags a cleaned-up src/, since an old leak can survive in compiled output).
Sanity-net grep, then read each hit in context:
grep -rniE "harness|premade|benchmark|the bad|the good|checkpoint|feature.?agent|refactor(ing)?.?agent|evaluat" \
<scope>/*/premades/*/*/{src,scripts,public}/* <scope>/*/premades/*/*/package.json 2>/dev/null
Workflow
- Scope from
$ARGUMENTS: a specific benchmark, or all of
benchmarks/*/*/premades/{good,bad}/* + tracked premade-prerefactor dirs +
archived-benchmarks/*/premade*.
- Fan out one subagent per benchmark (Agent tool) rather than reading serially — give
each the "why this matters"/"what counts" sections above, its variant directories,
and its
manifest.yaml + initial-prompt.txt + update-prompt.txt. Ask for exact
file/line + offending text, split "clear violations" vs "borderline".
- Report findings before editing, unless already told to fix as you go.
- Fix by rewording out the leaking clause, keeping any legitimate rationale alongside it.
- Re-run the grep and check
git status/git diff --stat to confirm scoped edits.
