菜单
Use when investigating why the Index Network opportunity evaluator scores matches wrong, or when asked to explain, diagnose, or report on matching-eval results — turning a matching-eval run report (packages/protocol/eval/matching/runs/*.json) into a root-cause and fix report.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Finish a pull request end-to-end from the canonical/root session: investigate PR health (base freshness, checks, review threads, version bumps, local builds), hand off any needed worktree changes to the worktree session as a fix prompt — never mutating the worktree from this session — merge the PR after explicit confirmation, verify post-merge GitHub/Railway deployment health, and close or update related GitHub and Linear issues. Use when the user says a PR is ready to finish, ship, merge, or close out.
Run the packages/protocol eval harnesses correctly — the live LLM harnesses (bun run eval:matching/hyde/premise/profile/opportunity/clarification, which need OPENROUTER_API_KEY auto-loaded from root .env.test) versus the provider-free CI gate (bun run eval:verify, which strips credentials and never calls a model). Use when asked to "run the evals", verify eval suites, add a new eval harness, update a baseline, or when eval:verify fails on an unlisted directory or a baseline-coverage spec after adding a corpus case.
Ship a change to the Edge-City agentvillage submodule end-to-end and bump the index monorepo's submodule pointer, without initializing the submodule in a worktree. Use after merging a PR in Edge-City/agentvillage (or when the pointer at packages/edge-city/agentvillage is stale) — covers the branch-in-submodule → PR-to-Edge-City → pointer-bump-PR-to-dev flow, the git update-index --cacheinfo 160000 plumbing trick for committing a gitlink from a worktree with an empty submodule dir, and restoring the root submodule to a clean detached checkout.
Ship and flip feature flags consistently across ALL env surfaces in the index monorepo — .env.example (committed, commented docs), root .env.development (gitignored local mirror of Railway dev), Railway dev service variables (Railway MCP), and startup.env.ts registration. Use when adding a new env-gated feature flag, enabling/disabling a flag on dev, or when a flag behaves differently locally vs on Railway. Covers the ship-dark→flip order, railway_set_variables gotchas (snake_case ids, map-shaped variables, auto-redeploy), and why the root-dev-guard warning on .env.development edits is safe to ignore.
Create and operate git worktrees in the index monorepo using the project's `bun run worktree:*` helpers instead of raw git. Use whenever you need an isolated branch checkout to make changes (the canonical root must stay on dev and is read-only for the assistant, enforced by the root-dev-guard extension), or when a worktree is missing env files, node_modules, or git hooks, or when a tool/bash call is blocked for touching the canonical root, or when `git commit`/`git rebase` fails with "gpg: signing failed: Inappropriate ioctl for device" in a non-interactive shell. Covers the dashed-folder / slashed-branch naming convention, why `worktree:setup` is mandatory after creating a worktree, and how to disable GPG signing worktree-locally without touching the user's repo-wide signing config.
Open a dated release branch from dev and create a GitHub pull request into main with a generated changelog from git log and git diff. Use when preparing a dev-to-main release PR or when the user asks to cut/open a release PR.
| name | diagnosing-matching-eval |
| description | Use when investigating why the Index Network opportunity evaluator scores matches wrong, or when asked to explain, diagnose, or report on matching-eval results — turning a matching-eval run report (packages/protocol/eval/matching/runs/*.json) into a root-cause and fix report. |
The matching eval harness scores the opportunity evaluator (invokeEntityBundle) against a
golden corpus. A run report (--report → eval/matching/runs/<ts>.json) records, per case
per run, each candidate's actual score, role, and the evaluator's own verbatim reasoning.
This skill turns that artifact into a report that explains why the evaluator fails and what to
change.
Core principle: The reasoning string is the evidence. Every claim about why a case fails
cites the evaluator's own words; every proposed fix cites a verified file:line. No guessing,
no memory.
packages/protocol: bun run eval:matching -- --report (add --runs 7 to cut noise).
Writes eval/matching/runs/<timestamp>.json (gitignored). Analyze the newest one.baselines/matching.baseline.json — reasoning is stripped from it. It has
scores only. You need a --report artifact for the "why".Read the reasoning, not just the scores. For each failing or flaky case, read every run's
candidates[].reasoning alongside the assertion detail lines. The reasoning is the model
explaining itself — that is your evidence.
Classify the cause before prescribing a fix. For each failure, decide which it is and say so:
minScore (matching.runner.ts:26, currently 30) and the
scorer counts a surfaced candidate as matched (matched = score > 0, matching.scorer.ts:30).
So if a case sets match:false but a scoreBand whose max is ≥ minScore, the band and the
match expectation cannot both pass — that is a corpus bug; fix the case, not the prompt.Separate signal from noise. Three runs is a small sample. An all-fail (0/N) or all-pass
(N/N) case is conclusive as-is. But a mixed result is flaky — before drawing any conclusion
from it, re-run just that rule to see where it settles: bun run eval:matching -- --rule <rule> --runs 7 --report. Flaky (mixed passes across runs) is not a regression (the harness's
own baseline diff). Never call sample-to-sample variance a regression.
Locate the responsible clause precisely — and in the RIGHT prompt. The evaluator has two
system prompts in src/opportunity/opportunity.evaluator.ts:
systemPrompt (~line 21) — strict, single-pair, ≥70 threshold.entityBundleSystemPrompt (~line 77) — permissive, batch, ≥30 threshold.The eval calls invokeEntityBundle, which uses entityBundleSystemPrompt only. A fix
applied to systemPrompt changes nothing in the eval. grep the rule, read it, cite the exact
file:line, and confirm it lives in entityBundleSystemPrompt. Check whether the same rule is
duplicated elsewhere. Attribute behavior to the right component: surfacing threshold = runner
minScore; matched = scorer (score > 0); scoring/role rules = the prompt.
Write the report, then verify every quoted clause and line number by opening the file — never paraphrase a prompt rule from memory.
reasoning) → cause classification (weakness vs miscalibration, with evidence) → fix with a
verified file:line in the correct prompt.--runs re-run
confirmed the failure is systematic rather than flaky.| Mistake | Fix |
|---|---|
| Analyzing the baseline JSON (no reasoning) | Generate a --report artifact first |
| Quoting a prompt rule from memory | Open the file; cite exact file:line; confirm the text |
| Citing a line range (e.g. "21–153") | A clause has ONE location — find it; check for duplicates |
Fixing systemPrompt | The eval uses entityBundleSystemPrompt — confirm which prompt |
| Defaulting to "fix the prompt" | First rule out a corpus / scoreBand-vs-minScore inconsistency |
| Calling 3-run variance a "regression" | Re-run with more iterations; flaky ≠ regression |
Attributing minScore to the scorer | minScore is the runner; matched = score>0 is the scorer |
reasoning.