メニュー
Run the RAG evaluation harness and report metric deltas — use after touching data/rag, the Evidence Agent, or Citation Guard, or whenever the user asks to check/update the eval numbers.
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/Juanpacol/SEPHIROTH --skill evalコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Add a new clinical guideline document to the RAG corpus with matching golden/paraphrase eval cases and tests — use when the user wants to expand data/rag/SEED_GUIDELINES with a new topic.
Run the full local gate (lint, tests+coverage, eval regression check, README/baseline consistency, clean tree) before pushing — use before opening a PR or when the user asks "is this ready to push?".
Launch SEPHIROTH locally and smoke-test the core flow (health, auth, one consultation) — this repo's project verify skill, with this machine's port/venv quirks baked in.
SKILL.md を表示中
| name | eval |
| description | Run the RAG evaluation harness and report metric deltas — use after touching data/rag, the Evidence Agent, or Citation Guard, or whenever the user asks to check/update the eval numbers. |
Two modes; pick based on whether a Gemini API key is available and whether the user asked for a quick check or a full refresh.
--mode ci, always safe, no API key needed)cd clinical-ai-copilot
PYTHONPATH=.:platform .venv/bin/python -m intelligence.evaluation.run --mode ci
Read the printed table. If it says Overall: FAIL, report which metric(s) dropped below threshold and by how much — check git diff on data/rag/__init__.py, src/sephiroth/verification/citation_guard.py, or intelligence/agents/__init__.py (EvidenceAgent prompt) for what likely caused it.
If it warns results/latest.json is missing or stale, the dataset or transcripts changed since the last full run — offer to run --mode full --record (below) if GEMINI_API_KEY is set, or tell the user it needs a refresh before merging.
--mode full --record, needs GEMINI_API_KEY — burns free-tier quota)Check the key is set first:
grep -q '^GEMINI_API_KEY=.' .env && echo "key present" || echo "GEMINI_API_KEY missing"
If present, run the real Evidence Agent against every golden case and refresh the committed baseline:
PYTHONPATH=.:platform .venv/bin/python -m intelligence.evaluation.run --mode full --record --skip-pubmed
(--skip-pubmed keeps transcripts reproducible — PubMed responses vary run to run.)
Omit --skip-pubmed only if the user explicitly wants live PubMed citations exercised too.
Add --model <name> to override the model (defaults to settings.gemini_model, i.e. gemini-flash-latest). Useful if the committed baseline was generated with a different model — check intelligence/evaluation/results/latest.json's run.model field to see what it was actually generated with, and flag it in your summary if it's a stand-in rather than the production model.
results/latest.json (git diff intelligence/evaluation/results/latest.json before staging) and summarize what moved and why.README.md's ## Evaluation section to match the new numbers exactly — a stale README table is worse than no table.results/latest.json and intelligence/evaluation/transcripts/*.json must be committed together (they're hash-linked — CI checks the pair for staleness). git add intelligence/evaluation/.