菜单
Run the RAG evaluation harness and report metric deltas — use after touching data/rag, the Evidence Agent, or Citation Guard, or whenever the user asks to check/update the eval numbers.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/Juanpacol/SEPHIROTH --skill eval命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
Add a new clinical guideline document to the RAG corpus with matching golden/paraphrase eval cases and tests — use when the user wants to expand data/rag/SEED_GUIDELINES with a new topic.
Run the full local gate (lint, tests+coverage, eval regression check, README/baseline consistency, clean tree) before pushing — use before opening a PR or when the user asks "is this ready to push?".
Launch SEPHIROTH locally and smoke-test the core flow (health, auth, one consultation) — this repo's project verify skill, with this machine's port/venv quirks baked in.
正在显示 SKILL.md
| name | eval |
| description | Run the RAG evaluation harness and report metric deltas — use after touching data/rag, the Evidence Agent, or Citation Guard, or whenever the user asks to check/update the eval numbers. |
Two modes; pick based on whether a Gemini API key is available and whether the user asked for a quick check or a full refresh.
--mode ci, always safe, no API key needed)cd clinical-ai-copilot
PYTHONPATH=.:platform .venv/bin/python -m intelligence.evaluation.run --mode ci
Read the printed table. If it says Overall: FAIL, report which metric(s) dropped below threshold and by how much — check git diff on data/rag/__init__.py, src/sephiroth/verification/citation_guard.py, or intelligence/agents/__init__.py (EvidenceAgent prompt) for what likely caused it.
If it warns results/latest.json is missing or stale, the dataset or transcripts changed since the last full run — offer to run --mode full --record (below) if GEMINI_API_KEY is set, or tell the user it needs a refresh before merging.
--mode full --record, needs GEMINI_API_KEY — burns free-tier quota)Check the key is set first:
grep -q '^GEMINI_API_KEY=.' .env && echo "key present" || echo "GEMINI_API_KEY missing"
If present, run the real Evidence Agent against every golden case and refresh the committed baseline:
PYTHONPATH=.:platform .venv/bin/python -m intelligence.evaluation.run --mode full --record --skip-pubmed
(--skip-pubmed keeps transcripts reproducible — PubMed responses vary run to run.)
Omit --skip-pubmed only if the user explicitly wants live PubMed citations exercised too.
Add --model <name> to override the model (defaults to settings.gemini_model, i.e. gemini-flash-latest). Useful if the committed baseline was generated with a different model — check intelligence/evaluation/results/latest.json's run.model field to see what it was actually generated with, and flag it in your summary if it's a stand-in rather than the production model.
results/latest.json (git diff intelligence/evaluation/results/latest.json before staging) and summarize what moved and why.README.md's ## Evaluation section to match the new numbers exactly — a stale README table is worse than no table.results/latest.json and intelligence/evaluation/transcripts/*.json must be committed together (they're hash-linked — CI checks the pair for staleness). git add intelligence/evaluation/.