菜单
Run OCR and translation quality evaluations across scripts and languages. Produces research-grade reports with MCR, cross-model agreement, embedding-space hallucination detection, and corpus readiness scores.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Run a Glass Bead Game session — a parlor game played against the Source Library corpus. Players connect verified passages from different books, times, and traditions into a growing string of 'beads,' with Claude as librarian, verifier, and scribe. Trigger when the user says 'glass bead game,' 'let's play beads,' or asks to start/resume/publish a game session. Claude runs the search-verify-display loop; humans do the noticing, choosing, and judging.
Stage-2 verification of first-translation flips using INDEPENDENT Claude subagents (subscription, not API key) with full evidence capture. Use to verify proposed badge changes (demotes/promotes) before writing any public claim — the binding step that catches AI-fabricated priors. Each subagent reports its actual search queries + sources, persisted to the first_translation_attempts provenance log.
Triage and respond to the user-feedback queue — pull open feedback, detect items already fixed but never marked, route translation-requests and metadata-corrections, mark items resolved, and (on approval) email submitters. Use when asked to check/triage/answer/clear feedback, "what are people saying?", or "respond to feedback."
Sweep Claude's private per-machine memory for team-relevant operational knowledge and propose moving it into the shared repo (or the private ops repo) as draft PRs. Use to promote lessons that accreted privately, or run on a schedule. Never auto-merges; never leaks.
One-shot audience & usage report — signups, DAU/MAU, time on-site, traffic, top content, search behavior, referrers, countries, and user journeys. Use when asked "what's our usage?", "how many signups?", "time on site?", "traffic?", or any audience/engagement metrics question.
Use this skill whenever choosing or evaluating a background image for the 'quote' section of a Source Library collection page — the wide banner where a sourced quotation is overlaid on an image drawn from the collection's plates or illustrations. Trigger for any request to pick, rank, or judge a background image behind quote text, select a plate for the quote banner, or decide whether a candidate image works as a text background. The goal is a beautiful, legible, on-theme backdrop that the quotation reads cleanly over.
| name | qa-eval |
| description | Run OCR and translation quality evaluations across scripts and languages. Produces research-grade reports with MCR, cross-model agreement, embedding-space hallucination detection, and corpus readiness scores. |
Run systematic quality evaluations on OCR and translation output across all scripts and languages in Source Library. Produces structured JSON results and markdown blog posts suitable for academic publication.
Issue: #1329
# Load env
set -a; source .env.production.local; set +a
# OCR consistency (run each model N times, compute Modal Consistency Rate)
node scripts/eval/qa-eval.mjs consistency --corpus=bhutan --sample=10 --models=flash,opus --runs=3
# Embedding-space evaluation (hallucination detection without ground truth)
node scripts/eval/qa-eval.mjs embedding --corpus=bhutan --sample=10
# Compare against ground truth (CER for OCR, BLEU/ROUGE for translation)
node scripts/eval/qa-eval.mjs compare --corpus=bhutan --against=ocr
# Readiness score for a corpus
node scripts/eval/qa-eval.mjs readiness bhutan
# Show all results
node scripts/eval/qa-eval.mjs report --latest
# Generate blog post from results
node scripts/eval/qa-eval.mjs report --corpus=bhutan --format=blog --save
/qa-eval # Show help and available corpora
/qa-eval --corpus=bhutan --sample=5 # Quick consistency check
/qa-eval --corpus=bhutan --blog # Full eval + blog post
/qa-eval consistency --corpus=bhutan --models=flash,opus --runs=3
/qa-eval embedding --corpus=bhutan --sample=20
/qa-eval compare --corpus=bhutan --against=translation
/qa-eval matrix # All corpora comparison table
/qa-eval readiness bhutan # Quick readiness score
/qa-eval consistency --corpus=bhutan --sample=10 --models=flash,opus --runs=3 --dry-run
Defined in scripts/eval/corpus-registry.json:
| Corpus | Script | Description |
|---|---|---|
| bhutan | Tibetan | 1,325 EAP manuscripts (dbu can + dbu med) |
| latin-alchemy | Latin | Printed alchemical texts (baseline) |
| fraktur | German | Pre-1800 Fraktur/blackletter |
| arabic | Arabic | Printed Naskh |
| hebrew | Hebrew | Hebrew + Rashi script |
| chinese | CJK | Woodblock-printed classical Chinese |
| sanskrit | Devanagari | Printed Sanskrit editions |
| greek-ancient | Greek | Aldine and early printed Greek |
| bph-manuscripts | Mixed | BPH high-quality manuscript scans |
| Alias | Full Model ID |
|---|---|
| flash | gemini-3-flash-preview |
| lite | gemini-3.1-flash-lite-preview |
| opus | claude-opus-4-6 |
| sonnet | claude-sonnet-4-6 |
| haiku | claude-haiku-4-5-20251001 |
Results are saved to scripts/eval/results/ as JSON and optionally as markdown blog posts in docs/.
scripts/eval/results/
bhutan-consistency-2026-04-23.json
bhutan-embedding-2026-04-23.json
matrix-2026-04-23.json
docs/
qa-eval-bhutan-2026-04-23.md
Place reference transcriptions and translations in scripts/eval/ground-truth/ as JSON:
{
"book_id": "abc123",
"page_number": 5,
"script": "tibetan",
"source": "BDRC etext",
"source_url": "https://library.bdrc.io/...",
"ocr_ground_truth": "...",
"translation_ground_truth": "...",
"translation_source": "Thurman 1994"
}
Sources: BDRC etexts, OpenPecha, Esukhia Derge Kangyur, Lotsawa House, scholarly editions.
Ground truth for ANY language is built from published etexts with build-reference-groundtruth.mjs, which generalizes the ctext system below. Per-language works files live in scripts/eval/reference-works/<language>.json (curated passages + provenance; sources registry in scripts/eval/reference-sources.json):
node scripts/eval/build-reference-groundtruth.mjs --language=hebrew # dry run
node scripts/eval/build-reference-groundtruth.mjs --language=hebrew --write # pin ground truth
node scripts/eval/qa-eval.mjs compare --corpus=hebrew --against=ocr # score one corpus
node scripts/eval/qa-eval.mjs scorecard # per-language table
Two hard-won rules (measured on Armenian/Latin, 2026-07-19 — see tests/unit/reference-ocr-guard.test.ts):
Orthography that varies BETWEEN EDITIONS (Latin u/v, i/j, long-s; Hebrew niqqud + maqaf; Greek polytonic marks; Armenian ew-ligature) is folded before comparison in normalizeForScript (lib/metrics.mjs) so it never counts as OCR error.
Current scorecard (2026-07-19): Armenian 91.8%* · Chinese ~98.5% · Greek 100% · Hebrew 100% · Latin ~99% (char accuracy on aligned canonical passages; run scorecard for current numbers).
* Armenian is a lower bound: manual audit of the page scan showed ~18/21 word errors are the 1805 printing's nomina-sacra abbreviations (ա՟ծ = Աստուած etc.) transcribed diplomatically by our OCR but silently expanded by the TITUS reference — true misread rate on the page is ~1%. See audit_note in ground-truth/armenian-zohrab-john-1.json. General lesson: when a scorecard number looks bad, diff the words and READ THE SCAN before believing it — reference transcription conventions (abbreviation expansion, normalized punctuation) masquerade as OCR error.
Canonical references (Genesis 1, Iliad I, John 1) measure OCR with the model's memory helping — the score is a memory-assisted upper bound. Since 2026-07-19 every language works file also carries non-canonical rows (non_canonical: true, memorization_risk: low|medium; ground-truth files mirror both under page_class): editor prefaces, biographical front matter, and mid-text passages of rarely digitized works, scored by the same guard on the same page classes. The canonical-vs-non-canonical score gap ≈ the memorization subsidy, and non-canonical scores are the numbers that transfer to the rare texts the library actually exists for.
Where the references come from (the trick that made this findable): etexts that preserve their source edition's print pagination map deterministically onto our scans once we hold the same edition — TITUS (Armenian: Xorenac'i Tiflis 1913, Eznik Venice 1826) and First1KGreek (Hero Teubner 1899, Simplicius CAG 9 1882, Philo Cohn 1896) both do this. Wikisource's Clementine Vulgate matches the 1566 Louvain edition it descends from; Sefaria covers non-liturgical Hebrew (Sha'arei Orah, Sefer HaYirah); Durham's Living Poets has the Donatus-auctus Vita Vergilii that 16th-c Virgils actually print (the modern critical Vita would fail the guard — memorized text actively hurts there, which is the point).
Rules for adding rows: (1) same pinning discipline as canonical rows — visual page audit before --write, page_class with canonical_text: false; (2) a "published transcription" is never zero-exposure (First1K is on GitHub) — record memorization_risk, don't claim zero; (3) cross-edition references (our Dioscorides 1549 vs Wellmann 1907; TITUS's 1959 Eznik orthography vs the 1826 printing) make char accuracy a LOWER BOUND — note it, don't "fix" the reference; (4) rejected candidates worth remembering: Zohar Cremona 1558 (Sefaria transcribes the Mantua/Vilna recension — score would conflate recension with OCR error), Zohrab 1805 front matter, Aldine Greek prefaces, Manutius' Virgil dedications (no open transcriptions exist — verified negatives, 2026-07-19). Works entries may pin book_id/page_number directly (copy-exact contrasts, brittle probes); build-reference-groundtruth.mjs --only=<slug-rx> scopes --write so it can't clobber hand-annotated files, and qa-eval.mjs scorecard --only=<rx> scopes paid model runs the same way (regex; scoped runs save as scorecard-<rx>-<date>.json so they never clobber the full scorecard).
RECITATION HAZARD (the inverse failure — a number that looks too GOOD). Two pinned rows were deleted on 2026-07-19 after page-scan audits proved the pipeline OCR recited the canonical passage from memory instead of reading the page: a 1450 Mishnah-commentary manuscript (OCR contained only the letter-perfect canonical Mishnah, none of the visible commentary) and a Daxue Huowen page (OCR half-read the real columns then emitted the whole Zhongyong opening, which is not printed there). A third row (Zhuangzi) had its reference trimmed because the OCR completed the famous sentence past the page edge. Pattern: pages that discuss a canonical text prime the model to recite it, and the identity guard cannot tell reading from reciting — it verifies the reference is in the OCR, not that it came from the page. Rules: (1) pin only pages that verifiably PRINT the passage — check the image, not the OCR; (2) prefer pages where the OCR also transcribes non-canonical page furniture (apparatus, commentary, adjacent text) at several times the reference length; (3) treat a perfect score on a degraded/manuscript source as a hallucination flag, not a triumph. Related: lesson_gemini_recitation_canonical_texts (Gemini's RECITATION safety block on canonical texts is the same memorization surfacing differently).
For the Chinese corpus, OCR ground truth is auto-built from ctext.org canonical transcriptions:
node scripts/eval/build-ctext-groundtruth.mjs # dry run — shows alignment + which works pass the guard
node scripts/eval/build-ctext-groundtruth.mjs --write # write pinned ground-truth files
node scripts/eval/qa-eval.mjs compare --corpus=chinese --against=ocr
Two things make this work where a naive CER fails:
subsequenceCER in lib/metrics.mjs). Most of our Chinese editions are commentary editions — the canonical main text is interleaved with small-character annotation that ctext's main-text-only transcription lacks. The reference is matched as an in-order subsequence of the OCR with extra (commentary) characters skipped free, so the metric scores OCR error on the canonical text only. A plain edit distance scored the Book of Odes at 6% when it was really 99%.compare fetches the exact book_id/page_number from each ground-truth file (no fuzzy title matching that could grab a same-phrase decoy). The generator only writes a file when the passage aligns below --threshold (default 0.30); anything above is skipped as a wrong book or a divergent recension (e.g. Zhu Xi's reordered Great Learning, the Shiji with 三家注) — reported, not counted as OCR error.Coverage caveat: ctext holds canonical printed texts only — manuscripts, tables, and rare/regional works (the actual OCR frontier) are out of scope and need MCR / cross-model / embedding checks instead. Baseline run (2026-06-25): 98.5% character accuracy across 7 canonical works.
scripts/eval/
qa-eval.mjs # CLI entrypoint
lib/
metrics.mjs # All metric functions
runners.mjs # Gemini + Claude model execution
sampling.mjs # MongoDB page sampling
report.mjs # JSON + Markdown output
embedding-eval.mjs # Embedding-space evaluation
corpus-registry.json # Known corpora
ground-truth/ # Reference data
results/ # Output
docs/blog-tibetan-ocr-benchmark.md_tmp-ocr-consistency.mjsgemini-embedding-2-preview (768d, matches production search)node scripts/eval/export-eval-dataset.mjs --version=vX.Y emits scripts/eval/dataset/<version>/ — pages (with MEASURED image resolution), license-gated reference texts (TITUS/ctext ship as sha256 pointers, not text), and raw scored runs. Datasheet + caveats in the version README. Resolution spans 0.64–17.4 MP across pinned pages and confounds naive language-level gaps — read caveat 2 before quoting numbers.