name

benchmark-ebible-translations

description

Design, create, validate, run, analyze, and preserve rigorous eBible translation benchmarks. Use when a user wants to test a translation prompting technique, retrieval strategy, context policy, model, representation, project workflow, or any new translation-quality hypothesis; compare benchmark arms; reproduce or extend a prior result; inspect stored benchmark evidence; or turn a conversational research question into a new executable benchmark.

eBible Benchmark Factory

Turn a translation-research question into a paired, leakage-audited experiment using the repository's ebiblebench library. Preserve the reasoning and raw evidence so a later agent can distinguish established results from exploratory clues.

Workflow

Inspect README.md, the nearest default benchmark, and ebiblebench runs before designing another experiment. Reuse prior controls and held-out sets when appropriate; do not accidentally present an old result as a new test.
Translate the user's idea into a concrete production decision, hypothesis, primary estimand, experimental unit, arms, practical threshold, and required confirmation. Read scientific-method.md before changing sample selection or inference.
Choose the nearest default pattern under benchmarks/. For a new question, run scripts/new_benchmark.py NAME --question "...", then edit the generated plan.yaml, benchmark.py, and research.md. Read benchmark-contract.md before implementing a custom builder.
Keep novel logic in the benchmark module. Keep corpus loading, deterministic sampling, retrieval, OpenRouter calls, retries, scoring, bootstrapping, and storage in ebiblebench unless the research question explicitly tests one of those components.
Run ebiblebench validate BENCHMARK_DIR. Treat validation errors as blockers. Inspect representative rendered prompts and confirm that only the intended fields differ, the final source is present, and the held-out target is absent.
Calculate calls as units × arms plus any frozen-draft preparation calls. Obtain user approval before paid or networked execution. Never print or save API keys; load OPENROUTER_API_KEY from the environment or .env.
Run a separate smoke experiment first: ebiblebench run BENCHMARK_DIR --smoke 5 --yes. Inspect parsing, provider identities, failures, truncation, and cost. Do not merge smoke rows into the full estimate.
Run the complete benchmark with --yes. Let the immutable run directory and JSONL checkpoint handle interruptions; use --resume RUN_DIR only when the plan and prompt signature still match.
Read result-interpretation.md, inspect manifest.json, trials.jsonl, summary.json, and report.md, then update the benchmark's research.md with the decision and limitations. Lead with effect sizes and intervals, not significance labels.

Guardrails

Pair arms on identical evaluation units and randomize call order.
Freeze approved pools, evaluation sets, seeds, prompts, and the primary metric before model calls.
Never build retrieval indexes, glossaries, transformations, or prompt examples from evaluation references.
Keep development and confirmation passages disjoint. Label post-hoc subsets and provider filters as sensitivity analyses.
Score exhausted failed calls as empty for intent-to-treat analysis; also report failure and format-compliance rates.
Record requested model, actual model, provider, prompt hash, corpus hashes, token usage, retries, and raw output.
Do not call an interval crossing zero proof of equivalence. Use a predeclared non-inferiority or equivalence margin.
Do not broaden a narrow result. A prompt result for one model, language, or project state does not establish universality.

Common commands

ebiblebench inspect-corpus SOURCE TARGET --vrefs VREFS
ebiblebench new my-question --question "Does X improve translation?"
ebiblebench validate benchmarks/my-question
ebiblebench run benchmarks/my-question --smoke 5 --yes
ebiblebench run benchmarks/my-question --yes
ebiblebench runs

eBible Benchmark Factory

Workflow

Inspect README.md, the nearest default benchmark, and ebiblebench runs before designing another experiment. Reuse prior controls and held-out sets when appropriate; do not accidentally present an old result as a new test.

Translate the user's idea into a concrete production decision, hypothesis, primary estimand, experimental unit, arms, practical threshold, and required confirmation. Read scientific-method.md before changing sample selection or inference.

Choose the nearest default pattern under benchmarks/. For a new question, run scripts/new_benchmark.py NAME --question "...", then edit the generated plan.yaml, benchmark.py, and research.md. Read benchmark-contract.md before implementing a custom builder.

Keep novel logic in the benchmark module. Keep corpus loading, deterministic sampling, retrieval, OpenRouter calls, retries, scoring, bootstrapping, and storage in ebiblebench unless the research question explicitly tests one of those components.

Run ebiblebench validate BENCHMARK_DIR. Treat validation errors as blockers. Inspect representative rendered prompts and confirm that only the intended fields differ, the final source is present, and the held-out target is absent.

Calculate calls as units × arms plus any frozen-draft preparation calls. Obtain user approval before paid or networked execution. Never print or save API keys; load OPENROUTER_API_KEY from the environment or .env.

Run a separate smoke experiment first: ebiblebench run BENCHMARK_DIR --smoke 5 --yes. Inspect parsing, provider identities, failures, truncation, and cost. Do not merge smoke rows into the full estimate.

Run the complete benchmark with --yes. Let the immutable run directory and JSONL checkpoint handle interruptions; use --resume RUN_DIR only when the plan and prompt signature still match.

Read result-interpretation.md, inspect manifest.json, trials.jsonl, summary.json, and report.md, then update the benchmark's research.md with the decision and limitations. Lead with effect sizes and intervals, not significance labels.

Guardrails

Pair arms on identical evaluation units and randomize call order.

Freeze approved pools, evaluation sets, seeds, prompts, and the primary metric before model calls.

Never build retrieval indexes, glossaries, transformations, or prompt examples from evaluation references.

Keep development and confirmation passages disjoint. Label post-hoc subsets and provider filters as sensitivity analyses.

Score exhausted failed calls as empty for intent-to-treat analysis; also report failure and format-compliance rates.

Record requested model, actual model, provider, prompt hash, corpus hashes, token usage, retries, and raw output.

Do not call an interval crossing zero proof of equivalence. Use a predeclared non-inferiority or equivalence margin.

Do not broaden a narrow result. A prompt result for one model, language, or project state does not establish universality.

Common commands

ebiblebench inspect-corpus SOURCE TARGET --vrefs VREFS ebiblebench new my-question --question "Does X improve translation?" ebiblebench validate benchmarks/my-question ebiblebench run benchmarks/my-question --smoke 5 --yes ebiblebench run benchmarks/my-question --yes ebiblebench runs