quoracle

Coordinates LiveBench benchmark runs. Reads a pre-built manifest, dispatches one solver per question by sequential index using batch_async, then scores all answers via score-run.sh and produces the benchmark report. Use when running LiveBench evaluations. Do NOT use for MMLU-Pro or general tasks.

Coordinates MMLU-Pro benchmark runs. Reads a pre-built manifest, dispatches one answerer per question by sequential index using batch_async, then scores all answers via score-run.sh and produces the benchmark report. Use when running MMLU-Pro evaluations. Do NOT use for general Q&A tasks.

Solves a single LiveBench benchmark question. Reads the question from a file and answers it directly. Use when spawned by livebench-coordinator. Do NOT use for MMLU-Pro or non-benchmark tasks.

Answers a single MMLU-Pro multiple-choice question. Reads the question from a file, follows the prompt exactly, and writes the answer to a file. Use when spawned by mmlu-coordinator during benchmark runs. Do NOT use for general Q&A or non-benchmark tasks.