// Use when working with umbrela CLI commands (judge, evaluate), backend selection (gpt, gemini, hf, os, ensemble), qrel handling (dl19-passage, dl20-passage, etc.), relevance labels (0–3), or introspection (doctor, describe, schema, validate). Covers all entry points, flags, and evaluation workflows.
Use when analyzing umbrela evaluation results — comparing nDCG@10 scores across backends, interpreting confusion matrices, computing kappa agreement, and comparing modified qrels against human judgments. Use after running evaluate to interpret results.
Set up an umbrela development environment — checks Python 3.11+, installs via uv or pip with cloud extras, and verifies with doctor. Use when someone is onboarding, setting up a fresh clone, or troubleshooting their environment.
Use when validating umbrela judge outputs — checks label range (0–3), qid/docid completeness, result_status consistency, backend metadata, and JSONL integrity. Wraps `umbrela validate` plus custom assertions. Use after running judge or evaluate to verify output correctness.
| name | umbrela-quickstart |
| description | Use when working with umbrela CLI commands (judge, evaluate), backend selection (gpt, gemini, hf, os, ensemble), qrel handling (dl19-passage, dl20-passage, etc.), relevance labels (0–3), or introspection (doctor, describe, schema, validate). Covers all entry points, flags, and evaluation workflows. |
Reference for the umbrela CLI — the tool for LLM-based relevance assessment of query-passage pairs, reproducing the Bing BRELA methodology with 4-class labels (0–3).
umbrela <command> [options]
| Command | Purpose |
|---|---|
judge | Run a single backend on direct JSON or batch JSONL input |
evaluate | Generate modified qrels and compute nDCG@10 from a result file |
| Command | Purpose |
|---|---|
doctor | Check Python version, API keys, backend readiness |
describe <cmd> | Machine-readable command contract |
schema <name> | Print JSON Schema for inputs/outputs |
validate <target> | Validate inputs without running models |
prompt list|show|render | Inspect and render prompt templates |
view <path> | Inspect existing artifact files |
# 1. Check environment
umbrela doctor
# 2. Judge query-passage pairs
umbrela judge --backend gpt --model gpt-4o \
--input-file pairs.jsonl --output-file judgments.jsonl
# 3. Evaluate against standard qrels
umbrela evaluate --backend gpt --model gpt-4o \
--qrel dl19-passage --result-file run.trec
Read these on demand for details:
references/cli-examples.md — Common invocations for each commandreferences/input-output-examples.md — JSONL format examples and judgment outputreferences/backends.md — Backend selection guide and provider configurationreferences/qrel-handling.md — Standard qrels, label meanings, and evaluation workflowgpt (OpenAI/Azure/OpenRouter), gemini (Vertex AI), hf (HuggingFace), os (FastChat), ensemble (majority voting, evaluate only)bing (BRELA-style, default) or basic--backend ensemble is only valid for evaluate, not judge.--execution-mode async is only supported for GPT backend.evaluate command requires pyserini for qrel loading and passage retrieval. Install with uv sync --extra pyserini.evaluate writes two artifacts: a modified qrel file in modified_qrels/ and a confusion matrix PNG in conf_matrix/.--judge-cat controls which relevance categories to evaluate. Default 0,1,2,3 judges all pairs. Use 2,3 to focus on relevant documents only.--min-judgment on judge filters output — pairs below the threshold go to --filtered-output-file.--few-shot-count controls few-shot examples in the prompt. Examples are drawn from the qrel dataset. Default is 0 (zero-shot)..umbrela.toml or ~/.config/umbrela/config.toml) can set defaults.