// Set up an umbrela development environment — checks Python 3.11+, installs via uv or pip with cloud extras, and verifies with doctor. Use when someone is onboarding, setting up a fresh clone, or troubleshooting their environment.
Use when analyzing umbrela evaluation results — comparing nDCG@10 scores across backends, interpreting confusion matrices, computing kappa agreement, and comparing modified qrels against human judgments. Use after running evaluate to interpret results.
Use when validating umbrela judge outputs — checks label range (0–3), qid/docid completeness, result_status consistency, backend metadata, and JSONL integrity. Wraps `umbrela validate` plus custom assertions. Use after running judge or evaluate to verify output correctness.
Use when working with umbrela CLI commands (judge, evaluate), backend selection (gpt, gemini, hf, os, ensemble), qrel handling (dl19-passage, dl20-passage, etc.), relevance labels (0–3), or introspection (doctor, describe, schema, validate). Covers all entry points, flags, and evaluation workflows.
| name | umbrela-install |
| description | Set up an umbrela development environment — checks Python 3.11+, installs via uv or pip with cloud extras, and verifies with doctor. Use when someone is onboarding, setting up a fresh clone, or troubleshooting their environment. |
Development environment setup for umbrela — LLM-based relevance assessment of query-passage pairs.
github.com:castorini)python3 --version # must be 3.11+
command -v uv # if present, use uv path; otherwise recommend uv
If uv is on PATH, use it silently. If not, ask the user once: install uv or proceed with pip.
If no pyproject.toml in cwd:
git clone git@github.com:castorini/umbrela.git && cd umbrela
uv venv --python 3.11
source .venv/bin/activate
uv sync --group dev --extra cloud
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[cloud]"
pip install pre-commit pytest mypy ruff
pip install umbrela
umbrela doctor --output json
umbrela --help
pre-commit install
references/extras.md — Optional dependency stacks (cloud, hf, fastchat, pyserini)--extra pyserini for evaluation workflows. The core umbrela package does not require Java.[dependency-groups] — only uv sync --group dev resolves them natively. With pip, install each package manually.disallow_untyped_defs = true. All new functions need type annotations.tests/ (with an s).