// Use when validating umbrela judge outputs — checks label range (0–3), qid/docid completeness, result_status consistency, backend metadata, and JSONL integrity. Wraps `umbrela validate` plus custom assertions. Use after running judge or evaluate to verify output correctness.
Use when analyzing umbrela evaluation results — comparing nDCG@10 scores across backends, interpreting confusion matrices, computing kappa agreement, and comparing modified qrels against human judgments. Use after running evaluate to interpret results.
Set up an umbrela development environment — checks Python 3.11+, installs via uv or pip with cloud extras, and verifies with doctor. Use when someone is onboarding, setting up a fresh clone, or troubleshooting their environment.
Use when working with umbrela CLI commands (judge, evaluate), backend selection (gpt, gemini, hf, os, ensemble), qrel handling (dl19-passage, dl20-passage, etc.), relevance labels (0–3), or introspection (doctor, describe, schema, validate). Covers all entry points, flags, and evaluation workflows.
| name | umbrela-verify |
| description | Use when validating umbrela judge outputs — checks label range (0–3), qid/docid completeness, result_status consistency, backend metadata, and JSONL integrity. Wraps `umbrela validate` plus custom assertions. Use after running judge or evaluate to verify output correctness. |
Validates umbrela judgment outputs for label correctness, completeness, and consistency.
umbrela judge — verify judgment output integrityumbrela evaluate — verify modified qrel completenessmodel, query, passage, judgment, and result_statusjudgment is integer 0–3result_status is 0 or 1 (1 = successfully parsed, 0 = fallback)result_status == 1modified_qrels/ directoryRun the verification script:
bash .claude/skills/umbrela-verify/scripts/verify.sh <artifact-path> [judge|qrel]
Or use the built-in validator first:
umbrela validate judge --input-file pairs.jsonl
umbrela validate evaluate --qrel dl19-passage --result-file run.trec
See scripts/verify.sh for the runnable verification wrapper.
umbrela validate checks input contracts. The verify script checks output artifacts.result_status == 0 means the LLM response couldn't be parsed into a 0–3 label — it falls back to 0. A high rate of result_status == 0 suggests prompt issues or model incompatibility.prediction field contains raw LLM text. The label is extracted by 39 regex patterns in common_utils.py. Check this if judgment distribution looks anomalous.{qrel}_{model}_{judge_cat}{few_shot}_{num_sample}.txt — verify the filename matches expectations.