// Use when working with nuggetizer CLI commands (create, assign, metrics), prompt templates, JSONL formats, or introspection (doctor, describe, schema, validate). Covers all entry points, flags, and the YAML prompt system.
Use when analyzing nuggetizer metrics outputs — comparing runs, computing agreement between models, reporting token usage and latency from trace fields, and building side-by-side evaluation tables. Use after running metrics to interpret and compare results.
Set up a nuggetizer development environment — checks Python 3.11+, installs via uv or pip, and verifies with doctor. Use when someone is onboarding, setting up a fresh clone, or troubleshooting their environment.
Use when validating nuggetizer batch outputs — checks JSONL integrity, duplicate qids, missing nugget fields, assignment label validity, and metrics consistency. Wraps `nuggetizer validate` plus custom assertions. Use after running create, assign, or metrics to verify output correctness.
| name | nuggetizer-quickstart |
| description | Use when working with nuggetizer CLI commands (create, assign, metrics), prompt templates, JSONL formats, or introspection (doctor, describe, schema, validate). Covers all entry points, flags, and the YAML prompt system. |
Reference for the nuggetizer CLI — the tool for creating, scoring, and assigning factual nuggets for RAG evaluation using LLM backends.
nuggetizer <command> [options]
| Command | Purpose |
|---|---|
create | Create and score nuggets from query + candidate passages |
assign | Assign nuggets to contexts (answers or retrieval results) |
metrics | Calculate per-query and global nugget-based scores |
| Command | Purpose |
|---|---|
doctor | Check Python version, API keys, backend readiness |
describe <cmd> | Machine-readable command contract (flags, defaults, examples) |
schema <name> | Print JSON Schema for inputs/outputs |
validate <target> | Validate inputs without running models |
prompt list|show|render | Inspect and render YAML prompt templates |
view <path> | Inspect existing artifact files |
# 1. Check environment
nuggetizer doctor
# 2. Create nuggets from a pool file
nuggetizer create --input-file pool.jsonl --output-file nuggets.jsonl --model gpt-4o
# 3. Assign nuggets to answers
nuggetizer assign --contexts answers.jsonl --nuggets nuggets.jsonl \
--input-kind answers --output-file assignments.jsonl --model gpt-4o
# 4. Calculate metrics
nuggetizer metrics --input-file assignments.jsonl --output-file metrics.jsonl
Read these on demand for details:
references/cli-examples.md — Common invocations for each commandreferences/input-output-examples.md — JSONL format examples (create input, create output, assign input/output, metrics output)references/prompt-templates.md — How the YAML prompt template system worksvital or okaysupport, partial_support, not_support) or 2-grade (support, not_support)--use-azure-openai), OpenRouter (--use-openrouter), vLLM (local)--resume (append, skip processed), --overwrite (truncate), --fail-if-existscreate has separate --creator-model and --scorer-model overrides. If only --model is set, both stages use it.assign --input-kind answers expects --contexts to be a JSONL with topic_id and answer fields. assign --input-kind retrieval expects the same format as create input.--nuggets is required for batch assign but not for direct input (nuggets are in the JSON payload).metrics operates on assign output only — it reads assignment labels, not raw nuggets.--execution-mode async is available but the default is sync.scripts/ translate to CLI calls with --resume by default.