// Use when working with nuggetizer CLI commands (create, assign, metrics), prompt templates, JSONL formats, or introspection (doctor, describe, schema, validate). Covers all entry points, flags, and the YAML prompt system.
| name | nuggetizer-quickstart |
| description | Use when working with nuggetizer CLI commands (create, assign, metrics), prompt templates, JSONL formats, or introspection (doctor, describe, schema, validate). Covers all entry points, flags, and the YAML prompt system. |
Reference for the nuggetizer CLI — the tool for creating, scoring, and assigning factual nuggets for RAG evaluation using LLM backends.
nuggetizer <command> [options]
| Command | Purpose |
|---|---|
create | Create and score nuggets from query + candidate passages |
assign | Assign nuggets to contexts (answers or retrieval results) |
metrics | Calculate per-query and global nugget-based scores |
| Command | Purpose |
|---|---|
doctor | Check Python version, API keys, backend readiness |
describe <cmd> | Machine-readable command contract (flags, defaults, examples) |
schema <name> | Print JSON Schema for inputs/outputs |
validate <target> | Validate inputs without running models |
prompt list|show|render | Inspect and render YAML prompt templates |
view <path> | Inspect existing artifact files |
# 1. Check environment
nuggetizer doctor
# 2. Create nuggets from a pool file
nuggetizer create --input-file pool.jsonl --output-file nuggets.jsonl --model gpt-4o
# 3. Assign nuggets to answers
nuggetizer assign --contexts answers.jsonl --nuggets nuggets.jsonl \
--input-kind answers --output-file assignments.jsonl --model gpt-4o
# 4. Calculate metrics
nuggetizer metrics --input-file assignments.jsonl --output-file metrics.jsonl
Read these on demand for details:
references/cli-examples.md — Common invocations for each commandreferences/input-output-examples.md — JSONL format examples (create input, create output, assign input/output, metrics output)references/prompt-templates.md — How the YAML prompt template system worksvital or okaysupport, partial_support, not_support) or 2-grade (support, not_support)--use-azure-openai), OpenRouter (--use-openrouter), vLLM (local)--resume (append, skip processed), --overwrite (truncate), --fail-if-existscreate has separate --creator-model and --scorer-model overrides. If only --model is set, both stages use it.assign --input-kind answers expects --contexts to be a JSONL with topic_id and answer fields. assign --input-kind retrieval expects the same format as create input.--nuggets is required for batch assign but not for direct input (nuggets are in the JSON payload).metrics operates on assign output only — it reads assignment labels, not raw nuggets.--execution-mode async is available but the default is sync.scripts/ translate to CLI calls with --resume by default.Use when analyzing nuggetizer metrics outputs — comparing runs, computing agreement between models, reporting token usage and latency from trace fields, and building side-by-side evaluation tables. Use after running metrics to interpret and compare results.
Set up a nuggetizer development environment — checks Python 3.11+, installs via uv or pip, and verifies with doctor. Use when someone is onboarding, setting up a fresh clone, or troubleshooting their environment.
Use when validating nuggetizer batch outputs — checks JSONL integrity, duplicate qids, missing nugget fields, assignment label validity, and metrics consistency. Wraps `nuggetizer validate` plus custom assertions. Use after running create, assign, or metrics to verify output correctness.
