// Use when working with the rank-llm CLI: rerank, evaluate, analyze, retrieve-cache, serve, validate, prompt, view, describe, schema, or doctor. Covers entry points, common flags, JSONL and TREC artifacts, and end-to-end retrieval plus reranking workflows.
Set up a rank_llm development environment. Use when someone is onboarding, setting up a fresh clone, choosing extras such as cloud, api, local, or pyserini, or troubleshooting whether the packaged rank-llm CLI is ready.
Use when analyzing rank_llm evaluation outputs across runs or models. Covers aggregated trec_eval JSONL files, response-analysis metrics, retrieval-cache handoff files, and side-by-side comparison of stored evaluation artifacts.
Use when validating rank_llm artifacts after rerank, retrieve-cache, or related CLI workflows. Checks JSONL integrity, TREC formatting, candidate shape, invocation-history structure, and duplicate query identifiers. Wraps rank-llm view plus custom assertions.
Reference for the packaged rank-llm CLI.
rank-llm <command> [options]
| Command | Purpose |
|---|---|
rerank | Run reranking from dataset retrieval, request files, or direct JSON input |
evaluate | Aggregate trec_eval metrics across stored rerank outputs |
analyze | Analyze stored responses and error counts |
retrieve-cache | Build cached retrieval JSON from an existing TREC run |
serve http | Start the HTTP server |
serve mcp | Start the MCP server |
| Command | Purpose |
|---|---|
doctor | Check Python version and optional dependency readiness |
describe <cmd> | Return structured command metadata |
schema <name> | Print JSON Schema for supported inputs and outputs |
validate rerank | Validate rerank inputs without executing a model |
prompt list|show|render | Inspect bundled prompt templates |
view <path> | Inspect rerank JSONL, request JSONL, TREC runs, and invocation histories |
# 1. Check environment
rank-llm doctor
# 2. Run retrieval + reranking
rank-llm rerank --model-path castorini/rank_zephyr_7b_v1_full --dataset dl20 \
--retrieval-method bm25 --top-k-candidates 100 \
--output-jsonl-file rerank_results.jsonl --output-trec-file rerank_results.trec
# 3. Inspect the artifact
rank-llm view rerank_results.jsonl
# 4. Aggregate trec_eval metrics from stored outputs
rank-llm evaluate --model-name castorini/rank_zephyr_7b_v1_full
# 5. Analyze invocation histories or stored response files
rank-llm analyze --files demo_outputs/inference_invocations_history.json --verbose
Read these on demand for details:
references/cli-examples.md - Common invocations for each commandreferences/input-output-examples.md - JSONL, TREC, and invocation-history artifact shapesreferences/workflows.md - Backend and workflow selection guidererank accepts dataset retrieval, request files, direct JSON payloads, or stdin.cloud; local model paths usually need local or a batched backend extra.rank-llm prompt.rerank requires one input source: --dataset, --requests-file, --input-json, or --stdin.rerank also requires --retrieval-method.rank-llm view detects .trec, .jsonl, and invocation-history .json artifacts, but it does not inspect arbitrary JSON files.evaluate operates on stored rerank outputs in a directory and writes trec_eval_aggregated_results_<model>.jsonl.analyze can return partial_success when a file mixes valid and malformed model outputs.serve http needs the api extra. serve mcp needs the mcp stack.