compare-models

name	compare-models
description	Quickly compare responses from different LLM models side-by-side. Use when exploring model behavior, comparing finetuned models to base models, testing prompts across models, or investigating how different system prompts affect responses.
allowed-tools	Bash, Read, Write

Compare Models

This project includes a CLI tool for quickly comparing LLM model responses. Use it to explore differences between models, test finetuned models, or see how system prompts affect behavior.

Quick Start

# Compare two models on a single prompt
echo "Explain recursion briefly" | uv run python -m compare --model gpt-4o --model gpt-3.5-turbo

# Multiple prompts (one per line)
printf "What is 2+2?\nExplain gravity" | uv run python -m compare --model gpt-4o

# With system prompts per model
echo "Say hello" | uv run python -m compare \
  --model gpt-4o --system "You are formal" \
  --model gpt-4o --system "You are casual"

Common Use Cases

Compare a finetuned model to its base

echo "Your test prompt" | uv run python -m compare \
  --model ft:gpt-4o:your-org:your-model:id \
  --model gpt-4o

Test multiple samples for variability

echo "Write a haiku" | uv run python -m compare --model gpt-4o --samples 5 --temperature 1.0

Run prompts from a file

uv run python -m compare --model gpt-4o --prompts test_prompts.jsonl --output results.jsonl

Use a local vLLM or other OpenAI-compatible server

# vLLM server running locally
echo "Hello" | uv run python -m compare \
  --model my-local-model \
  --base-url http://localhost:8000/v1

# With API key if required
echo "Hello" | uv run python -m compare \
  --model my-model \
  --base-url http://my-server:8000/v1 \
  --api-key sk-xxx

CLI Reference

Flag	Short	Description	Default
--model MODEL		Model identifier (repeatable)	Required
--system PROMPT		System prompt for preceding model	None
--prompts FILE		JSONL input file	stdin
--output FILE	-o	JSONL output file	terminal
--temperature	-t	Sampling temperature	0.7
--max-tokens		Max output tokens	1024
--samples	-n	Samples per prompt per model	1
--verbose	-v	Show progress bars	off
--base-url URL		OpenAI-compatible server URL	OpenAI
--api-key KEY		API key for custom server	'none'

Input Formats

• Plain text: One prompt per line via stdin
• JSONL with content: {"content": "your prompt"}
• JSONL with messages: {"messages": [{"role": "user", "content": "..."}]}

Output

Terminal output shows each prompt with all model responses grouped together. With --output, results are saved as JSONL with structure:

{"prompt": {"content": "..."}, "responses": {"model1": ["resp1", "resp2"], "model2": ["resp1"]}}

Tips

• Use --samples with higher --temperature to explore response variability
• Pipe output to files for later analysis: ... | tee comparison.txt
• The same model with different system prompts counts as separate entries