Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Flag-based architecture - Single service per agent, mode selected via environment variables
Type-safe constants - MCP server definitions in TypeScript
Isolated execution - Each pairing runs in its own Docker container
Architecture:
agent-schemas/ - Headless adapter JSON schemas
mcp-servers.ts - TypeScript MCP server constants
docker/entrypoint - Bun shell script for runtime config
scripts/ - Type-safe execution and comparison CLI tools
docker/ - Container infrastructure
Quick Commands
Run Evaluations
# Full dataset (151 prompts), k=5 — all 8 agent×provider scenarios
bun run trials
# Quick smoke test (5 random prompts, single trial)
bun run trials -- --count 5 -k 1
bun run trials -- --agent claude-code --search-provider
bun run trials -- --agent gemini --search-provider you
bun run trials -- --trial-type capability
bun run trials -- --trial-type regression
bun run trials -- -k 7
bun run trials -- -j 4
bun run trials -- --prompt-concurrency 4
docker compose run -- -e SEARCH_PROVIDER= claude-code
docker compose run -- -e SEARCH_PROVIDER=you -e PROMPT_COUNT=5 gemini
# Specific agent or provider
builtin
# Trial type presets
# k=10, deep exploration
# k=3, fast regression check
# Custom k value
# Control parallelism
# Limit to 4 containers
# 4 prompts per container
# Direct Docker (manual testing)
rm
builtin
rm
Compare Results
Comparisons are written to data/comparisons/YYYY-MM-DD/.
# Latest date auto-detected
bun run compare
# Statistical analysis with bootstrap confidence intervals
bun run compare:stat# Specific date or filter
bun run compare -- --run-date 2026-02-18
bun run compare -- --agent droid
bun run compare -- --search-provider builtin
bun run compare -- --trial-type capability
# View resultscat data/comparisons/2026-02-18/all-builtin-weighted.json | jq '.capability'cat data/comparisons/2026-02-18/builtin-vs-you-weighted.json | jq '.headToHead.capability'
Comparison strategies:
weighted (default) - Capability, reliability, and consistency weighted scoring
statistical - Bootstrap sampling with 95% confidence intervals
Generate Report
Generate a comprehensive REPORT.md from comparison results:
# Latest date auto-detected
bun run report
# Specific date
bun run report -- --run-date 2026-02-18
# Preview without writing
bun run report -- --dry-run
Report includes:
Executive summary with best capability, reliability, and performance
Quality rankings with pass@k and pass^k scores
Performance rankings (latency P50/P90/P99)
Flakiness analysis with top flaky prompts
MCP tool impact analysis (builtin vs MCP comparison)
Tool call statistics (P50/P90/P99/mean per provider)
Tool call distribution histograms
Failing prompts list (pass@k = 0%) with query text
Output:data/comparisons/YYYY-MM-DD/REPORT.md
Calibrate Grader
Interactive wizard to sample failures and review grader accuracy. Helps distinguish between agent failures (agent got it wrong) and grader bugs (agent was correct, grader too strict).
# Interactive calibration (recommended)
bun run calibrate
Interactive prompts:
Run date - Select from available dated runs
Agents - Multi-select via numbers or "all"
Search providers - Multi-select via numbers or "all"
Sample count - Number of failures to sample (default: 5)
The evaluation harness supports two-level parallelization for optimal performance:
Container-Level Concurrency (-j, --concurrency)
Controls how many Docker containers (agent×provider scenarios) run simultaneously.
bun run trials # Unlimited (default, all 8 scenarios at once)
bun run trials -- -j 4 # Limit to 4 containers
bun run trials -- -j 1 # Sequential (debugging)
Use cases:
Unlimited (default) - All scenarios at once, I/O-bound workload handles it fine
-j 4 - Limit concurrency if hitting API rate limits
-j 2 - Conservative, for low-resource machines
-j 1 - Sequential execution for debugging
Prompt-Level Concurrency (--prompt-concurrency)
Controls how many prompts run in parallel within each container.
bun run trials -- --prompt-concurrency 4 # 4 prompts (moderate parallelism)
bun run trials -- --prompt-concurrency 1 # Sequential (default, safest)
bun run trials -- --prompt-concurrency 8 # 8 prompts (high memory, CI only)
How it works:
Uses harness -j flag with --workspace-dir for isolation
Each prompt gets its own workspace directory
Web searches are I/O-bound — parallel prompts maximize network bandwidth
Performance comparison:
Config
Containers
Prompts/Container
Full (151 prompts, k=5)
Default
unlimited
1
~2.5 hrs
Faster
unlimited
4
~40 min
CI (high memory)
unlimited
8
~20 min
Warning: Stream-mode agents (claude-code, droid) use ~400-500MB RSS per prompt process. With --prompt-concurrency 8 that's 3-4GB per container — OOM kills likely in Docker (see issue #45)
Prompts
Prompts live in a flat data/prompts/ directory. The format differs by search provider:
Builtin mode: Just the query (e.g., "What are the best free icon libraries...")
MCP mode: "Use {server-name} and answer\n{query}" with MCP metadata
The entrypoint automatically selects the correct prompt file based on SEARCH_PROVIDER. To run a random subset, pass PROMPT_COUNT (or --count N via CLI):
bun run trials -- --count 5 # 5 random prompts from full dataset
Results
All trial results are written to flat dated directories:
