Create and manage AILANG eval benchmarks. Use when user asks to create benchmarks, fix benchmark issues, debug failing benchmarks, or analyze benchmark results.
Create and manage AILANG eval benchmarks. Use when user asks to create benchmarks, fix benchmark issues, debug failing benchmarks, or analyze benchmark results.
Benchmark Manager
Manage AILANG evaluation benchmarks with correct prompt integration, debugging workflows, and best practices learned from real benchmark failures.
Quick Start
Debugging a failing benchmark:
# 1. Show the full prompt that models see
.claude/skills/benchmark-manager/scripts/show_full_prompt.sh json_parse
# 2. Test a benchmark with a specific model
ailang eval-suite --models claude-haiku-4-5 --benchmarks json_parse
# 3. Check benchmark YAML for common issues
.claude/skills/benchmark-manager/scripts/check_benchmark.sh benchmarks/json_parse.yml
When to Use This Skill
Invoke this skill when:
User asks to create a new benchmark
User asks to debug/fix a failing benchmark
User wants to understand why models generate wrong code
User asks about benchmark YAML format
Benchmarks show 0% pass rate despite language support
CRITICAL: prompt vs task_prompt
This is the most important concept for benchmark management.
The Problem (v0.4.8 Discovery)
Benchmarks have TWO different prompt fields with VERY different behavior:
Field
Behavior
Use When
prompt:
REPLACES the teaching prompt entirely
Testing raw model capability (rare)
task_prompt:
APPENDS to teaching prompt
Normal benchmarks (99% of cases)
Why This Matters
# BAD - Model never sees AILANG syntax!prompt:|
Write a program that prints "Hello"
# GOOD - Model sees teaching prompt + tasktask_prompt:|
Write a program that prints "Hello"
With prompt:, models generate Python/pseudo-code because they never learn AILANG syntax.
id:my_benchmark# Unique identifier (snake_case)description:"Short description of what this tests"languages: ["python", "ailang"]
entrypoint:"main"# Function to callcaps: ["IO"] # Required capabilitiesdifficulty:"easy|medium|hard"expected_gain:"low|medium|high"task_prompt:|# ALWAYS use task_prompt, not prompt!Writeaprogramin<LANG>that:1.Doessomething2.PrintstheresultOutputonlythecode,noexplanations.expected_stdout:|# Exact expected outputexpectedoutputhere
Can models solve this with just the teaching prompt?
What's the expected output?
Step 2: Write the Benchmark
id:my_new_benchmarkdescription:"Test feature X capability"languages: ["python", "ailang"]
entrypoint:"main"caps: ["IO"]
difficulty:"medium"expected_gain:"medium"task_prompt:|
Write a program in <LANG> that:
1. Clear description of task
2. Another step
3. Print the result
Outputonlythecode,noexplanations.expected_stdout:|
exact expected output
Step 3: Validate and Test
# Check for issues
.claude/skills/benchmark-manager/scripts/check_benchmark.sh benchmarks/my_new_benchmark.yml
# Test with cheap model first
ailang eval-suite --models claude-haiku-4-5 --benchmarks my_new_benchmark
Debugging Failing Benchmarks
Symptom: 0% Pass Rate Despite Language Support
Check 1: Is it using task_prompt:?
grep -E "^prompt:" benchmarks/failing_benchmark.yml
# If this returns a match, change to task_prompt:
# After editing prompts/v0.x.x.md:
make quick-install # REQUIRED!
3. Putting Hints in Benchmarks
# WRONG - Hints in benchmarktask_prompt:|
Write code that prints 42.
Hint: Use print(show(42)) in AILANG.
# CORRECT - No hints; if models fail, fix the teaching prompttask_prompt:|
Write code that prints 42.
If models need AILANG-specific hints, the teaching prompt is incomplete. Use the prompt-manager skill to fix it.
4. Testing Too Many Models at Once
# WRONG - Expensive and slow for debugging
ailang eval-suite --full --benchmarks my_test
# CORRECT - Use one cheap model first
ailang eval-suite --models claude-haiku-4-5 --benchmarks my_test
