Skip to main content Home Creators synthetic-sciences openscience evaluating-code-models
evaluating-code-models Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.
Jump to install Skills Marketplace Discover and explore AI skills built by the community.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Show prompt details A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/synthetic-sciences/openscience --skill evaluating-code-modelsThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Download Zip Downloading... More from this repository
Related occupations SOC
Based on SOC occupation classification
name evaluating-code-models description Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards. category ml-training version 1.0.0 author Synthetic Sciences license MIT tags ["Evaluation","Code Generation","HumanEval","MBPP","MultiPL-E","Pass@k","BigCode","Benchmarking","Code Models"] dependencies ["bigcode-evaluation-harness","transformers>=4.25.1","accelerate>=0.13.2","datasets>=2.6.1"]
BigCode Evaluation Harness - Code Model Benchmarking
Quick Start
BigCode Evaluation Harness evaluates code generation models across 15+ benchmarks including HumanEval, MBPP, and MultiPL-E (18 languages).
Installation :
git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git
cd bigcode-evaluation-harness
pip install -e .
accelerate config
Evaluate on HumanEval :
accelerate launch main.py \
--model bigcode/starcoder2-7b \
--tasks humaneval \
--max_length_generation 512 \
--temperature 0.2 \
--n_samples 20 \
--batch_size 10 \
--allow_code_execution \
--save_generations
View available tasks :
python -c "from bigcode_eval.tasks import ALL_TASKS; print(ALL_TASKS)"
Common Workflows
Workflow 1: Standard Code Benchmark Evaluation
Evaluate model on core code benchmarks (HumanEval, MBPP, HumanEval+).
Checklist :
Code Benchmark Evaluation:
- [ ] Step 1: Choose benchmark suite
- [ ] Step 2: Configure model and generation
- [ ] Step 3: Run evaluation with code execution
- [ ] Step 4: Analyze pass@k results
Step 1: Choose benchmark suite
Python code generation (most common):
HumanEval : 164 handwritten problems, function completionHumanEval+ : Same 164 problems with 80× more tests (stricter)MBPP : 500 crowd-sourced problems, entry-level difficultyMBPP+ : 399 curated problems with 35× more tests
Multi-language (18 languages):
MultiPL-E : HumanEval/MBPP translated to C++, Java, JavaScript, Go, Rust, etc.
Advanced :
APPS : 10,000 problems (introductory/interview/competition)DS-1000 : 1,000 data science problems across 7 libraries
Step 2: Configure model and generation
accelerate launch main.py \
--model bigcode/starcoder2-7b \
--tasks humaneval \
--max_length_generation 512 \
--temperature 0.2 \
--do_sample True \
--n_samples 200 \
--batch_size 50 \
--allow_code_execution
accelerate launch main.py \
--model bigcode/starcoder2-7b \
--tasks humaneval \
--temperature 0.8 \
--n_samples 200 \
--batch_size 50 \
--allow_code_execution \
--save_generations \
--metric_output_path results/starcoder2-humaneval.json
Results in results/starcoder2-humaneval.json:
{
"humaneval" : {
"pass@1" : 0.354 ,
"pass@10" : 0.521 ,
"pass@100" : 0.689
} ,
"config" : {
"model" : "bigcode/starcoder2-7b" ,
"temperature" : 0.8 ,
"n_samples" : 200
}
}
Workflow 2: Multi-Language Evaluation (MultiPL-E) Evaluate code generation across 18 programming languages.
Multi-Language Evaluation:
- [ ] Step 1: Generate solutions (host machine)
- [ ] Step 2: Run evaluation in Docker (safe execution)
- [ ] Step 3: Compare across languages
Step 1: Generate solutions on host
accelerate launch main.py \
--model bigcode/starcoder2-7b \
--tasks multiple-py,multiple-js,multiple-java,multiple-cpp \
--max_length_generation 650 \
--temperature 0.8 \
--n_samples 50 \
--batch_size 50 \
--generation_only \
--save_generations \
--save_generations_path generations_multi.json
Step 2: Evaluate in Docker container
docker pull ghcr.io/bigcode-project/evaluation-harness-multiple
docker run -v $(pwd )/generations_multi.json:/app/generations.json:ro \
-it evaluation-harness-multiple python3 main.py \
--model bigcode/starcoder2-7b \
--tasks multiple-py,multiple-js,multiple-java,multiple-cpp \
--load_generations_path /app/generations.json \
--allow_code_execution \
--n_samples 50
Supported languages : Python, JavaScript, Java, C++, Go, Rust, TypeScript, C#, PHP, Ruby, Swift, Kotlin, Scala, Perl, Julia, Lua, R, Racket
Workflow 3: Instruction-Tuned Model Evaluation Evaluate chat/instruction models with proper formatting.
Instruction Model Evaluation:
- [ ] Step 1: Use instruction-tuned tasks
- [ ] Step 2: Configure instruction tokens
- [ ] Step 3: Run evaluation
Step 1: Choose instruction tasks
instruct-humaneval : HumanEval with instruction promptshumanevalsynthesize-{lang} : HumanEvalPack synthesis tasks Step 2: Configure instruction tokens
accelerate launch main.py \
--model codellama/CodeLlama-7b-Instruct-hf \
--tasks instruct-humaneval \
--instruction_tokens "<s>[INST],</s>,[/INST]" \
--max_length_generation 512 \
--allow_code_execution
Step 3: HumanEvalPack for instruction models
accelerate launch main.py \
--model codellama/CodeLlama-7b-Instruct-hf \
--tasks humanevalsynthesize-python,humanevalsynthesize-js \
--prompt instruct \
--max_length_generation 512 \
--allow_code_execution
Workflow 4: Compare Multiple Models Benchmark suite for model comparison.
Step 1: Create evaluation script
#!/bin/bash
MODELS=(
"bigcode/starcoder2-7b"
"codellama/CodeLlama-7b-hf"
"deepseek-ai/deepseek-coder-6.7b-base"
)
TASKS="humaneval,mbpp"
for model in "${MODELS[@]} " ; do
model_name=$(echo $model | tr '/' '-' )
echo "Evaluating $model "
accelerate launch main.py \
--model $model \
--tasks $TASKS \
--temperature 0.2 \
--n_samples 20 \
--batch_size 20 \
--allow_code_execution \
--metric_output_path results/${model_name} .json
done
Step 2: Generate comparison table
import json
import pandas as pd
models = ["bigcode-starcoder2-7b" , "codellama-CodeLlama-7b-hf" , "deepseek-ai-deepseek-coder-6.7b-base" ]
results = []
for model in models:
with open (f"results/{model} .json" ) as f:
data = json.load(f)
results.append({
"Model" : model,
"HumanEval pass@1" : f"{data['humaneval' ]['pass@1' ]:.3 f} " ,
"MBPP pass@1" : f"{data['mbpp' ]['pass@1' ]:.3 f} "
})
df = pd.DataFrame(results)
print (df.to_markdown(index=False ))
When to Use vs Alternatives Use BigCode Evaluation Harness when:
Evaluating code generation models specifically
Need multi-language evaluation (18 languages via MultiPL-E)
Testing functional correctness with unit tests (pass@k)
Benchmarking for BigCode/HuggingFace leaderboards
Evaluating fill-in-the-middle (FIM) capabilities
Use alternatives instead:
lm-evaluation-harness : General LLM benchmarks (MMLU, GSM8K, HellaSwag)EvalPlus : Stricter HumanEval+/MBPP+ with more test casesSWE-bench : Real-world GitHub issue resolutionLiveCodeBench : Contamination-free, continuously updated problemsCodeXGLUE : Code understanding tasks (clone detection, defect prediction)
Supported Benchmarks Benchmark Problems Languages Metric Use Case HumanEval 164 Python pass@k Standard code completion HumanEval+ 164 Python pass@k Stricter evaluation (80× tests) MBPP 500 Python pass@k Entry-level problems MBPP+ 399 Python pass@k Stricter evaluation (35× tests) MultiPL-E 164×18 18 languages pass@k Multi-language evaluation APPS 10,000 Python pass@k Competition-level DS-1000 1,000 Python pass@k Data science (pandas, numpy, etc.) HumanEvalPack 164×3×6 6 languages pass@k Synthesis/fix/explain Mercury 1,889 Python Efficiency Computational efficiency
Common Issues Issue: Different results than reported in papers
--n_samples 200
--temperature 0.8
--tasks humaneval
--max_length_generation 512
Issue: CUDA out of memory
--load_in_8bit
--load_in_4bit
--batch_size 1
--max_memory_per_gpu "20GiB"
Issue: Code execution hangs or times out
Use Docker for safe execution:
--generation_only --save_generations
docker run ... --allow_code_execution --load_generations_path ...
Issue: Low scores on instruction models
Ensure proper instruction formatting:
--tasks instruct-humaneval
--instruction_tokens "<s>[INST],</s>,[/INST]"
Issue: MultiPL-E language failures
Use the dedicated Docker image:
docker pull ghcr.io/bigcode-project/evaluation-harness-multiple
Command Reference Argument Default Description --model- HuggingFace model ID or local path --tasks- Comma-separated task names --n_samples1 Samples per problem (200 for pass@k) --temperature0.2 Sampling temperature --max_length_generation512 Max tokens (prompt + generation) --batch_size1 Batch size per GPU --allow_code_executionFalse Enable code execution (required) --generation_onlyFalse Generate without evaluation --load_generations_path- Load pre-generated solutions --save_generationsFalse Save generated code --metric_output_pathresults.json Output file for metrics --load_in_8bitFalse 8-bit quantization --load_in_4bitFalse 4-bit quantization --trust_remote_codeFalse Allow custom model code --precisionfp32 Model precision (fp32/fp16/bf16)
Hardware Requirements Model Size VRAM (fp16) VRAM (4-bit) Time (HumanEval, n=200) 7B 14GB 6GB ~30 min (A100) 13B 26GB 10GB ~1 hour (A100) 34B 68GB 20GB ~2 hours (A100)
Resources 