Skip to main content 首页 创作者 synthetic-sciences openscience evaluating-code-models
evaluating-code-models Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.
跳到安装 Skills Marketplace 发现并探索由社区构建的 Agent Skills
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/synthetic-sciences/openscience --skill evaluating-code-models命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
下载 Zip 下载中... 同仓库更多 Skills Diffusion-based molecular docking. Predict protein-ligand binding poses from PDB/SMILES, confidence scores, virtual screening, for structure-based drug design. Not for affinity prediction.
Fast inference and fine-tuning platform with serverless and on-demand GPU deployments. OpenAI-compatible API for chat completions, embeddings, function calling, vision, and structured output. Supports SFT, DPO, and RL fine-tuning. SOC2 + HIPAA compliant.
Serverless inference, fine-tuning, embeddings, image generation, and batch processing on 200+ open-source models via an OpenAI-compatible API. Use when you need fast, cost-effective access to open-source LLMs without managing infrastructure.
synthetic-sciences
synthetic-sciences/openscience
打开 GitHub 仓库 name evaluating-code-models description Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards. category ml-training version 1.0.0 author Synthetic Sciences license MIT tags ["Evaluation","Code Generation","HumanEval","MBPP","MultiPL-E","Pass@k","BigCode","Benchmarking","Code Models"] dependencies ["bigcode-evaluation-harness","transformers>=4.25.1","accelerate>=0.13.2","datasets>=2.6.1"]
BigCode Evaluation Harness - Code Model Benchmarking
Quick Start
BigCode Evaluation Harness evaluates code generation models across 15+ benchmarks including HumanEval, MBPP, and MultiPL-E (18 languages).
Installation :
git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git
cd bigcode-evaluation-harness
pip install -e .
accelerate config
Evaluate on HumanEval :
accelerate launch main.py \
--model bigcode/starcoder2-7b \
--tasks humaneval \
--max_length_generation 512 \
--temperature 0.2 \
--n_samples 20 \
--batch_size 10 \
--allow_code_execution \
--save_generations
View available tasks :
python -c "from bigcode_eval.tasks import ALL_TASKS; print(ALL_TASKS)"
Common Workflows
Workflow 1: Standard Code Benchmark Evaluation
Evaluate model on core code benchmarks (HumanEval, MBPP, HumanEval+).
Checklist :
Code Benchmark Evaluation:
- [ ] Step 1: Choose benchmark suite
- [ ] Step 2: Configure model and generation
- [ ] Step 3: Run evaluation with code execution
- [ ] Step 4: Analyze pass@k results
Step 1: Choose benchmark suite
Python code generation (most common):
HumanEval : 164 handwritten problems, function completionHumanEval+ : Same 164 problems with 80× more tests (stricter)MBPP : 500 crowd-sourced problems, entry-level difficultyMBPP+ : 399 curated problems with 35× more tests
Multi-language (18 languages):
MultiPL-E : HumanEval/MBPP translated to C++, Java, JavaScript, Go, Rust, etc.
Advanced :
APPS : 10,000 problems (introductory/interview/competition)DS-1000 : 1,000 data science problems across 7 libraries
Step 2: Configure model and generation
accelerate launch main.py \
--model bigcode/starcoder2-7b \
--tasks humaneval \
--max_length_generation 512 \
--temperature 0.2 \
--do_sample True \
--n_samples 200 \
--batch_size 50 \
--allow_code_execution
accelerate launch main.py \
--model bigcode/starcoder2-7b \
--tasks humaneval \
--temperature 0.8 \
--n_samples 200 \
--batch_size 50 \
--allow_code_execution \
--save_generations \
--metric_output_path results/starcoder2-humaneval.json
Results in results/starcoder2-humaneval.json:
{
"humaneval" : {
"pass@1" : 0.354 ,
"pass@10" : 0.521 ,
"pass@100" : 0.689
} ,
"config" : {
"model" : "bigcode/starcoder2-7b" ,
"temperature" : 0.8 ,
"n_samples" : 200
}
}
Workflow 2: Multi-Language Evaluation (MultiPL-E) Evaluate code generation across 18 programming languages.
Multi-Language Evaluation:
- [ ] Step 1: Generate solutions (host machine)
- [ ] Step 2: Run evaluation in Docker (safe execution)
- [ ] Step 3: Compare across languages
Step 1: Generate solutions on host
accelerate launch main.py \
--model bigcode/starcoder2-7b \
--tasks multiple-py,multiple-js,multiple-java,multiple-cpp \
--max_length_generation 650 \
--temperature 0.8 \
--n_samples 50 \
--batch_size 50 \
--generation_only \
--save_generations \
--save_generations_path generations_multi.json
Step 2: Evaluate in Docker container
docker pull ghcr.io/bigcode-project/evaluation-harness-multiple
docker run -v $(pwd )/generations_multi.json:/app/generations.json:ro \
-it evaluation-harness-multiple python3 main.py \
--model bigcode/starcoder2-7b \
--tasks multiple-py,multiple-js,multiple-java,multiple-cpp \
--load_generations_path /app/generations.json \
--allow_code_execution \
--n_samples 50
Supported languages : Python, JavaScript, Java, C++, Go, Rust, TypeScript, C#, PHP, Ruby, Swift, Kotlin, Scala, Perl, Julia, Lua, R, Racket
Workflow 3: Instruction-Tuned Model Evaluation Evaluate chat/instruction models with proper formatting.
Instruction Model Evaluation:
- [ ] Step 1: Use instruction-tuned tasks
- [ ] Step 2: Configure instruction tokens
- [ ] Step 3: Run evaluation
Step 1: Choose instruction tasks
instruct-humaneval : HumanEval with instruction promptshumanevalsynthesize-{lang} : HumanEvalPack synthesis tasks Step 2: Configure instruction tokens
accelerate launch main.py \
--model codellama/CodeLlama-7b-Instruct-hf \
--tasks instruct-humaneval \
--instruction_tokens "<s>[INST],</s>,[/INST]" \
--max_length_generation 512 \
--allow_code_execution
Step 3: HumanEvalPack for instruction models
accelerate launch main.py \
--model codellama/CodeLlama-7b-Instruct-hf \
--tasks humanevalsynthesize-python,humanevalsynthesize-js \
--prompt instruct \
--max_length_generation 512 \
--allow_code_execution
Workflow 4: Compare Multiple Models Benchmark suite for model comparison.
Step 1: Create evaluation script
#!/bin/bash
MODELS=(
"bigcode/starcoder2-7b"
"codellama/CodeLlama-7b-hf"
"deepseek-ai/deepseek-coder-6.7b-base"
)
TASKS="humaneval,mbpp"
for model in "${MODELS[@]} " ; do
model_name=$(echo $model | tr '/' '-' )
echo "Evaluating $model "
accelerate launch main.py \
--model $model \
--tasks $TASKS \
--temperature 0.2 \
--n_samples 20 \
--batch_size 20 \
--allow_code_execution \
--metric_output_path results/${model_name} .json
done
Step 2: Generate comparison table
import json
import pandas as pd
models = ["bigcode-starcoder2-7b" , "codellama-CodeLlama-7b-hf" , "deepseek-ai-deepseek-coder-6.7b-base" ]
results = []
for model in models:
with open (f"results/{model} .json" ) as f:
data = json.load(f)
results.append({
"Model" : model,
"HumanEval pass@1" : f"{data['humaneval' ]['pass@1' ]:.3 f} " ,
"MBPP pass@1" : f"{data['mbpp' ]['pass@1' ]:.3 f} "
})
df = pd.DataFrame(results)
print (df.to_markdown(index=False ))
When to Use vs Alternatives Use BigCode Evaluation Harness when:
Evaluating code generation models specifically
Need multi-language evaluation (18 languages via MultiPL-E)
Testing functional correctness with unit tests (pass@k)
Benchmarking for BigCode/HuggingFace leaderboards
Evaluating fill-in-the-middle (FIM) capabilities
Use alternatives instead:
lm-evaluation-harness : General LLM benchmarks (MMLU, GSM8K, HellaSwag)EvalPlus : Stricter HumanEval+/MBPP+ with more test casesSWE-bench : Real-world GitHub issue resolutionLiveCodeBench : Contamination-free, continuously updated problemsCodeXGLUE : Code understanding tasks (clone detection, defect prediction)
Supported Benchmarks Benchmark Problems Languages Metric Use Case HumanEval 164 Python pass@k Standard code completion HumanEval+ 164 Python pass@k Stricter evaluation (80× tests) MBPP 500 Python pass@k Entry-level problems MBPP+ 399 Python pass@k Stricter evaluation (35× tests) MultiPL-E 164×18 18 languages pass@k Multi-language evaluation APPS 10,000 Python pass@k Competition-level DS-1000 1,000 Python pass@k Data science (pandas, numpy, etc.) HumanEvalPack 164×3×6 6 languages pass@k Synthesis/fix/explain Mercury 1,889 Python Efficiency Computational efficiency
Common Issues Issue: Different results than reported in papers
--n_samples 200
--temperature 0.8
--tasks humaneval
--max_length_generation 512
Issue: CUDA out of memory
--load_in_8bit
--load_in_4bit
--batch_size 1
--max_memory_per_gpu "20GiB"
Issue: Code execution hangs or times out
Use Docker for safe execution:
--generation_only --save_generations
docker run ... --allow_code_execution --load_generations_path ...
Issue: Low scores on instruction models
Ensure proper instruction formatting:
--tasks instruct-humaneval
--instruction_tokens "<s>[INST],</s>,[/INST]"
Issue: MultiPL-E language failures
Use the dedicated Docker image:
docker pull ghcr.io/bigcode-project/evaluation-harness-multiple
Command Reference Argument Default Description --model- HuggingFace model ID or local path --tasks- Comma-separated task names --n_samples1 Samples per problem (200 for pass@k) --temperature0.2 Sampling temperature --max_length_generation512 Max tokens (prompt + generation) --batch_size1 Batch size per GPU --allow_code_executionFalse Enable code execution (required) --generation_onlyFalse Generate without evaluation --load_generations_path- Load pre-generated solutions --save_generationsFalse Save generated code --metric_output_pathresults.json Output file for metrics --load_in_8bitFalse 8-bit quantization --load_in_4bitFalse 4-bit quantization --trust_remote_codeFalse Allow custom model code --precisionfp32 Model precision (fp32/fp16/bf16)
Hardware Requirements Model Size VRAM (fp16) VRAM (4-bit) Time (HumanEval, n=200) 7B 14GB 6GB ~30 min (A100) 13B 26GB 10GB ~1 hour (A100) 34B 68GB 20GB ~2 hours (A100)
Resources 