hugging-face-evaluation

# Overview This skill provides tools to add structured evaluation results to Hugging Face model cards. It supports multiple methods for adding evaluation data: - Extracting existing evaluation tables from README content - Importing benchmark scores from Artificial Analysis - Running custom model evaluations with vLLM or accelerate backends (lighteval/inspect-ai) ## Integration with HF Ecosystem - **Model Cards**: Updates model-index metadata for leaderboard integration - **Artificial Analysis**: Direct API integration for benchmark imports - **Papers with Code**: Compatible with their model-index specification - **Jobs**: Run evaluations directly on Hugging Face Jobs with `uv` integration - **vLLM**: Efficient GPU inference for custom model evaluation - **lighteval**: HuggingFace's evaluation library with vLLM/accelerate backends - **inspect-ai**: UK AI Safety Institute's evaluation framework # Version 1.3.0 # Dependencies ## Core Dependencies - huggingface_hub>=0.26.0 - markdown-it-py>=3.0.0 - python-dotenv>=1.2.1 - pyyaml>=6.0.3 - requests>=2.32.5 - re (built-in) ## Inference Provider Evaluation - inspect-ai>=0.3.0 - inspect-evals - openai ## vLLM Custom Model Evaluation (GPU required) - lighteval[accelerate,vllm]>=0.6.0 - vllm>=0.4.0 - torch>=2.0.0 - transformers>=4.40.0 - accelerate>=0.30.0 Note: vLLM dependencies are installed automatically via PEP 723 script headers when using `uv run`. # IMPORTANT: Using This Skill ## ⚠️ CRITICAL: Check for Existing PRs Before Creating New Ones **Before creating ANY pull request with `--create-pr`, you MUST check for existing open PRs:** ```bash uv run scripts/evaluation_manager.py get-prs --repo-id "username/model-name" ``` **If open PRs exist:** 1. **DO NOT create a new PR** - this creates duplicate work for maintainers 2. **Warn the user** that open PRs already exist 3. **Show the user** the existing PR URLs so they can review them 4. Only proceed if the user explicitly confirms they want to create another PR This prevents spamming model repositories with duplicate evaluation PRs. --- **Use `--help` for the latest workflow guidance.** Works with plain Python or `uv run`: ```bash uv run scripts/evaluation_manager.py --help uv run scripts/evaluation_manager.py inspect-tables --help uv run scripts/evaluation_manager.py extract-readme --help ``` Key workflow (matches CLI help): 1) `get-prs` → check for existing open PRs first 2) `inspect-tables` → find table numbers/columns 3) `extract-readme --table N` → prints YAML by default 4) add `--apply` (push) or `--create-pr` to write changes # Core Capabilities ## 1. Inspect and Extract Evaluation Tables from README - **Inspect Tables**: Use `inspect-tables` to see all tables in a README with structure, columns, and sample rows - **Parse Markdown Tables**: Accurate parsing using markdown-it-py (ignores code blocks and examples) - **Table Selection**: Use `--table N` to extract from a specific table (required when multiple tables exist) - **Format Detection**: Recognize common formats (benchmarks as rows, columns, or comparison tables with multiple models) - **Column Matching**: Automatically identify model columns/rows; prefer `--model-column-index` (index from inspect output). Use `--model-name-override` only with exact column header text. - **YAML Generation**: Convert selected table to model-index YAML format - **Task Typing**: `--task-type` sets the `task.type` field in model-index output (e.g., `text-generation`, `summarization`) ## 2. Import from Artificial Analysis - **API Integration**: Fetch benchmark scores directly from Artificial Analysis - **Automatic Formatting**: Convert API responses to model-index format - **Metadata Preservation**: Maintain source attribution and URLs - **PR Creation**: Automatically create pull requests with evaluation updates ## 3. Model-Index Management - **YAML Generation**: Create properly formatted model-index entries - **Merge Support**: Add evaluations to existing model cards without overwriting - **Validation**: Ensure compliance with Papers with Code specification - **Batch Operations**: Process multiple models efficiently ## 4. Run Evaluations on HF Jobs (Inference Providers) - **Inspect-AI Integration**: Run standard evaluations using the `inspect-ai` library - **UV Integration**: Seamlessly run Python scripts with ephemeral dependencies on HF infrastructure - **Zero-Config**: No Dockerfiles or Space management required - **Hardware Selection**: Configure CPU or GPU hardware for the evaluation job - **Secure Execution**: Handles API tokens safely via secrets passed through the CLI ## 5. Run Custom Model Evaluations with vLLM (NEW) ⚠️ **Important:** This approach is only possible on devices with `uv` installed and sufficient GPU memory. **Benefits:** No need to use `hf_jobs()` MCP tool, can run scripts directly in terminal **When to use:** User working in local device directly when GPU is available ### Before running the script - check the script path - check uv is installed - check gpu is available with `nvidia-smi` ### Running the script ```bash uv run scripts/train_sft_example.py ``` ### Features - **vLLM Backend**: High-performance GPU inference (5-10x faster than standard HF methods) - **lighteval Framework**: HuggingFace's evaluation library with Open LLM Leaderboard tasks - **inspect-ai Framework**: UK AI Safety Institute's evaluation library - **Standalone or Jobs**: Run locally or submit to HF Jobs infrastructure # Usage Instructions The skill includes Python scripts in `scripts/` to perform operations. ### Prerequisites - Preferred: use `uv run` (PEP 723 header auto-installs deps) - Or install manually: `pip install huggingface-hub markdown-it-py python-dotenv pyyaml requests` - Set `HF_TOKEN` environment variable with Write-access token - For Artificial Analysis: Set `AA_API_KEY` environment variable - `.env` is loaded automatically if `python-dotenv` is installed ### Method 1: Extract from README (CLI workflow) Recommended flow (matches `--help`): ```bash # 1) Inspect tables to get table numbers and column hints uv run scripts/evaluation_manager.py inspect-tables --repo-id "username/model" # 2) Extract a specific table (prints YAML by default) uv run scripts/evaluation_manager.py extract-readme \ --repo-id "username/model" \ --table 1 \ [--model-column-index <column index shown by inspect-tables>] \ [--model-name-override "<column header/model name>"] # use exact header text if you can't use the index # 3) Apply changes (push or PR) uv run scripts/evaluation_manager.py extract-readme \ --repo-id "username/model" \ --table 1 \ --apply # push directly # or uv run scripts/evaluation_manager.py extract-readme \ --repo-id "username/model" \ --table 1 \ --create-pr # open a PR ``` Validation checklist: - YAML is printed by default; compare against the README table before applying. - Prefer `--model-column-index`; if using `--model-name-override`, the column header text must be exact. - For transposed tables (models as rows), ensure only one row is extracted. ### Method 2: Import from Artificial Analysis Fetch benchmark scores from Artificial Analysis API and add them to a model card. **Basic Usage:** ```bash AA_API_KEY="your-api-key" python scripts/evaluation_manager.py import-aa \ --creator-slug "anthropic" \ --model-name "claude-sonnet-4" \ --repo-id "username/model-name" ``` **With Environment File:** ```bash # Create .env file echo "AA_API_KEY=your-api-key" >> .env echo "HF_TOKEN=your-hf-token" >> .env # Run import python scripts/evaluation_manager.py import-aa \ --creator-slug "anthropic" \ --model-name "claude-sonnet-4" \ --repo-id "username/model-name" ``` **Create Pull Request:** ```bash python scripts/evaluation_manager.py import-aa \ --creator-slug "anthropic" \ --model-name "claude-sonnet-4" \ --repo-id "username/model-name" \ --create-pr ``` ### Method 3: Run Evaluation Job Submit an evaluation job on Hugging Face infrastructure using the `hf jobs uv run` CLI. **Direct CLI Usage:** ```bash HF_TOKEN=$HF_TOKEN \ hf jobs uv run hf-evaluation/scripts/inspect_eval_uv.py \ --flavor cpu-basic \ --secret HF_TOKEN=$HF_TOKEN \ -- --model "meta-llama/Llama-2-7b-hf" \ --task "mmlu" ``` **GPU Example (A10G):** ```bash HF_TOKEN=$HF_TOKEN \ hf jobs uv run hf-evaluation/scripts/inspect_eval_uv.py \ --flavor a10g-small \ --secret HF_TOKEN=$HF_TOKEN \ -- --model "meta-llama/Llama-2-7b-hf" \ --task "gsm8k" ``` **Python Helper (optional):** ```bash python scripts/run_eval_job.py \ --model "meta-llama/Llama-2-7b-hf" \ --task "mmlu" \ --hardware "t4-small" ``` ### Method 4: Run Custom Model Evaluation with vLLM Evaluate custom HuggingFace models directly on GPU using vLLM or accelerate backends. These scripts are **separate from inference provider scripts** and run models locally on the job's hardware. #### When to Use vLLM Evaluation (vs Inference Providers) | Feature | vLLM Scripts | Inference Provider Scripts | |---------|-------------|---------------------------| | Model access | Any HF model | Models with API endpoints | | Hardware | Your GPU (or HF Jobs GPU) | Provider's infrastructure | | Cost | HF Jobs compute cost | API usage fees | | Speed | vLLM optimized | Depends on provider | | Offline | Yes (after download) | No | #### Option A: lighteval with vLLM Backend lighteval is HuggingFace's evaluation library, supporting Open LLM Leaderboard tasks. **Standalone (local GPU):** ```bash # Run MMLU 5-shot with vLLM python scripts/lighteval_vllm_uv.py \ --model meta-llama/Llama-3.2-1B \ --tasks "leaderboard|mmlu|5" # Run multiple tasks python scripts/lighteval_vllm_uv.py \ --model meta-llama/Llama-3.2-1B \ --tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5" # Use accelerate backend instead of vLLM python scripts/lighteval_vllm_uv.py \ --model meta-llama/Llama-3.2-1B \ --tasks "leaderboard|mmlu|5" \ --backend accelerate # Chat/instruction-tuned models python scripts/lighteval_vllm_uv.py \ --model meta-llama/Llama-3.2-1B-Instruct \ --tasks "leaderboard|mmlu|5" \ --use-chat-template ``` **Via HF Jobs:** ```bash hf jobs uv run scripts/lighteval_vllm_uv.py \ --flavor a10g-small \ --secrets HF_TOKEN=$HF_TOKEN \ -- --model meta-llama/Llama-3.2-1B \ --tasks "leaderboard|mmlu|5" ``` **lighteval Task Format:** Tasks use the format `suite|task|num_fewshot`: - `leaderboard|mmlu|5` - MMLU with 5-shot - `leaderboard|gsm8k|5` - GSM8K with 5-shot - `lighteval|hellaswag|0` - HellaSwag zero-shot - `leaderboard|arc_challenge|25` - ARC-Challenge with 25-shot **Finding Available Tasks:** The complete list of available lighteval tasks can be found at: https://github.com/huggingface/lighteval/blob/main/examples/tasks/all_tasks.txt This file contains all supported tasks in the format `suite|task|num_fewshot|0` (the trailing `0` is a version flag and can be ignored). Common suites include: - `leaderboard` - Open LLM Leaderboard tasks (MMLU, GSM8K, ARC, HellaSwag, etc.) - `lighteval` - Additional lighteval tasks - `bigbench` - BigBench tasks - `original` - Original benchmark tasks To use a task from the list, extract the `suite|task|num_fewshot` portion (without the trailing `0`) and pass it to the `--tasks` parameter. For example: - From file: `leaderboard|mmlu|0` → Use: `leaderboard|mmlu|0` (or change to `5` for 5-shot) - From file: `bigbench|abstract_narrative_understanding|0` → Use: `bigbench|abstract_narrative_understanding|0` - From file: `lighteval|wmt14:hi-en|0` → Use: `lighteval|wmt14:hi-en|0` Multiple tasks can be specified as comma-separated values: `--tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5"` #### Option B: inspect-ai with vLLM Backend inspect-ai is the UK AI Safety Institute's evaluation framework. **Standalone (local GPU):** ```bash # Run MMLU with vLLM python scripts/inspect_vllm_uv.py \ --model meta-llama/Llama-3.2-1B \ --task mmlu # Use HuggingFace Transformers backend python scripts/inspect_vllm_uv.py \ --model meta-llama/Llama-3.2-1B \ --task mmlu \ --backend hf # Multi-GPU with tensor parallelism python scripts/inspect_vllm_uv.py \ --model meta-llama/Llama-3.2-70B \ --task mmlu \ --tensor-parallel-size 4 ``` **Via HF Jobs:** ```bash hf jobs uv run scripts/inspect_vllm_uv.py \ --flavor a10g-small \ --secrets HF_TOKEN=$HF_TOKEN \ -- --model meta-llama/Llama-3.2-1B \ --task mmlu ``` **Available inspect-ai Tasks:** - `mmlu` - Massive Multitask Language Understanding - `gsm8k` - Grade School Math - `hellaswag` - Common sense reasoning - `arc_challenge` - AI2 Reasoning Challenge - `truthfulqa` - TruthfulQA benchmark - `winogrande` - Winograd Schema Challenge - `humaneval` - Code generation #### Option C: Python Helper Script The helper script auto-selects hardware and simplifies job submission: ```bash # Auto-detect hardware based on model size python scripts/run_vllm_eval_job.py \