add-benchmark

name	add-benchmark
description	Add a new benchmark to sgl-eval by vendoring its NeMo-Skills dataset module and registering it in `_TABLE`. Use when the user asks to "add <benchmark>" / "support <benchmark>" / "register <benchmark>" inside the sgl-eval repo.

Adding a benchmark

Don't skip step 1. If NS doesn't have the benchmark, this becomes a heavier add-metrics-type task instead, not this skill.

1. Verify upstream coverage

SHA=$(python -c "import yaml; print(yaml.safe_load(open('sgl_eval/_vendored/nemo_skills/SOURCES.yaml'))['synced_from_sha'])")
gh api "repos/NVIDIA/NeMo-Skills/contents/nemo_skills/dataset/<name>?ref=$SHA" \
  --jq '.[] | .name'
gh api "repos/NVIDIA/NeMo-Skills/contents/nemo_skills/dataset/<name>/__init__.py?ref=$SHA" \
  --jq '.content' | base64 -d

Confirm:

• The directory exists.
• METRICS_TYPE is one of math, multichoice (the wired-up types). Otherwise stop and ask the user; this skill cannot proceed.
• GENERATION_ARGS contains a ++prompt_config=... (the prompt yaml basename will be auto-derived).

2. Add source files to SOURCES.yaml

Always append the dataset __init__.py:

- src: nemo_skills/dataset/<name>/__init__.py
  dst: dataset/<name>/__init__.py