nemo-gym-blade-analysis

name

nemo-gym-blade-analysis

description

Use when analyzing NeMo Gym benchmark rollouts for BLADE-style reports, writing benchmark methodology notes, checking whether a benchmark is BLADE-ready, comparing model runs, or explaining why a benchmark report passed, failed, or changed. Covers aggregate metrics, rollout evidence, report structure, root-cause taxonomy, judge expectations, and improvement recommendations. For generic reward profiling commands, prefer nemo-gym-reward-profiling; for failed infrastructure jobs, prefer nemo-gym-debugging.

NeMo Gym BLADE Analysis

Invocation Check

Use this skill when the user wants to turn NeMo Gym rollout outputs into an analysis report, benchmark card, model comparison, benchmark-improvement recommendation, or BLADE-ready benchmark package.

Load references/blade-benchmark-build-guide.md when the user asks how to build, validate, submit, or review a BLADE benchmark or asks whether a benchmark has all required BLADE deliverables.

Use the bundled public helper at scripts/blade_toolkit.py for package validation, draft anchor-fact extraction, shallow baseline generation, and local calibration when external BLADE tooling is not available in the target repository.

Do not load benchmark-specific examples by default. Load references/cvdp-report-example.md only when the user explicitly asks for a CVDP example, the original CVDP report layout, or this optional reference, or when the agent is confused about the goal and needs one concrete example to re-anchor on what a BLADE-style report is supposed to look like.

Nemotron-only golden analysis artifacts are available under references/nemotron-analysis-artifacts/ as original-CVDP example artifacts. Load them only when the user explicitly asks to study an example completed BLADE-style report, asks for CVDP artifacts, or the agent is confused about the goal and needs a concrete completed example. Do not load those files by default.

Inputs To Gather

Start by identifying the artifact set:

• rollout JSONL from ng_collect_rollouts
• aggregate metrics JSON, if present
• reward profile JSONL from ng_reward_profile, if present
• benchmark-specific report directory, if present
• optional golden analysis artifacts, if the user asks to compare against a curated report
• config paths, agent name, model name, repeat count, and sampling settings
• source dataset metadata, license, and known redaction limits

If artifacts are missing, state which claims cannot be supported rather than filling gaps from memory.

If the task is benchmark construction rather than report analysis, first build an inventory of BLADE deliverables: analysis skill, rollout data, and golden report packages with metrics and anchor facts. Current BLADE scoring is handled by the universal blade-judge; benchmark-local judge utilities are optional pre-checks, not required deliverables. Missing deliverables are blocking work items, not footnotes.

If external BLADE tools are not available, use the local helper script:

uv run python scripts/blade_toolkit.py validate --benchmark-dir <benchmark_dir>
uv run python scripts/blade_toolkit.py extract-anchor-facts --help
uv run python scripts/blade_toolkit.py make-shallow --help
uv run python scripts/blade_toolkit.py calibrate --help

Analysis Workflow

1. Count tasks, rollouts, completed rows, repeats per task, and missing rows.
2. Compute pass@1 and pass@k from rewards or reward profiles.
3. Build a workflow funnel appropriate to the benchmark.
4. Split tasks into always-pass, sometimes-pass, never-pass, and missing.
5. Compare passing and failing trajectories for sometimes-pass tasks.
6. Inspect representative never-pass trajectories in chronological order.
7. Separate model capability gaps, agent behavior issues, verifier/task issues, and data or infrastructure problems.
8. Map findings to concrete actions: data, prompts, agent workflow, verifier repair, environment reliability, SFT, RL, or benchmark documentation.

For multi-repeat benchmarks, sometimes-pass tasks are the highest-signal slice: they show the conditions under which the same task can succeed or fail.

Report Structure

Use this structure unless the benchmark already defines a report format:

# <Benchmark> BLADE Analysis Report

## Executive Summary
## Artifact Inventory
## Aggregate Results
## Workflow Funnel
## Task Outcome Buckets
## Dominant Failure Modes
## Sometimes-Pass Deep Dives
## Never-Pass Deep Dives
## Cross-Model Comparison
## Recommendations
## Reproducibility Notes

Keep the executive summary short and evidence-backed. A useful report explains what changed, why it changed, and what to do next.

Core Metrics

• pass@1: average rollout success rate.
• pass@k: fraction of tasks with at least one successful rollout across k repeats.
• consistency: fraction of tasks where every rollout succeeds.
• coverage: completed rollout rows divided by expected rows.
• retry value: pass@k minus pass@1, useful for spotting instability.
• variance across repeats, seeds, model versions, or task categories.

For benchmark-specific metrics, keep the original names and definitions. Do not rename verifier outputs unless the report includes a mapping table.

Evidence Rules

• Tie every major claim to row counts, task ids, rollout ids, logs, verifier messages, tool calls, or report files.
• Read trajectories in order. Avoid attributing an early failure to evidence that only appears later.
• Distinguish self-test success from verifier success.
• If an error appears before the model's consequential action, do not attribute that error to the later action; diagnose whether the model failed to retry, verify, or recover instead.
• Treat sometimes-pass tasks as primary diagnostic evidence, but inspect whether success came from real understanding, lucky output, or workflow variance.
• Treat missing rows, timeouts, and malformed outputs as first-class outcomes.
• Mark redacted or unavailable evidence explicitly.
• Do not include private source code, private endpoints, credentials, user names, or unreleased benchmark names in a shareable report.

Root-Cause Taxonomy

Use one primary label per failed or mixed task when possible:

• KG knowledge gap: the model lacks domain, API, tool, or verifier knowledge.
• UK unreliable knowledge: some repeats show the needed knowledge and others do not.
• BI behavioral issue: the model appears capable but skips key steps, gives up, thrashes, ignores feedback, or uses tools poorly.
• TI task/verifier issue: the task, harness, timeout, dependency, or expected answer is suspect.
• IR infrastructure reliability: failures come from service availability, sandbox startup, scheduler behavior, network, storage, or provider errors.
• DA data artifact: duplicated rows, bad metadata, prompt leakage, missing files, or inconsistent labels affect the result.

Prefer a mixed label only when the evidence genuinely requires it, such as BI+KG. Do not use a mixed label to avoid making a call.

Recommendation Mapping

• KG: add targeted examples, domain SFT, better task docs, or verifier-facing explanations.
• UK: add repeated rollout training, RL, self-checking prompts, or comparison data from passing trajectories.
• BI: shape the agent loop, add workflow checks, reward intermediate verification behavior, or simplify tool affordances.
• TI: repair the task, verifier, timeout, or dependency; rerun baselines after repair.
• IR: fix infrastructure, isolate flaky rows, and keep flake rates separate from model quality.
• DA: correct dataset metadata and regenerate artifacts before model comparison.

Quality Bar

A good BLADE analysis is not a metrics dump. It should identify the largest drop-off, prove the dominant failure mode with examples, explain sometimes-pass behavior, and end with an intervention plan that follows from the evidence.