skills-optimization

name	skills-optimization
description	Improve agent skills via the `nemo agents` plugin (evaluate-suite / analyze / optimize-skills). Use when the user wants to improve an agent's skills using a Harbor or NAT eval suite, run a batch of agentic tests, analyze why evals fail, or kick off an automated skill-optimization loop. Trigger keywords - optimize skills, evaluate agent suite, analyze eval batch, skills optimizer, harbor evals.

NeMo skills improvement workflow

The nemo agents plugin exposes three commands for improving an agent that already has an eval suite (Harbor task.toml or NAT workflow.yml tasks):

Command	Purpose
nemo agents evaluate-suite	Run a directory of containerized eval tasks against the agent
nemo agents analyze	Cluster failures, surface regressions, generate hypotheses
nemo agents optimize-skills	Full loop: run evals → analyze → have Claude edit skills → verify → keep or discard

All three are NemoJob groups with run / submit / explain verbs. The spec is supplied via --spec-file <path.yml> (YAML or JSON file) or --spec '{...}' (JSON inline). When both are given, --spec-file wins.

When to recommend each command

• "How are my agent's evals doing?" → evaluate-suite run (collect data) then analyze run (interpret it).
• "Why did these evals fail?" / "What's slow?" → analyze run on an existing batch directory.
• "Improve / optimize / fix my agent" → optimize-skills run (only after confirming a .agent-improver.yml exists or asking the user for the evals / agent / skills_path values).

Self-referential example: improve NeMo itself

The Platform repo ships a canonical .agent-improver.yml at its root. Running nemo agents optimize-skills run --spec-file .agent-improver.yml from the repo root improves the skills under .agents/skills/ based on the tests/agentic-use/ Harbor evals. This supplants the older standalone tools/self_improve/ package.

export ANTHROPIC_API_KEY='<key>'
export ANTHROPIC_BASE_URL='https://inference-api.nvidia.com'
nemo agents optimize-skills run --spec-file .agent-improver.yml

When the user wants to improve another agent, they copy plugins/nemo-agents/examples/agent-improver.example.yml into their agent's repo, retarget the paths, and run the same command.

Important constraints — surface these proactively

• The optimize-skills loop spawns claude --print as a subprocess. The loop strips CLAUDE_CODE_* env markers internally, so it can be invoked from inside an active Claude Code session — evaluate-suite, analyze, and optimize-skills all work the same way whether you're in CC or not. For long unattended runs (full suite, multiple iterations), wrap in tmux so the user can detach:

  tmux new -s improve
  nemo agents optimize-skills run --spec-file .agent-improver.yml
  # detach: Ctrl-B D
  # reattach: tmux attach -t improve
  

• Variance: the default verdict is single-trial. Cold-cache effects on containerized agentic runs commonly produce 20-40% wallclock variance trial-to-trial, so single-trial verdicts can commit noise as "improvement". For verdicts you can trust, set repeats: 3 in the YAML. Median aggregation + majority-vote pass/fail.

• The eval directory is immutable. The loop's post-edit guard reverts any change inside the evals path. If the user asks to "fix the eval", redirect them: the loop improves the agent, not the evals. Eval authoring is a separate workflow.

• open_pr is opt-in. Default behaviour is "verified diff producer" — the loop produces a local branch with a clear diff for human review. Recommend open_pr: true only when the user explicitly wants automation end-to-end. With open_pr: true, the loop detects gh (GitHub) or glab (GitLab) and dispatches accordingly.

Prerequisites — check / surface in the order most likely to fail

1. claude CLI on PATH and authenticated (only needed for optimize-skills).
2. docker daemon running and harbor CLI on PATH.
3. ANTHROPIC_API_KEY and ANTHROPIC_BASE_URL exported.
4. Eval directory exists with at least one task containing task.toml or workflow.yml plus instruction.md and tests/test_outputs.py.
5. For optimize-skills: the directory at skills_path exists inside agent.

The plugin's preflight checks fail fast with actionable error messages, so reading the first error in any failure is usually enough.

Reading the output

optimize-skills prints a JSON LoopState at the end. Key fields:

• iterations[].status: improved, regressed, neutral, or error
• iterations[].hypotheses[]: what the LLM analyzer proposed (root_cause, category, proposed_fix, confidence, expected_impact)
• iterations[].changes_made: files Claude actually modified
• iterations[].eval_results_before / _after: agent-time per affected eval
• iterations[].improvement_pct: aggregate duration delta
• iterations[].branch_name: the worktree's branch (kept locally on improvement; discarded on regress/neutral)
• iterations[].mr_url: set when open_pr=True and PR/MR creation succeeded

evaluate-suite run writes report.md, report.csv, report.json, and baselines.json to the batch directory. Per-eval trial data lands at <batch_dir>/<eval-name>__trials.json when repeats > 1.

Common command shapes

# Run a single eval to debug
nemo agents evaluate-suite run --spec '{
  "evals": "tests/agentic-use",
  "agent": ".",
  "filter_glob": "auth-authorization-cli",
  "concurrency": 1
}'

# Run the full suite with variance reduction — set `repeats: 3` in
# .agent-improver.yml, then:
nemo agents evaluate-suite run --spec-file .agent-improver.yml

# Analyze a previous batch
nemo agents analyze run --spec '{
  "batch": "./runs/batch-2026-04-30__09-10-42",
  "format": "md"
}'

# Scope to one eval — edit `filter_glob` / `iterations` in the YAML
# (or copy and edit a copy), then:
nemo agents optimize-skills run --spec-file .agent-improver.yml

# Full loop with auto-PR — set `open_pr: true` in the YAML, then:
nemo agents optimize-skills run --spec-file .agent-improver.yml

Don't do

• Don't suggest editing files under the evals directory. Strategies are scoped to write only under skills_path; eval files are reverted.
• Don't recommend optimize-skills without confirming the agent has skills to improve. If the agent is stateless / has no skill files, the loop has no writable surface.