optimize-skill-performance

name

optimize-skill-performance

description

Run task evals, analyze results, diagnose failures, apply targeted fixes, and re-run to verify improvements. Use when debugging evaluation scores, fixing failing or regressed criteria, analyzing why eval criteria pass or fail, reviewing eval rubric quality and redundant criteria, tracking before/after score improvements, editing plugin content to fix specific failing behaviors, or improving agent performance based on eval evidence.

Review Task Performance

You are an agent that runs task evals and automates the improvement cycle for Tessl plugins. The user has a plugin with eval results and wants to improve their scores. You handle the analysis, diagnosis, fixes, and re-run cycle.

Companion skill: If the user has no scenarios yet, point them to the setup-skill-performance skill which handles scenario generation.

Time expectations: Each re-run takes ~10–15 minutes per scenario per agent (each scenario runs baseline + with-context). Budget accordingly — if you have 3 scenarios, expect ~30–45 minutes per iteration.

Run labels

Every tessl eval run invocation MUST include --label <run-label> so the run is identifiable in tessl eval list. The label is a short, human-readable description of what the run is about — not a structured ID.

Compose <run-label> from whatever helps you recognise the run later when scanning the list. Typical ingredients:

• Eval type — e.g. activation, baseline, initial evals, verification
• What was being tested or changed — e.g. description rewrite, plan-solution fixes, clean scenario
• Model in parens, when comparing models — e.g. (haiku-4-5), (sonnet-4-6)
• Version when iterating — e.g. v0.5.0, v4

Examples:

• repro-clean-scenario
• task-prep v0.3.0 baseline
• task-prep v0.5.0 plan-solution fixes
• v4-final-verification
• skill-insights activation (haiku-4-5)
• skill-insights initial evals (haiku-4-5)

Keep it concise — what the run was about should be obvious without opening it.

---

Phase 0: Detect Starting Point

Before diving into analysis, determine what state the user is in.

0.1 Decide which eval types the user needs

Rather than routing to a single path (activation OR scored), reason about which combination fits the user's goal:

• Multi-skill plugin, no evals yet → run activation first, then scored evals only where routing is clean. This avoids wasting scored-eval time on scenarios that route to the wrong skill.
• Multi-skill plugin, user wants the full picture fast → run both activation and scored evals in parallel (two tessl eval run commands).
• Multi-skill plugin, scored evals already exist → run activation to check routing, then cross-reference with existing scored results.

To determine the plugin's skill count:

ls skills/*/SKILL.md 2>/dev/null | wc -l

To check for existing eval results:

tessl eval view --last --json 2>&1

If results exist, check whether the run skipped forced context activation:

tessl eval view --last --json | jq '.data.attributes.skipForcedContextActivation'

• false or field absent → proceed to Phase 1 (score-based analysis). If multi-skill, suggest also running activation to check routing.
• true → proceed to Phase 0.4 (activation analysis). Recommend a scored eval run next for scenarios where routing is clean.

0.2 If no results, check for scenarios on disk

Look for an evals/ directory:

ls evals/*/task.md 2>/dev/null

If scenarios exist on disk but no eval results → before running, quickly check whether the scenarios are missing fixtures or setup scripts the plugin clearly needs. Apply the signals from setup-skill-performance/references/phase3-fixtures-and-setup.md §"Detection rules" to each scenario's task.md + criteria.json + the plugin's SKILL.md.

If any scenario fires a signal but has no fixtures (or no setup / setup.sh when warranted), warn the user:

"Scenario <slug> looks like it needs a [fixture / setup script] but scenario.json doesn't declare one. Running the eval as-is will start in an empty workdir and likely fail. Want to jump back to setup-skill-performance Phase 3 to generate it, or run anyway?"

If they choose to generate, hand off to that procedure. If they choose to run anyway, proceed.

Then tell the user:

"I found scenarios on disk but no eval results yet. Want me to run evals now?"

If yes, run:

tessl eval run <path/to/plugin> --label <run-label>

Then poll for completion (see Phase 4.4) and proceed to Phase 1.

0.3 If no scenarios exist

Tell the user no scenarios were found and offer to invoke setup-skill-performance (which generates scenarios, downloads them, and runs baseline + with-context evals).

0.4 Activation results detected

The last eval used --skip-forced-context-activation --skip-scoring. This shows which skills fired per scenario — there are no baseline or with-context scores, so bucket classification does not apply.

1. Skill coverage summary: Report which skills in the plugin never fired across any scenario (see phase5-view-results.md §5.1b "Skill coverage summary").

2. Zero-activation analysis: For each scenario showing – (no skill activated), check if a score-based eval exists for the same scenario and use the delta to determine if it's a routing gap or an out-of-scope task (see phase5-view-results.md §5.1b "Zero-activation analysis").

3. Auto-suggest description rewrites: For any routing gap identified, read the skill's description and the scenario's task phrasing, then propose a minimal description edit (see phase5-view-results.md §5.1b "Auto-suggested description rewrites"). Apply approved rewrites before proceeding.

To continue with score-based optimization, run a full eval:

tessl eval run <path/to/plugin> --agent=claude:claude-sonnet-4-6 --label <run-label>

Then return to Phase 0.1 once complete.

---

Phase 1: Analyze Results

This phase applies to scored runs only. If the last eval used --skip-forced-context-activation --skip-scoring, go back to Phase 0.4.

1.1 Get the latest eval results

tessl eval view --last --json

The eval view gives you the detailed per-criterion scores.

Parse the JSON output. For each scenario, extract:

• Scenario name
• Each criterion's name, max score, baseline score, and with-context score
• The aggregate baseline → with-context delta for each scenario

1.2 Classify every criterion into one of four buckets

Bucket A — Working well (no action needed)

• With-context score is >= 80% of max AND significantly higher than baseline
• These are your plugin's strengths. Leave them alone.

Bucket B — Plugin gap (needs a fix)

• With-context score is < 80% of max AND baseline is also low
• The agent doesn't know this without your help, and your plugin isn't teaching it well enough yet.
• This is where fixes have the highest impact.

Bucket C — Redundant (consider removing)

• Baseline score is already >= 80% of max without any plugin context
• The agent already knows this. Your plugin isn't adding value for this criterion.
• Flag these to the user — the criterion may be unnecessary, or the task is too easy.

Bucket D — Regression (needs investigation)

• With-context score is LOWER than baseline
• Your plugin is actively confusing the agent on this point. This is the highest priority to fix.

1.3 Present the analysis

Show the user a summary table:

Eval Analysis for: <plugin-name>

Scenario: <name> (baseline: XX% -> with-plugin: YY%)

  Bucket B — Plugin Gaps (fix these):
    - "Exponential backoff" — 0/9 (baseline also 0/9)
      Diagnosis: Plugin never mentions backoff timing pattern
      File to fix: skills/onboard/SKILL.md
      Suggested fix: Add "retry with exponential backoff: 1s, 2s, 4s" to Step 1

  Bucket D — Regressions (investigate):
    - "Auth URL capture" — 4/8 (baseline was 6/8)
      Diagnosis: Recent edit may have muddied the auth instructions
      Files to check: skills/onboard/SKILL.md, rules/onboarding-guide.md

  Bucket C — Redundant:
    - "Step-by-step structure" — baseline 10/10, plugin 10/10
      Note: Agents already do this naturally. Consider removing this criterion.

  Bucket A — Working well (5 criteria): [collapsed]

Ask the user: "Want me to fix the Bucket B and D items? I'll show you each change before committing."

---

Phase 2: Diagnose Root Causes

For each Bucket B and Bucket D criterion:

2.1 Read the criterion details

Open the scenario's criteria.json to understand exactly what the rubric checks for.

2.2 Read the relevant plugin files

Read:

• skills/*/SKILL.md — skill instructions
• rules/*.md — rules loaded into agent context
• docs/*.md — reference documentation

2.3 Find the gap

For each failing criterion, determine:

• What the rubric wants: The specific behavior or content the judge is looking for
• What the plugin says: What guidance the plugin files currently provide (or don't)
• The gap: What's missing, vague, or contradictory

2.4 Check for contradictions

Scan across ALL plugin files for statements that contradict each other. Common patterns:

• Skill says "retry 3 times" but rules say "retry with backoff" without specifying count
• Docs describe a different flow order than the skill's steps
• Rules say something is optional but the skill treats it as required

Flag any contradictions to the user even if they aren't related to failing criteria — they can cause future regressions.

---

Phase 3: Apply Fixes

For each fix, follow this sequence:

3.1 Propose the change

Show the user:

• Which file you'll edit
• What you'll add/change (the actual text)
• Why this should improve the specific criterion

3.2 Apply the edit

Make the change to the file. Keep edits minimal and targeted — don't rewrite sections that are already working.

Rules for good fixes:

• Be explicit. If the criterion wants "exponential backoff: 1s, 2s, 4s", write exactly that. Don't write "use appropriate backoff."
• Match the rubric's language. If criteria.json checks for the phrase "safe and reversible", use those exact words in your plugin.
• Don't bloat the plugin. Add the minimum needed. Every token of context costs attention.
• Preserve what works. Don't restructure sections that score well in Bucket A.

3.3 Lint after each fix

tessl plugin lint <plugin-path>

Check that the plugin is still valid and token costs haven't ballooned. If front-loaded tokens increased significantly, consider moving content to docs (on-demand) instead of rules (always loaded).

3.4 Handle Bucket C (redundant criteria)

For criteria where baseline is already high, ask the user:

"The criterion '' scores % even without your plugin. Options:

1. Remove it from criteria.json (agents already know this)
2. Make the task harder so it actually tests your plugin's value
3. Keep it as a sanity check

What do you prefer?"

If the user chooses to remove, edit the scenario's criteria.json and redistribute the weight to remaining criteria.

3.5 Handle Bucket D (regressions)

For regressions, the fix often isn't adding content — it's clarifying or removing content that confused the agent. Check for:

• Ambiguous instructions that could be interpreted multiple ways
• Contradictory statements between files
• Overly verbose sections where the key point gets buried
• Recent additions that conflict with existing guidance

Show the user the contradiction or ambiguity, then propose a clarification.

---

Phase 4: Re-run and Verify

4.1 Summarize all changes

Before committing, show the user a summary:

Changes made:
  1. skills/onboard/SKILL.md — Added exponential backoff timing (1s, 2s, 4s) to Step 1
  2. rules/onboarding-guide.md — Clarified that repo eval is always optional
  3. evals/error-recovery/criteria.json — Removed redundant "network retry" criterion

Expected impact:
  - "Exponential backoff" should go from 0/9 -> 9/9
  - "Repo eval is optional" should go from 0/8 -> 8/8
  - Regression on "Auth URL capture" should resolve (removed contradictory instruction)

Commit and re-run evals?

4.2 Commit (with user approval)

git add <files-you-changed>
git commit -m "Improve plugin: <brief description of fixes>"

Only stage the files you actually changed. Don't stage unrelated files.

4.3 Re-run evals

tessl eval run <path/to/plugin> --label <run-label>

If the eval doesn't pick up your changes, make sure you've committed them first.

4.4 Poll for completion

tessl eval list --mine --limit 1

Wait until status shows completed. If status shows failed, run tessl eval retry <id>.

Otherwise, get results:

tessl eval view --last

4.5 Report the before/after

Show the user:

Before -> After:

  CLI setup automation:        87% -> 96%  (+9)
  Skill scaffolding:           88% -> 88%  (no change)
  Output file generation:     100% -> 100% (no change)
  Error recovery:              91% -> 99%  (+8)
  User interaction:           100% -> 100% (no change)

  Average:                     93% -> 97%  (+4)

Remaining gaps:
  - "Exponential backoff" still at 0/9 — may need a different approach

If gaps remain, ask: "Want me to take another pass at the remaining gaps?"

---

Phase 5: Scenario Quality Review (Bonus)

If the user asks, or if you notice issues during Phase 2, review the scenarios themselves:

5.1 Check task realism

Read each task.md and flag:

• Tasks that are unrealistically specific (testing memorization, not understanding)
• Tasks that are too vague to produce consistent results
• Tasks that don't match real-world use cases for this plugin

5.2 Check criteria quality

Read each criteria.json and flag:

• Criteria with equal weights (should important ones weigh more?)
• Criteria that are too strict (exact string matching when intent matching would be better)
• Missing criteria for important behaviors the plugin teaches
• Criteria that test the agent's general ability, not the plugin's value

5.3 Check for external service coverage

If the skill involves external service calls — APIs, databases, MCPs, third-party tools — check whether any scenarios exercise those paths. If not, suggest adding at least one scenario that tests the integration.

Remind the user that tessl eval run --env-file <path> injects secrets into the sandbox, so external calls can be exercised in evals without hardcoding credentials.

5.4 Suggest improvements

Propose specific edits to task.md or criteria.json files. Show diffs and explain why.

---

When to stop

Stop iterating when:

• All with-context scores are >= 85% and no regressions exist
• Remaining low scores are on criteria the user has reviewed and accepted
• The user says they're satisfied
• Further improvements would require restructuring the plugin significantly (suggest this as a separate effort)