recursive-agent-improvement

name	recursive-agent-improvement
description	Use when an agent repeatedly fails at a narrow measurable subtask, the user wants a trainable specialist tool instead of more prompting, or the task needs an evaluator, reranker, classifier, detector, verifier, or small PyTorch model exposed to an agent.
version	1.0.0
author	Hasan Ali
license	MIT
metadata	{"tags":["ai-agents","recursive-improvement","specialist-tools","ml-pipeline","evaluation"],"related_skills":["tier-1-2-3-skill-system"],"related_repositories":["https://github.com/H-Ali13381/tier-1-2-3-skill-system"]}

Recursive Agent Improvement

Overview

Build durable agent capabilities by turning repeated LLM failures into narrow, measurable tools. Use the LLM for planning and orchestration; use specialist models or deterministic tools for perception, ranking, scoring, detection, extraction, and verification.

Related Skill

Use tier-1-2-3-skill-system first when deciding whether a workflow should stay text-only, become script-backed, or escalate to an ML/specialist pipeline. Use this skill after the Tier 3 decision is justified: it turns the measurable failure into a specialist tool, evaluator, reranker, classifier, detector, verifier, or training pipeline.

Repo: https://github.com/H-Ali13381/tier-1-2-3-skill-system

When to Use

Trigger this skill on any of these signals:

• Repeated LLM failure: the agent keeps making the same bounded mistake despite clearer instructions.
• Tool-over-prompt request: the user asks to add capability through a tool, model, evaluator, scorer, verifier, or skill instead of modifying base model weights.
• Specialist architecture fit: the weak spot looks like classification, ranking, detection, scoring, extraction, calibration, anomaly detection, perception, or consistency checking.
• Measurable artifact exists: inputs, expected outputs, success metrics, and failure cases can be written down before training.
• Agent-control decision: the specialist output would change what the agent does next, such as accept/reject, rerank, abstain, ask for review, retry, or route to another tool.
• Reusable pipeline need: the user wants a template for PyTorch training, hard-negative evaluation, Optuna sweeps, model cards, or JSON tool packaging.

Good user-language triggers include:

• “LLMs are bad at X; can we build a tool for it?”
• “Train a small model/classifier/reranker/evaluator for this agent weakness.”
• “Make the agent verify/score/detect this before claiming success.”
• “Package this neural model as a Hermes tool/MCP/CLI with JSON output.”
• “Create a reusable specialist training pipeline.”

Do not use this when:

• The goal is vague, like “make the LLM smarter generally.” Narrow the failure first.
• A deterministic script, schema, regex, database query, or existing library solves the problem without training.
• There is no realistic way to collect or label data, define a baseline, or evaluate held-out performance.
• The task requires broad open-ended reasoning rather than a bounded specialist judgment.

Core Pattern

1. Define the recurring agent failure.
2. Specify inputs, outputs, metrics, and baseline.
3. Build or collect data with train/val/test splits.
4. Add hard negatives or adversarial cases.
5. Train or implement the specialist.
6. Evaluate against the baseline and hard cases.
7. Package the result as a JSON-returning tool.
8. Write a skill explaining when to call it and how to interpret output.

PyTorch Template

Clone the starter pipeline:

cp -a assets/pytorch-training-pipeline ~/Documents/my-specialist-tool
cd ~/Documents/my-specialist-tool

Fill these before implementation:

1. docs/TASK_SPEC.md
2. docs/EVAL_PLAN.md
3. docs/DATASET_CARD.md
4. docs/TOOL_CONTRACT.md
5. docs/MODEL_CARD.md
6. SKILL_TEMPLATE.md -> final SKILL.md

Training Defaults

Prefer:

• real data over guesses
• explicit baselines
• hard-negative evaluation
• minimax/worst-case objectives when defaults must generalize
• quick/default mode for normal runs
• thorough/Optuna mode only when needed
• checkpoints, logs, and resumable studies
• memory-safe PyTorch defaults: num_workers=0, explicit cleanup, no weights in Optuna attrs

Available Hardware

Fill this in for the machine running training:

• CPU: replace-me CPU model and thread count
• RAM: replace-me system RAM
• GPU: replace-me GPU model and VRAM
• CUDA/accelerator stack: replace-me CUDA, ROCm, MPS, CPU-only, or other runtime

Use these values to estimate feasible model size, batch size, data volume, Optuna sweep budget, and whether a task fits local training or needs cloud hardware. Keep the committed GitHub version generic; the user's agent should replace these placeholders in a local copy.

Tool Contract

Specialist tools should return structured JSON:

{
  "success": true,
  "label": null,
  "score": null,
  "confidence": null,
  "threshold": null,
  "abstained": false,
  "reasons": [],
  "recommended_action": null,
  "metadata": {}
}

Low confidence means abstain or verify another way. Treat the specialist result as evidence, not final truth.

Error Handling

• If the failure is vague, stop and narrow it before creating data or training.
• If a deterministic script, schema, regex, database query, or existing library solves the weakness, use a Tier 2 script/tool instead of training.
• If there is no baseline, held-out split, or measurable metric, define the evaluation plan before implementation.
• If confidence is low or the input is outside the evaluated distribution, abstain and route to another verification path.
• If packaging fails, preserve the JSON contract and return a structured error instead of prose.

Common Mistakes

• Training before the task and eval spec are clear enough.
• Optimizing average score when one catastrophic subtask failure matters.
• Trusting confidence outside the evaluated distribution.
• Skipping leakage checks.
• Shipping a model without a tool contract or skill.
• Treating a narrow specialist as a general reasoning upgrade.

Verification Checklist

• Task is narrow and measurable.
• Baseline exists.
• Splits and leakage checks are documented.
• Hard cases are included.
• Specialist beats baseline on held-out data.
• Tool returns structured JSON.
• Skill documents when to use, when not to use, and how to interpret output.