skill-os-learning-skill-curation-self-evolving-agents

name	skill-os-learning-skill-curation-self-evolving-agents
description	Use when implementing agent skill curation, self-evolving agent systems, or RL-based skill management. SkillOS framework for autonomous skill library optimization.
version	1.0.0
author	Hermes Agent
license	MIT
platforms	["linux","macos","windows"]
metadata	{"hermes":{"tags":["agent","skill-curation","self-evolving","rl","skill-management","agent-architecture"],"related_skills":["hermes-agent","claude-code"]}}

SkillOS: Learning Skill Curation for Self-Evolving Agents

Overview

SkillOS (arXiv:2605.06614) proposes a skill curation learning framework that enables agents to autonomously learn and optimize their skill library from experience, achieving self-evolution. The core insight is that agent performance improves not just from better models, but from better skill management over time.

The framework formalizes skill curation as a reinforcement learning problem, where a separate Skill Curator module learns to manage (insert/update/delete) skills in a SkillRepo based on delayed downstream feedback from related tasks.

Key insight: Training a small curator (8B parameters) with RL can surpass using a frontier model (Gemini-2.5-Pro) directly for skill decisions.

When to Use

Appropriate for:

• Building agents that improve over time through experience
• Implementing skill management systems with delayed feedback
• Designing multi-task agent curricula where skills transfer across tasks
• Creating self-evolving systems that avoid skill library bloat

Not appropriate for:

• Single-shot tasks with no skill reuse potential
• Situations requiring immediate skill deployment (use static skill libraries instead)
• When all tasks are independent with no transferable knowledge

Core Architecture

SkillOS uses a three-component modular design:

┌─────────────────────────────────────────────────────────┐
│                     Skill Curator (trainable)            │
│  Observes execution trajectories → decides insert/update/delete │
└─────────────────────────────────────────────────────────┘
         ↑                    ↑                    ↓
         │                    │                    │
┌────────┴────────┐  ┌───────┴────────┐  ┌───────┴────────┐
│  SkillRepo       │  │ Agent Executor │  │  Environment   │
│  (Markdown)      │←─┤ (frozen)       │→ │  (tasks)        │
└─────────────────┘  └────────────────┘  └────────────────┘

1. SkillRepo (Skill Repository)

External skill library stored as Markdown files. Each skill contains:

• YAML frontmatter: name, use scenario description
• Markdown body: workflow, constraints, reusable heuristics

Format follows Anthropic Skills SKILL.md convention — skills are readable by both agents and humans.

2. Agent Executor (Frozen)

Given a task, the executor:

1. Retrieves relevant skills via BM25
2. Combines task context with retrieved skills
3. Executes actions

The executor is frozen (not trained) — all learning happens in the curator.

3. Skill Curator (Trainable)

Observes execution trajectories and decides operations:

• insert_skill — add new skill from successful trajectory
• update_skill — refine existing skill based on new evidence
• delete_skill — remove obsolete/incorrect skills

Key Innovations

Task Grouping for Delayed Feedback

Traditional RL struggles with skill curation because feedback is delayed and indirect. SkillOS solves this by grouping related tasks — early tasks produce skills that later tasks validate.

Task Group: [Task A] → [Task B] → [Task C]
                ↓
           Skill learned from A is validated/refined by B and C

Grouping strategy (from paper):

• Use Gemini-2.5-Pro to generate task tags (topic, common pitfalls)
• Group by tag similarity
• Tasks within the same group share non-trivial skill dependencies

Composite Reward Design

The reward function combines four signals:

Reward Component	Meaning	Weight
r_task	Task result reward	λ_f=1.0
r_fc	Function call validity (format + execution)	λ_u=0.1
r_cnt	Content quality (abstraction, reusability, actionability, faithfulness)	λ_c=0.05
r_comp	Compression (encourage distillation over verbatim storage)	variable

GRPO Optimization

Grouped Reward Policy Optimization (GRPO) trains the curator:

• Computes advantage across multiple rollouts within a task group
• Advantage signal distributed uniformly across all tokens
• Key finding: Dropping the KL divergence term actually helps exploration

Skill Content Quality Dimensions

External judge (Qwen3-32B) evaluates skills on four dimensions:

1. ABSTRACTION — Captures generalizable patterns, not specific IDs/numbers

   • Replace specific values with variables/generic concepts

2. REUSABILITY — Atomic and modular, describes one coherent capability

   • Should be triggerable by future related tasks, not bundled steps

3. ACTIONABILITY — Markdown body provides concrete guidance

   • Includes workflow, conditions, When NOT to Use
   • Executor can act directly, not vague suggestions

4. FAITHFULNESS — All claims supported by trajectory

   • No fabricated facts, tools, or environment behaviors

Experimental Results

ALFWorld Multi-Step Interactive Tasks

Executor Configuration	Method	Success Rate	Steps
Qwen3-8B	No Memory	47.2%	21.1
Qwen3-8B	ReasoningBank	55.7%	20.1
Qwen3-8B	SkillOS	61.2%	18.9
Gemini-2.5-Pro	No Memory	66.4%	17.7
Gemini-2.5-Pro	SkillOS	80.2%	14.8

Key findings:

• RL-trained 8B curator surpasses frontier model (Gemini-2.5-Pro) used directly
• Both efficiency (steps) and effectiveness (success rate) improve simultaneously
• Benefits scale with executor capability (+9.5% with Gemini-2.5-Pro)

Implementation Guide

Phase 1: SkillRepo Setup

Create a Markdown-based skill repository:

---
name: example-skill
description: "Use when [trigger scenario]"
---

# Example Skill

## When to Use
Brief trigger description.

## Workflow
Step-by-step actions.

## Constraints
What to avoid.

## When NOT to Use
Counter-indications.

Phase 2: Executor Implementation

The executor retrieves skills using BM25:

# Pseudocode for skill retrieval
def retrieve_skills(task_description, skill_repo, top_k=5):
    query = f"{task_description}"
    retrieved = bm25.retrieve(query, skill_repo.all_skills(), k=top_k)
    return retrieved

Then combine with task context:

def execute_task(task, retrieved_skills):
    context = {
        "task": task,
        "skills": [skill.markdown for skill in retrieved_skills]
    }
    return executor.run(context)

Phase 3: Curator Training

Start with SkillOS-base (random curator) before RL training:

# Training configuration (from paper)
config = {
    "learning_rate": 1e-6,
    "batch_size": 32,
    "group_size": 8,  # tasks per group
    "num_gpu": 16,    # H100 GPUs
    "training_steps": 30000,  # ~3 days
}

Training progression:

1. SkillOS-base: Initial curator without RL, establishes format/flow
2. GRPO training: Learn from delayed downstream feedback
3. Optional KL fine-tuning: For more conservative behavior

Task Grouping Strategy

Implement task grouping to enable delayed feedback learning:

def group_tasks(tasks):
    # Use LLM to generate task tags
    tagged_tasks = []
    for task in tasks:
        tags = llm.generate_tags(task)  # {topic, common_pitfalls}
        tagged_tasks.append({**task, "tags": tags})
    
    # Group by similarity
    groups = cluster_by_similarity(tagged_tasks, metric="cosine")
    return groups

Alternative approaches:

• Task type metadata (e.g., ALFWorld task categories)
• Domain knowledge graphs
• Embedding similarity clustering

Composite Reward Computation

def compute_reward(trajectory, task_group, skill_repo):
    # r_task: downstream task success from same group
    r_task = compute_downstream_success(trajectory, task_group)
    
    # r_fc: function call validity
    r_fc = 1.0 if curator_action.format_valid else 0.0
    
    # r_cnt: content quality via external judge
    r_cnt = judge.evaluate(skill_content, dimensions=[
        "abstraction", "reusability", "actionability", "faithfulness"
    ])
    
    # r_comp: compression ratio
    r_comp = compression_ratio(trajectory, skill_content)
    
    return (1.0 * r_task) + (0.1 * r_fc) + (0.05 * r_cnt) + r_comp

Engineering Considerations

Frozen Executor Strategy

Keeping the executor frozen provides:

• Training stability (no co-adaptation issues)
• Isolation of skill management learning
• Ability to swap executors without retraining curator

Delete Operations are Essential

Most memory systems only handle insert, but delete is critical:

• Prevents invalid/obsolete skills from polluting the repo
• Enables skill evolution as environment changes
• Requires long-horizon feedback to learn correctly

Version Control for SkillRepo

Treat skills as code:

• Store in Git for complete audit trail
• Enable rollback on corrupted skills
• Support collaborative skill development
• Track skill lineage across curator updates

Curator Model Size

Key insight from paper: 8B curator > frontier model directly

Selection criteria:

• Must understand complex trajectories
• Must generate high-quality skill content
• Can be smaller than executor (specialized vs. general)
• Language understanding capacity matters most

Common Pitfalls

1. Ignoring delete operations — Focus only on insert leads to skill bloat

2. Short-horizon feedback only — Without task grouping, curator can't learn long-term impact

3. Training executor and curator together — Causes co-adaptation; keep executor frozen

4. Verbatim trajectory storage — Skills should abstract, not copy; use compression reward

5. KL term too restrictive early — Prevents exploration of destructive but necessary operations

6. Poor task grouping — Random task grouping provides weak learning signals; invest in grouping strategy

Verification Checklist

• SkillRepo follows Anthropic SKILL.md format with YAML frontmatter
• Executor uses BM25 retrieval (not semantic search) for consistency
• Task grouping produces groups with non-trivial skill dependencies
• Reward function includes all four components (r_task, r_fc, r_cnt, r_comp)
• Curator trained separately from frozen executor
• Training uses GRPO with advantage computed across task groups
• KL divergence initially dropped; added only for fine-tuning
• Delete operations enabled (not just insert/update)
• Skills version-controlled in Git for auditability
• Skill quality evaluated on four dimensions (abstraction, reusability, actionability, faithfulness)

One-Shot Recipe: Minimal SkillOS Implementation

# 1. Create skill repository directory
mkdir -p skill_repo/skills

# 2. Initialize with seed skills (optional)
echo '# Seed Skill\n...' > skill_repo/skills/seed.md

# 3. Implement executor with BM25 retrieval
# (see Phase 2 above)

# 4. Implement curator with insert/update/delete actions
# (see Phase 3 above)

# 5. Generate task groups with LLM tagging
# (see Task Grouping above)

# 6. Train with GRPO
python train_curator.py --config skillos_config.yaml

# 7. Evaluate on held-out tasks
python eval.py --curator trained_curator --executor frozen_executor

Further Reading

• Paper: arXiv:2605.06614
• Related: Self-Evolving Agents survey, Memento-Skills (agent external memory)