# AdaSkill
**From Multi-Agent to Single-Agent: When Is Skill Distillation Beneficial?**
> Binyan Xu¹², Dong Fang²†, Haitao Li², Kehuan Zhang¹
>
> ¹ The Chinese University of Hong Kong · ² LIGHTSPEED, Tencent
>
> *NeurIPS 2026 (under review)*
>
> [Paper](https://arxiv.org/abs/2604.01608)
---
## Overview
Multi-agent systems (MAS) decompose tasks into specialized sub-agents, but incur coordination overhead, context fragmentation, and brittle phase ordering. Distilling a MAS into a single-agent skill can bypass these costs — but when does it actually help?
We introduce **Metric Freedom** (*F*), the first *a priori*, metric-level predictor of skill utility. *F* measures how tightly output variation couples with score variation (via a Mantel test). Guided by *F*, we propose a **two-stage adaptive distillation framework**:
- **Stage 1 (Adaptive Extraction)**: Selectively distills tools and knowledge; discards restrictive pipeline structures on "free" metrics (*F* > 0.65).
- **Stage 2 (Targeted Refinement)**: Optional 4-agent iterative optimization loop, activated only for "rigid" metrics (*F* < 0.6).
**Key results** across 4 tasks, 11 datasets, 6 metrics:
- *F* strongly predicts skill lift direction (*r* = −0.85, *p* < 0.0001)
- Our adaptive agent matches or exceeds original MAS at **1.4–8× lower cost** and up to **15× lower latency**
- Identical agent trajectories yield *opposite* skill lifts under rigid vs. free metrics
---
## Repository Structure
```
AdaSkill/
├── benchmarks/
│ ├── freedom_pipeline/ # Metric Freedom (F) computation pipeline
│ │ ├── core.py # Mantel test, F_out, F_trace computation
│ │ ├── runner.py # Benchmark runner with approach sampling
│ │ ├── adapters/ # Task-specific adapters (T2SQL, CE, CD, FE)
│ │ └── configs/ # YAML configs per task
│ ├── causal_discovery/ # Causal Discovery benchmark runner
│ ├── causal_estimate/ # Causal Estimation benchmark runner
│ ├── text2sql/ # Text-to-SQL benchmark runner (BIRD, Spider)
│ ├── feature_engineering/ # Feature Engineering benchmark runner
│ └── skill_utils.py # Shared skill loading/unloading utilities
├── skills/
│ ├── adaptive_text2sql_bird/ # Adaptive skill for Text-to-SQL / BIRD (based on APEX-SQL)
│ ├── adaptive_text2sql_spider/ # Adaptive skill for Text-to-SQL / Spider (based on APEX-SQL)
│ ├── adaptive_causal_estimate/ # Adaptive skill for Causal Estimation (based on CAIS)
│ ├── adaptive_causal_discovery/ # Adaptive skill for Causal Discovery (based on MATMCD)
│ ├── adaptive_feature_engineering/ # Adaptive skill for Feature Engineering (based on FELA)
│ ├── adaptive_mas_converter/ # Meta-skill: adaptive MAS → skill converter
│ └── mas-compiler_baseline/ # Meta-skill: MAS-Compiler baseline (arXiv 2601.04748)
├── skill_converter/ # Stage 1: MAS → Skill converter
├── skill_iterator/ # Stage 2: Iterative skill refinement
├── scripts/
│ └── download_data.py # Dataset download helper
└── requirements.txt
```
---
## Installation
```bash
git clone https://github.com/<YOUR_ORG>/AdaSkill.git
cd AdaSkill
pip install -r requirements.txt
```
You also need a Claude Code CLI installed and configured with your API key:
```bash
# Install Claude Code (https://claude.ai/code)
# Set your API key according to your provider's instructions
```
---
## Data Setup
### Quick Start
```bash
# Auto-download Causal Discovery datasets (Sachs, Asia, Child)
python scripts/download_data.py --task causal_discovery
# See instructions for all datasets
python scripts/download_data.py --task all
```
See [DATASET_SETUP.md](DATASET_SETUP.md) for detailed instructions.
### Dataset Summary
| Task | Dataset | Download | Auto-download |
|------|---------|----------|---------------|
| Causal Discovery | Sachs, Asia, Child | [bnlearn](https://www.bnlearn.com/) | ✓ |
| Causal Discovery | AutoMPG, DWDClimate | [MATMCD](https://github.com/mas-takayama/LLM-and-SCD) | ✗ |
| Causal Estimation | QRData / Synthetic / Real | [CauSciBench](https://github.com/causalNLP/causal-agent/tree/main/data) | ✗ |
| Text-to-SQL | BIRD | [BIRD Benchmark](https://bird-bench.github.io/) | ✗ (direct link) |
| Text-to-SQL | Spider | [Spider](https://yale-lily.github.io/spider) | ✗ |
| Feature Engineering | Taobao | [Tianchi](https://tianchi.aliyun.com/dataset/56) | ✗ (registration) |
| Feature Engineering | Dia | [Kaggle](https://www.kaggle.com/datasets/mathchi/diabetes-data-set) | ✗ |
---
## Quick Start
### Step 1: Compute Metric Freedom
```bash
# Run n=10 raw-agent baseline runs on a task
python benchmarks/freedom_pipeline/runner.py \
--task causal_estimate \
--config benchmarks/freedom_pipeline/configs/causal_estimate.yaml \
--n-approaches 10 \
--output-dir results/freedom/
# Compute F_out and F_trace
python benchmarks/freedom_pipeline/core.py \
--results-dir results/freedom/causal_estimate/
```
### Step 2: Run a Benchmark with Adaptive Skill
First, install the skill to your Claude Code skills directory:
```bash
cp -r skills/adaptive_causal_estimate ~/.claude/skills/adaptive_causal_estimate
```
Then run the benchmark:
```bash
# Causal Estimation with adaptive skill
python benchmarks/causal_estimate/run_claude_direct.py \
--dataset real \
--skill adaptive_causal_estimate
# Causal Discovery with adaptive skill
python benchmarks/causal_discovery/runner/run_causal_discovery.py \
--datasets Sachs Asia Child \
--skill adaptive_causal_discovery
# Text-to-SQL / BIRD with adaptive skill
python benchmarks/text2sql/run.py \
--dataset bird-147 \
--skill adaptive_text2sql_bird
# Text-to-SQL / Spider with adaptive skill
python benchmarks/text2sql/run.py \
--dataset spider \
--skill adaptive_text2sql_spider
# Feature Engineering with adaptive skill
python benchmarks/feature_engineering/run.py \
--dataset taobao \
--skill adaptive_feature_engineering
```
### Step 3: (Optional) Stage 2 — Iterative Skill Refinement
Recommended only for low-*F* metrics (*F* < 0.6):
```bash
python skill_iterator/run.py \
--benchmark causal_estimate \
--base-skill skills/adaptive_causal_estimate \
--train-cases benchmarks/causal_estimate/data/train.json \
--val-cases benchmarks/causal_estimate/data/val.json
```
### Step 4: (Optional) Convert Your Own MAS
```bash
python skill_converter/convert.py \
--codebase /path/to/your/mas/repo \
--skill-name my_adaptive_skill \
--output-dir skills/
```
---
## Skill Structure
Each skill in `skills/` contains:
- `SKILL.md` — Instructions and domain knowledge for the Claude Code agent
- `tools/` — Python modules callable from Bash within the agent session
The skill is loaded by copying it to `~/.claude/skills/<skill_name>/` before a benchmark run (handled automatically by `skill_utils.py`).
---
## Reproducing Paper Results
| Experiment | Command |
|-----------|---------|
| Figure 1 (F vs. lift correlation) | `python benchmarks/freedom_pipeline/runner.py --task all` |
| Table 2 (T2SQL) | `python benchmarks/text2sql/run.py --dataset bird-147 --methods raw compiled adaptive` |
| Table 3 (CE) | `python benchmarks/causal_estimate/run_claude_direct.py --dataset all --methods raw compiled adaptive` |
| Table 4 (CD) | `python benchmarks/causal_discovery/runner/run_causal_discovery.py --datasets all` |
| Table 5 (FE) | `python benchmarks/feature_engineering/run.py --dataset taobao --n-seeds 3` |
---
## Citation
```bibtex
@article{xu2026adaptive,
title = {From Multi-Agent to Single-Agent: When Is Skill Distillation Beneficial?},
author = {Xu, Binyan and Fang, Dong and Li, Haitao and Zhang, Kehuan},
journal = {Advances in Neural Information Processing Systems},
year = {2026},
url = {https://arxiv.org/abs/2604.01608}
}
```
---
## License
MIT License. See [LICENSE](LICENSE).
## Acknowledgements
We thank the authors of [APEX-SQL](https://arxiv.org/abs/2602.16720), [CAIS](https://arxiv.org/abs/2501.12321), [MATMCD](https://arxiv.org/abs/2412.13667), and [FELA](https://arxiv.org/abs/2510.25223) for their open-source MAS baselines, which form the foundation for our distillation experiments.