| name | dev-skill-load-test |
| description | Use when testing skill loading effectiveness, verifying subagent compliance with skill requirements, or conducting batch skill validation tests. Triggers when you need to validate that skills load correctly across multiple test cases. |
dev-skill-load-test
Batch-test whether a target skill is actually loaded by subagents under realistic task prompts.
Core insight: Subagents often load a skill without explicitly saying so. If AGENTS.md enforces skill call logs, read the log directly from the task output. Otherwise, resume the session and ask.
Cost insight: Each subagent session that runs to completion (full design, full implementation) wastes tokens and time. The test only needs the agent to reach the skill-loading decision point — not to finish the task.
Workflow
digraph {
rankdir=TB;
node [shape=box];
prerequisite [label="0. Prerequisite check"];
design [label="1. Design test matrix"];
baseline [label="2. Baseline run (A)"];
experiment [label="3. Experiment run (B)"];
probe [label="4. Probe every session"];
compare [label="5. Compare A vs B"];
report [label="6. Write report"];
prerequisite -> design -> baseline -> experiment -> probe -> compare -> report;
}
Step 0 — Prerequisite check
Before any test, verify the target skill is visible to the system:
Ask a subagent: "List every skill name you can see in available_skills."
If the skill is missing, stop — no amount of testing will work. New skills require a session restart to appear in available_skills.
Step 1 — Design test matrix
Pick N task prompts (recommend N >= 10) that should trigger the target skill.
Guidelines:
- Cover a range of difficulty / specificity (simple, medium, complex, edge-case).
- Each prompt should be a natural user request, not a meta-instruction about skills.
- Define
expected_skills: the skill name(s) the agent should load for each prompt.
- Add an early-stop constraint to every prompt (see below) — do not let agents run to completion.
Early-stop constraint
The skill-loading decision happens in the first 1-2 replies (brainstorming phase). Everything after that (design, implementation, verification) is irrelevant to the test and wastes tokens. Append a stopping condition to every prompt:
在提出第一个澄清问题后停下来,等待用户回复。不要继续执行后续步骤。
Or in English:
Stop after asking your first clarifying question. Do not proceed further until the user replies.
This ensures the agent loads all relevant skills (which happens before or during the first reply), then halts — giving you a clean observable result without running the full task.
Example (target skill = rtl-coding-style):
| # | prompt | expected_skills |
|---|
| 1 | "Design a sync FIFO. Stop after asking your first clarifying question." | rtl-coding-style, design-microarch |
| 2 | "Design an edge detector. Stop after asking your first clarifying question." | rtl-coding-style, design-microarch |
| … | … | … |
Step 2 — Baseline run (A group)
Run N prompts without the modification you want to test (original skill version, or target skill absent).
Launch all N prompts in parallel in a single message (up to 100 concurrent task() calls). Collect all task_ids before proceeding.
Important: Do NOT ask about skills in the task prompt — let the agent behave naturally.
Step 3 — Experiment run (B group)
Apply the modification (e.g., new skill version, changed description, added flow step), then repeat the same N prompts.
Step 4 — Collect skill call logs
If AGENTS.md enforces skill call logs (RULE #2):
Each task output already contains a --- skill call log --- block at the end. Read it directly — no probe needed. Extract the cumulative list and check whether the target skill appears.
累计调用:
#1. brainstorming
#2. rtl-coding-style
#3. design-microarch
← design-workflow missing → FAIL
The log also reveals call order, which is critical for diagnosing why a skill was missed (e.g., brainstorming loaded in the same batch as other skills, so its instructions arrived too late to influence that batch).
If skill call logs are not available:
Resume each session and ask directly:
task(
task_id = "<session_id>",
prompt = "请直接回答,不要执行任何操作:\n1. 你加载了哪些skill?列出完整名称。\n2. 是否加载了 <target_skill>?\n3. 如果没有,为什么?"
)
Why resume instead of a new session: The agent retains its tool-call history. It can truthfully report what it loaded. A new session has no memory.
Probe all sessions in parallel in a single message, same as execution.
Step 5 — Compare A vs B
Fill in the comparison table:
| # | prompt | A: loaded? | B: loaded? |
|---|--------|-----------|-----------|
| 1 | … | no | yes |
| 2 | … | no | yes |
Compute:
load_rate(group) = count(loaded) / N
lift = load_rate(B) - load_rate(A)
Step 6 — Write report
Save to test_results/skill_load_test_<skill_name>_<date>.md:
# Skill Load Test: <target_skill>
**Date**: … **Cases**: N
## Modification tested
<one-sentence description of what changed between A and B>
## Results
| Metric | A (baseline) | B (experiment) |
|--------|-------------|----------------|
| Load rate | X% | Y% |
| Lift | — | +Z% |
## Per-case detail
| # | prompt | A | B |
…
## Conclusion
…
Lessons learned (from 74 real tests)
These are general principles, not specific to any one skill:
-
Visibility is prerequisite — If a newly created skill is not in available_skills, load rate is always 0%. The system reads the skill list at startup; a session restart is required after adding skills.
-
Passive observation underestimates load rate — In one 20-case batch, only 60% of agents explicitly mentioned the loaded skill. After probing, actual load rate was 100%. Always probe.
-
Description wording matters, but only when visible — Changing a skill description from passive ("coding style guide") to imperative ("CRITICAL: Load immediately for ANY RTL task") had zero effect when the skill wasn't in available_skills, but worked once visible.
-
LLMs are process-oriented — Agents decide to load skills based on "what am I doing right now?" not "what domain is this task in?". A skill positioned as "design-phase guidance" loads earlier than one positioned as "coding-phase rules".
-
Upstream flow skills have leverage — Modifying a flow skill (e.g., brainstorming) to include a "load domain skills" step is more effective than relying on the target skill's own description to self-advocate.
-
One variable at a time — When multiple changes are applied simultaneously (new skill + modified description + flow change), a positive result doesn't tell you which change mattered. Prefer isolated A/B tests.
Common mistakes
| Mistake | Why it's wrong | Fix |
|---|
| Judge by response text only | Agents load skills silently ~40% of the time | Read skill call log, or probe via resumed session |
| Let agents run to completion | Wastes tokens; skill loading happens in first reply | Add early-stop constraint to every prompt |
| Test 1-2 cases and conclude | Insufficient sample size | Minimum 10, recommend 20 |
| Skip prerequisite check | Invisible skills always show 0% | Verify visibility first |
| Change multiple things between A and B | Can't attribute causation | Isolate one variable |
| Forget to save task_ids | Can't probe later | Record every task_id immediately |
| Run all cases sequentially | Wastes time | Launch all in parallel (up to 100) |