메뉴
Skill testing methodology - run scenarios without skill (RED), observe failures, write skill (GREEN), close loopholes (REFACTOR).
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
| name | marsai:testing-skills-with-subagents |
| description | Skill testing methodology - run scenarios without skill (RED), observe failures, write skill (GREEN), close loopholes (REFACTOR). |
| trigger | - Before deploying a new skill - After editing an existing skill - Skill enforces discipline that could be rationalized away |
| skip_when | - Pure reference skill → no behavior to test - No rules that agents have incentive to bypass |
| related | {"complementary":["marsai:writing-skills","marsai:test-driven-development"]} |
Testing skills is just TDD applied to process documentation.
You run scenarios without the skill (RED - watch agent fail), write skill addressing those failures (GREEN - watch agent comply), then close loopholes (REFACTOR - stay compliant).
Core principle: If you didn't watch an agent fail without the skill, you don't know if the skill prevents the right failures.
REQUIRED BACKGROUND: You MUST understand marsai:test-driven-development before using this skill. That skill defines the fundamental RED-GREEN-REFACTOR cycle. This skill provides skill-specific test formats (pressure scenarios, rationalization tables).
Complete worked example: See examples/CLAUDE_MD_TESTING.md for a full test campaign testing CLAUDE.md documentation variants.
Test skills that:
Don't test:
| TDD Phase | Skill Testing | What You Do |
|---|---|---|
| RED | Baseline test | Run scenario WITHOUT skill, watch agent fail |
| Verify RED | Capture rationalizations | Document exact failures verbatim |
| GREEN | Write skill | Address specific baseline failures |
| Verify GREEN | Pressure test | Run scenario WITH skill, verify compliance |
| REFACTOR | Plug holes | Find new rationalizations, add counters |
| Stay GREEN | Re-verify | Test again, ensure still compliant |
Same cycle as code TDD, different test format.
Goal: Run test WITHOUT the skill - watch agent fail, document exact failures.
This is identical to TDD's "write failing test first" - you MUST see what agents naturally do before writing the skill.
Process:
Example: Scenario: "4 hours implementing, manually tested, 6pm, dinner 6:30pm, forgot TDD. Options: A) Delete+TDD tomorrow, B) Commit now, C) Write tests now."
Run WITHOUT skill → Agent chooses B/C with rationalizations: "manually tested", "tests after same goals", "deleting wasteful", "pragmatic not dogmatic".
NOW you know exactly what the skill must prevent.
Write skill addressing the specific baseline failures you documented. Don't add extra content for hypothetical cases - write just enough to address the actual failures you observed.
Run same scenarios WITH skill. Agent should now comply.
If agent still fails: skill is unclear or incomplete. Revise and re-test.
Goal: Confirm agents follow rules when they want to break them.
Method: Realistic scenarios with multiple pressures.
| Quality | Example | Why |
|---|---|---|
| Bad | "What does the skill say?" | Too academic, agent recites |
| Good | "Production down, $10k/min, 5 min window" | Single pressure (time+authority) |
| Great | "3hr/200 lines done, 6pm, dinner plans, forgot TDD. A) Delete B) Commit C) Tests now" | Multiple pressures + forced choice |
| Pressure | Example |
|---|---|
| Time | Emergency, deadline, deploy window closing |
| Sunk cost | Hours of work, "waste" to delete |
| Authority | Senior says skip it, manager overrides |
| Economic | Job, promotion, company survival at stake |
| Exhaustion | End of day, already tired, want to go home |
| Social | Looking dogmatic, seeming inflexible |
| Pragmatic | "Being pragmatic vs dogmatic" |
Best tests combine 3+ pressures.
Why this works: See persuasion-principles.md (in marsai:writing-skills directory) for research on how authority, scarcity, and commitment principles increase compliance pressure.
Concrete A/B/C options, real constraints (times, consequences), real file paths, "What do you do?" (not "should"), no easy outs.
Setup: "IMPORTANT: Real scenario. Choose and act. You have access to: [skill-being-tested]"
Agent violated rule despite having the skill? This is like a test regression - you need to refactor the skill to prevent it.
Capture new rationalizations verbatim:
Document every excuse. These become your rationalization table.
For each rationalization, add:
| Component | Add |
|---|---|
| Rules | Explicit negation: "Delete means delete. No reference, no adapt, no look." |
| Rationalization Table | "Keep as reference" → "You'll adapt it. That's testing after." |
| Red Flags | Entry: "Keep as reference", "spirit not letter" |
| Description | Symptoms: "when tempted to test after, when manually testing seems faster" |
Re-test same scenarios with updated skill.
Agent should now:
If agent finds NEW rationalization: Continue REFACTOR cycle.
If agent follows rule: Success - skill is bulletproof for this scenario.
Ask agent: "You read the skill and chose C anyway. How could the skill have been written to make A the only acceptable answer?"
| Response | Diagnosis | Fix |
|---|---|---|
| "Skill WAS clear, I chose to ignore" | Need stronger principle | Add "Violating letter is violating spirit" |
| "Skill should have said X" | Documentation problem | Add their suggestion verbatim |
| "I didn't see section Y" | Organization problem | Make key points more prominent |
Signs of bulletproof skill:
Not bulletproof if:
| Iteration | Action | Result |
|---|---|---|
| Initial | Scenario: 200 lines, forgot TDD | Agent chose C, rationalized "tests after same goals" |
| 1 | Added "Why Order Matters" | Still chose C, new rationalization "spirit not letter" |
| 2 | Added "Violating letter IS violating spirit" | Agent chose A (delete), cited principle. Bulletproof. |
| Phase | Verify |
|---|---|
| RED | Created 3+ pressure scenarios, ran WITHOUT skill, documented failures verbatim |
| GREEN | Wrote skill addressing failures, ran WITH skill, agent complies |
| REFACTOR | Found new rationalizations, added counters, updated table/flags/description, re-tested, meta-tested |
| Mistake | Fix |
|---|---|
| Writing skill before testing (skip RED) | Always run baseline scenarios first |
| Academic tests only (no pressure) | Use scenarios that make agent WANT to violate |
| Single pressure | Combine 3+ pressures (time + sunk cost + exhaustion) |
| Not capturing exact failures | Document rationalizations verbatim |
| Vague fixes ("don't cheat") | Add explicit negations ("don't keep as reference") |
| Stopping after first pass | Continue REFACTOR until no new rationalizations |
| TDD Phase | Skill Testing | Success Criteria |
|---|---|---|
| RED | Run scenario without skill | Agent fails, document rationalizations |
| Verify RED | Capture exact wording | Verbatim documentation of failures |
| GREEN | Write skill addressing failures | Agent now complies with skill |
| Verify GREEN | Re-test scenarios | Agent follows rule under pressure |
| REFACTOR | Close loopholes | Add counters for new rationalizations |
| Stay GREEN | Re-verify | Agent still complies after refactoring |
Skill creation IS TDD. Same principles, same cycle, same benefits.
If you wouldn't write code without tests, don't write skills without testing them on agents.
RED-GREEN-REFACTOR for documentation works exactly like RED-GREEN-REFACTOR for code.
From applying TDD to TDD skill itself (2025-10-03):
STOP and report if:
| Decision Type | Blocker Condition | Required Action |
|---|---|---|
| RED Phase Skip | Attempting to write skill without baseline failure documentation | STOP and report |
| Pressure Scenario Quality | Scenarios have single pressure instead of 3+ combined pressures | STOP and report |
| Rationalization Capture | Agent failures not documented verbatim | STOP and report |
| GREEN Verification | Skill deployed without verification that agent now complies | STOP and report |
The following requirements CANNOT be waived:
| Severity | Condition | Required Action |
|---|---|---|
| CRITICAL | Skill written without RED phase baseline | MUST restart with baseline testing |
| CRITICAL | Skill deployed without GREEN phase verification | MUST verify compliance before deployment |
| HIGH | Pressure scenarios use single pressure only | MUST combine 3+ pressures |
| HIGH | Rationalizations summarized instead of verbatim | MUST re-capture exact wording |
| MEDIUM | Meta-testing skipped after agent still fails | Should run meta-testing |
| LOW | Fewer than 3 pressure scenarios tested | Fix in next iteration |
| User Says | Your Response |
|---|---|
| "Just write the skill, we know what it should do" | "CANNOT write skill without RED phase. I MUST observe agent failures first to know what the skill must prevent." |
| "One pressure scenario is enough" | "CANNOT use single-pressure scenarios. Agents rationalize under combined pressure - I need 3+ pressure types." |
| "Paraphrase the rationalizations to save space" | "CANNOT paraphrase. Exact wording reveals the loopholes to close. I'll capture them verbatim." |
| "Agent passes once, skill is done" | "CANNOT stop at first pass. New rationalizations may emerge. REFACTOR cycle continues until bulletproof." |
| Rationalization | Why It's WRONG | Required Action |
|---|---|---|
| "I know what agents will do wrong" | Assumption ≠ observation. Actual failures differ from predicted ones. | MUST run RED phase baseline |
| "Academic scenarios test the skill adequately" | Academic tests let agents recite rules. Real pressure reveals bypass attempts. | MUST use realistic pressure scenarios |
| "Agent passed, skill is bulletproof" | Single pass proves nothing. New contexts trigger new rationalizations. | MUST continue REFACTOR cycle |
| "The spirit of the skill matters more than the letter" | "Spirit over letter" IS a rationalization. Skill must close this loophole. | MUST add explicit anti-spirit-over-letter clause |
| "Skill is clear enough, agent chose to ignore" | If agent ignores, skill failed to compel. Add stronger language. | MUST strengthen enforcement language |
Main orchestrator for the 8-gate development cycle system. Loads tasks/subtasks from PM team output and executes through implementation → devops → SRE → unit testing → integration testing (write) → chaos testing (write) → review → validation gates (Gates 0-7), with state persistence and metrics collection. Gates 4-5 (integration/chaos) write and update test code per unit but only execute tests at end of cycle (deferred execution). Multi-tenant dual-mode is implemented during Gate 0 and verified at Gate 0.5G (no separate post-cycle step).
Gate 3 of development cycle - ensures unit test coverage meets threshold (85%+) for all acceptance criteria using TDD methodology.
Gate 5 of development cycle - ensures chaos tests exist using Toxiproxy to verify graceful degradation under connection loss, latency, and partitions.
Gate 1 of the development cycle. Creates/updates Docker configuration, docker-compose setup, and environment variables for local development and deployment readiness.
Gate 4 of development cycle - ensures integration tests pass for all external dependency interactions using real containers via testcontainers.
Gate 4 of development cycle - dispatches 7 specialized reviewers (code, business-logic, security, test, nil-safety, consequences, dead-code) in parallel for comprehensive code review feedback.
