Menu
Automated compliance measurement for skills, rules, and agent definitions
The review-first prompt is selected by default. You can switch to a direct command or download a local copy.
Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/andrem-sec/psc-comet --skill skill-complyThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Based on SOC occupation classification
Showing SKILL.md
| name | skill-comply |
| description | Automated compliance measurement for skills, rules, and agent definitions |
| version | 0.1.0 |
| level | 3 |
| triggers | ["measure compliance","does claude follow this skill","test skill effectiveness","compliance check"] |
| context_files | ["context/learnings.md"] |
| steps | [{"name":"Spec Generation","description":"Extract behavioral spec from target .md file"},{"name":"Scenario Creation","description":"Generate 3 scenarios with varying prompt strictness"},{"name":"Execution","description":"Run claude -p capturing tool call traces"},{"name":"Classification","description":"Classify tool calls against spec steps using LLM"},{"name":"Compliance Scoring","description":"Calculate compliance percentage and identify violations"},{"name":"Report Generation","description":"Output findings with specific violation examples"}] |
Automated compliance measurement for skills, rules, and agents. Verifies Claude actually follows defined behaviors.
Without compliance testing, you:
Skill comply measures what actually happens, not what should happen.
Extract behavioral specification from any .md file (skill, rule, agent definition).
For Skills:
For Rules:
For Agents:
Spec Format:
{
"target": "tdd",
"type": "skill",
"required_behaviors": [
"write test before implementation",
"run test and verify it fails",
"implement minimal code to pass test"
],
"prohibited_behaviors": [
"implement without test"
Generate 3 test scenarios with decreasing prompt strictness:
Supportive Scenario: Prompt explicitly invokes skill: "Use TDD skill to add a calculateTotal function"
Neutral Scenario: Prompt mentions task without skill: "Add a calculateTotal function"
Competing Scenario: Prompt contradicts skill: "Quickly implement calculateTotal, skip tests for now"
Compliance threshold:
Run scenarios via claude -p (programmatic mode) capturing tool call traces:
claude -p --trace-file trace.jsonl << EOF
$(cat scenario_prompt.txt)
EOF
Trace format (JSONL):
{"timestamp": "2026-03-28T19:00:00Z", "tool": "Write", "args": {"file_path": "test_total.py", "content": "..."}}
{"timestamp": "2026-03-28T19:00:05Z", "tool": "Bash", "args": {"command": "pytest test_total.py"}}
{"timestamp": "2026-03-28T19:00:10Z", "tool": "Write", "args": {"file_path": "calculator.py", "content": "..."}}
Run each scenario 3 times (measure consistency).
Use LLM to classify each tool call against behavioral spec:
Classifier Prompt:
Behavioral spec:
- REQUIRED: Write test before implementation
- REQUIRED: Run test and verify it fails
- PROHIBITED: Implement without test
Tool call trace:
1. Write(test_total.py)
2. Bash(pytest test_total.py)
3. Write(calculator.py)
Classify each call:
1. COMPLIANT (matches required: write test first)
2. COMPLIANT (matches required: run test, verify fail)
3. COMPLIANT (implementation after test)
Compliance: 3/3 (100%)
Classifier model: Haiku (fast, cheap, good enough for binary classification)
Ambiguous cases:
Calculate compliance percentage per scenario:
Formula:
Compliance% = (COMPLIANT + 0.5 * PARTIAL_COMPLIANT) / TOTAL_RELEVANT_CALLS * 100
Report per scenario:
Overall verdict:
Output structured findings:
# Compliance Report: tdd skill
**Date:** 2026-03-28
**Scenarios:** 3 (Supportive, Neutral, Competing)
**Runs per scenario:** 3
**Overall Verdict:** DEGRADED
## Summary
| Scenario | Compliance | Threshold | Status |
|----------|------------|-----------|--------|
| Supportive | 90% | ≥90% | ✓ PASS |
| Neutral | 65% | ≥70% | ✗ FAIL |
| Competing | 45% | ≥50% | ✗ FAIL |
## Violations
**Neutral Scenario (Run 2):**
- Tool call #3: `Write(calculator.py)` before test written
- Expected: Write test first
- Actual: Implementation written directly
**Competing Scenario (Run 1):**
- Tool call #1: `Write(calculator.py)` (no test)
- Skill: Test-first required
- User prompt: "skip tests for now"
- Result: Skill ignored, user request followed
## Recommendations
1. Strengthen TDD skill preamble: emphasize test-first is non-negotiable
2. Add CLAUDE.md rule: "Never write implementation without test"
3. Consider PreToolUse hook: block Write to src/ if no test exists
4. Retest after changes to verify improvement
Skills (.claude/skills/):
Rules (.claude/rules/):
Agents (.claude/agents/):
Testing skills in isolation: Skills interact. Test tdd + code-review together, not separately.
Ignoring Neutral/Competing scenarios: Skills that only work when explicitly invoked aren't proactive enough.
Single-run testing: Run each scenario 3x to measure consistency. One success out of three is 33% compliance, not 100%.
No action on failures: Compliance reports that don't lead to skill refinement are wasted effort.
Over-testing: Don't test every skill monthly. Prioritize high-impact skills (tdd, security-gate, prd).