andrem-sec
andrem-sec/psc-comet
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/andrem-sec/psc-comet --skill skill-comply命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
基于 SOC 职业分类
正在显示 SKILL.md
| name | skill-comply |
| description | Automated compliance measurement for skills, rules, and agent definitions |
| version | 0.1.0 |
| level | 3 |
| triggers | ["measure compliance","does claude follow this skill","test skill effectiveness","compliance check"] |
| context_files | ["context/learnings.md"] |
| steps | [{"name":"Spec Generation","description":"Extract behavioral spec from target .md file"},{"name":"Scenario Creation","description":"Generate 3 scenarios with varying prompt strictness"},{"name":"Execution","description":"Run claude -p capturing tool call traces"},{"name":"Classification","description":"Classify tool calls against spec steps using LLM"},{"name":"Compliance Scoring","description":"Calculate compliance percentage and identify violations"},{"name":"Report Generation","description":"Output findings with specific violation examples"}] |
Automated compliance measurement for skills, rules, and agents. Verifies Claude actually follows defined behaviors.
Without compliance testing, you:
Skill comply measures what actually happens, not what should happen.
Extract behavioral specification from any .md file (skill, rule, agent definition).
For Skills:
For Rules:
For Agents:
Spec Format:
{
"target": "tdd",
"type": "skill",
"required_behaviors": [
"write test before implementation",
"run test and verify it fails",
"implement minimal code to pass test"
],
"prohibited_behaviors": [
"implement without test"
Generate 3 test scenarios with decreasing prompt strictness:
Supportive Scenario: Prompt explicitly invokes skill: "Use TDD skill to add a calculateTotal function"
Neutral Scenario: Prompt mentions task without skill: "Add a calculateTotal function"
Competing Scenario: Prompt contradicts skill: "Quickly implement calculateTotal, skip tests for now"
Compliance threshold:
Run scenarios via claude -p (programmatic mode) capturing tool call traces:
claude -p --trace-file trace.jsonl << EOF
$(cat scenario_prompt.txt)
EOF
Trace format (JSONL):
{"timestamp": "2026-03-28T19:00:00Z", "tool": "Write", "args": {"file_path": "test_total.py", "content": "..."}}
{"timestamp": "2026-03-28T19:00:05Z", "tool": "Bash", "args": {"command": "pytest test_total.py"}}
{"timestamp": "2026-03-28T19:00:10Z", "tool": "Write", "args": {"file_path": "calculator.py", "content": "..."}}
Run each scenario 3 times (measure consistency).
Use LLM to classify each tool call against behavioral spec:
Classifier Prompt:
Behavioral spec:
- REQUIRED: Write test before implementation
- REQUIRED: Run test and verify it fails
- PROHIBITED: Implement without test
Tool call trace:
1. Write(test_total.py)
2. Bash(pytest test_total.py)
3. Write(calculator.py)
Classify each call:
1. COMPLIANT (matches required: write test first)
2. COMPLIANT (matches required: run test, verify fail)
3. COMPLIANT (implementation after test)
Compliance: 3/3 (100%)
Classifier model: Haiku (fast, cheap, good enough for binary classification)
Ambiguous cases:
Calculate compliance percentage per scenario:
Formula:
Compliance% = (COMPLIANT + 0.5 * PARTIAL_COMPLIANT) / TOTAL_RELEVANT_CALLS * 100
Report per scenario:
Overall verdict:
Output structured findings:
# Compliance Report: tdd skill
**Date:** 2026-03-28
**Scenarios:** 3 (Supportive, Neutral, Competing)
**Runs per scenario:** 3
**Overall Verdict:** DEGRADED
## Summary
| Scenario | Compliance | Threshold | Status |
|----------|------------|-----------|--------|
| Supportive | 90% | ≥90% | ✓ PASS |
| Neutral | 65% | ≥70% | ✗ FAIL |
| Competing | 45% | ≥50% | ✗ FAIL |
## Violations
**Neutral Scenario (Run 2):**
- Tool call #3: `Write(calculator.py)` before test written
- Expected: Write test first
- Actual: Implementation written directly
**Competing Scenario (Run 1):**
- Tool call #1: `Write(calculator.py)` (no test)
- Skill: Test-first required
- User prompt: "skip tests for now"
- Result: Skill ignored, user request followed
## Recommendations
1. Strengthen TDD skill preamble: emphasize test-first is non-negotiable
2. Add CLAUDE.md rule: "Never write implementation without test"
3. Consider PreToolUse hook: block Write to src/ if no test exists
4. Retest after changes to verify improvement
Skills (.claude/skills/):
Rules (.claude/rules/):
Agents (.claude/agents/):
Testing skills in isolation: Skills interact. Test tdd + code-review together, not separately.
Ignoring Neutral/Competing scenarios: Skills that only work when explicitly invoked aren't proactive enough.
Single-run testing: Run each scenario 3x to measure consistency. One success out of three is 33% compliance, not 100%.
No action on failures: Compliance reports that don't lead to skill refinement are wasted effort.
Over-testing: Don't test every skill monthly. Prioritize high-impact skills (tdd, security-gate, prd).