| name | rule-quality-evaluator |
| description | Evaluate the quality of existing agent instruction rule sets — CLAUDE.md, AGENTS.md, .cursorrules, copilot-instructions.md, or any coding-agent context file. Use this skill whenever someone asks to audit, score, or improve their agent rules; wants to know if their instructions are effective; or describes rules that aren't changing agent behavior. Trigger on phrases like 'are my rules good?', 'why is the agent ignoring my instructions?', 'score my CLAUDE.md', 'audit my agent rules', 'are these instructions effective?', 'review my copilot-instructions.md', 'my rules aren't working', 'evaluate my context file', or any situation where a human has an existing instruction file and wants to know whether it will actually improve agent behavior. Also trigger when someone has just run agent-instruction-forge and wants to verify the output before committing. |
Rule Quality Evaluator
A rule that cannot be violated cannot be followed. Most instruction files fail because of unfalsifiable guidance, redundant noise, and implicit knowledge that never made it to paper. Score rules against seven properties, identify structural weaknesses, optionally verify with live tasks.
Two-Phase Overview
- Phase 1 — Static Critic (always): Read rules, score against Seven Properties, detect structural issues, produce scorecard. No agent execution.
- Phase 2 — Behavioral Test (opt-in): Generate coding tasks, derive assertions, hand off to context-eval, synthesize report.
Interview-only fallback: If no instruction file accessible, ask user to paste rules. Flag at end: "I can't verify codebase alignment — code pattern checks need manual verification."
Phase 1 — Static Critic
Step 1: Ingest
Read instruction file(s). Detect format (Markdown prose, structured rules list, YAML front matter, fenced sections). Parse into individual rules — one rule = one discrete behavioral instruction. Split on bullets, numbered items, or paragraph breaks starting an imperative.
Optionally scan codebase for linter configs (.eslintrc, pyproject.toml, ruff.toml), type configs (tsconfig.json), CI configs (.github/workflows/). Enables redundancy detection.
Step 2: Score Each Rule — The Seven Properties
Score 0–7 by counting properties satisfied (each binary).
| Property | ID | Score 0 | Score 1 |
|---|
| Specific and falsifiable | P1 | "Write clean code" | "Every API handler must return Result<T, AppError>, never throw" |
| Encodes WHY | P2 | "Don't use console.log" | "Don't use console.log — use src/lib/logger.ts. Bypasses correlation IDs, breaks Datadog traces" |
| Born from real failure | P3 | "Handle errors carefully" | "Never add indexes to reservations without DBA approval — compound index locked table 47 min in Q2 2024" |
| Scoped to right level | P4 | Rule about billing API in root file | Same rule in src/billing/CLAUDE.md |
| Points to canonical example | P5 | "Follow our API patterns" | "New endpoints follow src/api/reservations/create.ts — handler → validation → service → response" |
| Includes anti-pattern | P6 | "Use the service layer" | "We do NOT use the repository pattern. Each service calls Prisma directly" |
| Token-efficient | P7 | "When writing tests, please make sure to always use Vitest and not Jest" | "Tests: Vitest, never Jest. Config: vitest.config.ts" |
Score ≤ 2 = weak — rewrite/remove candidate.
P1 = 0 = noise — agent can't verify, can't steer.
Step 3: Structural Analysis
Redundancy: Flag rules duplicating linter (ESLint, Ruff, Pylint), type system (TypeScript strict, mypy), or CI (test gates, build gates). Redundant rules waste budget and dilute signal.
Scope: "Does this rule apply to ALL code agent will see in this directory?" Flag:
- Over-scoped: package-specific rule in root file
- Under-scoped: identical rules duplicated across subdirs that should move up
Coverage — map to nine categories:
- C1 Architecture & Boundaries
- C2 Domain Model & Business Rules
- C3 Conventions & Patterns
- C4 Integrations & External Dependencies
- C5 Operations & Deployment
- C6 Testing Philosophy & Strategy
- C7 Security Model
- C8 Performance Constraints
- C9 Historical Decisions & Tech Debt
Mark ● (covered), ◐ (partial), ○ (missing).
Token budget:
- Claude Code: root <4,000 tokens, subdir <1,000
- Copilot: root <1,000 lines, first 4,000 chars read per code review
- Cursor / Windsurf / AGENTS.md: similar to Claude Code
Step 4: Produce Scorecard
Output, then ask whether to run Phase 2.
Rule Scorecard — [file path]
| Rule (truncated) | Score | Weakest Property | Flags |
|---|
| "…" | N/7 | P? | weak / noise / redundant / over-scoped |
Summary Stats
- Total rules: N
- Mean score: X.X / 7
- Strong (5–7): N | Adequate (3–4): N | Weak (0–2): N
- Noise rules (P1 = 0): N
- Redundant with linter/CI/types: N
- Over-scoped: N | Under-scoped: N
Coverage Map
C1[●] C2[○] C3[●] C4[○] C5[◐] C6[●] C7[○] C8[○] C9[○]
● covered ◐ partial ○ missing
Token Budget
- Current: ~N tokens / N chars
- Limit: [system-appropriate]
- Headroom: [remaining / over by N]
Top 3 Improvements
- [Most impactful — which rule, which property, what to add]
- [Second]
- [Third]
Phase 1 Verdict
| Threshold | Verdict |
|---|
| Mean ≥ 5 | STRONG — specific, grounded, efficient |
| Mean 3–4 | ADEQUATE — usable but with gaps |
| Mean < 3 | WEAK — won't reliably steer behavior |
Run Phase 2 — Behavioral Testing? Generates real coding tasks, measures whether rules change agent behavior. Requires context-eval.
Phase 2 — Behavioral Test
Step 5: Generate Coding Tasks
Generate 3 coding tasks. Each:
- ~50 lines of code to produce
- Touches 2+ rules (discriminating grading)
- Prioritize: rules scoring 3-5 (uncertain), C7 Security or C2 Domain (high-stakes), incident-born rules (P3 = 1)
Per task: which rules exercised, what correct behavior looks like.
Step 6: Generate Assertions
One testable assertion per rule:
- Observable in agent output (no execution required)
- Pass/fail check
Write to evals/evals.json:
{
"harness_name": "[instruction file name]",
"harness_type": "coding agent instruction rules",
"harness_path": "[path to instruction file]",
"evals": [
{
"id": 1,
"prompt": "[the coding task prompt]",
"expected_output": "[description of correct output under the rules]",
"files": [],
"assertions": [
"[rule → assertion, e.g., 'Uses src/lib/logger.ts, not console.log']",
"[second rule → assertion]"
]
}
]
}
Step 7: Hand Off to context-eval
Harness under test = instruction file. Baseline = same tasks without the file.
→ context-eval: evaluate harness at [instruction file path] using evals/evals.json
context-eval runs with/without harness, grades, computes benefit delta.
Step 8: Synthesize Report
Combined Report
Phase 1 Static: [mean] / 7 → [STRONG / ADEQUATE / WEAK]
Phase 2 Behavioral Delta: +[delta] pass rate over baseline
Per-Rule Behavioral Confirmation
| Rule | P1 Verdict | P2 Delta | Confirmed? |
|---|
| "…" | score/7 | +X% | yes / partial / no |
Final Verdict
| Condition | Verdict |
|---|
| Static ≥ 5 AND delta ≥ 0.25 | STRONG — well-formed, measurable improvement |
| Static ≥ 3 AND delta ≥ 0.10 | ADEQUATE — works, room to improve |
| Static < 3 OR delta < 0.10 | WEAK — not reliably changing behavior |
| Delta < 0 | HARMFUL — degrading performance |
Delta = pass-rate diff between with-rules and without-rules (0.0–1.0; 0.25 = 25 pp).
Discrepancies
Rules high in Phase 1 with no behavioral delta (or vice versa) — most actionable findings. High-score + no delta = either not reachable in eval tasks or not being read. Low-score + high delta = encodes something valuable scoring missed.
Calibration Rules
- Phase 1 always first. Scorecard catches most failures without tooling.
- P1 is the gatekeeper. Unfalsifiable rules can't be improved by other properties. Fix specificity first.
- Redundancy is the easiest win. Lint/type/CI duplicates can be deleted immediately.
- Behavioral testing is gold standard, not default. Phase 2 needs infrastructure and time. Most rule sets benefit more from targeted Phase 1 rewrites.
- Score what you see. Literal text, not charitable interpretation. Rules requiring generous reading fail in practice.
Composes With
context-gap-analyzer → agent-instruction-forge → rule-quality-evaluator → context-eval
