scoring

# NLPM Quality Scoring Rubric 100-point quality scale for all NL programming artifacts. Apply penalties deterministically. Use calibration examples to anchor judgment on borderline cases. --- ## Scoring Formula ``` base_score = 100 adjustments = sum of all applicable penalties (all penalties are negative) final_score = max(0, min(100, base_score + adjustments)) ``` Penalties stack. The floor is 0; the ceiling is 100. No bonuses — the default assumption is that an artifact is well-formed, and quality is measured by what is missing or wrong. --- ## Penalty Tables The tables in this file cover every artifact type a Claude Code project contains. Tables for the remaining artifact types live in `references/`, one file per group. When the artifact you are scoring is listed below, Read that file and apply its tables as if they appeared here — they carry the same weight, and a finding cited to one of their rows is a rubric finding. | Artifact being scored | Read | |---|---| | Codex hook events; `.codex-plugin/plugin.json`; `.agents/plugins/marketplace.json`; `agents/openai.yaml`; `.codex/config.toml` | [`references/codex.md`](references/codex.md) | | Antigravity / Gemini-lineage hook events; `gemini-extension.json`; `.gemini/commands/*.toml` | [`references/antigravity.md`](references/antigravity.md) | | Memory files (`~/.claude/projects/*/memory/*.md`); agent workflow programs (project-root `program.md`-style files) | [`references/memory-and-workflow.md`](references/memory-and-workflow.md) | The universal Hooks checks below apply to Codex and Antigravity hook configs too. ### Skills | Rule | Check | Condition | Penalty | |------|-------|-----------|---------| | -- | `name` present | Missing | -25 | | -- | `name` matches parent directory | Frontmatter `name:` value does not equal parent directory name (per `nlpm:conventions` §5 — open spec MUST) | -15 | | R04 | `description` present | Missing | -25 | | R04 | Trigger quality | Description is generic (≤1 specific phrase) | -15 | | R04 | Description length | Description 500–800 chars | -5 | | R04 | Description length | Description >800 chars | -10 | | R05 | Body length | 400–500 lines | -5 | | R05 | Body length | >500 lines | -10 | | R06 | Code examples | Complex concepts with no examples | -5 | | R06 | Code examples | No examples at all in a technical skill | -10 | | R06 | `<example>` blocks | Zero `<example>` blocks on a user-invocable skill (`user-invocable` absent or `true`) | -10 | | R07 | Scope note | No scope note / cross-references | -3 | > **Scope-note discipline:** R07 means "scope note when related skills exist." > Do NOT apply R07 to missing example blocks — that is the new R06 row above > (penalty -10, not -15). The 2026-05-13 lijigang/ljg-skills audit applied > R07 + −15 fourteen times for missing example blocks; both labels were > wrong (R07 is not example-related, and -15 is the agents penalty, not > the skills penalty). The validator at `auditor/scripts/validate-rule-ids.py` > catches this kind of drift in CI. > **`<example>`-block counting discipline** (added 2026-08-01, origin: > xiaolai/cc-suite v1.3.1 remediation): an `<example>` block counts only when > it sits outside fenced code blocks — in the body or in a frontmatter > description block scalar. Tags inside a fenced template (` ```markdown … ``` `) > are illustrative content, and prose that names the string `` `<example>` `` > is a mention, not a block. A 2026-07-31 scoring pass credited a skill with > example blocks that existed only inside a fenced template, hiding a real R06 > violation across 13 files. Verify by reading the file, not by grepping for > the tag. > **`name` matches parent directory** (added 2026-05-25, audit: > google/skills): the open Agent Skills spec at agentskills.io makes this > a MUST. Mismatch is deterministic, high-confidence, and reproducible by > single-line diff (frontmatter `name:` vs `basename($(dirname FILE))`). > Mark such findings `confidence: high` per the manifest-vs-disk-diff > principle in `agents/scorer.md` step 6. Note: this penalty did not > exist before 2026-05-25 — re-scoring past audits will yield slightly > lower scores for any corpus containing this defect, but no contribute > outcomes are retroactively affected since no PRs were ever opened > against a name-mismatch finding under the prior rubric. --- ### Agents | Rule | Check | Condition | Penalty | |------|-------|-----------|---------| | R09 | `description` present | Missing | -25 | | R09 | `<example>` blocks | Zero `<example>` blocks in the description | -15 | | R09 | Exclusion clause | Description names no situation the agent is not for (a "Not for …"/"Do not use …" sentence, or an example where the assistant declines or routes elsewhere) | -5 | | R09 | Description length | Description value over 1,200 characters, examples included | -5 | | R10 | `model` declared | Not declared | -5 | | R10 | `model` appropriate | Wrong tier for task (e.g. opus for parsing) | -5 | | R11 | `tools` declared | Not declared | -5 | | R11 | Unused tools | Each tool declared but not used in body | -3 each | | R12 | Output format | No output format spec in body | -10 | | R11 | Write on read-only | Audit/review/scan agent declares Write or Edit | -10 | > **R09 example budget** (changed in 1.4.0): one `<example>` block is full > credit. An agent's `description` sits in the Agent tool's text on every turn > and is the only thing Claude sees when choosing an agent, so examples stay in > the description — moving them into the body hides them from routing. One > well-chosen example plus an explicit exclusion ("Not for …") carries the > routing signal at a fraction of the tokens; extra examples are allowed but > cost always-on context, which is why R09 deducts 5 points when the > description exceeds 1,200 characters. > **`min_examples` override** (`R09: { min_examples: N }` in the project's > `nlpm.local.md`, see `nlpm:conventions` §6): with N greater than 1, a > description that has at least one but fewer than N `<example>` blocks costs > -5 per missing block, capped at -15 so that a partial set never costs more > than none; zero blocks stays -15. The exclusion-clause and length rows are > unchanged. Without the override, N is 1 and the rows above apply as written. --- ### Commands | Rule | Check | Condition | Penalty | |------|-------|-----------|---------| | -- | `description` present | Missing | -25 | | R18 | `argument-hint` present | Command takes input but no hint | -5 | | R14 | Steps numbered | Multi-step body with no numbered steps | -10 | | R15 | Empty input handling | No handling for empty/missing input | -10 | | R16 | Output format | No output format defined | -10 | | R17 | Error paths | No error handling for missing files or bad data | -5 | --- ### Shared Partials | Rule | Check | Condition | Penalty | |------|-------|-----------|---------| | R19 | `user-invocable: false` | Missing or set to true | -25 | | R20 | Purpose clear | Description doesn't state it's a partial | -10 | --- ### Rules | Rule | Check | Condition | Penalty | |------|-------|-----------|---------| | R21 | `description` present | Missing frontmatter description | -10 | | R21 | Format: bold imperative | No bold imperative opening | -5 | | R21 | Format: rationale | No rationale following the imperative | -10 | | R22 | Enforceability | Rule is not specific/testable | -10 | | R23 | Budget | Rule file over 500 lines | -15 | | R26 | Conflicts with other rules | Direct contradiction with another rule in same set | -20 | | R24 | Duplicates tooling | Re-states what eslint/ruff/clippy already catches | -10 | --- ### Hooks — universal checks (apply to all tools) | Rule | Check | Condition | Penalty | |------|-------|-----------|---------| | -- | Valid syntax | Hook config file fails to parse (JSON or TOML per tool) | -25 | | R29 | Scripts exist | Referenced script file does not exist | -20 | | -- | Command safety | Hook command contains dangerous patterns (`rm -rf`, `git push --force`, `DROP TABLE`) | -15 | | -- | Matcher regex valid | Matcher pattern doesn't compile as valid regex | -10 | | -- | Timeout reasonable | Hook specifies timeout > 30s (likely hangs) | -5 | ### Hooks (Claude Code — Tier 2-Claude only) Authoritative event list: `nlpm:conventions-claude` §7 together with its extended allow-list in `conventions-claude/reference.md`. Per the multi-tool design (`analysis/multi-tool-design-2026-05.md` decision #4), Claude / Codex / Antigravity hook event vocabularies are NOT 1:1 mappable — three separate tables, no translation. The Codex and Antigravity tables are in `references/codex.md` and `references/antigravity.md`. | Rule | Check | Condition | Penalty | |------|-------|-----------|---------| | R27 | Event names valid (Claude) | Uses an event name that is in neither the `nlpm:conventions-claude` §7 table nor its extended allow-list (`conventions-claude/reference.md`, Hook Events). Those two lists together are the confirmed set (33 events, including `SubagentStop`, `SubagentStart`, `PreCompact` and `Notification`); a name missing from both still passes if `code.claude.com/docs/en/hooks.md` documents it | -15 | | R27 | Case correct (Claude) | Event name has wrong case (e.g. `pretooluse`) | -10 | | -- | Hook type valid (Claude) | Uses unrecognized `type` value — confirmed Claude types: `command`, `http`, `mcp_tool`, `prompt`, `agent` | -10 | | -- | MCP matcher format (Claude) | Matcher targets MCP tool but doesn't use `mcp__<server>__<tool>` pattern | -5 | --- ### plugin.json (Claude Code — `.claude-plugin/plugin.json`) | Check | Condition | Penalty | |-------|-----------|---------| | `name` present | Missing | -25 | | `version` is semver | Present but not valid semver | -10 | | `description` present | Missing | -5 | --- ### .claude-plugin/marketplace.json (Claude Code — Tier 2-Claude) Schema reference: `nlpm:conventions-claude` §17 and its `reference.md` (Plugin Distribution). | Check | Condition | Penalty | |-------|-----------|---------| | Valid JSON | File fails JSON parse | -25 | | `name` present | Missing | -25 | | `owner.name` present | `owner` missing, or it has no `name` | -10 | | `plugins` array present | Missing or empty | -10 | | Per-plugin `name` and `source` | Either missing | -10 each | | Per-plugin `source` valid | A relative path that doesn't start with `./` (bare names are valid only under `metadata.pluginRoot`), or an object whose `source.source` isn't one of the six object types in `reference.md` or that lacks that type's required field | -10 each | | Per-plugin `version` in step | Differs from the `version` in the `plugin.json` it describes, when that file is in the same repository (`plugin.json` wins at load, so the entry is stale) | -5 each | | Per-plugin `description` present | Missing (the `/plugin` browser shows it) | -3 each | --- ### .mcp.json (Claude Code — `.mcp.json` at repo root) | Check | Condition | Penalty | |-------|-----------|---------| | Valid JSON | File fails JSON parse | -25 | | Server `command` present | MCP server entry missing `command` field | -15 | --- ### .lsp.json (Claude Code LSP — Tier 2-Claude) Schema details: `nlpm:conventions-claude` §12. Stable in 2026. | Check | Condition | Penalty | |-------|-----------|---------| | Valid JSON | File fails JSON parse | -25 | --- ### monitors/monitors.json (Claude Code monitors — Tier 2-Claude) Schema details: `nlpm:conventions-claude` §13. Stable in 2026. | Check | Condition | Penalty | |-------|-----------|---------| | Valid JSON | File fails JSON parse | -25 | --- ### Settings Files (.claude/settings.json, .claude/settings.local.json) | Check | Condition | Penalty | |-------|-----------|---------| | Valid JSON | File fails JSON parse | -25 | | No hardcoded secrets | Contains API keys, tokens, or passwords | -25 | | Permission mode sanity | `bypassPermissions` enabled in a shared project settings file (not .local) | -15 |