| name | night-market-diagnostics-toolkit |
| description | Run and interpret repo diagnostic scripts (ratchets, validators, token stats). Use when measuring health. Do not use to run tests; use night-market-operations. |
Night Market Diagnostics Toolkit
Measure instead of eyeballing. Every claim about the health of this
repo ("the skill graph is clean", "descriptions fit the budget",
"no compromised dependencies") has a script that produces a number
or a PASS/FAIL. This skill catalogs those scripts, their exact
invocations, and how to read their output.
Conventions for everything below:
- Run all commands from the repo root
(
/home/alext/claude-night-market).
- Plain
python3 is enough for every script in the table except
mutation testing, which needs uv.
- All checks here are read-only. None of them mutate the repo.
The ratchet-baseline concept
Several checks are ratchets. A ratchet check counts a category of
existing debt (for example, SKILL.md files missing an Exit Criteria
section), compares the live count against a frozen number in a
baseline JSON file, and:
- passes while the live count is at or below the baseline
(pre-existing debt is tolerated),
- fails the moment a NEW violation pushes the count above the
baseline (debt may not grow),
- nudges you to lower the baseline number when the live count drops
(locking in the win so the debt cannot silently return).
The goal is a baseline that only shrinks, eventually to zero. The
two ratchet baselines live at scripts/skill_graph_baseline.json
(keys max_dangling_bugs, max_uncalled_libraries) and
scripts/skill_exit_criteria_baseline.json
(key max_missing_exit_criteria). When a ratchet prints a
"dropped to N (baseline M). Lower ... to lock the win." line, edit
the baseline JSON down to N in the same PR.
Tool table
Jargon used in the table: "dangling Skill() ref" means a
Skill(plugin:name) reference in a skill/command/agent file whose
target skill does not exist. "Uncalled library" means a
library-role skill no other skill invokes (the risk targeted by
.claude/rules/shared-utility-consumer-rule.md).
| Tool | Invocation | What it measures | How to interpret | When to run |
|---|
| Plugin structure validator | python3 plugins/abstract/scripts/validate_plugin.py plugins/<name> | plugin.json validity, kebab-case naming, directory layout | Exit 0 with "Plugin validation passed". Any FAIL line names the broken file | Before committing plugin changes. make validate-all loops it over every plugin |
| Skill-graph drift ratchet | python3 scripts/check_skill_graph_drift.py | Dangling Skill() refs and uncalled libraries vs skill_graph_baseline.json | Exit 0 at/below baseline. Nonzero exit lists the new dangling refs. "Lower ... to lock the win" means shrink the baseline | After adding, renaming, or deleting skills or Skill() refs |
| Exit-criteria drift ratchet | python3 scripts/check_skill_exit_criteria_drift.py | SKILL.md files under plugins/ missing an ## Exit Criteria heading vs skill_exit_criteria_baseline.json | Same ratchet semantics as above | After adding or editing any SKILL.md |
| Pinned-version checker | python3 scripts/check_pinned_versions.py | GitHub-sourced tool pins vs the latest upstream release (needs network) | "N pin(s) current" is healthy. A "holding X at ..." line documents an intentional hold with its reason (for example bandit 1.8.6 for the Python 3.9 hook floor) | In pre-commit, and whenever a CI tool-setup step breaks |
| Capabilities sync | bash scripts/capabilities-sync-check.sh | plugin.json registrations vs the generated capabilities reference in book/src/ | "PASSED: All capabilities are in sync" plus counts. On drift, run /sanctum:sync-capabilities --fix | After changing any skill, command, or agent registration |
| Supply-chain scan | python3 scripts/supply_chain_scan.py | Lockfiles vs the known-compromised-versions blocklist, plus known malicious artifacts | Two [OK] lines is clean. Any hit must be resolved before release | When adding dependencies, during incidents, before releases |
| Token stats | python3 plugins/abstract/scripts/context_optimizer.py stats plugins/ --format json |
Interpretation notes
- context_optimizer stats prints nothing in the default text
format. This is a verified quirk:
stats on a directory with
no --format flag produces empty output and exit 0. Always pass
--format json (clean JSON) rather than --format table, which
dumps a raw Python dict.
- Ratchet failures name the new violations. A ratchet that
fails after your change means your change added debt. Fix the
new ref or add the missing section. Never raise a baseline number
to make a check pass, with one documented exception: a brand-new
library skill legitimately starts uncalled, and
scripts/check_skill_graph_drift.py itself (plus the _comment
in scripts/skill_graph_baseline.json) instructs you to raise
max_uncalled_libraries to record the 30-day consumer grace
period that .claude/rules/shared-utility-consumer-rule.md
grants (the rule grants the grace period; the script defines the
baseline mechanics). night-market-architecture-contract records
the same exception in its ratchet invariants.
- Mutation exit codes are the contract. CI
(
.github/workflows/mutation-testing.yml) treats exit 0 and
exit 2 as success and posts survivors to the step summary. Only
exits other than 0 and 2 fail the workflow. So a green mutation
run does NOT mean zero survivors. Read the report.
- check_pinned_versions needs network access. It queries GitHub
for latest releases. Offline, expect failures that are
environmental, not real findings.
Health snapshot in one command
scripts/health-snapshot.sh (in this skill directory) runs the six
cheap checks and prints a PASS/FAIL table. It is read-only,
needs only bash and python3, requires no network, and completes in
about 6 seconds (measured 2026-07-02).
bash .claude/skills/night-market-diagnostics-toolkit/scripts/health-snapshot.sh
It runs: plugin structure validation (all plugins), capabilities
sync, supply-chain scan, both drift ratchets, and the description
budget. Exit 0 means all six passed. Exit 1 means at least one
failed, and the failing check's full output is printed to stderr
above the table. Run it at session start, before a PR, and after
any bulk skill edit.
Blocking vs advisory
Which failures stop a merge and which are signals:
| Check | Gate | Effect on failure |
|---|
| validate_plugin.py | pre-commit (19 hook entries) | Blocks commit |
| Skill-graph drift ratchet | pre-commit | Blocks commit |
| Exit-criteria drift ratchet | pre-commit | Blocks commit |
| check_noqa.py | pre-commit | Blocks commit |
| check_docstring_quality.py | pre-commit | Blocks commit |
| check-markdown-links.py | pre-commit | Blocks commit |
| validate_budget.py | pre-commit | Blocks commit |
| check-json-utils-drift.sh | pre-commit | Blocks commit |
| check_pinned_versions.py | pre-commit | Blocks commit |
| capabilities-sync-check.sh | pre-commit and CI (capabilities-sync.yml) | Blocks commit and PR |
| Mutation testing | CI (weekly and manual dispatch) | Advisory: survivors reported, workflow stays green |
| supply_chain_scan.py | manual / make supply-chain-scan (not wired to CI) | Advisory day-to-day, treat as blocking before any release |
| context_optimizer, skrills analyze, clawhub stats, framework_detect | manual | Advisory: measurement only |
The dimension-level policy lives in .claude/quality_gates.json:
enforce_blocking is true with max_critical_issues: 3, and only
the security dimension sets block_on_violation: true. The
performance (20KB / 5,000-token file limits, 60-line functions,
complexity under 12), maintainability (debt ratio under 0.3,
nesting under 5), and compliance dimensions are advisory
(block_on_violation: false). So: security findings block,
size/complexity findings warn.
When NOT to use
- Running the test suite, lint, typecheck, or a release:
use night-market-operations (command anatomy and artifacts).
- You already have a failure and need symptom-to-cause triage:
use night-market-debugging-playbook.
- You need the meaning and default of a config knob rather than a
measurement: use night-market-config-catalog.
- You are deciding whether evidence is sufficient to claim work
complete: use night-market-validation-and-qa.
Exit Criteria
Provenance and maintenance
Compiled 2026-07-02 against repo v1.9.15 on branch
discussions-fix-1.9.14. Volatile numbers observed that day:
197 skills, 134 commands, 56 agents registered; description budget
73,135 of 90,000 chars used (81.3%); ratchet live counts 5 dangling
refs (baseline 31), 7 uncalled libraries (baseline 8), 1 missing
Exit Criteria (baseline 127); 13 version pins current with bandit
intentionally held at 1.8.6.
Re-verification one-liners:
python3 scripts/check_skill_graph_drift.py; python3 scripts/check_skill_exit_criteria_drift.py
bash scripts/capabilities-sync-check.sh | tail -3; python3 plugins/abstract/scripts/validate_budget.py | tail -3
rg -n "check_skill_graph_drift|check_noqa|validate_budget|check_pinned_versions" .pre-commit-config.yaml
rg -n "exit_code" .github/workflows/mutation-testing.yml
python3 -c "import json;print(json.load(open('.claude/quality_gates.json')))"
