ecosystem-health

# Ecosystem Health ## MANDATORY: docs-management Delegation > **CRITICAL:** This skill follows the anti-duplication principle. **ALL component details MUST be queried from docs-management at runtime.** ### What This Skill Hardcodes (Static - Changes Rarely) | Data | Why Static | | ---- | ---------- | | Tier structure (1-6) | Design decision, architectural choice | | Audit commands (`/audit-*`) | OUR commands in claude-ecosystem plugin | | Audit types (automated/manual/documentation) | Classification policy | | Scoring/prioritization logic | Policy decisions | ### What MUST Be Delegated (Dynamic - Changes With Releases) | Data | Why Dynamic | How to Get | | ---- | ----------- | ---------- | | CLI flags list | New flags every release | Query: `docs-management: cli-reference.md CLI flags` | | Environment variables | New env vars frequently | Query: `docs-management: settings.md environment variables` | | Authentication methods | New providers added | Query: `docs-management: iam.md authentication methods` | | Permission modes | Modes evolve | Query: `docs-management: iam.md permission modes` | | Cloud providers | New providers added | Query: `docs-management: setup.md cloud providers` | | IDE integrations | New integrations added | Query: `docs-management: third-party-integrations.md IDE` | | Change keywords | Terminology evolves | Derive from docs-management index keywords | | File patterns | Locations can change | Query official docs for current patterns | ### Delegation Rules 1. **NEVER use hardcoded lists** for CLI flags, env vars, auth methods, cloud providers, IDE integrations, permission modes, or any frequently-changing data 2. **ALWAYS query docs-management** when you need component details 3. **Use claude-code-guide agent** for live verification during `--discover` and `--check` modes 4. **If docs-management returns empty**, that's a signal to check if the component still exists ### Query Patterns for docs-management ```text # Tier 3: Environment & CLI "cli-reference.md CLI flags" → Get current CLI flags list "settings.md environment variables" → Get current env vars list "iam.md permission modes" → Get current permission modes # Tier 4: Authentication & Access "iam.md authentication methods" → Get current auth methods "iam.md configuring permissions" → Get permission rule patterns "iam.md credential management" → Get credential features # Tier 5: Session & Runtime "cli-reference.md session features" → Get session features (resume, checkpoints) "security.md sandbox configuration" → Get sandbox settings # Tier 6: Integration "third-party-integrations.md IDE" → Get IDE integrations list "setup.md cloud providers" → Get cloud provider list "common-workflows.md CI/CD" → Get CI/CD platforms # Changelog for change categorization "CHANGELOG recent changes" → Get changelog entries ``` --- ## Overview This skill tracks Claude Code ecosystem health across **ALL extensibility points** - not just plugin components. It monitors **27 component types** across **6 tiers**. **Schema v2.2** introduces the **Tiered Validation Model** for changelog-triggered audit staleness: - **Minor changes** (features, deprecations) → Targeted keyword validation (cheap) - **Major changes** (behavior_change, security) → Full audit required (no shortcuts) - **Bugfixes** → No validation needed This approach provides **60-96% token savings** while maintaining strict accuracy requirements. ### Component Tiers | Tier | Category | Components | Audit Type | Description | | ---- | -------- | ---------- | ---------- | ----------- | | 1 | Core Configuration | 4 | Mixed | User, project, and enterprise settings | | 2 | Plugin Components | 12 | Automated | Components packaged in plugins | | 3 | Environment & CLI | 3 | Documentation | Env vars, CLI flags, permission modes | | 4 | Authentication & Access | 3 | Mixed | Auth methods, permission rules | | 5 | Session & Runtime | 2 | Mixed | Session features, sandbox config | | 6 | Integration | 3 | Documentation | IDEs, cloud providers, CI/CD | ### Audit Types Explained | Type | Description | Has Audit Command? | Tracking Method | | ---- | ----------- | ------------------ | --------------- | | `automated` | Full audit via `/audit-*` | Yes | Pass rate, component count | | `manual` | Requires human review | No | Human review tracking | | `documentation` | Tracks doc coverage only | No | Doc coverage via docs-management queries | --- ## Tiered Validation Model (v2.2) Schema v2.2 introduces a **three-tier validation model** that automatically invalidates audits when changelog changes affect components, while using the most token-efficient validation method appropriate for each change type. ### Change Severity Classification | Change Type | Severity | Validation Requirement | Rationale | | ----------- | -------- | ---------------------- | --------- | | `feature` | Minor | Targeted validation (keyword check) | New features can be verified by checking keywords exist | | `deprecation` | Minor | Targeted validation (keyword check) | Deprecations can be verified by checking warnings documented | | `bugfix` | None | No validation needed | Bugfixes don't affect plugin documentation/compliance | | `behavior_change` | **Major** | **Full audit required** | Behavior changes may have wide-ranging impacts | | `security` | **Major** | **Full audit required** | Security changes require comprehensive review | ### Validation Tiers #### Tier 1: Targeted Validation (Minor Changes) For `feature` and `deprecation` changes: - **Method:** Grep-based keyword verification - **Evidence:** File path, line number, matched text - **Cost:** ~100-500 tokens per change - **Status on pass:** `VALIDATED` **Algorithm:** ```text 1. Load validation spec from granular_changelog change entry 2. For each keyword in validation.keywords: a. Run grep against target_skill directory b. If match found, record evidence (file, line, match text) 3. If matches >= required_matches: a. Set validation_result.validated = true b. Set confidence based on match count: - 1 match = "medium" - 2+ matches = "high" 4. Update component_coverage validation tracking ``` #### Tier 2: Full Audit (Major Changes) For `behavior_change` and `security` changes: - **Method:** Spawns auditor agent(s) - **Audit command:** Specified in validation.audit_command - **Cost:** ~3,000-8,000 tokens - **Status on pass:** `AUDITED` **Important:** Major changes **cannot** be validated via targeted validation. The system must enforce this - no shortcuts. #### Tier 3: Periodic Review Regardless of validation status: - **Threshold:** >90 days since last full audit - **Action:** Triggers full audit - **Rationale:** Ensures nothing drifts over time ### Validation Spec Schema (v2.2) Each change in `granular_changelog` can have a validation spec: ```yaml validation: method: "keyword_check" # keyword_check | full_audit | manual | none target_skill: "hook-management" # Which skill to validate (null for memory files) keywords: # Manual list - NEVER auto-extracted - "additionalContext" - "additional context" - "PreToolUse.*additionalContext" # Regex supported required_matches: 1 # How many keywords must match reason: "..." # Optional explanation (esp. for method: none) # For full_audit method: validation: method: "full_audit" target_skill: "permission-management" audit_command: "/audit-settings" reason: "Security fix - targeted validation insufficient" ``` ### Validation Result Schema After validation runs, results are recorded: ```yaml validation_result: validated: true # Did validation pass? validated_date: "2026-01-16" # When validated evidence: # For keyword_check method - file: "path/to/file.md" line: 142 match: "matched text" audit_performed: true # For full_audit method audit_date: "2026-01-12" # When audit was run confidence: "high" # high | medium | low ``` ### Status Calculation (Conservative) Component status is determined by this priority order (first match wins): | Priority | Condition | Status | | -------- | --------- | ------ | | 1 | Has `pending_major_changes` | `NEEDS AUDIT` ⚠️ | | 2 | Has `pending_minor_changes` | `NEEDS VALIDATION` | | 3 | `days_since(last_audit) > 90` | `STALE` | | 4 | `validation_version == latest AND validation_confidence == "high"` | `VALIDATED` | | 5 | `last_audit` recent AND no pending changes | `OK` | | 6 | Never audited | `UNKNOWN` | **Conservative Rule:** If ANY doubt exists, escalate to higher requirement. --- ## Validation Accuracy Rules (CRITICAL) The system must **NEVER** report "validated" unless certain. These rules are non-negotiable. ### Rule 1: Keyword Matches Must Be Contextually Correct ```text ❌ Finding "context" when looking for "context: fork" is NOT a match ✅ Must match exact keyword or regex pattern ``` ### Rule 2: Multiple Evidence Preferred | Evidence Count | Confidence Level | | -------------- | ---------------- | | 0 matches | `validation_failed: true` | | 1 match | `confidence: "medium"` | | 2+ matches | `confidence: "high"` | ### Rule 3: Human Confirmation for Edge Cases If confidence is `"low"`: - Flag for manual review - Do NOT auto-mark as validated - Include in `--apply` output for user decision ### Rule 4: Evidence is Mandatory - No validation without captured evidence - Evidence must include file path and line number - Empty evidence = validation failed ### Rule 5: Major Changes Cannot Use Targeted Validation ```text ❌ security change → keyword_check (FORBIDDEN) ❌ behavior_change → keyword_check (FORBIDDEN) ✅ security change → full_audit (REQUIRED) ✅ behavior_change → full_audit (REQUIRED) ``` The system must reject attempts to use targeted validation for major changes. The skill provides: 1. **Parsing changelogs** to identify new features and changes 2. **Tracking audit coverage** across all 27 component types 3. **Documentation coverage** for non-auditable components (queried from docs-management) 4. **Identifying pending updates** needed for compliance 5. **Orchestrating audits** efficiently (avoiding token waste) 6. **Helping apply updates** from new Claude Code versions ## When to Use This Skill Use this skill when: - Checking if ANY Claude Code extensibility component is up-to-date - Getting an overview of audit coverage and staleness across all tiers - Tracking documentation coverage for non-auditable components - Planning which audits to run (token-efficient approach) - Applying updates after Claude Code releases new versions - Preparing a plugin release - Detecting new/deprecated Claude Code features ## Tracking File **Location:** `.claude/ecosystem-health.yaml` This file persists ecosystem health state across sessions. It stores **audit metadata ONLY** - not component details (which are delegated to docs-management). **Schema v2.1 Structure:** ```yaml schema_version: "2.1" last_check: date: "YYYY-MM-DD" claude_code_version: "X.Y.Z" changelog_hash: "sha256:..." component_coverage: # Tier 1: Core Configuration tier1_configuration: user_settings: last_audit: null components_audited: 0 pass_rate: null audit_type: "manual" # Query docs-management for: settings.md user configuration project_settings: last_audit: "YYYY-MM-DD" components_audited: N pass_rate: 0.XX audit_type: "automated" audit_command: "/audit-settings" managed_settings: { ... } memory_system: { ... } # Tier 2: Plugin Components (12 components) tier2_plugins: skills: { audit_type: "automated", audit_command: "/audit-skills" } agents: { audit_type: "automated", audit_command: "/audit-agents" } # ... (12 components, each with audit_type and audit_command) # Tier 3-6: Documentation-tracked (DELEGATE to docs-management) tier3_environment: environment_variables: audit_type: "documentation" # DELEGATE: Query docs-management for: settings.md environment variables cli_flags: { ... } permission_modes: { ... } tier4_authentication: { ... } tier5_session: { ... } tier6_integration: { ... } changelog_versions_checked: - version: "X.Y.Z" checked_date: "YYYY-MM-DD" changes_applied: true/false pending_updates: - feature: "feature name" since_version: "X.Y.Z" affects: ["skills", "commands"] status: "pending" | "applied" | "skipped" last_discovery: date: "YYYY-MM-DD" docs_scanned: [...] changelog_version: "X.Y.Z" components_detected: 27 tiers_scanned: 6 gaps_found: 0 ``` **Key Design Decision:** The tracking file does NOT contain `tracked_*` arrays (no hardcoded lists of env vars, CLI flags, auth methods, etc.). All such data must be queried from docs-management at runtime. ## Changelog Access **MANDATORY:** Access changelog via `docs-management` skill.