measure-ai-proficiency

# Measure AI Proficiency Assess repository context engineering maturity and provide actionable recommendations for improving AI collaboration. This skill works with Claude Code, GitHub Copilot, Cursor, and OpenAI Codex (via the [Agent Skills](https://agentskills.io/) open standard). ## Prerequisites Install the measure-ai-proficiency tool: ```bash pip install measure-ai-proficiency ``` ## Workflow ### 1. Choose Your Scanning Method #### Option A: Scan GitHub Directly (No Cloning Required!) Scan GitHub repositories without cloning them: ```bash # Scan a single GitHub repository measure-ai-proficiency --github-repo owner/repo # Scan entire GitHub organization measure-ai-proficiency --github-org your-org-name # Limit number of repos scanned measure-ai-proficiency --github-org your-org-name --limit 50 # Output to file measure-ai-proficiency --github-org your-org --format json --output report.json ``` **Requirements:** [GitHub CLI (gh)](https://cli.github.com/) authenticated with `gh auth login` **How it works:** - Uses GitHub API to fetch repository file tree - Downloads only AI proficiency files (CLAUDE.md, .cursorrules, skills, etc.) - Scans in temporary directories - Cleans up automatically - Much faster than cloning! #### Option B: Discover Then Clone (Traditional Method) For organizations wanting more control, first discover which repos have AI context artifacts: ```bash # Find active repos (commits in last 90 days) with AI context files ./scripts/find-org-repos.sh your-org-name # JSON output for automation ./scripts/find-org-repos.sh your-org-name --json > repos.json ``` **What you get:** - Total repos in organization - Active repos (with recent commits) - Repos with AI context artifacts (CLAUDE.md, AGENTS.md, .cursorrules, etc.) - Percentage baseline for your org - List of repos to scan **Requirements:** [GitHub CLI (gh)](https://cli.github.com/) and [jq](https://stedolan.github.io/jq/) Then clone and scan the identified repos. #### Option C: Scan Local Repositories ```bash # Scan current directory measure-ai-proficiency # Scan specific repository measure-ai-proficiency /path/to/repo # Scan multiple repositories measure-ai-proficiency /path/to/repo1 /path/to/repo2 # Scan all repos in directory (cloned org) measure-ai-proficiency --org /path/to/org-repos ``` ### 2. Run Assessment Most common commands: ```bash # Local scan measure-ai-proficiency # GitHub scan (recommended for orgs) measure-ai-proficiency --github-org your-org-name ``` ### 3. Interpret Results **Maturity Levels (aligned with Steve Yegge's 8-stage model):** | Level | Name | Yegge Stage | Indicators | |-------|------|-------------|------------| | 1 | Zero AI | Stage 1 | No AI-specific files (baseline) | | 2 | Basic Instructions | Stage 2 | CLAUDE.md, .cursorrules exist | | 3 | Comprehensive Context | Stage 3 | Architecture, conventions documented | | 4 | Skills & Automation | Stage 4 | Hooks, commands, memory files, skills | | 5 | Multi-Agent Ready | Stage 5 | Specialized agents, MCP configs | | 6 | Fleet Infrastructure | Stage 6 | Beads, shared context, workflows | | 7 | Agent Fleet | Stage 7 | Governance, scheduling, 10+ agents | | 8 | Custom Orchestration | Stage 8 | Gas Town, meta-automation, frontier | **Score interpretation:** File count matters more than percentage. The tool includes hundreds of patterns for comprehensive detection. ### Understanding Quality Scoring Each AI instruction file is scored 0-10 based on quality indicators: | Symbol | Indicator | What It Means | Points | |--------|-----------|---------------|--------| | § | Sections | Markdown headers (`##`) | 0-2 | | ⌘ | Paths | File/dir paths (`/src/`) | 0-2 | | $ | Commands | CLI in backticks | 0-2 | | ! | Constraints | never/avoid/don't | 0-2 | | ↻N | Commits | Git history (N commits) | 0-2 | **Commit scoring:** Files with 5+ commits get full points (indicates active maintenance). 3-4 commits = 1pt. ### Cross-Reference Detection The tool detects links between your AI instruction files: - **Markdown links:** `[architecture](ARCHITECTURE.md)` - **File mentions:** `"CONVENTIONS.md"` or `` `TESTING.md` `` - **Relative paths:** `./docs/API.md` - **Directory refs:** `skills/`, `.claude/commands/` Resolution tracking shows if referenced files exist (helps find broken links). **Bonus points:** Up to +10 points from cross-references (5 pts) + quality (5 pts). ### Content Validation The tool validates that your documentation references real files: - **Missing references:** Files mentioned in docs that don't exist - **Stale references:** References to deleted files (detected via git history) - **Template markers:** Uncustomized content (TODO, PLACEHOLDER, etc.) **Validation penalty:** Up to -4 points for validation issues. **Skip false positives:** If your docs contain example file names (meta-tools, templates), configure `skip_validation_patterns` in `.ai-proficiency.yaml`: ```yaml skip_validation_patterns: - "COMPLIANCE.md" # Example mentioned in docs - ".mcp.json" # Best practice not yet adopted - "examples/*" # All files under examples/ ``` ### 4. Provide Recommendations After assessment, offer to create missing high-priority files: **Level 2 gaps:** Create CLAUDE.md, .cursorrules, or .github/copilot-instructions.md **Level 3 gaps:** Create ARCHITECTURE.md, CONVENTIONS.md, or TESTING.md **Level 4 gaps:** - Create skills directories: `.claude/skills/`, `.github/skills/`, or `.cursor/skills/` - Add `.claude/commands/` for slash commands - Create MEMORY.md or LEARNINGS.md - Consider SOUL.md or IDENTITY.md (ClawdBot pattern) for agent personality - **Boris Cherny's key insight:** Add verification loops (tests, linters) - this 2-3x quality **Level 5 gaps:** - Create specialized agents in `.github/agents/` or `.claude/agents/` - Set up `.mcp.json` at root (Boris Cherny pattern) for team-shared tool configs - Add agents/HANDOFFS.md for agent coordination **Level 6 gaps:** Beads memory system, shared context, workflow pipelines **Level 7 gaps:** - Add GOVERNANCE.md for agent permissions and policies - Set up convoys/ or molecules/ (Gas Town work decomposition) - Consider swarm/, wisps/, polecats/ for advanced agent patterns **Level 8 gaps:** - Build custom orchestration in orchestration/ - Consider .gastown/ for Kubernetes-like agent management - Add protocols: MAIL_PROTOCOL.md, FEDERATION.md, ESCALATION.md, watchdog/ ### 5. Create Missing Files When creating context files, include: **CLAUDE.md structure:** - Project overview (what it does, who it's for) - Directory structure and key files - Important conventions and patterns - Common tasks and how to perform them - Things to avoid **ARCHITECTURE.md structure:** - System overview and purpose - Key components and responsibilities - Data flow between components - Important design decisions **CONVENTIONS.md structure:** - Naming conventions - Code organization patterns - Error handling approach - Testing conventions ## Quick Reference Common triggers for this skill: - "Assess my AI proficiency" - "How mature is my context engineering?" - "What context files should I add?" - "Help me improve for AI coding" - "Check my CLAUDE.md setup" - "Am I ready for AI-assisted development?" ## Customization Use the **customize-measurement** skill for guided configuration: ``` "Customize measurement for my repo" ``` Or see the manual guide: https://github.com/pskoett/measuring-ai-proficiency/blob/main/CUSTOMIZATION.md