confidence-check-skills

# Confidence Check Skills **Purpose**: Quantified pre-implementation validation system that prevents wasted effort by requiring ≥90% confidence before coding begins. **Critical Use Case**: Spend 100-200 tokens on validation to save 5,000-50,000 tokens on wrong-direction work. Proven results: 100% precision/recall in production testing. **Used By**: All agents before implementation - especially implementor, developer-agent, frontend-ui-developer, ml-model-implementor --- ## When to Use Confidence Check **MANDATORY before**: - ✅ Implementing new features or functionality - ✅ Major refactoring or architectural changes - ✅ Bug fixes (except critical production emergencies) - ✅ Adding new libraries, frameworks, or dependencies - ✅ Database schema changes - ✅ API endpoint creation - ✅ Authentication/security implementations **OPTIONAL/SKIP for**: - ❌ Trivial documentation updates - ❌ Simple typo fixes - ❌ Critical production hotfixes (time-sensitive) - ❌ Exploratory research (no code changes) --- ## The 5-Factor Assessment Model ### Overview Each factor contributes a weighted score to calculate overall confidence: | Factor | Weight | Purpose | Tools Used | |--------|--------|---------|------------| | **Duplicate Detection** | 25% | Prevent re-implementing existing solutions | Glob, Grep, ChromaDB | | **Architecture Alignment** | 25% | Verify tech stack compatibility | Read, Grep | | **Documentation Review** | 20% | Ensure official docs consulted | Glob, Read | | **OSS Reference** | 15% | Find proven working implementations | WebSearch | | **Root Cause Analysis** | 15% | Verify problem understanding | Read, analysis | **Formula**: ``` confidence = (duplicate × 0.25) + (architecture × 0.25) + (docs × 0.20) + (oss × 0.15) + (rootcause × 0.15) ``` **Result**: 0.0 to 1.0 (0% to 100%) --- ## Factor 1: Duplicate Detection (25%) **Question**: Does this functionality already exist in the codebase? ### Automated Checks ```javascript // Step 1: File pattern search const similarFiles = Glob({ pattern: `**/*${featureKeyword}*{.ts,.js,.py,.rs,.md}`, path: "." }); // Step 2: Code pattern search const similarCode = Grep({ pattern: coreLogicPattern, // e.g., "function authenticateUser" glob: "**/*.{ts,js,py,rs}", output_mode: "files_with_matches" }); // Step 3: ChromaDB semantic search (if available) const semanticMatches = mcp__chroma__query_documents({ collection_name: "codebase_features_all", query_texts: [featureDescription], n_results: 5, where: { "status": "implemented" } }); // Step 4: Scoring logic let duplicateScore = 1.0; // Default: PASS (no duplicates) if (similarFiles.length > 0) { duplicateScore = 0.0; // FAIL: Files with similar names found } else if (similarCode.length > 0) { duplicateScore = 0.5; // PARTIAL: Similar code patterns found } else if (semanticMatches.distances[0][0] < 0.3) { duplicateScore = 0.3; // PARTIAL: Semantically similar feature exists } return duplicateScore; // 0.0, 0.3, 0.5, or 1.0 ``` ### Pass Criteria - ✅ **PASS (1.0)**: No similar files, code, or semantic matches - ⚠️ **PARTIAL (0.5)**: Similar code patterns but different purpose - ⚠️ **PARTIAL (0.3)**: Semantic similarity but different implementation - ❌ **FAIL (0.0)**: Duplicate functionality already exists ### Example Scenarios **Scenario 1: JWT Authentication** ``` Feature: "Implement JWT authentication middleware" Checks: - Glob: **/*auth*.{ts,js} → Found: src/auth/jwt-middleware.ts ❌ - Grep: "function.*authenticate|jwt.*verify" → 3 matches ❌ - ChromaDB: "jwt authentication" → Distance 0.08 (highly similar) ❌ Score: 0.0 (FAIL - duplicate exists) Recommendation: Use existing src/auth/jwt-middleware.ts ``` **Scenario 2: Rate Limiter** ``` Feature: "Create API rate limiter" Checks: - Glob: **/*rate*limit*.{ts,js} → None found ✅ - Grep: "rateLimit|rate.*limit" → None found ✅ - ChromaDB: "rate limiting middleware" → Distance 0.82 (dissimilar) ✅ Score: 1.0 (PASS - no duplicates) Recommendation: Proceed with implementation ``` --- ## Factor 2: Architecture Alignment (25%) **Question**: Is this compatible with the current tech stack and patterns? ### Automated Checks ```javascript // Step 1: Read architecture documentation const claudeMd = Read({ file_path: "CLAUDE.md" }); const readmeMd = Read({ file_path: "README.md" }); const packageJson = Read({ file_path: "package.json" }); // Node.js const cargoToml = Read({ file_path: "Cargo.toml" }); // Rust const requirementsTxt = Read({ file_path: "requirements.txt" }); // Python // Step 2: Extract tech stack const techStack = { language: detectLanguage([packageJson, cargoToml, requirementsTxt]), frameworks: extractFrameworks(claudeMd, packageJson), patterns: extractPatterns(claudeMd, readmeMd), database: detectDatabase(claudeMd, packageJson) }; // Step 3: Verify compatibility let architectureScore = 1.0; // Default: PASS // Example checks: if (proposedTech.language !== techStack.language) { architectureScore = 0.0; // FAIL: Wrong language } else if (!techStack.frameworks.includes(proposedTech.framework)) { architectureScore = 0.5; // PARTIAL: New framework (may be acceptable) } else if (violatesPatterns(proposedTech, techStack.patterns)) { architectureScore = 0.3; // PARTIAL: Pattern violation } return architectureScore; // 0.0, 0.3, 0.5, or 1.0 ``` ### Pass Criteria - ✅ **PASS (1.0)**: Fully compatible with documented tech stack - ⚠️ **PARTIAL (0.5)**: Compatible but introduces new framework/library - ⚠️ **PARTIAL (0.3)**: Compatible but violates documented patterns - ❌ **FAIL (0.0)**: Incompatible (wrong language, framework, database) ### Example Scenarios **Scenario 1: Python Library in Rust Project** ``` Proposal: "Add pandas for data processing" Tech Stack: Rust (from Cargo.toml) Check: - Cargo.toml exists → Rust project ✅ - Proposed: Python pandas library ❌ Score: 0.0 (FAIL - language mismatch) Recommendation: Use Rust polars crate instead ``` **Scenario 2: New Framework Addition** ``` Proposal: "Use Tailwind CSS for styling" Tech Stack: React + plain CSS (from package.json) Check: - package.json shows React ✅ - Currently using plain CSS (no Tailwind) ⚠️ - CLAUDE.md doesn't prohibit Tailwind ✅ Score: 0.5 (PARTIAL - new framework, acceptable) Recommendation: Proceed, but document Tailwind addition ``` --- ## Factor 3: Documentation Review (20%) **Question**: Have relevant documentation and guides been consulted? ### Automated Checks ```javascript // Step 1: Find documentation const docsFound = Glob({ pattern: "**/{docs,documentation,README,CLAUDE}*.md", path: "." }); // Step 2: Search for relevant sections const relevantDocs = []; for (const docFile of docsFound) { const content = Read({ file_path: docFile }); // Check if doc is relevant to feature if (content.toLowerCase().includes(featureKeyword.toLowerCase())) { relevantDocs.push({ file: docFile, excerpts: extractRelevantSections(content, featureKeyword) }); } } // Step 3: Scoring logic let docsScore = 0.0; // Default: FAIL (no docs) if (relevantDocs.length === 0) { docsScore = 0.0; // FAIL: No relevant docs found } else if (relevantDocs.length >= 1) { docsScore = 1.0; // PASS: Relevant docs found and should be reviewed } // Special case: If docs directory doesn't exist at all if (docsFound.length === 0) { docsScore = 0.5; // PARTIAL: No docs exist (not agent's fault) } return docsScore; // 0.0, 0.5, or 1.0 ``` ### Pass Criteria - ✅ **PASS (1.0)**: Relevant documentation found and reviewed - ⚠️ **PARTIAL (0.5)**: No documentation exists in project (not agent's fault) - ❌ **FAIL (0.0)**: Documentation exists but not consulted ### Example Scenarios **Scenario 1: Documented Authentication Pattern** ``` Feature: "Implement OAuth2 flow" Check: - Glob: **/docs/**/*.md → Found: docs/authentication-guide.md ✅ - Read: docs/authentication-guide.md → Contains "OAuth2" section ✅ - Reviewed: Yes (agent read the OAuth2 section) ✅ Score: 1.0 (PASS - docs found and reviewed) Recommendation: Follow documented OAuth2 pattern ``` **Scenario 2: No Documentation** ``` Feature: "Create data export feature" Check: - Glob: **/docs/**/*.md → None found ⚠️ - Glob: **/README.md → Found but no mention of data export ⚠️ Score: 0.5 (PARTIAL - no docs exist) Recommendation: Proceed, but create docs after implementation ``` --- ## Factor 4: OSS Reference (15%) **Question**: Is there a proven, working implementation we can reference? ### Automated Checks ```javascript // Step 1: Search for existing implementations const githubSearch = WebSearch({ query: `${techStack} ${featureName} implementation site:github.com`, allowed_domains: ["github.com"] }); const npmSearch = WebSearch({ // For Node.js query: `${featureName} site:npmjs.com`, allowed_domains: ["npmjs.com"] }); const cratesSearch = WebSearch({ // For Rust query: `${featureName} site:crates.io`, allowed_domains: ["crates.io"] }); // Step 2: Parse quality metrics const references = parseSearchResults(githubSearch, npmSearch, cratesSearch); // Step 3: Scoring based on quality let ossScore = 0.0; // Default: FAIL if (references.some(ref => ref.stars >= 1000 && ref.maintained)) { ossScore = 1.0; // PASS: High-quality reference (1K+ stars, maintained) } else if (references.some(ref => ref.stars >= 100)) { ossScore = 0.7; // PARTIAL: Medium-quality reference (100+ stars) } else if (references.length > 0) { ossScore = 0.4; // PARTIAL: Low-quality reference (exists but unverified) } return ossScore; // 0.0, 0.4, 0.7, or 1.0 ``` ### Pass Criteria - ✅ **PASS (1.0)**: High-quality reference (1K+ stars, actively maintained) - ⚠️ **PARTIAL (0.7)**: Medium-quality reference (100+ stars) - ⚠️ **PARTIAL (0.4)**: Low-quality reference (exists but unverified) - ❌ **FAIL (0.0)**: No working implementation found ### Example Scenarios **Scenario 1: Express.js Rate Limiting** ``` Feature: "API rate limiter for Express" Tech Stack: Node.js + Express WebSearch: "express rate limiting npm" Results: - express-rate-limit: 3.2K stars, maintained ✅ - rate-limiter-flexible: 2.8K stars, maintained ✅ Score: 1.0 (PASS - multiple high-quality references) Recommendation: Use express-rate-limit (most popular) ``` **Scenario 2: Custom Algorithm** ``` Feature: "Implement custom sorting algorithm for trades" WebSearch: "custom trade sorting algorithm" Results: - No high-quality libraries found ❌ - Academic papers exist (not production code) ⚠️ Score: 0.0 (FAIL - no proven implementation) Recommendation: Implement custom, but add extensive tests ``` --- ## Factor 5: Root Cause Analysis (15%) **Question**: Is the underlying problem clearly understood? ### Manual Evaluation This check requires human judgment but follows a structured approach: ```javascript // Step 1: Analyze problem description const problemDescription = extractProblemFromRequest(userRequest); // Step 2: Check clarity indicators const clarityChecks = { symptomsDescribed: problemDescription.includes("error") || problemDescription.includes("fails") || problemDescription.includes("doesn't work"), rootCauseIdentified: problemDescription.includes("because") || problemDescription.includes("due to") || problemDescription.includes("caused by"), reproductionSteps: problemDescription.match(/\d+\.\s+/g)?.length >= 2, // Numbered steps expectedVsActual: problemDescription.includes("expected") && problemDescription.includes("actual"), contextProvided: problemDescription.length > 100 // Sufficient detail }; // Step 3: Scoring logic