confidence-check-skills

name	confidence-check-skills
description	Pre-implementation validation framework requiring ≥90% confidence before coding. Prevents wrong-direction work by assessing duplicates, architecture alignment, documentation, OSS references, and root cause understanding. Use before implementing features, fixes, or refactoring to save 5K-50K tokens per prevented error.
license	MIT

Confidence Check Skills

Purpose: Quantified pre-implementation validation system that prevents wasted effort by requiring ≥90% confidence before coding begins.

Critical Use Case: Spend 100-200 tokens on validation to save 5,000-50,000 tokens on wrong-direction work. Proven results: 100% precision/recall in production testing.

Used By: All agents before implementation - especially implementor, developer-agent, frontend-ui-developer, ml-model-implementor

---

When to Use Confidence Check

MANDATORY before:

• ✅ Implementing new features or functionality
• ✅ Major refactoring or architectural changes
• ✅ Bug fixes (except critical production emergencies)
• ✅ Adding new libraries, frameworks, or dependencies
• ✅ Database schema changes
• ✅ API endpoint creation
• ✅ Authentication/security implementations

OPTIONAL/SKIP for:

• ❌ Trivial documentation updates
• ❌ Simple typo fixes
• ❌ Critical production hotfixes (time-sensitive)
• ❌ Exploratory research (no code changes)

---

The 5-Factor Assessment Model

Overview

Each factor contributes a weighted score to calculate overall confidence:

Factor	Weight	Purpose	Tools Used
Duplicate Detection	25%	Prevent re-implementing existing solutions	Glob, Grep, ChromaDB
Architecture Alignment	25%	Verify tech stack compatibility	Read, Grep
Documentation Review	20%	Ensure official docs consulted	Glob, Read
OSS Reference	15%	Find proven working implementations	WebSearch
Root Cause Analysis	15%	Verify problem understanding	Read, analysis

Formula:

confidence = (duplicate × 0.25) + (architecture × 0.25) + (docs × 0.20)
           + (oss × 0.15) + (rootcause × 0.15)

Result: 0.0 to 1.0 (0% to 100%)

---

Factor 1: Duplicate Detection (25%)

Question: Does this functionality already exist in the codebase?

Automated Checks

// Step 1: File pattern search
const similarFiles = Glob({
  pattern: `**/*${featureKeyword}*{.ts,.js,.py,.rs,.md}`,
  path: "."
});

// Step 2: Code pattern search
const similarCode = Grep({
  pattern: coreLogicPattern,  // e.g., "function authenticateUser"
  glob: "**/*.{ts,js,py,rs}",
  output_mode: "files_with_matches"
});

// Step 3: ChromaDB semantic search (if available)
const semanticMatches = mcp__chroma__query_documents({
  collection_name: "codebase_features_all",
  query_texts: [featureDescription],
  n_results: 5,
  where: { "status": "implemented" }
});

// Step 4: Scoring logic
let duplicateScore = 1.0;  // Default: PASS (no duplicates)

if (similarFiles.length > 0) {
  duplicateScore = 0.0;  // FAIL: Files with similar names found
} else if (similarCode.length > 0) {
  duplicateScore = 0.5;  // PARTIAL: Similar code patterns found
} else if (semanticMatches.distances[0][0] < 0.3) {
  duplicateScore = 0.3;  // PARTIAL: Semantically similar feature exists
}

return duplicateScore;  // 0.0, 0.3, 0.5, or 1.0

Pass Criteria

• ✅ PASS (1.0): No similar files, code, or semantic matches
• ⚠️ PARTIAL (0.5): Similar code patterns but different purpose
• ⚠️ PARTIAL (0.3): Semantic similarity but different implementation
• ❌ FAIL (0.0): Duplicate functionality already exists

Example Scenarios

Scenario 1: JWT Authentication

Feature: "Implement JWT authentication middleware"

Checks:
- Glob: **/*auth*.{ts,js} → Found: src/auth/jwt-middleware.ts ❌
- Grep: "function.*authenticate|jwt.*verify" → 3 matches ❌
- ChromaDB: "jwt authentication" → Distance 0.08 (highly similar) ❌

Score: 0.0 (FAIL - duplicate exists)
Recommendation: Use existing src/auth/jwt-middleware.ts

Scenario 2: Rate Limiter

Feature: "Create API rate limiter"

Checks:
- Glob: **/*rate*limit*.{ts,js} → None found ✅
- Grep: "rateLimit|rate.*limit" → None found ✅
- ChromaDB: "rate limiting middleware" → Distance 0.82 (dissimilar) ✅

Score: 1.0 (PASS - no duplicates)
Recommendation: Proceed with implementation

---

Factor 2: Architecture Alignment (25%)

Question: Is this compatible with the current tech stack and patterns?

Automated Checks

// Step 1: Read architecture documentation
const claudeMd = Read({ file_path: "CLAUDE.md" });
const readmeMd = Read({ file_path: "README.md" });
const packageJson = Read({ file_path: "package.json" });  // Node.js
const cargoToml = Read({ file_path: "Cargo.toml" });      // Rust
const requirementsTxt = Read({ file_path: "requirements.txt" });  // Python

// Step 2: Extract tech stack
const techStack = {
  language: detectLanguage([packageJson, cargoToml, requirementsTxt]),
  frameworks: extractFrameworks(claudeMd, packageJson),
  patterns: extractPatterns(claudeMd, readmeMd),
  database: detectDatabase(claudeMd, packageJson)
};

// Step 3: Verify compatibility
let architectureScore = 1.0;  // Default: PASS

// Example checks:
if (proposedTech.language !== techStack.language) {
  architectureScore = 0.0;  // FAIL: Wrong language
} else if (!techStack.frameworks.includes(proposedTech.framework)) {
  architectureScore = 0.5;  // PARTIAL: New framework (may be acceptable)
} else if (violatesPatterns(proposedTech, techStack.patterns)) {
  architectureScore = 0.3;  // PARTIAL: Pattern violation
}

return architectureScore;  // 0.0, 0.3, 0.5, or 1.0

Pass Criteria

• ✅ PASS (1.0): Fully compatible with documented tech stack
• ⚠️ PARTIAL (0.5): Compatible but introduces new framework/library
• ⚠️ PARTIAL (0.3): Compatible but violates documented patterns
• ❌ FAIL (0.0): Incompatible (wrong language, framework, database)

Example Scenarios

Scenario 1: Python Library in Rust Project

Proposal: "Add pandas for data processing"
Tech Stack: Rust (from Cargo.toml)

Check:
- Cargo.toml exists → Rust project ✅
- Proposed: Python pandas library ❌

Score: 0.0 (FAIL - language mismatch)
Recommendation: Use Rust polars crate instead

Scenario 2: New Framework Addition

Proposal: "Use Tailwind CSS for styling"
Tech Stack: React + plain CSS (from package.json)

Check:
- package.json shows React ✅
- Currently using plain CSS (no Tailwind) ⚠️
- CLAUDE.md doesn't prohibit Tailwind ✅

Score: 0.5 (PARTIAL - new framework, acceptable)
Recommendation: Proceed, but document Tailwind addition

---

Factor 3: Documentation Review (20%)

Question: Have relevant documentation and guides been consulted?

Automated Checks

// Step 1: Find documentation
const docsFound = Glob({
  pattern: "**/{docs,documentation,README,CLAUDE}*.md",
  path: "."
});

// Step 2: Search for relevant sections
const relevantDocs = [];
for (const docFile of docsFound) {
  const content = Read({ file_path: docFile });

  // Check if doc is relevant to feature
  if (content.toLowerCase().includes(featureKeyword.toLowerCase())) {
    relevantDocs.push({
      file: docFile,
      excerpts: extractRelevantSections(content, featureKeyword)
    });
  }
}

// Step 3: Scoring logic
let docsScore = 0.0;  // Default: FAIL (no docs)

if (relevantDocs.length === 0) {
  docsScore = 0.0;  // FAIL: No relevant docs found
} else if (relevantDocs.length >= 1) {
  docsScore = 1.0;  // PASS: Relevant docs found and should be reviewed
}

// Special case: If docs directory doesn't exist at all
if (docsFound.length === 0) {
  docsScore = 0.5;  // PARTIAL: No docs exist (not agent's fault)
}

return docsScore;  // 0.0, 0.5, or 1.0

Pass Criteria

• ✅ PASS (1.0): Relevant documentation found and reviewed
• ⚠️ PARTIAL (0.5): No documentation exists in project (not agent's fault)
• ❌ FAIL (0.0): Documentation exists but not consulted

Example Scenarios

Scenario 1: Documented Authentication Pattern

Feature: "Implement OAuth2 flow"

Check:
- Glob: **/docs/**/*.md → Found: docs/authentication-guide.md ✅
- Read: docs/authentication-guide.md → Contains "OAuth2" section ✅
- Reviewed: Yes (agent read the OAuth2 section) ✅

Score: 1.0 (PASS - docs found and reviewed)
Recommendation: Follow documented OAuth2 pattern

Scenario 2: No Documentation

Feature: "Create data export feature"

Check:
- Glob: **/docs/**/*.md → None found ⚠️
- Glob: **/README.md → Found but no mention of data export ⚠️

Score: 0.5 (PARTIAL - no docs exist)
Recommendation: Proceed, but create docs after implementation

---

Factor 4: OSS Reference (15%)

Question: Is there a proven, working implementation we can reference?

Automated Checks

// Step 1: Search for existing implementations
const githubSearch = WebSearch({
  query: `${techStack} ${featureName} implementation site:github.com`,
  allowed_domains: ["github.com"]
});

const npmSearch = WebSearch({  // For Node.js
  query: `${featureName} site:npmjs.com`,
  allowed_domains: ["npmjs.com"]
});

const cratesSearch = WebSearch({  // For Rust
  query: `${featureName} site:crates.io`,
  allowed_domains: ["crates.io"]
});

// Step 2: Parse quality metrics
const references = parseSearchResults(githubSearch, npmSearch, cratesSearch);

// Step 3: Scoring based on quality
let ossScore = 0.0;  // Default: FAIL

if (references.some(ref => ref.stars >= 1000 && ref.maintained)) {
  ossScore = 1.0;  // PASS: High-quality reference (1K+ stars, maintained)
} else if (references.some(ref => ref.stars >= 100)) {
  ossScore = 0.7;  // PARTIAL: Medium-quality reference (100+ stars)
} else if (references.length > 0) {
  ossScore = 0.4;  // PARTIAL: Low-quality reference (exists but unverified)
}

return ossScore;  // 0.0, 0.4, 0.7, or 1.0

Pass Criteria

• ✅ PASS (1.0): High-quality reference (1K+ stars, actively maintained)
• ⚠️ PARTIAL (0.7): Medium-quality reference (100+ stars)
• ⚠️ PARTIAL (0.4): Low-quality reference (exists but unverified)
• ❌ FAIL (0.0): No working implementation found

Example Scenarios

Scenario 1: Express.js Rate Limiting

Feature: "API rate limiter for Express"
Tech Stack: Node.js + Express

WebSearch: "express rate limiting npm"
Results:
- express-rate-limit: 3.2K stars, maintained ✅
- rate-limiter-flexible: 2.8K stars, maintained ✅

Score: 1.0 (PASS - multiple high-quality references)
Recommendation: Use express-rate-limit (most popular)

Scenario 2: Custom Algorithm

Feature: "Implement custom sorting algorithm for trades"

WebSearch: "custom trade sorting algorithm"
Results:
- No high-quality libraries found ❌
- Academic papers exist (not production code) ⚠️

Score: 0.0 (FAIL - no proven implementation)
Recommendation: Implement custom, but add extensive tests

---

Factor 5: Root Cause Analysis (15%)

Question: Is the underlying problem clearly understood?

Manual Evaluation

This check requires human judgment but follows a structured approach:

// Step 1: Analyze problem description
const problemDescription = extractProblemFromRequest(userRequest);

// Step 2: Check clarity indicators
const clarityChecks = {
  symptomsDescribed: problemDescription.includes("error") ||
                     problemDescription.includes("fails") ||
                     problemDescription.includes("doesn't work"),

  rootCauseIdentified: problemDescription.includes("because") ||
                       problemDescription.includes("due to") ||
                       problemDescription.includes("caused by"),

  reproductionSteps: problemDescription.match(/\d+\.\s+/g)?.length >= 2,  // Numbered steps

  expectedVsActual: problemDescription.includes("expected") &&
                    problemDescription.includes("actual"),

  contextProvided: problemDescription.length > 100  // Sufficient detail
};

// Step 3: Scoring logic