forge-ai-audit

name	forge-ai-audit
description	Review AI/LLM integrations - client configurations, prompt quality, and fallback safety. Trigger when user types /forge-ai-audit or requests an AI audit/review.

/forge-ai-audit

Review AI/LLM integrations, prompts, and configurations using AI reasoning.

Command

/forge-ai-audit [path] [--quick|--deep] [--markdown|--json] [--verbose]

Options

• [path] — Target directory (default: current directory)
• --quick — Fast scan, focus on critical issues only
• --deep — Thorough scan, check all patterns
• --markdown — Output as markdown (default)
• --json — Output as JSON
• --verbose — Include detailed evidence

Purpose

Analyze source code for:

• AI Client SDK configuration issues (temperature, max_tokens, retry policies, error-handling)
• Prompt quality, formatting, Socratic Tutor pitfalls, and output parsing safety
• Fallback mechanisms and content security/filtering for LLM responses

Workflow

Step 1: Understand Project

Use shared/analyzer/ to detect:

• Repository type
• Primary language
• Framework
• Project structure

Step 2: Run AI Integration Analysis

Execute analysis prompts in order:

1. Client SDK Configuration (prompts/client-config.md)
2. Prompt Quality & Format (prompts/prompt-quality.md)
3. Fallback & Content Security (prompts/fallback-safety.md)

Step 3: Classify Findings

For each finding:

• Assign severity: Critical, High, Medium, Low, Info
• Assign confidence: High, Medium, Low
• Include evidence (file, line, code snippet)

Step 4: Generate Report

Use shared/report/ to generate:

• AI Integration Score (0-100)
• Severity-grouped findings
• Recommendations for AI/LLM stability and accuracy

Output Format

Follow the markdown report format generated by shared/report/markdown.ts. It MUST use the following layout:

AI Integration Audit Report

Project: [Project Name]
Path: [Project Path]
Language: [Primary Language]
Framework: [Detected Framework (optional)]
Date: [Timestamp]

Score

[Score]/100 (Grade: [Grade])

Severity	Count
🔴 Critical	[count]
🟠 High	[count]
🟡 Medium	[count]
🔵 Low	[count]
⚪ Info	[count]

Executive Summary

Found [total] issues: [count] critical, [count] high, [count] medium, [count] low, [count] info.

🔍 Detailed Findings

[Severity Group (e.g. 🟠 High)]

Title	Location	Description	Impact
[Finding Title]	[file-name]:[line]	[Description]	[Impact]

📂 Code Evidence

Finding #[number]: [Finding Title]

[code snippet]