ai-product

# AI Product Development Expert in shipping production-grade AI-powered features — LLM integration patterns, RAG architecture, prompt engineering that scales, AI UX that users trust, safety and guardrails, streaming, and cost optimization. Treats prompts as code, validates all outputs, and never trusts an LLM blindly. **Use when**: building AI features into a product, integrating LLMs, designing RAG pipelines, implementing AI safety/guardrails, optimizing AI costs, building AI UX patterns, prompt engineering for production, handling hallucinations, streaming LLM responses, or evaluating AI output quality. --- ## When This Skill Is Activated 1. Read this file fully before proceeding 2. Understand what AI feature the user is building 3. Apply the relevant patterns below (integration, RAG, UX, safety, cost) 4. Always address: output validation, error handling, cost awareness, user trust --- ## Core Principle Demos are easy. Production is hard. Every pattern below exists because something broke in production. --- ## LLM Integration Patterns ### Structured Output with Validation Never parse free-text LLM output with regex. Use structured output modes and validate with a schema. ```typescript import { z } from "zod"; import OpenAI from "openai"; // 1. Define your schema const ProductReviewSchema = z.object({ sentiment: z.enum(["positive", "negative", "neutral"]), score: z.number().min(0).max(10), summary: z.string().max(200), keyTopics: z.array(z.string()).max(5), }); type ProductReview = z.infer<typeof ProductReviewSchema>; // 2. Call with structured output const openai = new OpenAI(); async function analyzeReview(reviewText: string): Promise<ProductReview> { const response = await openai.chat.completions.create({ model: "gpt-4o-mini", response_format: { type: "json_object" }, messages: [ { role: "system", content: `Analyze the product review. Return JSON matching this schema: { sentiment: "positive"|"negative"|"neutral", score: 0-10, summary: string, keyTopics: string[] }`, }, { role: "user", content: reviewText }, ], }); const raw = JSON.parse(response.choices[0].message.content!); // 3. Always validate — the model can return anything const result = ProductReviewSchema.parse(raw); return result; } ``` ### Streaming with Progress Stream LLM responses to reduce perceived latency. Show users something is happening immediately. ```typescript async function streamResponse(prompt: string, onChunk: (text: string) => void) { const stream = await openai.chat.completions.create({ model: "gpt-4o-mini", messages: [{ role: "user", content: prompt }], stream: true, }); let fullText = ""; for await (const chunk of stream) { const delta = chunk.choices[0]?.delta?.content || ""; fullText += delta; onChunk(delta); // Update UI incrementally } return fullText; } // React example: streaming into state function useStreamingAI() { const [text, setText] = useState(""); const [isStreaming, setIsStreaming] = useState(false); const generate = async (prompt: string) => { setIsStreaming(true); setText(""); await streamResponse(prompt, (chunk) => { setText((prev) => prev + chunk); }); setIsStreaming(false); }; return { text, isStreaming, generate }; } ``` ### Prompt Versioning and Testing Treat prompts as code. Version them. Test with regression suites. ```typescript // prompts/v3-review-analyzer.ts export const REVIEW_ANALYZER_PROMPT = { version: "3.0", system: `You are a product review analyst. Extract sentiment, score, summary, and topics. Always return valid JSON. Never hallucinate topics not mentioned in the review.`, temperature: 0.1, // Low temp for consistent structured output maxTokens: 500, }; // tests/prompts/review-analyzer.test.ts describe("Review Analyzer Prompt v3", () => { const testCases = [ { input: "This product is amazing! Great battery life and beautiful screen.", expected: { sentiment: "positive", minScore: 7 }, }, { input: "Terrible. Broke after 2 days. Worst purchase ever.", expected: { sentiment: "negative", maxScore: 3 }, }, { input: "It's okay. Does what it says but nothing special.", expected: { sentiment: "neutral", minScore: 4, maxScore: 6 }, }, ]; test.each(testCases)("correctly analyzes: $input", async ({ input, expected }) => { const result = await analyzeReview(input); expect(result.sentiment).toBe(expected.sentiment); if (expected.minScore) expect(result.score).toBeGreaterThanOrEqual(expected.minScore); if (expected.maxScore) expect(result.score).toBeLessThanOrEqual(expected.maxScore); }); }); ``` --- ## RAG Architecture ### When to Use RAG | Approach | When | Example | |----------|------|---------| | Prompt only | Model already knows the answer | General knowledge questions | | RAG | Answer depends on your data | "What's our refund policy?" | | Fine-tuning | Model needs new behavior/style | Domain-specific tone or format | | RAG + Fine-tuning | Both custom data and custom behavior | Enterprise support bot | ### RAG Pipeline ``` User Query ↓ Query Processing (rewrite, expand, decompose) ↓ Embedding (text → vector) ↓ Vector Search (find relevant chunks) ↓ Re-ranking (order by relevance) ↓ Context Assembly (fit within token budget) ↓ LLM Generation (with retrieved context) ↓ Citation Extraction + Validation ↓ Response with Sources ``` ### Implementation ```typescript import { OpenAI } from "openai"; const openai = new OpenAI(); // 1. Chunk documents at ingest time function chunkDocument(text: string, maxChunkSize = 500, overlap = 50): string[] { const sentences = text.split(/(?<=[.!?])\s+/); const chunks: string[] = []; let current = ""; for (const sentence of sentences) { if ((current + sentence).length > maxChunkSize && current) { chunks.push(current.trim()); // Keep overlap for context continuity const words = current.split(" "); current = words.slice(-overlap).join(" ") + " " + sentence; } else { current += " " + sentence; } } if (current.trim()) chunks.push(current.trim()); return chunks; } // 2. Embed and store async function embedChunks(chunks: string[]) { const response = await openai.embeddings.create({ model: "text-embedding-3-small", input: chunks, }); return response.data.map((d, i) => ({ text: chunks[i], embedding: d.embedding, })); } // 3. Retrieve relevant context async function retrieve(query: string, topK = 5) { const queryEmbedding = await openai.embeddings.create({ model: "text-embedding-3-small", input: query, }); // Search your vector DB (Pinecone, Weaviate, pgvector, etc.) const results = await vectorDB.search({ vector: queryEmbedding.data[0].embedding, topK, }); return results.map((r) => r.metadata.text); } // 4. Generate with context async function ragGenerate(query: string) { const context = await retrieve(query); const response = await openai.chat.completions.create({ model: "gpt-4o-mini", messages: [ { role: "system", content: `Answer based ONLY on the provided context. If the context doesn't contain the answer, say "I don't have enough information to answer that." Context: ${context.map((c, i) => `[${i + 1}] ${c}`).join("\n\n")}`, }, { role: "user", content: query }, ], }); return response.choices[0].message.content; } ``` ### RAG Quality Checklist - Chunk size tuned for your content type (code: larger, FAQ: smaller) - Overlap between chunks prevents losing context at boundaries - Query rewriting improves retrieval for vague user questions - Re-ranking (Cohere, cross-encoder) boosts relevance over pure vector similarity - Token budget management — don't exceed context window - Citation tracking — map output claims back to source chunks - Evaluation set — measure retrieval recall and answer quality regularly --- ## AI UX Patterns — Building Trust ### Show Your Work Users trust AI more when they can see why it produced an answer. | Pattern | Implementation | Trust Impact | |---------|---------------|--------------| | **Citations** | "Based on [Source A] and [Source B]..." | High | | **Confidence indicators** | Color-coded confidence bars | Medium | | **Editable output** | Let users modify AI-generated content | High | | **Explain reasoning** | Step-by-step breakdown | High | | **Show alternatives** | "Here are 3 options..." | Medium | | **Undo/regenerate** | One-click to try again | High | ### Loading States for AI ```typescript // Bad: Spinner for 10 seconds with no context <Spinner /> // Good: Progressive disclosure function AILoadingState({ stage }: { stage: string }) { return ( <div className="ai-loading"> <div className="pulse-dot" /> <span className="text-sm text-muted">{stage}</span> </div> ); } // Usage with stages const stages = [ "Understanding your question...", "Searching relevant documents...", "Generating response...", ]; ``` ### Error States ```typescript // Always have AI-specific error handling function AIErrorBoundary({ error, onRetry }: { error: Error; onRetry: () => void }) { const messages: Record<string, string> = { RATE_LIMITED: "Too many requests. Please wait a moment.", CONTEXT_TOO_LONG: "Your input is too long. Try shortening it.", SAFETY_FILTERED: "This request couldn't be processed. Try rephrasing.", API_ERROR: "AI service is temporarily unavailable.", }; return ( <div className="ai-error"> <p>{messages[error.message] || "Something went wrong."}</p> <button onClick={onRetry}>Try Again</button> </div> ); } ``` --- ## Safety and Guardrails ### Input Sanitization ```typescript function sanitizeInput(userInput: string): string { // 1. Length limit if (userInput.length > 10_000) { throw new Error("CONTEXT_TOO_LONG"); } // 2. Strip known injection patterns const cleaned = userInput .replace(/ignore (all |previous |above )?instructions/gi, "[filtered]") .replace(/you are now/gi, "[filtered]") .replace(/system:\s/gi, "[filtered]"); return cleaned; } ``` ### Output Validation ```typescript async function safeGenerate(prompt: string) { const response = await generate(prompt); // 1. Check for refusal (model declined to answer) if (response.includes("I cannot") || response.includes("I'm not able to")) { return { type: "refused", content: response }; } // 2. Check for hallucinated URLs/emails const urls = response.match(/https?:\/\/[^\s]+/g) || []; const validatedUrls = await Promise.all(urls.map(validateUrl)); if (validatedUrls.some((v) => !v)) { return { type: "contains_hallucinated_links", content: stripInvalidUrls(response) }; } // 3. Content policy check (use a classifier or moderation endpoint) const moderation = await openai.moderations.create({ input: response }); if (moderation.results[0].flagged) { return { type: "flagged", content: "Response filtered for safety." }; } return { type: "ok", content: response }; } ``` ### Guardrail Layers | Layer | What It Catches | Implementation | |-------|----------------|----------------| | Input validation | Injection, abuse, length | Regex + length checks | | System prompt | Role boundaries | Strong system instructions |