Use this skill when building production LLM applications, implementing guardrails, evaluating model outputs, or deciding between prompting and fine-tuning. Triggers on LLM app architecture, AI guardrails, output evaluation, model selection, embedding pipelines, vector databases, fine-tuning, function calling, tool use, and any task requiring production AI application design.
Use this skill when building production LLM applications, implementing guardrails, evaluating model outputs, or deciding between prompting and fine-tuning. Triggers on LLM app architecture, AI guardrails, output evaluation, model selection, embedding pipelines, vector databases, fine-tuning, function calling, tool use, and any task requiring production AI application design.
When this skill is activated, always start your first response with the 🧢 emoji.
LLM App Development
Building production LLM applications requires more than prompt engineering - it
demands the same reliability, observability, and safety thinking applied to any
critical system. This skill covers the full stack: architecture, guardrails,
evaluation pipelines, RAG, function calling, streaming, and cost optimization.
It emphasizes when patterns apply and what to do when they fail, not just
happy-path implementation.
When to use this skill
Trigger this skill when the user:
Designs the architecture for a new LLM-powered application or feature
Implements content filtering, PII detection, or schema validation on model I/O
Builds or improves an evaluation pipeline (automated evals, human review, A/B tests)
Sets up a RAG pipeline (chunking, embedding, retrieval, reranking)
Adds function calling or tool use to an agent or chat interface
Streams LLM responses to a client (SSE, token-by-token rendering)
Optimizes inference cost or latency (caching, model routing, prompt compression)
Decides whether to fine-tune a model or improve prompting instead
Do NOT trigger this skill for:
Pure ML research, model training from scratch, or academic benchmarking
Questions about a specific AI framework API (use the framework's own skill, e.g., mastra)
Key principles
Evaluate before you ship - A feature without evals is a feature you cannot
safely iterate on. Define success metrics and build automated checks before the
first production deployment.
Guardrails are non-negotiable - Validate both input and output on every
production request. Content filtering, PII scrubbing, and schema validation
belong in your request path, not as optional post-processing.
Start with prompting before fine-tuning - Fine-tuning is expensive, slow to
iterate, and hard to maintain. Exhaust systematic prompt engineering, few-shot
examples, and RAG before considering fine-tuning.
Design for failure and fallback - LLM calls fail: timeouts, rate limits,
malformed outputs, hallucinations. Every integration needs retry logic, output
validation, and a fallback response.
Cost-optimize from day one - Track token usage per feature. Cache deterministic
outputs. Route cheap queries to smaller models. Set hard budget limits.
Core concepts
LLM app stack
User input
-> Input guardrails (safety, PII, token limits)
-> Prompt construction (system prompt, context, few-shots, retrieved docs)
-> Model call (streaming or batch)
-> Output guardrails (schema validation, content check, hallucination detection)
-> Post-processing (formatting, citations, structured extraction)
-> Response to user
Every layer is an independent failure point and must be observable.
Embedding / vector DB architecture
Documents are chunked into overlapping segments, embedded into dense vectors,
and stored in a vector database. At query time the user message is embedded,
similar chunks are retrieved via ANN search, optionally reranked by a cross-encoder,
and injected into the context window. Chunk quality determines retrieval quality
more than model choice.
Caching strategies
Layer
What to cache
TTL
Exact cache
Identical prompt+params hash
Hours to days
Semantic cache
Fuzzy-match on embedding similarity
Minutes to hours
Embedding cache
Vectors for known documents
Until doc changes
KV prefix cache
Shared system prompt prefix (provider-side)
Session
Common tasks
Design LLM app architecture
Key decisions before writing code:
Decision
Options
Guide
Context strategy
Long context vs RAG
RAG if >50% of context is static documents
Output mode
Free text, structured JSON, tool calls
Use structured output for any downstream processing
State
Stateless, session, persistent memory
Default stateless; add memory only when proven necessary
Always validate output against a Zod or JSON Schema
Fine-tuning as first resort
Weeks of work, costly, hard to update; usually unnecessary
Exhaust few-shot prompting and RAG first
Ignoring token costs in dev
Small test prompts hide 10x token usage in production
Log token counts per call from day one; set usage alerts
Single monolithic prompt
Hard to test or improve any individual step
Decompose into a pipeline of smaller, testable prompt steps
No fallback on LLM failure
Rate limits or downtime = user-facing 500 errors
Retry with exponential backoff; fall back to smaller model or cached response
Gotchas
Streaming guardrails can only run post-completion - You cannot validate a streamed response mid-stream for content policy or schema compliance. The full text is only available after the last token. Run output guardrails after the stream ends, and design your client to handle a late rejection (e.g., replace streamed content with an error state) rather than assuming the stream is always valid.
JSON mode does not guarantee valid JSON on all providers - OpenAI's response_format: { type: "json_object" } reduces but does not eliminate parse errors, especially on long outputs that hit max_tokens. Always wrap JSON.parse() in a try/catch and treat a parse failure as a retriable error, not a crash.
RAG retrieval quality is dominated by chunk boundaries, not embedding models - Switching from text-embedding-3-small to text-embedding-3-large rarely fixes poor retrieval. Poor recall almost always traces to chunks that split mid-sentence or mid-concept. Fix chunking strategy (overlapping windows, semantic boundaries) before upgrading the embedding model.
Tool call loops can exceed maxSteps silently on some SDKs - If the model keeps calling tools without emitting a stop finish reason, some SDK wrappers will retry indefinitely. Always set an explicit maxSteps cap and treat a loop-exceeded condition as a hard error, not a retry.
Semantic caches can return stale or incorrect answers for slightly rephrased queries - A semantic cache that matches "What is the capital of France?" to "Tell me the capital of France" is fine. But caches with broad similarity thresholds can match unrelated questions with similar wording. Set cosine similarity thresholds conservatively (0.97+) for factual queries; use exact caching only for truly deterministic prompts.
References
For detailed content on specific sub-domains, load the relevant reference file:
Only load a reference file when the task specifically requires it - they are
long and will consume significant context.
Companion check
On first activation of this skill in a conversation: check which companion skills are installed by running ls ~/.claude/skills/ ~/.agent/skills/ ~/.agents/skills/ .claude/skills/ .agent/skills/ .agents/skills/ 2>/dev/null. Compare the results against the recommended_skills field in this file's frontmatter. For any that are missing, mention them once and offer to install:
