LLM Integration
Patterns for integrating LLMs into production applications: tool use, streaming, local inference, and fine-tuning. Each category has individual rule files in rules/ loaded on-demand.
Quick Reference
| Category | Rules | Impact | When to Use |
|---|
| Function Calling | 3 | CRITICAL | Tool definitions, parallel execution, input validation |
| Streaming | 3 | HIGH | SSE endpoints, structured streaming, backpressure handling |
| Local Inference | 3 | HIGH | Ollama setup, model selection, GPU optimization |
| Fine-Tuning | 3 | HIGH | LoRA/QLoRA training, dataset preparation, evaluation |
| Context Optimization | 2 | HIGH | Window management, compression, caching, budget scaling |
| Evaluation | 2 | HIGH | LLM-as-judge, RAGAS metrics, quality gates, benchmarks |
| Prompt Engineering | 4 | HIGH | CoT, few-shot, versioning, DSPy optimization, ReAct, cost optimization |
Total: 20 rules across 7 categories
Quick Start
tools = [{
"type": "function",
"function": {
"name": "search_documents",
"description": "Search knowledge base",
"strict": True,
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query"},
"limit": {"type": "integer", "description": "Max results"}
},
"required": ["query", "limit"],
"additionalProperties": False
}
}
}]
@app.get("/chat/stream")
async def stream_chat(prompt: str):
async def generate():
async for token in async_stream(prompt):
yield {"event": "token", "data": token}
yield {"event": "done", "data": ""}
return EventSourceResponse(generate())
llm = ChatOllama(
model="deepseek-r1:70b",
base_url="http://localhost:11434",
temperature=0.0,
num_ctx=32768,
)
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/Meta-Llama-3.1-8B",
max_seq_length=2048, load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(model, r=16, lora_alpha=32)
Function Calling
Enable LLMs to use external tools and return structured data. Use strict mode schemas (2026 best practice) for reliability. Limit to 5-15 tools per request, validate all inputs with Pydantic/Zod, and return errors as tool results.
calling-tool-definition.md -- Strict mode schemas, OpenAI/Anthropic formats, LangChain bindingcalling-parallel.md -- Parallel tool execution, asyncio.gather, strict mode constraintscalling-validation.md -- Input validation, error handling, tool execution loops
Streaming
Deliver LLM responses in real-time for better UX. Use SSE for web, WebSocket for bidirectional. Handle backpressure with bounded queues.
streaming-sse.md -- FastAPI SSE endpoints, frontend consumers, async iteratorsstreaming-structured.md -- Streaming with tool calls, partial JSON parsing, chunk accumulationstreaming-backpressure.md -- Backpressure handling, bounded buffers, cancellation
Local Inference
Run LLMs locally with Ollama for cost savings (93% vs cloud), privacy, and offline development. Pre-warm models, use provider factory for cloud/local switching.
local-ollama-setup.md -- Installation, model pulling, environment configurationlocal-model-selection.md -- Model comparison by task, hardware profiles, quantizationlocal-gpu-optimization.md -- Apple Silicon tuning, keep-alive, CI integration
Fine-Tuning
Customize LLMs with parameter-efficient techniques. Fine-tune ONLY after exhausting prompt engineering and RAG. Requires 1000+ quality examples.
tuning-lora.md -- LoRA/QLoRA configuration, Unsloth training, adapter mergingtuning-dataset-prep.md -- Synthetic data generation, quality validation, deduplicationtuning-evaluation.md -- DPO alignment, evaluation metrics, anti-patterns
Context Optimization
Manage context windows, compression, and attention-aware positioning. Optimize for tokens-per-task.
context-window-management.md -- Five-layer architecture, anchored summarization, compression triggerscontext-caching.md -- Just-in-time loading, budget scaling, probe evaluation, CC 2.1.32+
Evaluation
Evaluate LLM outputs with multi-dimension scoring, quality gates, and benchmarks.
evaluation-metrics.md -- LLM-as-judge, RAGAS metrics, hallucination detectionevaluation-benchmarks.md -- Quality gates, batch evaluation, pairwise comparison
Prompt Engineering
Design, version, and optimize prompts for production LLM applications.
prompt-design.md -- Chain-of-Thought, few-shot learning, pattern selection guideprompt-testing.md -- Langfuse versioning, DSPy optimization, A/B testing, self-consistencyprompt-react-pattern.md -- ReAct loop for tool-using agents, thought-action-observation formatprompt-optimization.md -- Token reduction, cost optimization, model tiering, prompt spec format
Upstream coverage (do not restate)
These topics are covered by their vendors' own documentation. This skill points
at them instead of teaching them; the rules above keep only our floors, ceilings
and scars. Our delta on all of it is in references/ork-delta.md.
| Topic | First-party source |
|---|
| Strict-mode tool schemas, structured outputs | https://platform.openai.com/docs/guides/function-calling |
Anthropic input_schema / tool_use | https://docs.claude.com/en/docs/agents-and-tools/tool-use/overview |
| SSE client mechanics, reconnection, cancellation | https://developer.mozilla.org/en-US/docs/Web/API/EventSource |
| LoRA / QLoRA config, target modules, adapter merging | https://huggingface.co/docs/peft/developer_guides/lora |
| Unsloth training loop, 4-bit loading | https://docs.unsloth.ai/get-started/fine-tuning-llms-guide |
| DPO, preference pairs, beta tuning, RLHF comparison | https://huggingface.co/docs/trl/dpo_trainer |
| SFT dataset formats (Alpaca, ChatML) | https://huggingface.co/docs/trl/sft_trainer |
| Embedding similarity for dataset dedup | https://sbert.net/ |
| Fine-tune vs prompt vs RAG decision framework | https://platform.openai.com/docs/guides/optimizing-llm-accuracy |
| Vendor token pricing (never hardcode it here) | https://platform.openai.com/docs/pricing |
Supporting Files
references/ork-delta.md -- our delta: scars, house ceilings, retired-file provenancereferences/model-selection.md -- local model comparison by task and hardwarescripts/create-lora-config.md -- LoRA config scaffold with auto-detected model type
Key Decisions
| Decision | Recommendation |
|---|
| Tool schema mode | strict: true (2026 best practice) |
| Tool count | 5-15 max per request |
| Streaming protocol | SSE for web, WebSocket for bidirectional |
| Buffer size | 50-200 tokens |
| Local model (reasoning) | deepseek-r1:70b |
| Local model (coding) | qwen2.5-coder:32b |
| Fine-tuning approach | LoRA/QLoRA (try prompting first) |
| LoRA rank | 16-64 typical |
| Training epochs | 1-3 (more risks overfitting) |
| Context compression | Anchored iterative (60-80%) |
| Compress trigger | 70% utilization, target 50% |
| Judge model | claude-haiku-4-5-20251001 (cost tier) or gpt-5.5 |
| Quality threshold | 0.7 production, 0.6 drafts |
| Few-shot examples | 3-5 diverse, representative |
| Prompt versioning | Langfuse with labels |
| Auto-optimization | DSPy MIPROv2 |
Related Skills
ork:rag-retrieval -- Embedding patterns, when RAG is better than fine-tuningagent-loops -- Multi-step tool use with reasoningllm-evaluation -- Evaluate fine-tuned and local modelslangfuse-observability -- Track training experiments
Capability Details
function-calling
Keywords: tool, function, define tool, tool schema, function schema, strict mode, parallel tools
Solves:
- Define tools with clear descriptions and strict schemas
- Execute tool calls in parallel with asyncio.gather
- Validate inputs and handle errors in tool execution loops
streaming
Keywords: streaming, SSE, Server-Sent Events, real-time, backpressure, token stream
Solves:
- Stream LLM tokens via SSE endpoints
- Handle tool calls within streams
- Manage backpressure with bounded queues
local-inference
Keywords: Ollama, local, self-hosted, model selection, GPU, Apple Silicon
Solves:
- Set up Ollama for local LLM inference
- Select models based on task and hardware
- Optimize GPU usage and CI integration
fine-tuning
Keywords: LoRA, QLoRA, fine-tune, DPO, synthetic data, PEFT, alignment
Solves:
- Configure LoRA/QLoRA for parameter-efficient training
- Generate and validate synthetic training data
- Align models with DPO and evaluate results
