Comprehensive patterns for building production RAG systems. Each category has individual rule files in rules/ loaded on-demand.
House thresholds, fusion ordering, and the latency and quality budgets we assert live in
House delta below. Vendor documentation is linked, not restated (see
Upstream coverage).
Fetch these from the source instead of restating them here. Where a row says the house
subset stays, that named file carries only the tuned values and the reason, not a tutorial.
Topic
Source
pgvector install, operators, index build syntax, "why isn't my index used"
https://github.com/pgvector/pgvector; house subset (HNSW m=16, ef_construction=64, halfvec, binary quantize) stays in rules/pgvector-indexing.md and rules/pgvector-schema.md
Retrieval metric definitions (precision@k, recall@k, MRR, nDCG) and building the query set
ork skill golden-dataset
Search endpoint shape, pagination, error bodies
ork skill api-design
Tracing and dashboarding search calls
ork skill monitoring-observability
Writing the integration test that runs these assertions
ork skill testing-integration
Common Mistakes
No citation tracking (unverifiable answers)
Context too large (dilutes relevance)
Single retrieval method (misses keyword matches)
Not chunking long documents (context gets lost)
Embedding queries differently than documents
No fallback path in agentic RAG (workflow hangs)
Infinite rewrite loops (no retry limit)
Using wrong similarity metric (cosine vs euclidean)
Not caching embeddings (recomputing unchanged content)
Missing image captions in multimodal RAG (limits text search)
Evaluations
See test-cases.json for 30 test cases across all categories.
There is no pass-rate, precision, recall, or MRR floor for this skill. Earlier
wording here promised those numbers; they were never measured and are written down
nowhere. They are left unset rather than filled in with plausible-looking values,
because a threshold nobody measured is worse than an absent one: it gets cited in
review as though it means something, and passes or fails changes for no defensible
reason. test-cases.json is the right substrate for establishing real ones, and a
baseline run over those 30 cases would produce numbers worth asserting.
The one budget this skill does assert is latency, in House delta.
Inlined rather than placed in references/: this skill carries its house knowledge
in rules/ (32 files) and has never had a references/ directory, so a lone
delta file there would be the only occupant.
Embedding-model choice, chunking algorithms, vector-index tuning, reranker APIs and
the query-rewriting literature are vendor and upstream territory; see
Upstream coverage for where each lives. What
follows is only what OrchestKit adds, contradicts, or has to warn about.
Retrieval latency budget: p95 under 500ms end to end
Source: checklists/rag-quality.md:57, the only retrieval budget this skill has ever
actually asserted. It covers the whole path a user waits on (embed the query, search,
rerank, assemble context), not one stage in isolation. A pipeline that clears 500ms
per stage and blows it in aggregate has failed this budget.
Measure at p95, not mean. Retrieval latency is dominated by tail behaviour (cold index
shards, reranker queueing), and a mean hides exactly the requests users complain about.
Corpus-specific tuning does not transfer
Fusion weights, rerank depth and hybrid alpha are properties of a corpus, not of the
technique. A weighting that lifts recall on prose documentation regularly hurts it on
code or tabular data, so values copied between projects are noise rather than a
starting point. Re-derive them per corpus against that corpus's own eval set.
Apply metadata boosting after RRF fusion, never before
Why: boost factors multiply onto the RRF score of an already-fused result set and the list is then re-sorted, using section-title match 1.5x, section-path match 1.15x, and code block on a technical query 1.2x; boosting per-method scores before fusion destroys the property RRF depends on (it fuses by rank, not by score) and lets a lone strong keyword hit outrank a document both methods agreed on, and this exact order is what produced the +6% MRR credited to boosting in rules/pgvector-metadata.md (distilled from the retired checklists/search-implementation-checklist.md and examples/examples/orchestkit-retrieval.md; no traced incident).
Upstream: https://learn.microsoft.com/en-us/azure/search/index-add-scoring-profiles
Stop raising the RRF fetch multiplier past 3x
Why: measured pass rate on the reference corpus went 87.2% at 1x, 89.5% at 2x, 91.1% at 3x, 91.3% at 4x, so the fourth multiple bought 0.2 points for a third more rows scanned per method, which is why 3x is the house default hard-coded in rules/core-hybrid-search.md and rules/pgvector-hybrid-search.md; raise it only with a golden-set number that beats this curve (distilled from the retired examples/examples/orchestkit-retrieval.md; no traced incident).
Upstream: https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion
Gate every retrieval change on the golden-set thresholds
Why: the house bar before a retrieval change ships is pass rate at least 90% on the golden query set, precision@10 at least 0.70, recall@10 at least 0.85, and MRR at least 0.65, floors set just under the reference pipeline's observed 91.6% pass rate and 0.777 overall MRR (0.695 on the hard slice) so that a real regression trips them instead of a noisy run (distilled from the retired checklists/search-implementation-checklist.md and examples/examples/orchestkit-retrieval.md; no traced incident).
Upstream: ork skill golden-dataset
Assert search latency per stage, not just end to end
Why: the integration test asserts vector search under 100ms, keyword search under 50ms, and fused hybrid under 150ms separately, against an observed 415-chunk baseline of P50 15ms / P95 32ms / P99 62ms, because a single total-latency assertion stays green while one stage silently doubles, which is precisely the shape of an index that quietly stopped being used (distilled from the retired checklists/search-implementation-checklist.md and examples/examples/orchestkit-retrieval.md; no traced incident).
Upstream: https://www.postgresql.org/docs/current/using-explain.html
Budget the embedding call before tuning the index
Why: in the reference measurement the query embedding was P50 8ms of a 15ms end-to-end search against 2ms for vector search and 3ms for keyword search, and it was the throughput ceiling at roughly 120 requests per second, so cache and batch embeddings first because index tuning below that ceiling moves the total by single-digit percent (distilled from the retired examples/examples/orchestkit-retrieval.md; no traced incident).
Upstream: https://docs.voyageai.com/docs/embeddings
