rag

name	rag
description	Use for retrieval and grounding tasks: build or debug BM25, dense, hybrid, FAISS, metadata-filtered, or RRF retrieval pipelines; rank documents for queries; evaluate NDCG/MAP/Recall; and produce grounded outputs with document IDs, scores, ranks, and source traces.

Retrieval and Grounding

When To Use

Use this skill when the task requires finding evidence or documents for a query, combining sparse and dense search, reranking, evaluating retrieval quality, or ensuring generated answers cite retrieved context.

First Pass

1. Load corpus, queries, embeddings, metadata, and relevance judgments with stable IDs.
2. Define tokenization and normalization once and reuse it for indexing and queries.
3. Keep rankings as (doc_id, score, rank) records so metrics and outputs use the same order.
4. Separate retrieval quality from answer generation. A retrieval task should be correct even without an LLM.

Implementation Patterns

BM25

import math
import re
from collections import Counter, defaultdict

def tokenize(text):
    return re.findall(r"[a-z0-9]+", text.lower())

doc_tokens = [tokenize(doc["title"] + " " + doc["text"]) for doc in corpus]
doc_lens = [len(t) for t in doc_tokens]
avgdl = sum(doc_lens) / len(doc_lens)
df = Counter()
for toks in doc_tokens:
    df.update(set(toks))

def bm25(query, k1=1.2, b=0.75):
    scores = defaultdict(float)
    for term in tokenize(query):
        if term not in df:
            continue
        idf = math.log((len(corpus) - df[term] + 0.5) / (df[term] + 0.5) + 1.0)
        for i, toks in enumerate(doc_tokens):
            tf = toks.count(term)
            if tf == 0:
                continue
            denom = tf + k1 * (1 - b + b * doc_lens[i] / avgdl)
            scores[i] += idf * (tf * (k1 + 1)) / denom
    return sorted(scores.items(), key=lambda x: (-x[1], x[0]))

Dense FAISS Search

Use inner product for L2-normalized embeddings.

import faiss
import numpy as np

emb = np.load("embeddings.npy").astype("float32")
index = faiss.IndexFlatIP(emb.shape[1])
index.add(emb)

def normalize(v):
    norm = np.linalg.norm(v)
    return v / norm if norm else v

scores, idxs = index.search(query_vectors.astype("float32"), 100)

If the task defines a synthetic query vector, follow that definition exactly, including zero-vector fallback.

Reciprocal Rank Fusion

def rrf(bm25_ids, dense_ids, k=60, missing_rank=1000):
    ranks_a = {doc: r for r, doc in enumerate(bm25_ids, start=1)}
    ranks_b = {doc: r for r, doc in enumerate(dense_ids, start=1)}
    docs = set(ranks_a) | set(ranks_b)
    scored = []
    for doc in docs:
        score = 1 / (k + ranks_a.get(doc, missing_rank))
        score += 1 / (k + ranks_b.get(doc, missing_rank))
        scored.append((doc, score))
    return sorted(scored, key=lambda x: (-x[1], x[0]))

Metadata Filters In Batch Queries

When a batch retrieval API takes both query_embeddings and filters, two shapes are valid: a single filter dict broadcast across every query, or a list of per-query filter dicts. Disambiguate by type, not by length:

• If filters is a dict (or None), apply the same filter to all queries.
• If filters is a list, require len(filters) == len(query_embs) and raise a clear error otherwise. Do not silently truncate, pad, or recycle.

if isinstance(filters, list):
    if len(filters) != len(query_embs):
        raise ValueError(
            f"filters length {len(filters)} must match queries {len(query_embs)}"
        )
    per_query_filters = filters
else:
    per_query_filters = [filters] * len(query_embs)

This keeps single-dict broadcast and list-per-query alignment unambiguous and prevents off-by-one filter-to-query misassignment.

Metrics

Compute metrics per query, then macro-average. Use qrels by query_id; do not evaluate against all corpus IDs.

Validation

• Confirm corpus row index and doc_id mapping before scoring.
• Check that top-k results have deterministic tie-breaking.
• Validate metric formulas on a tiny hand-built query.
• Ensure every output query has exactly the required number of ranked results.
• For grounded answers, verify each cited source was actually retrieved.

Common Failures

• Using log(N / df) instead of BM25 IDF with +0.5 and +1.
• Forgetting document length normalization.
• Mixing corpus row number and public doc_id.
• Letting FAISS return arbitrary ties without stable sorting.
• Reporting best config from the wrong metric or unrounded values.