agentsop-multi-tenant-rag

# Multi-Tenant RAG · Security-First Isolation SOP > Third-person operating model for a coder agent that owns retrieval correctness > across tenant boundaries. The audience is the LLM agent writing or reviewing > the code — not the end user. > **One sentence**: *Isolation lives at the vector store query boundary, not > at the model. Anything that reaches the LLM's context window has already > leaked.* --- ## 1. 何时激活 (Activation Rules) Activate this skill whenever **any** of the following holds: 1. The codebase contains a retrieval call (`vector_store.query`, `query_points`, `similarity_search`, `as_retriever().retrieve(...)`, raw `pgvector` `ORDER BY embedding <-> $1`) **and** the corpus serves more than one tenant, customer, organisation, workspace, user, or permission scope. 2. The user mentions any of: multi-tenant RAG, namespace, tenant, workspace, `tenant_id`, `org_id`, `user_id`, "cross-customer", "shared index", "knowledge base per team". 3. A bug report says "User A saw User B's document", "wrong company's data surfaced", "the assistant cited a doc I don't have access to", or anything that smells like cross-context bleed. 4. PR review: any new code calling a vector store **without** a tenant-scoped filter argument, or filtering only on the returned `nodes` / `documents` list after retrieval. 5. A new RAG endpoint is about to ship and tenant scoping has not been explicitly audited. Do **not** activate when: - The corpus is fully public and there is no per-tenant view (e.g. open-data Q&A). - The retrieval pipeline already enforces a **physically separate** index / collection per tenant via infrastructure the application code cannot override (e.g. one Pinecone index per customer with credentials issued per-tenant). In that case the isolation lives in IAM, not in this skill. --- ## 2. 核心心智模型 (Core Mental Model) Three principles. If a design violates any of them the system is exploitable, regardless of how good the LLM prompt is. ### Principle 1 — Isolation lives at the boundary, not at the model The boundary is the vector store query call. **Whatever crosses that boundary is in the trust set.** An LLM, a reranker, or a postprocessor that "filters out" foreign-tenant chunks is operating *inside* the breach: those tokens have already been embedded, retrieved, scored, and exposed to attacker-controlled prompts. CVE-2024-41892 (Pinecone, 2024) is the canonical demonstration — RBAC checks executed *after* retrieval allowed sentinel data to cross namespace boundaries before the access check fired. (See *we45*, *CSO Online* references.) > **Operational corollary**: any code shaped `results = vs.query(...); results > = [r for r in results if r.metadata["tenant"] == ctx.tenant]` is a security > defect, not a style nit. The leak already happened — you only hid it from > the user. ### Principle 2 — Embed the tenant key at ingestion, not just at query A query-time filter is only enforceable if every vector carries the key. The two failure shapes: - *Forgotten field*: a single document ingested without `tenant_id` becomes a shared global record returned to all tenants — it matches no filter predicate that requires equality. - *Mutable field*: `tenant_id` derived from the document body / LLM extraction rather than from the request's authenticated session — attacker can craft document content that re-labels itself. The tenant key **must** come from the authenticated session at ingestion time and be written into the immutable payload / metadata column. Treat it like a foreign key to your tenants table. ### Principle 3 — Defence in depth: namespace **and** filter, not namespace **or** filter Most production-grade stacks layer two mechanisms: | Layer | Mechanism | What it stops | |---|---|---| | Storage | Per-tenant namespace / shard / collection / RLS policy | Operator bugs, mis-routed queries, ops mistakes | | Query | `filter={tenant_id: $session.tenant}` on every call | App bugs, namespace selection mistakes, cross-shard joins | Pinecone's official guidance is "namespace per tenant in serverless"; Weaviate mandates a tenant handle on every CRUD when `multiTenancyConfig.enabled=true`; Qdrant marks the field as `is_tenant=true` *and* requires `Filter.must` on read; pgvector pairs schema partitioning with RLS. The principle is the same: one layer is one bug away from failing open. --- ## 3. SOP 工作流 (Agentic Protocol) Eight stages. Each gates the next. Stop and remediate at the first failure. ### Stage 0 — Threat-model the boundary Before any code change, write down: 1. **Tenant key**: what identifier separates customers? (`tenant_id`, `org_id`, `workspace_id`, `user_id`, `acl_label`). Pick one — do not mix. 2. **Trust source**: where does the request-time tenant key come from? It must be from the authenticated session / JWT claim / signed header — **never from a query param, prompt, or document body**. 3. **Cardinality**: how many tenants? (<100 → namespace/collection viable; >10k → payload partition + index is mandatory). Drives Stage 2 choice. 4. **Blast radius**: if a single vector leaks, what's the worst document? (PII / regulated / IP / public). Raises bar on Stages 4-6. Artifact: a 5-line `THREAT_MODEL.md` snippet co-located with the retriever. ### Stage 1 — Pick the isolation primitive (vendor-specific) | Vendor | Recommended primitive | When | |---|---|---| | **Pinecone** | One **namespace** per tenant in a serverless index | Default; cheapest query (1 RU per 1 GB / namespace); offboarding = `delete_namespace` | | **Weaviate** | `multiTenancyConfig.enabled=True` on the collection | Native, per-shard isolation; mandatory tenant handle on CRUD | | **Qdrant** | Single collection, payload key with `is_tenant=true`, `keyword` index | Scales to millions of tenants; physical co-location, logical isolation | | **Chroma** | `where={"tenant_id": "..."}` on every `query`/`get`/`update`/`delete` | Smallest stacks; cookbook calls this the "naive multi-tenancy strategy" — accept its limits | | **pgvector** | Postgres schema or row-level security (RLS) on `tenant_id` column | When the rest of the app already lives in Postgres | | **Milvus** | `partition_key` on tenant field | Same shape as Qdrant `is_tenant` | Decision rule: **prefer the strongest physical primitive your tenant cardinality can afford**, then always add the filter at query time anyway (Principle 3). ### Stage 2 — Ingestion: write the key, lock the schema ```python # Canonical shape — vendor-agnostic node.metadata = { "tenant_id": session.tenant_id, # MUST come from authenticated session "doc_id": doc.id, "source": doc.uri, "ingested_at": now_utc(), # ... domain fields } assert node.metadata["tenant_id"], "refuse to embed without tenant_id" ``` Hard rules: - Refuse to write a vector if `tenant_id` is missing or empty — fail loud, do not default. - `tenant_id` is **never** taken from document content; **never** mutable post-ingestion. - Schema-validate metadata before embedding (Pydantic / JSON Schema) — type drift breaks `$eq` silently. - For Qdrant: create the payload index *before* the first upsert (`create_payload_index(field="tenant_id", schema="keyword", is_tenant=True)`). - For Weaviate: create the tenant (`collection.tenants.create([Tenant(name=t)])`) before any write. ### Stage 3 — Query-time filter (the only place the rule applies) **Every** retrieval call must accept a tenant key from the session and pass it into the vendor's filter argument. Templates in OP-04 through OP-08. ```python # Generic LlamaIndex shape — works across Pinecone, Qdrant, Weaviate, Chroma from llama_index.core.vector_stores import ( MetadataFilter, MetadataFilters, FilterOperator, ) filters = MetadataFilters(filters=[ MetadataFilter(key="tenant_id", value=session.tenant_id, operator=FilterOperator.EQ), ]) retriever = index.as_retriever(similarity_top_k=8, filters=filters) ``` If the framework or vendor SDK does not expose a query-time filter, **switch the framework or vendor — do not patch with a post-filter**. ### Stage 4 — Cross-tenant property test (the gate) Before merging, the PR must include — and CI must run — a test that: 1. Ingests N≥3 documents per tenant for at least 2 tenants (A and B). 2. Issues a query as tenant A whose semantic top-1 in the *unfiltered* search would be a tenant-B document (craft the query against B's content). 3. Asserts zero tenant-B results returned. 4. Repeats symmetrically as tenant B. If the unfiltered top-1 is *not* a foreign-tenant doc, the test is dishonest — rewrite the corpus to make foreign-tenant semantically closer, otherwise the test always passes vacuously. See OP-09 for a runnable scaffold. ### Stage 5 — Defense in depth (Principle 3) Layer at least one of: - Per-tenant namespace / shard / collection on top of the metadata filter. - Per-tenant API key / scoped credential at the vector store layer (Pinecone API key project scope, Postgres role per tenant). - DB-level enforcement (pgvector RLS policy `USING (tenant_id = current_setting('app.current_tenant')::uuid)`). A single layer is one config bug away from open. ### Stage 6 — Audit logging Every retrieval call logs: ``` {ts, request_id, session.tenant_id, query_hash, vs.namespace_or_collection, filter_clause, returned_count, returned_tenant_ids_set} ``` Then add a synchronous assertion in the request path: ```python assert returned_tenant_ids_set.issubset({session.tenant_id, GLOBAL_TENANT}) ``` This converts a silent leak into a loud 500 — the right failure mode. ### Stage 7 — Cache / rerank / postprocessing audit Walk every component **after** retrieval and confirm none of them: - Caches results across tenants under a tenant-agnostic key (e.g. `cache_key = hash(query)` is a leak; correct is `cache_key = hash((tenant_id, query))`). - Returns "fallback" results from a global pool when the tenant pool is empty. - Logs full retrieved content to a shared monitoring store readable by other tenants' staff. Reranker / NodePostprocessor / synthesizer can only safely *narrow* the set; they never restore foreign tenants and never invent context — confirm by reading the code path. ### Stage 8 — Ongoing verification - Add the cross-tenant property test to nightly CI, not just merge gate. - On schema migration: re-run with the new chunking / embedding model. - On vendor upgrade: re-read the changelog for filter / namespace semantics (this changes more than vendors admit — see Pinecone disk-based metadata filtering, Qdrant 1.16 tiered multitenancy). --- ## 4. 操作模型 (Operation Models) Format: **Trigger / Action / Output / Evidence**. Vendor-specific filter syntax canonicalised against current docs (May 2026). ### OP-01 EmbedTenantAtIngest - **Trigger**: New ingestion pipeline; any system not currently writing `tenant_id` into metadata. - **Action**: Add `tenant_id` (from authenticated session) to every node's metadata at chunk creation. Validate non-empty before `add()` / `upsert()`. Make the field part of the ingestion schema. - **Output**: 100% of vectors carry the immutable tenant key; ingestion refuses to write otherwise. - **Evidence**: Qdrant *Multitenancy* docs; Pinecone *Implement multitenancy* guide; we45 RAG leakage post-mortem. ### OP-02 ChooseIsolationPrimitive - **Trigger**: Greenfield RAG, or migration from single-tenant. - **Action**: Use the cardinality / latency / cost matrix in Stage 1. Default ladder: <100 tenants → namespace/collection per tenant; 100-10k → payload partition + index (`is_tenant`); >10k → tiered (hot collection + cold archive). Pair with query filter (Principle 3). - **Output**: A documented primitive + the one-line justification. - **Evidence**: Pinecone *Namespaces vs. metadata filtering*; Qdrant *Multitenancy and custom sharding*; Weaviate *Multi-tenancy operations*. ### OP-03 LockTenantSourceToSession - **Trigger**: Code review reveals `tenant_id = request.json["tenant_id"]` or similar untrusted source. - **Action**: Bind `tenant_id` to the authenticated principal (JWT claim, session row, signed header). Pass through one chokepoint (`Context` / `RequestState`) — never read user input again downstream. - **Output**: A single `get_tenant_id(ctx)` accessor used everywhere; no string-typed tenant ids floating through function args. - **Evidence**: OWASP LLM01 (prompt injection) + LLM06 (sensitive info disclosure); Christian Schneider *RAG security: the forgotten attack surface*. ### OP-04 PineconeFilterAtQuery - **Trigger**: Pinecone-backed RAG; multi-tenant. - **Action**: One namespace per tenant; **and** pass `filter={"tenant_id": {"$eq": tenant_id}}`. Avoid `$in` lists >10,000 (hard cap). ```python index.query( namespace=tenant_id, # primary isolation vector=embedding, top_k=8, filter={"tenant_id": {"$eq": tenant_id}}, # defence-in-depth include_metadata=True, ) ``` - **Output**: Two-layer isolation; `delete_namespace(tenant_id)` for offboarding. - **Evidence**: `docs.pinecone.io/guides/index-data/implement-multitenancy`, `docs.pinecone.io/troubleshooting/namespaces-vs-metadata-filtering`. ### OP-05 WeaviateTenantHandle - **Trigger**: Weaviate-backed RAG. - **Action**: Enable `multiTenancyConfig(enabled=True)` on the collection; create one tenant per customer; pass `tenant=` on every read/write. ```python coll = client.collections.get("Docs").with_tenant(session.tenant_id) coll.query.near_vector(vector=emb, limit=8) ``` Optionally `auto_tenant_activation=True` if tenant set is sparse. - **Output**: Each tenant on a separate shard; cross-tenant reads physically impossible from a single client handle. - **Evidence**: `docs.weaviate.io/weaviate/manage-collections/multi-tenancy`; *Rethinking Vector Search at Scale* (Weaviate blog). ### OP-06 QdrantIsTenantPayload - **Trigger**: Qdrant-backed RAG with ≥hundreds of tenants. - **Action**: Single collection; create payload index with `is_tenant=True`; filter on every search. ```python client.create_payload_index( collection_name="docs", field_name="tenant_id", field_schema=models.KeywordIndexParams( type="keyword", is_tenant=True)) client.query_points( collection_name="docs", query=emb, limit=8, query_filter=models.Filter(must=[ models.FieldCondition( key="tenant_id", match=models.MatchValue(value=session.tenant_id))]), ) ``` - **Output**: Per-tenant sub-indexes co-located on shared shards; scales to millions of tenants. - **Evidence**: `qdrant.tech/documentation/manage-data/multitenancy/`; `qdrant.tech/articles/multitenancy/`; `qdrant.tech/documentation/examples/llama-index-multitenancy/`. ### OP-07 ChromaWhereFilter - **Trigger**: Chroma-backed RAG (often local / smaller scale). - **Action**: Filter on every call. ```python collection.query(