pinecone-full-text-search

# Pinecone Full-Text Search > **Requires `pinecone` Python SDK ≥ 10.0.0** (`pip install pinecone>=10.0.0`). The document-schema API graduated out of `pinecone.preview` in 10.0.0 — it is now a first-class, SemVer-covered part of the SDK, reachable directly off `pc` (`pc.indexes`, `pc.index(...)`). If you land on this skill from an older habit of importing `pinecone.preview`, stop: that package is deleted outright in 10.0.0 (`ModuleNotFoundError`, no shim). The packaged helper script pins `pinecone==10.0.0` via PEP 723 inline metadata; if you're writing your own code against this skill, pin at least that version. The wire API version is `2026-07`. > **Authoritative reference (last resort).** If you hit a question this skill and its `references/*.md` files don't answer, the official Pinecone FTS docs are at <https://docs.pinecone.io/guides/search/full-text-search>. Prefer this skill's content for anything covered here — the docs may describe surfaces (e.g. classic vector API, or the older `pinecone.preview` shape) that don't apply to the graduated document-schema path. Consult the link only when you're genuinely stuck. > **Tell the user up front:** "This skill ships a helper at `scripts/ingest.py` that handles bulk ingestion safely (batched upsert, error inspection, readiness polling). When we get to the ingest step, I'll use it." Surface this at the start of the conversation so the user knows the helper exists. Query construction is hand-written `documents.search(...)` per the **Querying** section below — there is no query helper. A workflow skill for building a Pinecone full-text-search index with the graduated document-schema API (`pc.indexes`, `pc.index(name)`, API version `2026-07`). Covers schema design (text, dense vector, sparse vector, filterable metadata), ingestion (including async indexing and polling), and query construction (`text` / `query_string` / `dense_vector` / `sparse_vector` scoring; `$match_phrase` / `$match_all` / `$match_any` text-match filters; `$eq` / `$in` / `$gte` / `$exists` / `$and` / `$or` / `$not` metadata filters). ## Scope — this skill is for the document-schema FTS API only This skill covers `pc.indexes.create(..., schema=...)`, `pc.index(name)`, `idx.documents.upsert(...)` / `idx.documents.batch_upsert(...)` / `idx.documents.search(...)`. If you find yourself reaching for any of the following, **stop** — those are different Pinecone APIs and this skill's guidance and helpers won't apply: - **Classic vector / records API**: `pc.Index(name)`, `index.upsert(vectors=[...])`, `index.query(vector=..., sparse_vector=...)`, `pc.create_index(dimension=..., metric=..., spec=ServerlessSpec(...))`. This is the *deprecated sugar* path in 10.0.0 — it still runs, but it creates a schemaless index served by the vector data plane, addressing the vector by the reserved `_values` field. It cannot hold `full_text_search` fields. - **Integrated-embedding / records indexes**: `pc.create_index_for_model(...)` / `pc.indexes.create_for_model(...)` with `embed={...}`. Pinecone vectorizes text server-side, and the resulting `semantic_text` field is served by the **records** API (`upsert_records` / `search_records`), not the documents API. Different upsert/search shapes. A `semantic_text` field cannot be combined with `full_text_search` fields in the same index. If the user already has a non-document-schema index, they can stand up a separate document-schema index alongside it — the two are independent — but you can't add FTS fields to a classic or integrated-embedding index after the fact, and a document-schema index only ever serves reads and writes through `index.documents.*` — never `index.upsert` / `index.query` / `index.upsert_records` (those calls are refused with "This index has a document schema, so writes must go through the documents API"). ## Querying — construct `documents.search(...)` calls For any task that asks you to query an FTS index, you write a `documents.search(...)` call directly. The schema is authoritative — describe the index live before constructing the call so you know which fields are FTS-enabled, which are filterable, and which are vectors. **Workflow:** 1. **Discover the schema.** Call `pc.indexes.describe(<index>)` and read the `schema.fields` dict. Each field's class indicates its type (`StringField`, `FloatField`, `DenseVectorField`, etc.); attributes tell you whether it's FTS-enabled (`full_text_search`), filterable, or carries a `dimension`. Skip this step only if you've already seen the schema in this conversation. 2. **Construct the call** matching the rules below — one scoring type per request, hard requirements in `filter`, ranking signals in `score_by`, `include_fields` explicit on every call. 3. **Execute** with `idx = pc.index(name=<index>); resp = idx.documents.search(...)` and read `resp.matches`. **Canonical shapes:** ```python # Pure BM25 keyword search resp = idx.documents.search( namespace="__default__", top_k=10, score_by=[{"type": "text", "field": "body", "query": "machine learning"}], filter={"year": {"$gt": 2024}, "category": {"$eq": "ai"}}, # optional include_fields=["*"], # always pass explicitly ) # Hybrid: dense ranking with a lexical filter (one type in score_by + filter narrows) resp = idx.documents.search( namespace="__default__", top_k=10, score_by=[{"type": "dense_vector", "field": "embedding", "values": query_embedding}], filter={"body": {"$match_all": "TensorFlow"}, "year": {"$gt": 2024}}, include_fields=["*"], ) ``` **Key rules** (the server enforces these; following them locally keeps the agent loop tight): - `score_by` is a list of clauses, but **exactly one scoring type per request** (server rejects mixed types). Multi-field BM25 is the one exception: multiple `text` clauses, or one `query_string` with `fields: [...]`. To combine BM25 + dense signals, restrict the dense search with a text-match filter (`$match_all` / `$match_phrase` / `$match_any`); do NOT mix scoring types in `score_by`. - `filter` keys are field names (must exist in schema, or be an auto-indexed metadata field from upserted documents — see **Filterable metadata isn't declared in the schema** below) OR logical operators (`$and`, `$or`, `$not`). Field values are operator dicts (`{"$gt": 5}`, NOT bare values). - `include_fields` is required on every call. Pass `["*"]` for all stored fields, `[]` for ids+score only, or a list of names. Omitting it on some SDK/backend builds 400s. **Clause shapes** (for `score_by`): | `type` | Required keys | When to pick this | |---|---|---| | `text` | `field` (string FTS), `query` | Open-ended keyword search; BM25 ranking on one field | | `query_string` | `query` (Lucene), `fields` optional | Lucene boost (`^N`), proximity (`~N`), cross-field boolean, phrase prefix | | `dense_vector` | `field` (dense_vector), `values` (list of floats) | Semantic / mood / topic ranking | | `sparse_vector` | `field` (sparse_vector), `sparse_values` ({indices, values}) | Custom sparse-encoder ranking | `text` / `dense_vector` / `sparse_vector` use singular `field`. Only `query_string` accepts a `fields` array (and also accepts singular `field` as an alias). `sparse_vector` uses `sparse_values` (NOT `values`) — distinct from dense. **Filter operators by field type:** | Field type | Legal operators | |---|---| | `string` with FTS | `$match_phrase`, `$match_all`, `$match_any` | | filterable metadata (string / auto-indexed) | `$eq`, `$ne`, `$in`, `$nin`, `$exists` | | `string_list` filterable (auto-indexed, not schema-declared) | `$in`, `$nin`, `$exists` | | `float` filterable (auto-indexed, not schema-declared) | `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$exists` | | `boolean` filterable (auto-indexed, not schema-declared) | `$eq`, `$exists` | | logical wrappers | `$and: [filters]`, `$or: [filters]`, `$not: filter` | **Match shape on response:** ```python for m in resp.matches: m._id # document id m._score # match score (NOT `score`) m.to_dict() # full doc payload (when include_fields includes the field) ``` For deeper coverage — multi-field BM25, Lucene patterns, hybrid composition, RRF merges, common error symptoms — see `references/querying.md`. For schema field types and what they enable on the query side, see `references/schema-design.md`. ## Ingesting — use the packaged helper For **any task that asks you to bulk-ingest a JSONL file into an existing FTS index**, the canonical path is to invoke the bundled helper, NOT to hand-write a Python script. **Do not read the script's source** — everything you need is in this section. The script does three things bare-LLM ingest code reliably skips, each of which corresponds to a silent production failure: 1. **Bulk-upserts in batches.** No per-doc `upsert` loops. 2. **Inspects every batch result.** `batch_upsert` returns 202 even when individual documents fail; the failures live in `result.errors` / `result.has_errors`. Without inspection, "100 docs ingested" silently becomes "73 docs ingested + 27 lost." 3. **Polls until searchable.** After upsert, Pinecone is still building the inverted index. A `documents.search` call during that window returns empty. Without the poll, the user debugs their *query* code for an hour without finding the indexing race. You provide a prepared, schema-conformant JSONL file and the index name; the script does the rest. Schema validation is upstream concerns (your prep pipeline, or `prepare_documents.py` when it lands) — `ingest.py` trusts what you hand it. **Invocation:** ```bash uv run --script scripts/ingest.py \ --data processed.jsonl \ --index <index_name> \ --sentinel-field <fts_field> ``` **Flags:** | Flag | Short | Required | Purpose | |---|---|---|---| | `--data` | `-d` | yes | Path to JSONL file with prepared documents (one per line) | | `--index` | `-i` | yes | Pinecone index name (must already exist) | | `--sentinel-field` | `-f` | yes | An FTS-enabled field on the index, used for the readiness-poll query. Pick the longest free-text field on your schema. | | `--namespace` | `-n` | no | Default `__default__` | | `--batch-size` | `-b` | no | Default 50 (matches the SDK's own `batch_upsert` default). **Reduce for large dense vectors.** A 50-doc batch with 3072-dim float vectors lands ~5-10 MB and can be rejected; drop to `--batch-size 25` (or lower) at high dimensions. | | `--max-concurrency` | — | no | Default 4. Parallel HTTP connections used to upload batches. | | `--poll-deadline` | — | no | Default 300 (seconds). Time to wait for documents to become searchable before giving up. | | `--sentinel` | `-s` | no | Token used for the readiness-poll query. Default: first whitespace-separated token of `doc[0][sentinel-field]`. | **What the script prints:** ``` Loading processed.jsonl ... Loaded 5000 document(s). Sentinel: body='The' Upserting in batches of 50 ... batch @ 0: 50 docs in 0.31s (total: 50/5000) batch @ 50: 50 docs in 0.29s (total: 100/5000) ... Upsert complete: 5000 doc(s) in 21.4s. Polling for searchability (deadline 300s) ... Searchable after 12.3s (3 probe(s)). Done — total 33.7s. ``` If a batch fails, the script prints every error message and exits non-zero. If the poll deadline expires, the script prints a hint about why (sentinel field isn't FTS-enabled, deadline too tight, docs structurally upserted but rejected by the inverted-index builder) and exits non-zero. **Don't suppress these errors** — they're surfacing real problems with the data or the index. **When you should NOT use the script:** - The user is doing per-doc patch updates. Use `documents.update(...)` for partial field updates (see **Updating documents** in `references/ingestion.md`) — the script is for bulk loads, not per-record operations. - The user is ingesting from a non-JSONL source (CSV, Parquet, Postgres dump). Convert to JSONL first; the script doesn't parse other formats. - The user explicitly asks you to write the ingestion code from scratch (teaching context). Honor the request and follow the canonical pattern: `documents.batch_upsert` + `result.has_errors` inspection + `documents.search` polling with sentinel and deadline. The script lives at `scripts/ingest.py` relative to this skill directory. PEP 723 inline-metadata script — `uv run --script` installs `typer` and `pinecone` automatically on first invocation. No setup needed. ## Use cases Three concrete shapes to model your task on. Match the user's request to the closest one and follow its steps; improvise if the task is genuinely a hybrid. ### UC-1: Index a new corpus end-to-end **Trigger.** "Index this CSV / JSONL / folder for search," "build a search backend over [my articles / products / tickets / transcripts]," "make my [dataset] searchable." **For unprocessed / messy data, load the onboarding walkthrough first.** If the user is showing up with raw data (unclear field types, possibly long text fields exceeding FTS limits, comma-separated tag strings, dates as strings, possibly duplicate IDs, etc.) and they haven't given you an explicit schema, **read `references/onboarding-walkthrough.md` and follow it stage-by-stage.** It's a conversational guide — meet the data, surface the processing decisions to the user, propose a schema, confirm before creating, then process+ingest+verify together. The walkthrough exists because schemas are immutable and "onboarding a new corpus" is a high-stakes flow that benefits from explicit user buy-in at each decision point. If the user already gave you a clean JSONL + a schema spec, follow the abbreviated steps below. **Steps (when data is already prepared and the schema is decided):** 1. Inspect the corpus shape — text fields, structured metadata, do you also need a vector? Match it to one of the canonical shapes in `references/schema-design.md` (articles, products, tickets, image library, code).