elasticsearch-migration

name	elasticsearch-migration
description	Migrate from Elasticsearch to Vespa — map ES indices and mappings to Vespa schemas, translate Query DSL to YQL, plan reindexing, and bridge ranking differences. Use when the user mentions migrating from Elasticsearch, ES→Vespa, porting an ES index, or replacing Elasticsearch with Vespa.

Elasticsearch → Vespa Migration

Overview

This skill guides a migration from an existing Elasticsearch (or OpenSearch) deployment to Vespa.ai. It is the high-level coordinator: it covers the end-to-end flow, the conceptual mapping between the two systems, and the points where divergence is unavoidable (most notably ranking). Detailed work in each phase delegates to the other Vespa skills in this plugin:

• schema-authoring — writing .sd schemas, indexing pipelines, field types
• app-package — services.xml, deployment.xml, embedders, app layout
• query-builder — YQL, rank profiles, grouping, ML model integration
• feed-operations — bulk feeding, partial updates, conditional writes
• vespa-cli — deploy, target config, auth, CI/CD
• pyvespa — Python-driven equivalent of the above

Do not copy Elasticsearch or Vespa documentation into this skill. Both projects move quickly and embedded docs go stale. Instead, fetch fresh docs on demand using the patterns in Fetching fresh docs and the dedicated docs/fetching-docs.md.

When to use this skill

Trigger this skill when the user says something like:

• "Help me migrate from Elasticsearch to Vespa."
• "I have an ES index — how do I port it to Vespa?"
• "Translate this ES query to Vespa."
• "We're replacing Elasticsearch with Vespa."
• "Map my ES mapping to a Vespa schema."

For pure schema, query, or feeding tasks where the user is not coming from Elasticsearch, use the corresponding focused skill (schema-authoring, query-builder, feed-operations) directly instead.

The migration phases

The canonical Vespa migration guide walks through the flow. We break it into five operational phases below for clarity (the official guide groups some of these together). Fetch the latest version via https://docs.vespa.ai/en/learn/migrating-from-elastic-search.html.md (note the .md suffix — see Fetching fresh docs).

Phase	Goal	Tool / Skill
1. Export	Pull ES mapping + documents	mapping: curl /<index>/_mapping; documents: any tool that produces NDJSON (elasticdump, ES scroll API, _reindex-to-snapshot). For continuous side-by-side cutover, use Logstash with logstash-output-vespa_feed instead
2. Convert	Translate ES mapping → Vespa schema (.sd) + sample app package	ES_Vespa_parser.py (from Vespa sample-apps) as a starting point; manual review required → see schema-authoring
3. Generate app package	Wire schemas into a deployable application package	app-package skill
4. Deploy	Push the app package to Vespa (Cloud or self-hosted)	vespa-cli skill
5. Feed + verify	Load docs into the new app, validate queries and ranking	vespa feed CLI (recommended — fastest path, wraps the async HTTP/2 vespa-feed-client). Logstash for streaming; vespa-feed-client / pyvespa / /document/v1/ for programmatic. See feed-operations + query-builder skills

Treat each phase as a checkpoint. Validate the schema deploys cleanly before feeding; validate feeding succeeds before tuning ranking; validate functional query parity before attempting score parity (which is not always achievable — see Ranking divergence).

For the export step, see docs/parser-and-tooling.md for the Logstash pipeline, the dump-to-disk alternatives, and the parser's known limitations.

Concept mapping (cheat sheet)

Elasticsearch	Vespa
index	schema (.sd file under schemas/)
mapping (field types + options)	schema fields + indexing: directives
_id (flat string)	hierarchical document id: id:namespace:doctype:[group]:user-key
_source (raw doc echo)	indexing: summary on each field that should be returned
(no ES equivalent)	namespace — purely a logical id-collision separator in Vespa; pick any short constant string
index name / _index	the schema name (which is also the document-type segment in the id)
Query DSL (JSON)	YQL: select … from <schema> where …
analyzers / tokenizers / filters	match modes (text, exact, word, gram) + linguistics config
keyword field	string field with indexing: attribute + match: word (or exact)
text field	string field with indexing: index (+ summary if returned)
numeric / date / boolean	int, long, float, double, byte, bool, string (ISO date) — all need attribute to filter/sort/rank on
nested field	array<struct> in schema with explicit struct field declarations
object field	flatten to top-level fields or model as a struct
aliases	document selections + multi-schema queries (no aliasing layer)
bulk API (_bulk, NDJSON metadata+data lines)	no bulk endpoint — single-doc ops to /document/v1/ over HTTP/2; throughput via multiplexing in vespa feed CLI / vespa-feed-client (Java) / pyvespa (async)
reindex API	redeploy schema + vespa visit to refeed (or use Vespa's reindexing)
function_score / Painless scoring	rank profile with first-phase / second-phase / global-phase expressions
dense_vector + kNN	tensor<float>(x[N]) field + nearestNeighbor operator with HNSW index
sparse_vector / ELSER	sparse tensor + dotProduct ranking, or use a colbert/splade embedder
pipelines (ingest)	document-processing cluster in services.xml + custom doc processors

The full expansion with example before/after snippets lives in docs/concept-mapping.md. Load it when the user asks about a specific field type or feature.

Indexing directives — the critical conceptual shift

Elasticsearch maps a field's type (e.g. text, keyword) to a default set of inverted-index / doc-values behaviors. Vespa makes those choices explicit per field via the indexing: pipeline:

Directive	Enables
index	Tokenized inverted index → text search, userQuery(), BM25, weakAnd
attribute	Column store → filtering, sorting, grouping, ranking features, fast match
summary	Field is returned in search results

A field can have any combination. A field with no indexing: is stored but invisible to queries, which is the most common first-time-migrator surprise. Two rules of thumb:

• ES text field → Vespa string with indexing: index | summary
• ES keyword field used for filters/aggregations → Vespa string with indexing: attribute | summary and match: word (or exact for code-style ids)

For the full indexing pipeline grammar, match modes, and linguistics, load the schema-authoring skill.

Ranking divergence

This is the section to read carefully before promising stakeholders "the same results".

• Elasticsearch's default _score is BM25 over the matched terms in text fields, with optional function_score / script scoring on top.
• Vespa scores are whatever you write in a rank profile — there is no implicit default. The closest equivalent is a rank profile that uses bm25(field) as its first-phase expression, but the per-field normalization, length normalization parameters, and tie-breaking behavior are not identical.

How close can you actually get? The November 2024 Vespa vs Elasticsearch benchmark measured top-10 overlap with deliberately apples-to-apples configurations: 0.79 for lexical (WAND), 0.94 for vector (ANN), 0.81 for hybrid. The residual lexical gap is mostly down to linguistics (tokenization on dots, stemmer choice — porter2 came closest to Vespa's default), not BM25 math. Translation: vector parity is achievable; BM25 parity needs Lucene Linguistics on the Vespa side.

Practical implications:

1. Do not expect bit-for-bit score parity. Aim for result-set parity first (same top-K documents for representative queries), then rank-correlation parity (Spearman / NDCG), then absolute score parity only if a downstream system depends on the score values. ~0.8 lexical overlap with default linguistics is roughly the ceiling.
2. Rewrite custom scoring as rank profiles. Painless scripts → ranking expressions. function_score modifiers → second-phase or global-phase math over query and document features. The query-builder skill has the full rank-profile reference.
3. Vespa makes multi-phase ranking explicit. Most ES users do not realize they are doing a single-phase rank with no rescoring. Use the migration as an opportunity to introduce a cheap first-phase (BM25 or nativeRank) and an expressive second-phase (cross-encoder, LightGBM, learned-to-rank).

Gotchas

1. ES_Vespa_parser.py is a starter, not a finished tool. It produces a valid skeleton but does not infer good indexing directives, does not model nested or object fields well, and never produces a useful rank profile. Expect to hand-edit every generated .sd file. See docs/parser-and-tooling.md.

2. Document IDs are hierarchical, not flat. _id: "abc123" becomes id:mynamespace:mydoctype::abc123. The namespace is a logical id-collision separator — it has no functional role beyond that and can be any short constant string (per the Vespa documents reference); the doc-type segment must match the schema name. Picking namespaces before feeding is much easier than rewriting IDs afterward.

3. keyword vs text mapping is not symmetric. ES keyword fields are case-sensitive exact-match by default; in Vespa, get this with match: word on an attribute-backed string. ES text fields are tokenized and lowercased — Vespa's indexing: index does the same by default but the tokenizer is different (linguistics-aware, not ES analyzer-aware). Plan to re-tune analyzers.

   If you need to preserve ES analyzer behavior (custom token filters, language-specific stemmers, edge n-grams, etc.), look at Vespa's Lucene Linguistics component — it replaces Vespa's default linguistics with Apache Lucene's, supporting 40 languages and making analyzer parity much easier. Docs: https://docs.vespa.ai/en/lucene-linguistics.html. Sample apps (Minimal, Advanced, Going-Crazy, Non-Java): https://github.com/vespa-engine/sample-apps under examples/lucene-linguistics/. A dedicated "lucene-linguistics-migration" skill may follow as a phase-2 extension.

4. Nested and object fields need explicit modeling. Vespa has no implicit nested analog; use array<struct> for repeated nested objects and reference subfields with field.subfield syntax in YQL. Some users prefer to split nested data into a separate parent/child schema with referenced joins.

5. Ranking is not portable. Translating an ES query is mechanical; translating its scoring is not. Budget time for it explicitly.

6. The feed APIs work very differently — and the performance picture is asymmetric. ES uses the Bulk API with NDJSON payloads where each operation is a metadata line + a data line (no metadata line for deletes). Vespa has no bulk endpoint — clients write documents individually via /document/v1/, and throughput comes from HTTP/2 multiplexing in the underlying vespa-feed-client (which is what vespa feed and the Logstash output both wrap). The benchmark numbers (blog.vespa.ai, Nov 2024) are predictable, not random:

   Workload	Winner	Margin (per CPU core)
   Initial bootstrap write (empty → 1M docs)	Elasticsearch	~3x
   Refeed (rewrite all docs while serving queries)	Vespa	~comparable
   Partial updates (e.g. price/stock)	Vespa	~4x (up to 9x absolute throughput)
   Mixed write + query workload	Vespa	~2.5x write, also faster queries

   Why: ES Lucene segments are immutable, so partial updates do read-update-write (always); Vespa's attribute store is mutable, so most partial updates are in-place. If your real workload is heavy on price/inventory updates (typical e-commerce), expect Vespa to be substantially more CPU-efficient than ES even though raw bootstrap throughput favors ES. For tuning, see connections / max-streams-per-connection on the feed client — and the feed-operations skill.

7. Vespa is real-time; Elasticsearch is near-real-time. This is the biggest behavioral change most migrators don't anticipate. When vespa feed receives an ack, the document is already visible in searches — write-to-visibility is effectively zero. Elasticsearch acks once the doc is in the translog, but visibility waits for the next refresh (default 1s, but often pushed much higher in production — Vinted went from 300s refresh in ES to 5s end-to-end in Vespa after migrating). Practical impact:

   • Test code that does feed → refresh → query against ES can drop the refresh step in Vespa.
   • Code that relies on refresh-interval lag (e.g. waiting for an indexing pipeline to "settle") will see Vespa behave differently.
   • For very high-throughput indexing into ES, tuning refresh_interval is a common optimization; in Vespa there is no equivalent knob — visibility is always immediate.

8. Aliases don't exist. ES alias-based blue/green index swaps map onto Vespa's application package versioning (redeploy with a new schema and let convergence handle the cutover) or onto multi-schema queries (select … from sources schema_v1, schema_v2 where …). The ES "nightly reindex into a new index + force-merge + alias-swap" pattern doesn't translate — Vespa has no force-merge step (no segment-merge cost) and no alias layer, but it also has no nightly-batch performance windfall to recover. Plan the cutover as a re-deploy, not an index swap.

9. Date math, scripts, and runtime fields don't translate 1:1. ES "now-1d", painless filters, and runtime fields each need a Vespa-side analog: relative dates → use the container's current_time or compute in the client; painless filters → ranking expressions or document selections; runtime fields → derived fields in the indexing pipeline.

For deeper before/after examples, load docs/concept-mapping.md and docs/query-translation.md.

Fetching fresh docs (LLM-friendly resources)

Critical: prefer fetching live docs over recalling what you remember. Both Elasticsearch and Vespa evolve faster than any model's training cut-off. Use the patterns below.

Vespa documentation (LLM-friendly, well-supported)

Resource	URL	When to use
llms.txt index	https://docs.vespa.ai/llms.txt	Navigate to a specific page when you don't know its URL
llms-full.txt	https://docs.vespa.ai/llms-full.txt	Bulk-load when the user wants a broad survey (large — token-heavy)
Per-page markdown	Any …/page.html URL → append .md	First choice: you already know the page. The migration guide itself is https://docs.vespa.ai/en/learn/migrating-from-elastic-search.html.md

Default workflow: if you know the page, fetch its .html.md form via WebFetch. Otherwise grab llms.txt, find the page, then fetch its .html.md. Only reach for llms-full.txt for broad multi-topic questions.

Elasticsearch documentation (no llms.txt available)

Elastic.co does not currently publish an llms.txt. Fallbacks, in order of preference:

1. Source markdown on GitHub. Elasticsearch reference docs live in https://github.com/elastic/elasticsearch/tree/main/docs/reference — fetch raw markdown via https://raw.githubusercontent.com/elastic/elasticsearch/main/docs/reference/<path>. This is the authoritative source that the published HTML is built from.
2. WebFetch on www.elastic.co/guide/en/elasticsearch/reference/current/…. Returns HTML; the model has to interpret it but the content is current.
3. Probe with llms-txt-support first in case Elastic publishes one in future — cheap insurance and the skill is already on most Claude installs.

Never paste ES doc content into this skill. Fetch on demand, summarize for the user, and link to the source.

See docs/fetching-docs.md for concrete WebFetch invocations and worked examples for both projects.

Worked starting point

When the user kicks off a migration, the typical first turn looks like:

1. Ask for (or fetch) the ES mapping JSON for the index in question.
2. Walk through the mapping field-by-field, producing a .sd schema — defer to schema-authoring for syntax.
3. Generate a minimal services.xml + app-package layout — defer to app-package.
4. Feed a small sample of documents into the new app package and confirm select * from <schema> where true returns them — defer to feed-operations. Catch schema-deploy and ingest issues before pivoting to queries; iterating on the schema is cheaper than iterating on YQL against an empty index.
5. Pick one representative query from their current ES workload and translate it to YQL — defer to query-translation.md and query-builder.

Resist the urge to do everything in one turn. Each phase has its own gotchas that are easier to surface in isolation.

---

For deeper detail, load docs/concept-mapping.md (field-by-field mapping), docs/query-translation.md (Query DSL → YQL), docs/parser-and-tooling.md (Logstash + ES_Vespa_parser.py), or docs/fetching-docs.md (LLM-friendly doc-fetching patterns) from this skill's directory as needed.

Related skills: schema-authoring (for the .sd you'll be writing), app-package (for services.xml / deployment), query-builder (for YQL and rank profiles), feed-operations (for bulk feeding), vespa-cli (for deploy), pyvespa (for a Python-driven workflow).