| name | vector-db-storage |
| description | Storage backends and indexing for vector databases in Elixir. Use when choosing persistence, implementing exact vs approximate k-NN, or integrating pgvector, file, or ETS for vector storage. |
Vector DB Storage and Indexing
Storage Backends
| Backend | Use case | Persistence | Scale |
|---|
| ETS | Prototype, small index, low latency | No | < ~100k–500k vectors in memory |
| File | Simple persistence, no DB | Yes (binary/term) | Same as ETS, plus disk |
| PostgreSQL + pgvector | Production, SQL + vectors | Yes | Large; use IVFFlat/HNSW indexes |
| Qdrant (client) | Dedicated vector engine | Yes (external) | Very large |
In-Memory (ETS)
- :set table, key = vector id.
- Value:
{id, vector, metadata}; vector as list or binary.
- No persistence: load from file on GenServer init if needed (see File below).
- Exact k-NN: iterate all rows, compute distance, sort, take k. O(n) per query.
File Persistence
- On shutdown: GenServer
terminate/2 or :gen_server.cast(pid, :persist) → write ETS contents to file (e.g. :erlang.term_to_binary(entries) or custom binary format).
- On start: read file, recreate ETS and insert all entries.
- Keep dimension and metadata format in header or config so reload is consistent.
PostgreSQL + pgvector
Use pgvector and Ecto for persistent, indexed storage.
- Schema: table with
id, embedding (pgvector type), metadata (jsonb/map).
- Index: create HNSW or IVFFlat index on
embedding for approximate k-NN.
- Queries: use pgvector’s distance operators (
<=>, <->) in raw SQL or Ecto fragments; limit k.
# Example: nearest neighbors (cosine distance)
# fragment("embedding <=> ?::vector", ^Nx.to_flat_list(query))
- Handles large datasets; persistence and backups via Postgres.
Exact vs Approximate k-NN
| Method | Accuracy | Speed | When |
|---|
| Exact (scan all) | Exact | O(n) per query | Small n (e.g. < 100k), correctness critical |
| IVFFlat (pgvector) | Approximate | Fast | Medium/large, good recall with tuning |
| HNSW (pgvector) | Approximate | Very fast | Large, high recall |
For in-memory ETS, start with exact k-NN; move to pgvector (or Qdrant) when n grows or persistence is required.
Design Checklist
