add-vector-db

name	add-vector-db
description	Wire Twinfolio's retrieval to a VECTOR DATABASE the user owns, instead of the default local-vectors file or Gemini File Search. Use when the user says "/add-vector-db", "use Supabase/pgvector", "Pinecone", "Qdrant", "Weaviate", "MongoDB Atlas Vector Search", "Convex vector", "store my embeddings in a database", or "host the vectors in my own DB". Scaffolds a RagBackend adapter + an ingest step against the existing seam, in THEIR fork, with THEIR provider.
allowed-tools	Read Edit Write Bash(npm *) Bash(npx *) Bash(node *) Bash(git status)

Add a vector-database backend

Twinfolio retrieves grounding passages through one tiny seam — RagBackend — and ships three implementations: local vectors (default, content/.vectors.json), Gemini File Search, and None. This skill adds a fourth that the user owns: their corpus embeddings stored + searched in a real vector database.

Keep the project's spirit: this is optional and additive — the default stays no-database, the keyless demo must keep working, and the new backend must degrade gracefully (return [], never throw) when its env isn't set. The provider SDK is a dependency the USER adds to their fork — their choice, not core.

0. Orient — read the seam first (don't guess; the repo may have evolved)

• lib/rag/types.ts — the RagBackend interface (retrieve(query, {topK}) => RagChunk[], ready).
• lib/types.ts — RagChunk = { id; sourceName: string | null; content; similarity: number | null }.
• lib/rag/index.ts — getRagBackend(): the switch (config.rag.backend) you'll add a case to.
• lib/rag/embeddings.ts — embedQuery(text) => number[], embedDocuments(texts) => number[][], EMBED_DIM (768), DEFAULT_EMBED_MODEL. Use these so vectors match.
• lib/rag/local-vector.ts + lib/rag/local-index.ts — the reference retrieve (cosine top-K) and ingest (read corpus → chunk → embed → store) to mirror.
• lib/rag/chunk.ts — chunkDocument() for splitting docs before embedding.
• lib/rag/corpus-store.ts — readCorpusDocs() to load the corpus for ingest.
• lib/config/schema.ts — the rag.backend enum you'll extend.

1. Pick the provider (one short round)

Ask the user which they want, and confirm the trade. Well-known options:

Provider	What it is	Notes
Supabase / Postgres + pgvector	SQL + a vector column	The reference recipe below. Most common; free tier.
Pinecone	managed vector DB	Create an index, dimension 768, cosine.
Qdrant	open-source / cloud	Collection with 768-dim, cosine.
Weaviate	open-source / cloud	Class with a vector index.
MongoDB Atlas	$vectorSearch aggregation	Atlas Search vector index, 768 dims.
Convex	vectorIndex in a Convex function	Different paradigm (Convex functions).

Confirm with them: (a) this installs the provider's SDK in their fork (a new dependency — their call), (b) they need that DB's connection creds, (c) the default backend (local / File Search) is unchanged; this is opt-in via rag.backend. Pick a short backend id (e.g. supabase, pinecone, qdrant).

2. Scaffold the adapter — lib/rag/<id>.ts

Implement RagBackend. retrieve embeds the query with embedQuery, runs the provider's vector search for top-K, and maps results to RagChunk. ready reflects whether the env/creds are present. Never throw — on a missing config or a provider error, log and return [].

import type { RagBackend, RagChunk } from "@/lib/rag/types";
import { embedQuery } from "@/lib/rag/embeddings";
import { config } from "@/lib/config";

export class MyDbBackend implements RagBackend {
  get ready(): boolean {
    return Boolean(process.env.MYDB_URL /* + key */);
  }
  async retrieve(query: string, opts?: { topK?: number }): Promise<RagChunk[]> {
    if (!this.ready) return [];
    try {
      const vector = await embedQuery(query);            // 768-dim, matches ingest
      const topK = opts?.topK ?? config.rag.topK;
      const rows = await /* provider vector search(vector, topK) */ [];
      return rows.map((r) => ({
        id: String(r.id),
        sourceName: r.source ?? null,
        content: r.content,
        similarity: r.score ?? null,
      }));
    } catch (err) {
      console.error("[rag/<id>] retrieve failed:", err);
      return [];
    }
  }
}

3. Scaffold the ingest — scripts/ingest-<id>.ts (+ an npm script)

Mirror buildLocalIndex: readCorpusDocs() → chunkDocument() each → embedDocuments() the chunks → upsert { id, source, content, embedding } into the DB. Add "ingest:<id>": "tsx scripts/ingest-<id>.ts" to package.json scripts. This is the equivalent of npm run embed, but it writes to their database.

4. Wire it in (4 small edits)

1. lib/config/schema.ts — add the id to rag.backend's z.enum([...]).
2. lib/rag/index.ts — add a case "<id>": return new MyDbBackend(); to getRagBackend().
3. .env.local — add the connection vars; document them in .env.example with a comment.
4. (Optional) /admin Settings backend dropdown — add the option to the BackendField list.

5. Install the SDK

npm install <provider-sdk>   # e.g. @supabase/supabase-js, @pinecone-database/pinecone, @qdrant/js-client-rest

Tell the user this is the one new dependency, added to their fork.

Reference recipe — Supabase / pgvector

Run once in the Supabase SQL editor (embedding dim 768 to match gemini-embedding-001):

create extension if not exists vector;

create table twin_documents (
  id text primary key,
  source text,
  content text,
  embedding vector(768)
);

-- cosine-distance top-K
create or replace function match_twin_documents(query_embedding vector(768), match_count int)
returns table (id text, source text, content text, similarity float)
language sql stable as $$
  select id, source, content, 1 - (embedding <=> query_embedding) as similarity
  from twin_documents
  order by embedding <=> query_embedding
  limit match_count;
$$;

Adapter retrieve calls the RPC with @supabase/supabase-js: supabase.rpc("match_twin_documents", { query_embedding: vector, match_count: topK }), then maps rows to RagChunk. Ingest upserts rows into twin_documents. Env: SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY (server-only). For other providers, the shape is identical — only the create-index / upsert / query calls differ.

6. Verify

• npm run typecheck and npm run lint pass.
• Run npm run ingest:<id>, then start the app with rag.backend set to your id and ask a question that should hit the corpus — confirm a grounded answer + citations.
• Confirm degrade: with the env unset, ready is false and retrieval returns [] (the twin answers from persona only, never crashes).

Keep it honest in any docs you touch: this is an advanced, opt-in path — for a portfolio-sized corpus, the default local vectors or File Search is simpler.