spring-ai-rag-media-pgvector

# Spring AI RAG Media PgVector ## Workflow 1. Define knowledge scope: - Enumerate media asset types (movie metadata, synopsis, production notes, tags, reviews, scripts). - Define source-of-truth systems and ingestion frequency. 2. Provision database and extension: - Apply `references/pgvector-schema.sql` via Flyway. - Confirm `vector(1536)` dimension matches chosen embedding model. 3. Build ingestion pipeline: - Normalize source records into JSONL chunks with stable IDs. - Validate chunk payload using `scripts/validate_chunks.py`. - Persist raw document metadata and vectorized chunks. 4. Embed and index: - Use Spring AI embedding model to generate vectors. - Upsert chunk vectors in batches. - Rebuild IVFFlat index only after bulk ingestion. 5. Implement retrieval priority: - Execute vector similarity search against internal chunks first. - If top score is below threshold or result count is low, fallback to model-only answer path. - Keep both retrieved evidence and fallback reason in response metadata. 6. Enforce grounded generation: - Use retrieval context and source citations in prompt. - If no reliable internal evidence is found, explicitly state uncertainty. ## Guardrails - Keep chunk size stable (recommended 300-800 Chinese chars; overlap 50-120). - Version embedding model IDs; avoid mixing vectors from different dimensions in one table. - Use metadata filters (`asset_type`, `year`, `language`) before distance ranking when possible. - Log retrieval scores for online quality monitoring. ## Resources - `references/pgvector-schema.sql`: DB and index template for media RAG. - `references/rag-priority-strategy.md`: retrieval-first orchestration strategy. - `references/application-yml-template.md`: Spring AI and datasource baseline config. - `scripts/validate_chunks.py`: chunk payload validator before embedding ingestion.