rag-document-ingestion-pipeline

license	Apache-2.0
name	rag-document-ingestion-pipeline
description	Build production document ingestion pipelines with chunking, embedding, and vector DB storage. Activate on: document ingestion, chunking strategy, embedding pipeline, vector DB ingestion, RAG indexing. NOT for: LLM prompt design (prompt-engineer), retrieval query logic (ai-engineer), or vector DB ops/migration (vector-database-migration-tool).
allowed-tools	Read,Write,Edit,Bash(python:*,pip:*,npm:*,npx:*)
category	AI & Machine Learning
tags	["rag","embeddings","vector-database","document-processing","chunking"]
pairs-with	[{"skill":"ai-engineer","reason":"RAG retrieval and query logic sits downstream of ingestion"},{"skill":"data-pipeline-engineer","reason":"ETL patterns for source extraction before chunking"},{"skill":"vector-database-migration-tool","reason":"Schema and index design for the target vector store"}]

RAG Document Ingestion Pipeline

Build production-grade document ingestion pipelines that chunk, embed, and store documents in vector databases for retrieval-augmented generation.

Activation Triggers

Activate on: "document ingestion", "chunking strategy", "embedding pipeline", "vector DB ingestion", "RAG indexing", "ingest PDFs", "build knowledge base", "semantic chunking", "recursive chunking"

NOT for: LLM prompt design or retrieval query tuning (prompt-engineer, ai-engineer), vector DB operational migration (vector-database-migration-tool), or fine-tuning data preparation (fine-tuning-dataset-curator)

Quick Start

1. Identify sources — PDFs, HTML, Markdown, databases, APIs. Use unstructured or docling for parsing.
2. Choose chunking strategy — Recursive character splitting for general text, semantic chunking for domain-specific content, or document-structure-aware chunking for technical docs.
3. Select embedding model — text-embedding-3-large (OpenAI), embed-v4 (Cohere), or BAAI/bge-m3 (local). Match dimensionality to your vector DB plan.
4. Configure vector DB — Pinecone (managed), Qdrant (self-hosted or cloud), Weaviate (multi-tenant), or pgvector (Postgres-native).
5. Run ingestion with observability — Batch embed, upsert with metadata, validate retrieval quality on a test set.

Core Capabilities

Domain	Technologies	Notes
Document Parsing	unstructured, docling, PyMuPDF, markitdown	Handles PDF, DOCX, HTML, Markdown, images with OCR
Chunking	LangChain splitters, semantic-chunkers, chonkie	Recursive, semantic, markdown-header, code-aware
Embedding Models	OpenAI text-embedding-3, Cohere embed-v4, BGE-M3, Nomic Embed	Local or API; 256-3072 dimensions
Vector Databases	Pinecone, Qdrant, Weaviate, pgvector, Milvus	Managed or self-hosted; HNSW or IVF indexing
Orchestration	LangChain, LlamaIndex, Haystack, custom Python	Pipeline DAGs with retry and checkpointing

Architecture Patterns