| license | Apache-2.0 |
| name | rag-document-ingestion-pipeline |
| description | Build production document ingestion pipelines with chunking, embedding, and vector DB storage. Activate on: document ingestion, chunking strategy, embedding pipeline, vector DB ingestion, RAG indexing. NOT for: LLM prompt design (prompt-engineer), retrieval query logic (ai-engineer), or vector DB ops/migration (vector-database-migration-tool). |
| allowed-tools | Read,Write,Edit,Bash(python:*,pip:*,npm:*,npx:*) |
| category | AI & Machine Learning |
| tags | ["rag","embeddings","vector-database","document-processing","chunking"] |
| pairs-with | [{"skill":"ai-engineer","reason":"RAG retrieval and query logic sits downstream of ingestion"},{"skill":"data-pipeline-engineer","reason":"ETL patterns for source extraction before chunking"},{"skill":"vector-database-migration-tool","reason":"Schema and index design for the target vector store"}] |
RAG Document Ingestion Pipeline
Build production-grade document ingestion pipelines that chunk, embed, and store documents in vector databases for retrieval-augmented generation.
Activation Triggers
Activate on: "document ingestion", "chunking strategy", "embedding pipeline", "vector DB ingestion", "RAG indexing", "ingest PDFs", "build knowledge base", "semantic chunking", "recursive chunking"
NOT for: LLM prompt design or retrieval query tuning (prompt-engineer, ai-engineer), vector DB operational migration (vector-database-migration-tool), or fine-tuning data preparation (fine-tuning-dataset-curator)
Quick Start
- Identify sources — PDFs, HTML, Markdown, databases, APIs. Use
unstructured or docling for parsing.
- Choose chunking strategy — Recursive character splitting for general text, semantic chunking for domain-specific content, or document-structure-aware chunking for technical docs.
- Select embedding model —
text-embedding-3-large (OpenAI), embed-v4 (Cohere), or BAAI/bge-m3 (local). Match dimensionality to your vector DB plan.
- Configure vector DB — Pinecone (managed), Qdrant (self-hosted or cloud), Weaviate (multi-tenant), or pgvector (Postgres-native).
- Run ingestion with observability — Batch embed, upsert with metadata, validate retrieval quality on a test set.
Core Capabilities
| Domain | Technologies | Notes |
|---|
| Document Parsing | unstructured, docling, PyMuPDF, markitdown | Handles PDF, DOCX, HTML, Markdown, images with OCR |
| Chunking | LangChain splitters, semantic-chunkers, chonkie | Recursive, semantic, markdown-header, code-aware |
| Embedding Models | OpenAI text-embedding-3, Cohere embed-v4, BGE-M3, Nomic Embed | Local or API; 256-3072 dimensions |
| Vector Databases | Pinecone, Qdrant, Weaviate, pgvector, Milvus | Managed or self-hosted; HNSW or IVF indexing |
| Orchestration | LangChain, LlamaIndex, Haystack, custom Python | Pipeline DAGs with retry and checkpointing |
Architecture Patterns
