Menu
Use semantic boundaries and embedding similarity to chunk text for higher-relevance retrieval.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Use this skill when building, debugging, or improving Retrieval-Augmented Generation systems, including chunking, vector database selection, hybrid search, reranking, multimodal RAG, code documentation RAG, retrieval latency, and production RAG architecture.
Chunk nested documents into parent-child levels so retrieval can move from broad sections to fine-grained passages.
Route RAG chunking decisions across semantic, hierarchical, sliding-window, contextual-header, and framework-selection strategies.
Use overlapping windows to preserve context across chunk boundaries while controlling retrieval size.
Reduce retrieval latency with caching, batching, and index-level optimization.
Route RAG performance work for latency, caching, indexing, filtering, batching, and query optimization.
| name | semantic-chunking |
| title | Semantic Chunking |
| description | Use semantic boundaries and embedding similarity to chunk text for higher-relevance retrieval. |
| allowed-tools | ["Read","Grep","Glob","Bash"] |
| category | chunking |
| tags | ["semantic","nlp","sentence-boundary","context-preservation"] |
Semantic chunking divides documents into segments based on natural language boundaries and semantic meaning rather than fixed character counts. This approach preserves context integrity and improves retrieval relevance by keeping related ideas together.
Fixed-size chunking (e.g., 1000 tokens) often cuts through mid-sentence, mid-paragraph, or mid-concept, resulting in:
Use sentence boundary detection rather than simple period splitting. This handles abbreviations, decimal numbers, and other edge cases.
Why: Simple period/regex splitting fails on "Dr. Smith", "3.14", "U.S.A.", etc.
Group sentences into chunks based on semantic cohesion rather than fixed counts.
Why: Similarity-based chunking adapts to document density - dense technical sections get smaller chunks, narrative sections get larger ones.
Leverage document structure (headings, markdown, HTML tags) to identify natural sections.
Why: Document structure often encodes semantic boundaries (chapters, sections, subsections).
Adjust chunk sizes based on content characteristics (code blocks, lists, tables).
Why: Different content types need different chunking strategies for optimal retrieval.
For practical implementations, compare Sentence-BERT, LlamaIndex Semantic Chunker, LangChain RecursiveCharacterTextSplitter, and spaCy sentence segmentation.
