Lightweight Rust-based text chunking library for RAG pipelines providing 10+ chunking strategies, pipeline orchestration, and vector database integrations with Python and JavaScript APIs. Use when building document ingestion pipelines for retrieval-augmented generation, splitting text into meaningful chunks, or constructing CHOMP workflows.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Lightweight Rust-based text chunking library for RAG pipelines providing 10+ chunking strategies, pipeline orchestration, and vector database integrations with Python and JavaScript APIs. Use when building document ingestion pipelines for retrieval-augmented generation, splitting text into meaningful chunks, or constructing CHOMP workflows.
Chonkie is a lightweight, high-performance text chunking library designed for retrieval-augmented generation (RAG) pipelines. It provides 12 chunking strategies ranging from simple token-based splitting to neural and agentic approaches, all unified behind a consistent interface. Built with Python (3.10+) and JavaScript support, Chonkie follows the CHOMP architecture — Chef (preprocess), Hchunk, Overlap/Refine, Merge/export, Port/store — to orchestrate end-to-end document ingestion workflows.
Key characteristics:
Lightweight: 505 KB wheel size vs 1–12 MB for alternatives. Base install pulls only ~4 core dependencies (tqdm, numpy, chonkie-core, tenacity).
Fast: SIMD-accelerated chunking via chonkie-core Rust extension. Benchmarks show 33x faster token chunking than the slowest competitor and 2x faster sentence chunking than LlamaIndex.
Modular dependencies: Install only what you need via optional extras ([semantic], [code], [neural], etc.).
Unified Chunk type: Since v1.3.0, all chunkers return the same base Chunk dataclass with fields: text, start_index, end_index, token_count, optional context, and optional embedding.
Async support: Every chunker provides achunk(), achunk_batch(), and achunk_document() methods out of the box using asyncio.to_thread.
Needing fast, lightweight chunking without the overhead of LangChain or LlamaIndex
Building self-hosted REST APIs for chunking services via chonkie serve
Core Concepts
Chunk: The fundamental output unit. A dataclass with text, start_index, end_index, token_count, optional context, and optional embedding. All chunkers return this same type.
Chunker: A component that splits text into Chunk objects. Chonkie provides 12 chunkers: Token, Fast, Sentence, Recursive, Semantic, Late, Code, Neural, Slumber, Table, TeraflopAI, and SDPM (legacy).
Refinery: A post-processing component that enhances chunks. OverlapRefinery adds context from neighboring chunks. EmbeddingsRefinery computes and attaches embedding vectors.
Pipeline: A fluent, chainable interface for building multi-step workflows following the CHOMP architecture: Fetcher → Chef → Chunker → Refinery → Porter/Handshake. Pipelines auto-reorder components into correct execution order.
Chef: Text preprocessing component. TextChef cleans and normalizes text. MarkdownChef extracts tables and code blocks from markdown.
Porter: Exports chunks to file formats. JSONPorter writes JSON. DatasetsPorter pushes to HuggingFace Datasets.
Handshake: Connects chunks directly to vector databases for embedding and storage in one step (ChromaDB, Qdrant, Pinecone, Weaviate, pgvector, MongoDB, Elasticsearch, Milvus, Turbopuffer).
Genie: Interface to LLM providers for advanced chunking strategies. Supports Gemini, OpenAI, Azure OpenAI, Groq, and Cerebras.
Recipe: Pre-configured chunker settings loaded from HuggingFace Hub. Use from_recipe() for language-specific or document-type-specific chunking (e.g., RecursiveChunker.from_recipe("markdown", lang="en")).
Installation / Setup
Basic Python Installation
pip install chonkie
Or with uv:
uv add chonkie
This provides TokenChunker, FastChunker, SentenceChunker, RecursiveChunker, TableChunker, OverlapRefinery, and basic tokenizers (character, word, byte).
Optional Features
Install specific capabilities as needed:
# Semantic chunking (SemanticChunker, LateChunker) with Model2Vec embeddings
pip install "chonkie[semantic]"# Code-aware chunking (CodeChunker) with tree-sitter
pip install "chonkie[code]"# Neural chunking (NeuralChunker) with transformers + torch
pip install "chonkie[neural]"# LLM-based chunking (SlumberChunker) via Genie interface
pip install "chonkie[genie]"# Visualization tools
pip install "chonkie[viz]"# HuggingFace Hub recipes
pip install "chonkie[hub]"# All features (not recommended for production)
pip install "chonkie[all]"# Multiple features combined
pip install "chonkie[semantic,code,viz]"
JavaScript Installation
npm install @chonkiejs/core # Local chunking (Token + Recursive)
npm install @chonkiejs/cloud # API client for cloud chunking
npm install @chonkiejs/token # Custom tokenizers for JS
Logging Control
export CHONKIE_LOG=off # Disable all loggingexport CHONKIE_LOG=warning # Warnings and errors (default)export CHONKIE_LOG=info # More verboseexport CHONKIE_LOG=debug # Everything
Usage Examples
Basic Chunking
from chonkie import RecursiveChunker
chunker = RecursiveChunker(chunk_size=512)
chunks = chunker("Your document text here...")
for chunk in chunks:
print(f"Text: {chunk.text[:50]}...")
print(f"Tokens: {chunk.token_count}")
Pipeline Workflow
from chonkie import Pipeline
doc = (Pipeline()
.chunk_with("recursive", chunk_size=512)
.refine_with("overlap", context_size=100)
.run(texts="Your document text here..."))
for chunk in doc.chunks:
print(chunk.text)
All Chunkers: Detailed guide to all 12 chunking strategies including Token, Fast, Sentence, Recursive, Semantic, Late, Code, Neural, Slumber, Table, and TeraflopAI → Chunkers
Refineries and Pipeline API: OverlapRefinery, EmbeddingsRefinery, CHOMP architecture, pipeline methods, validation rules, and best practices → Refineries and Pipeline
API Server and JavaScript SDK: Self-hosted REST API with chonkie serve, Docker deployment, pipeline persistence, and the @chonkiejs JavaScript packages → API Server and JavaScript
