Chonkie 1.6.2
Overview
Chonkie is a lightweight, high-performance text chunking library designed for retrieval-augmented generation (RAG) pipelines. It provides 12 chunking strategies ranging from simple token-based splitting to neural and agentic approaches, all unified behind a consistent interface. Built with Python (3.10+) and JavaScript support, Chonkie follows the CHOMP architecture — Chef (preprocess), Hchunk, Overlap/Refine, Merge/export, Port/store — to orchestrate end-to-end document ingestion workflows.
Key characteristics:
- Lightweight: 505 KB wheel size vs 1–12 MB for alternatives. Base install pulls only ~4 core dependencies (tqdm, numpy, chonkie-core, tenacity).
- Fast: SIMD-accelerated chunking via
chonkie-core Rust extension. Benchmarks show 33x faster token chunking than the slowest competitor and 2x faster sentence chunking than LlamaIndex.
- Modular dependencies: Install only what you need via optional extras (
[semantic], [code], [neural], etc.).
- Unified
Chunk type: Since v1.3.0, all chunkers return the same base Chunk dataclass with fields: text, start_index, end_index, token_count, optional context, and optional embedding.
- Async support: Every chunker provides
achunk(), achunk_batch(), and achunk_document() methods out of the box using asyncio.to_thread.
- 32+ integrations: Tokenizers (tiktoken, HuggingFace tokenizers), embedding providers (OpenAI, Cohere, Jina, Voyage AI, sentence-transformers, Model2Vec, LiteLLM), LLM genies (Gemini, OpenAI, Groq, Cerebras), and vector database handshakes (ChromaDB, Qdrant, Pinecone, Weaviate, pgvector, MongoDB, Elasticsearch, Milvus, Turbopuffer).
When to Use
- Building RAG pipelines that need high-quality text chunking
- Splitting documents into semantically coherent chunks for vector database ingestion
- Processing code files with structure-aware chunking (AST-based CodeChunker)
- Running end-to-end ingestion workflows: fetch → clean → chunk → refine → embed → store
- Needing fast, lightweight chunking without the overhead of LangChain or LlamaIndex
- Building self-hosted REST APIs for chunking services via
chonkie serve
Core Concepts
Chunk: The fundamental output unit. A dataclass with text, start_index, end_index, token_count, optional context, and optional embedding. All chunkers return this same type.
Chunker: A component that splits text into Chunk objects. Chonkie provides 12 chunkers: Token, Fast, Sentence, Recursive, Semantic, Late, Code, Neural, Slumber, Table, TeraflopAI, and SDPM (legacy).
Refinery: A post-processing component that enhances chunks. OverlapRefinery adds context from neighboring chunks. EmbeddingsRefinery computes and attaches embedding vectors.
Pipeline: A fluent, chainable interface for building multi-step workflows following the CHOMP architecture: Fetcher → Chef → Chunker → Refinery → Porter/Handshake. Pipelines auto-reorder components into correct execution order.
Chef: Text preprocessing component. TextChef cleans and normalizes text. MarkdownChef extracts tables and code blocks from markdown.
Porter: Exports chunks to file formats. JSONPorter writes JSON. DatasetsPorter pushes to HuggingFace Datasets.
Handshake: Connects chunks directly to vector databases for embedding and storage in one step (ChromaDB, Qdrant, Pinecone, Weaviate, pgvector, MongoDB, Elasticsearch, Milvus, Turbopuffer).
Genie: Interface to LLM providers for advanced chunking strategies. Supports Gemini, OpenAI, Azure OpenAI, Groq, and Cerebras.
Recipe: Pre-configured chunker settings loaded from HuggingFace Hub. Use from_recipe() for language-specific or document-type-specific chunking (e.g., RecursiveChunker.from_recipe("markdown", lang="en")).
Installation / Setup
Basic Python Installation
pip install chonkie
Or with uv:
uv add chonkie
This provides TokenChunker, FastChunker, SentenceChunker, RecursiveChunker, TableChunker, OverlapRefinery, and basic tokenizers (character, word, byte).
Optional Features
Install specific capabilities as needed:
pip install "chonkie[semantic]"
pip install "chonkie[code]"
pip install "chonkie[neural]"
pip install "chonkie[genie]"
pip install "chonkie[viz]"
pip install "chonkie[hub]"
pip install "chonkie[all]"
pip install "chonkie[semantic,code,viz]"
JavaScript Installation
npm install @chonkiejs/core
npm install @chonkiejs/cloud
npm install @chonkiejs/token
Logging Control
export CHONKIE_LOG=off
export CHONKIE_LOG=warning
export CHONKIE_LOG=info
export CHONKIE_LOG=debug
Usage Examples
Basic Chunking
from chonkie import RecursiveChunker
chunker = RecursiveChunker(chunk_size=512)
chunks = chunker("Your document text here...")
for chunk in chunks:
print(f"Text: {chunk.text[:50]}...")
print(f"Tokens: {chunk.token_count}")
Pipeline Workflow
from chonkie import Pipeline
doc = (Pipeline()
.chunk_with("recursive", chunk_size=512)
.refine_with("overlap", context_size=100)
.run(texts="Your document text here..."))
for chunk in doc.chunks:
print(chunk.text)
RAG Ingestion Pipeline
from chonkie import Pipeline
docs = (Pipeline()
.fetch_from("file", dir="./knowledge_base", ext=[".txt", ".md"])
.process_with("text")
.chunk_with("semantic", threshold=0.8, chunk_size=1024)
.refine_with("overlap", context_size=100)
.store_in("qdrant", collection_name="knowledge", url="http://localhost:6333")
.run())
Async Chunking
import asyncio
from chonkie import SemanticChunker
async def process():
chunker = SemanticChunker(chunk_size=512)
chunks = await chunker.achunk("Document text...")
return chunks
asyncio.run(process())
Advanced Topics
All Chunkers: Detailed guide to all 12 chunking strategies including Token, Fast, Sentence, Recursive, Semantic, Late, Code, Neural, Slumber, Table, and TeraflopAI → Chunkers
Refineries and Pipeline API: OverlapRefinery, EmbeddingsRefinery, CHOMP architecture, pipeline methods, validation rules, and best practices → Refineries and Pipeline
Integrations: Embedding providers (9+), LLM Genies (5+), Vector database handshakes (9+), Porters, Tokenizers, Chefs, Fetchers → Integrations
API Server and JavaScript SDK: Self-hosted REST API with chonkie serve, Docker deployment, pipeline persistence, and the @chonkiejs JavaScript packages → API Server and JavaScript
