Lightweight Rust-based text chunking library for RAG pipelines providing 10+ chunking strategies, pipeline orchestration, and vector database integrations with Python and JavaScript APIs. Use when building document ingestion pipelines for retrieval-augmented generation, splitting text into meaningful chunks, or constructing CHOMP workflows.
Instrucciones de origen · Vista previa de solo lectura
name
chonkie-1-6-2
description
Lightweight Rust-based text chunking library for RAG pipelines providing 10+ chunking strategies, pipeline orchestration, and vector database integrations with Python and JavaScript APIs. Use when building document ingestion pipelines for retrieval-augmented generation, splitting text into meaningful chunks, or constructing CHOMP workflows.
Chonkie is a lightweight, high-performance text chunking library designed for retrieval-augmented generation (RAG) pipelines. It provides 12 chunking strategies ranging from simple token-based splitting to neural and agentic approaches, all unified behind a consistent interface. Built with Python (3.10+) and JavaScript support, Chonkie follows the CHOMP architecture — Chef (preprocess), Hchunk, Overlap/Refine, Merge/export, Port/store — to orchestrate end-to-end document ingestion workflows.
Key characteristics:
Lightweight: 505 KB wheel size vs 1–12 MB for alternatives. Base install pulls only ~4 core dependencies (tqdm, numpy, chonkie-core, tenacity).
Fast: SIMD-accelerated chunking via chonkie-core Rust extension. Benchmarks show 33x faster token chunking than the slowest competitor and 2x faster sentence chunking than LlamaIndex.
Modular dependencies: Install only what you need via optional extras ([semantic], [code], [neural], etc.).
Unified Chunk type: Since v1.3.0, all chunkers return the same base Chunk dataclass with fields: text, start_index, end_index, token_count, optional context, and optional embedding.
Async support: Every chunker provides achunk(), achunk_batch(), and achunk_document() methods out of the box using asyncio.to_thread.
Needing fast, lightweight chunking without the overhead of LangChain or LlamaIndex
Building self-hosted REST APIs for chunking services via chonkie serve
Core Concepts
Chunk: The fundamental output unit. A dataclass with text, start_index, end_index, token_count, optional context, and optional embedding. All chunkers return this same type.
Chunker: A component that splits text into Chunk objects. Chonkie provides 12 chunkers: Token, Fast, Sentence, Recursive, Semantic, Late, Code, Neural, Slumber, Table, TeraflopAI, and SDPM (legacy).
Refinery: A post-processing component that enhances chunks. OverlapRefinery adds context from neighboring chunks. EmbeddingsRefinery computes and attaches embedding vectors.
Pipeline: A fluent, chainable interface for building multi-step workflows following the CHOMP architecture: Fetcher → Chef → Chunker → Refinery → Porter/Handshake. Pipelines auto-reorder components into correct execution order.
Chef: Text preprocessing component. TextChef cleans and normalizes text. MarkdownChef extracts tables and code blocks from markdown.
Porter: Exports chunks to file formats. JSONPorter writes JSON. DatasetsPorter pushes to HuggingFace Datasets.
Handshake: Connects chunks directly to vector databases for embedding and storage in one step (ChromaDB, Qdrant, Pinecone, Weaviate, pgvector, MongoDB, Elasticsearch, Milvus, Turbopuffer).
Genie: Interface to LLM providers for advanced chunking strategies. Supports Gemini, OpenAI, Azure OpenAI, Groq, and Cerebras.
Recipe: Pre-configured chunker settings loaded from HuggingFace Hub. Use from_recipe() for language-specific or document-type-specific chunking (e.g., RecursiveChunker.from_recipe("markdown", lang="en")).
Installation / Setup
Basic Python Installation
pip install chonkie
Or with uv:
uv add chonkie
This provides TokenChunker, FastChunker, SentenceChunker, RecursiveChunker, TableChunker, OverlapRefinery, and basic tokenizers (character, word, byte).
Optional Features
Install specific capabilities as needed:
# Semantic chunking (SemanticChunker, LateChunker) with Model2Vec embeddings
pip install "chonkie[semantic]"# Code-aware chunking (CodeChunker) with tree-sitter
pip install "chonkie[code]"# Neural chunking (NeuralChunker) with transformers + torch
pip install "chonkie[neural]"# LLM-based chunking (SlumberChunker) via Genie interface
pip install "chonkie[genie]"# Visualization tools
pip install "chonkie[viz]"# HuggingFace Hub recipes
pip install "chonkie[hub]"# All features (not recommended for production)
pip install "chonkie[all]"# Multiple features combined
pip install "chonkie[semantic,code,viz]"
JavaScript Installation
npm install @chonkiejs/core # Local chunking (Token + Recursive)
npm install @chonkiejs/cloud # API client for cloud chunking
npm install @chonkiejs/token # Custom tokenizers for JS
Logging Control
export CHONKIE_LOG=off # Disable all loggingexport CHONKIE_LOG=warning # Warnings and errors (default)export CHONKIE_LOG=info # More verboseexport CHONKIE_LOG=debug # Everything
Usage Examples
Basic Chunking
from chonkie import RecursiveChunker
chunker = RecursiveChunker(chunk_size=512)
chunks = chunker("Your document text here...")
for chunk in chunks:
print(f"Text: {chunk.text[:50]}...")
print(f"Tokens: {chunk.token_count}")
Pipeline Workflow
from chonkie import Pipeline
doc = (Pipeline()
.chunk_with("recursive", chunk_size=512)
.refine_with("overlap", context_size=100)
.run(texts="Your document text here..."))
for chunk in doc.chunks:
print(chunk.text)
All Chunkers: Detailed guide to all 12 chunking strategies including Token, Fast, Sentence, Recursive, Semantic, Late, Code, Neural, Slumber, Table, and TeraflopAI → Chunkers
Refineries and Pipeline API: OverlapRefinery, EmbeddingsRefinery, CHOMP architecture, pipeline methods, validation rules, and best practices → Refineries and Pipeline
API Server and JavaScript SDK: Self-hosted REST API with chonkie serve, Docker deployment, pipeline persistence, and the @chonkiejs JavaScript packages → API Server and JavaScript
