Lightweight text chunking library for RAG pipelines providing 12+ chunkers, pipeline API, refineries, vector DB handshakes, and a self-hosted REST API. Use when building document ingestion pipelines for retrieval-augmented generation, splitting text into meaningful chunks, or constructing CHOMP workflows (fetch, clean, chunk, refine, store).
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Instruções da origem · Visualização somente leitura
name
chonkie-1-6-4
description
Lightweight text chunking library for RAG pipelines providing 12+ chunkers, pipeline API, refineries, vector DB handshakes, and a self-hosted REST API. Use when building document ingestion pipelines for retrieval-augmented generation, splitting text into meaningful chunks, or constructing CHOMP workflows (fetch, clean, chunk, refine, store).
Chonkie 1.6.4
Overview
Chonkie is a lightweight, high-performance text chunking library designed for retrieval-augmented generation (RAG) pipelines. It provides 12 chunking strategies ranging from simple token-based splitting to neural and agentic approaches, all unified behind a consistent interface. Built with Python (3.10+) and JavaScript support, Chonkie follows the CHOMP architecture — Chef (preprocess), Hchunk, Overlap/Refine, Merge/export, Port/store — to orchestrate end-to-end document ingestion workflows.
Key characteristics:
Lightweight: 505 KB wheel size vs 1–12 MB for alternatives. Base install pulls only ~4 core dependencies (tqdm, numpy, chonkie-core, tenacity).
Fast: SIMD-accelerated chunking via chonkie-core Rust extension (100+ GB/s for FastChunker). Benchmarks show 33x faster token chunking than the slowest competitor and 2x faster sentence chunking than LlamaIndex.
Modular dependencies: Install only what you need via optional extras ([semantic], [code], [neural], etc.).
Unified Chunk type: Since v1.3.0, all chunkers return the same base Chunk dataclass with fields: text, start_index, end_index, token_count, optional context, and optional .
embedding
Async support: Every chunker provides achunk(), achunk_batch(), and achunk_document() methods out of the box using asyncio.to_thread.
Needing fast, lightweight chunking without the overhead of LangChain or LlamaIndex
Building self-hosted REST APIs for chunking services via chonkie serve
Processing markdown tables with header-preserving TableChunker
Domain-specific text segmentation via TeraflopAI integration
Core Concepts
Chunk: The fundamental output unit. A dataclass with text, start_index, end_index, token_count, optional context, and optional embedding. All chunkers return this same type.
Chunker: A component that splits text into Chunk objects. Chonkie provides 12 chunkers: Token, Fast, Sentence, Recursive, Semantic, Late, Code, Neural, Slumber, Table, TeraflopAI, and SDPM (legacy/deprecated).
Refinery: A post-processing component that enhances chunks. OverlapRefinery adds context from neighboring chunks. EmbeddingsRefinery computes and attaches embedding vectors.
Pipeline: A fluent, chainable interface for building multi-step workflows following the CHOMP architecture: Fetcher → Chef → Chunker → Refinery → Porter/Handshake. Pipelines auto-reorder components into correct execution order.
Chef: Text preprocessing component. TextChef cleans and normalizes text. MarkdownChef extracts tables and code blocks from markdown. TableChef processes CSV/Excel into markdown tables.
Porter: Exports chunks to file formats. JSONPorter writes JSON. DatasetsPorter pushes to HuggingFace Datasets.
Handshake: Connects chunks directly to vector databases for embedding and storage in one step (ChromaDB, Qdrant, Pinecone, Weaviate, pgvector, MongoDB, Elasticsearch, Milvus, Turbopuffer, LanceDB).
Genie: Interface to LLM providers for advanced chunking strategies. Supports Gemini, OpenAI, Azure OpenAI, Groq, and Cerebras.
Recipe: Pre-configured chunker settings loaded from HuggingFace Hub. Use from_recipe() for language-specific or document-type-specific chunking (e.g., RecursiveChunker.from_recipe("markdown", lang="en")).
Installation / Setup
Basic Python Installation
pip install chonkie
Or with uv:
uv add chonkie
This provides TokenChunker, FastChunker, SentenceChunker, RecursiveChunker, TableChunker, OverlapRefinery, and basic tokenizers (character, word, byte).
Optional Features
Install specific capabilities as needed:
# Semantic chunking (SemanticChunker, LateChunker) with Model2Vec embeddings
pip install "chonkie[semantic]"# Code-aware chunking (CodeChunker) with tree-sitter
pip install "chonkie[code]"# Neural chunking (NeuralChunker) with transformers + torch
pip install "chonkie[neural]"# LLM-based chunking (SlumberChunker) via Genie interface
pip install "chonkie[genie]"# Groq genie (fast inference)
pip install "chonkie[groq]"# Cerebras genie (fastest inference)
pip install "chonkie[cerebras]"# Visualization tools
pip install "chonkie[viz]"# HuggingFace Hub recipes
pip install "chonkie[hub]"# All features (not recommended for production)
pip install "chonkie[all]"# Multiple features combined
pip install "chonkie[semantic,code,viz]"
JavaScript Installation
npm install @chonkiejs/core # Local chunking (Token, Sentence, Recursive, Fast, Table, Semantic, Code)
npm install @chonkiejs/cloud # API client for cloud chunking
npm install @chonkiejs/token # Custom tokenizers for JS
Logging Control
export CHONKIE_LOG=off # Disable all loggingexport CHONKIE_LOG=warning # Warnings and errors (default)export CHONKIE_LOG=info # More verboseexport CHONKIE_LOG=debug # Everything
Usage Examples
Basic Chunking
from chonkie import RecursiveChunker
chunker = RecursiveChunker(chunk_size=512)
chunks = chunker("Your document text here...")
for chunk in chunks:
print(f"Text: {chunk.text[:50]}...")
print(f"Tokens: {chunk.token_count}")
Pipeline Workflow
from chonkie import Pipeline
doc = (Pipeline()
.chunk_with("recursive", chunk_size=512)
.refine_with("overlap", context_size=100)
.run(texts="Your document text here..."))
for chunk in doc.chunks:
print(chunk.text)
All Chunkers: Detailed guide to all 12 chunking strategies including Token, Fast, Sentence, Recursive, Semantic, Late, Code, Neural, Slumber, Table, and TeraflopAI → Chunkers
Refineries and Pipeline API: OverlapRefinery, EmbeddingsRefinery, CHOMP architecture, pipeline methods, validation rules, and best practices → Refineries and Pipeline
API Server and JavaScript SDK: Self-hosted REST API with chonkie serve, Docker deployment, pipeline persistence, and the @chonkiejs JavaScript packages → API Server and JavaScript
