Menu
Use when user needs to split documents into chunks for RAG or search. Triggers on: chunking, split, chunk size, text splitter, token limit, overlap.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Use when user needs to group similar items together. Triggers on: clustering, group similar, topic modeling, user segmentation, categorization, automatic classification, unsupervised grouping.
Use when user needs to find duplicate or similar content. Triggers on: duplicate, deduplication, plagiarism detection, similar content, near-duplicate, similarity detection, content dedup, find copies.
Use when user needs long-term memory for chatbots. Triggers on: chat memory, conversation history, long-term memory, chatbot memory, memory retrieval, persistent memory, remember conversations.
Use when user wants to build image search or similar image finding. Triggers on: image search, similar image, visual search, image retrieval, CLIP, reverse image search, image matching, find similar photos.
Use when user needs RAG on documents with images and text. Triggers on: multimodal RAG, image-text mixed, document with images, PDF with charts, visual RAG, visual Q&A, documents with figures.
Use when user needs to search images using natural language descriptions. Triggers on: text to image, describe and find, natural language image search, image caption search, find image by description, describe to find.
| name | chunking |
| description | Use when user needs to split documents into chunks for RAG or search. Triggers on: chunking, split, chunk size, text splitter, token limit, overlap. |
Split long documents into smaller chunks suitable for vectorization and retrieval.
| Scenario | chunk_size | overlap | Notes |
|---|---|---|---|
| Precise Q&A | 256-512 | 50 | More precise matching |
| Summarization | 1024-2048 | 100 | More complete context |
| Code documentation | By function/class | 0 | Keep code complete |
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=50,
separators=["\n\n", "\n", ".", " ", ""]
)
text = "Your long document content..."
chunks = splitter.split_text(text)
from langchain.text_splitter import TokenTextSplitter
splitter = TokenTextSplitter(
chunk_size=500,
chunk_overlap=50
)
chunks = splitter.split_text(text)
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.schema import Document
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=50
)
# Document with metadata
doc = Document(
page_content="Document content...",
metadata={"source": "doc.pdf", "page": 1}
)
chunks = splitter.split_documents([doc])
# Each chunk retains original metadata
from langchain.text_splitter import MarkdownHeaderTextSplitter
headers = [
("#", "h1"),
("##", "h2"),
("###", "h3"),
]
splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers)
chunks = splitter.split_text(markdown_text)
from langchain.text_splitter import (
Language,
RecursiveCharacterTextSplitter
)
splitter = RecursiveCharacterTextSplitter.from_language(
language=Language.PYTHON,
chunk_size=1000,
chunk_overlap=100
)
chunks = splitter.split_text(code)
def analyze_chunks(chunks):
sizes = [len(c) for c in chunks]
print(f"Total chunks: {len(chunks)}")
print(f"Average size: {sum(sizes)/len(sizes):.0f}")
print(f"Min: {min(sizes)}, Max: {max(sizes)}")
analyze_chunks(chunks)
After chunking:
core:embeddingcore:indexing