菜单
Use when user needs to split documents into chunks for RAG or search. Triggers on: chunking, split, chunk size, text splitter, token limit, overlap.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
| name | chunking |
| description | Use when user needs to split documents into chunks for RAG or search. Triggers on: chunking, split, chunk size, text splitter, token limit, overlap. |
Split long documents into smaller chunks suitable for vectorization and retrieval.
| Scenario | chunk_size | overlap | Notes |
|---|---|---|---|
| Precise Q&A | 256-512 | 50 | More precise matching |
| Summarization | 1024-2048 | 100 | More complete context |
| Code documentation | By function/class | 0 | Keep code complete |
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=50,
separators=["\n\n", "\n", ".", " ", ""]
)
text = "Your long document content..."
chunks = splitter.split_text(text)
from langchain.text_splitter import TokenTextSplitter
splitter = TokenTextSplitter(
chunk_size=500,
chunk_overlap=50
)
chunks = splitter.split_text(text)
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.schema import Document
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=50
)
# Document with metadata
doc = Document(
page_content="Document content...",
metadata={"source": "doc.pdf", "page": 1}
)
chunks = splitter.split_documents([doc])
# Each chunk retains original metadata
from langchain.text_splitter import MarkdownHeaderTextSplitter
headers = [
("#", "h1"),
("##", "h2"),
("###", "h3"),
]
splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers)
chunks = splitter.split_text(markdown_text)
from langchain.text_splitter import (
Language,
RecursiveCharacterTextSplitter
)
splitter = RecursiveCharacterTextSplitter.from_language(
language=Language.PYTHON,
chunk_size=1000,
chunk_overlap=100
)
chunks = splitter.split_text(code)
def analyze_chunks(chunks):
sizes = [len(c) for c in chunks]
print(f"Total chunks: {len(chunks)}")
print(f"Average size: {sum(sizes)/len(sizes):.0f}")
print(f"Min: {min(sizes)}, Max: {max(sizes)}")
analyze_chunks(chunks)
After chunking:
core:embeddingcore:indexingUse when user needs to group similar items together. Triggers on: clustering, group similar, topic modeling, user segmentation, categorization, automatic classification, unsupervised grouping.
Use when user needs to find duplicate or similar content. Triggers on: duplicate, deduplication, plagiarism detection, similar content, near-duplicate, similarity detection, content dedup, find copies.
Use when user needs long-term memory for chatbots. Triggers on: chat memory, conversation history, long-term memory, chatbot memory, memory retrieval, persistent memory, remember conversations.
Use when user wants to build image search or similar image finding. Triggers on: image search, similar image, visual search, image retrieval, CLIP, reverse image search, image matching, find similar photos.
Use when user needs RAG on documents with images and text. Triggers on: multimodal RAG, image-text mixed, document with images, PDF with charts, visual RAG, visual Q&A, documents with figures.
Use when user needs to search images using natural language descriptions. Triggers on: text to image, describe and find, natural language image search, image caption search, find image by description, describe to find.