LangChain Data Handling — Loaders and Splitters (Python)
Overview
You have a RAG system over a Python docs site. A user asks "what does
trim_messages do?" and the retriever returns this chunk:
### `trim_messages(strategy="last", include_system=True)`
Trim a message history to fit a token budget. The newest messages are kept;
older messages are dropped. Pass `include_system=True` to preserve the system
...and that's it. The chunk ends there. The code example showing the function
body — the actual thing the user wanted — is in a different chunk, retrieved
with a lower similarity score and dropped before the LLM sees it. The model
then hallucinates the function's behavior from the signature alone.
This is pain-catalog entry P13. RecursiveCharacterTextSplitter's default
separators are ["\n\n", "\n", " ", ""]. It splits on any blank line — including
inside triple-backtick code fences in Markdown. The fix is a one-line swap
to RecursiveCharacterTextSplitter.from_language(Language.MARKDOWN), which
treats the fence as an atomic unit, but you have to know the bug exists.
The sibling failures this skill prevents:
- P49 —
PyPDFLoader splits by page. A 5-row financial table that spans
a page break gets torn in half; rows 1-3 go in one chunk, rows 4-5 in another
with no header. A RAG answer sourced from the second chunk misquotes the
numbers because the column meanings are in the first chunk. Fix: use
PyMuPDFLoader or UnstructuredPDFLoader, which detect tables and emit
them as distinct structured elements.
- P50 —
WebBaseLoader's default User-Agent is python-requests/2.x.
Cloudflare-protected sites flag this as a bot and return a 403 interstitial
HTML page ("Checking your browser...") instead of real content. The crawler
indexes the challenge page. You notice weeks later when every retrieval from
that source returns the same Cloudflare text. Fix: set a realistic
header_template={"User-Agent": "Mozilla/5.0 ..."}, respect robots.txt,
and rate-limit per-host to 1 req/sec.
Pinned versions: langchain-core 1.0.x, langchain-community 1.0.x,
langchain-text-splitters 1.0.x, pymupdf, unstructured.
Pain-catalog anchors: P13, P49, P50, P15.
This skill is the upstream half of the RAG pipeline — load and chunk.
For the downstream half (embedding, scoring, reranking) see the pair skill
langchain-embeddings-search, which covers score semantics (P12), dim guards
(P14), and reranker filtering (P15). Do not re-implement chunking there.
Prerequisites
- Python 3.10+
langchain-core >= 1.0, < 2.0 and langchain-community >= 1.0, < 2.0langchain-text-splitters >= 1.0, < 2.0- PDF support:
pip install pymupdf unstructured[pdf]
- Web loading:
pip install beautifulsoup4 requests
- For corpus dedup (optional):
pip install datasketch
Instructions
Step 1 — Choose a loader by source format
Loader selection is the first decision — get it wrong and no amount of
splitter tuning will recover. Use the decision table:
| Source | Use | NOT | Why |
|---|
| PDF with tables | PyMuPDFLoader or UnstructuredPDFLoader | PyPDFLoader | Tables torn by page splits (P49) |
| PDF text-only | PyPDFLoader | — | Simple, fast, OK when no tables |
| Web page | WebBaseLoader(header_template=...) | Default UA | Cloudflare 403 (P50) |
| Markdown docs | UnstructuredMarkdownLoader | Plain text read | Preserves heading structure |
| HTML long-form | WebBaseLoader + HTMLHeaderTextSplitter | Plain text | Keeps <h1>/<h2> context |
| Code repo | GenericLoader with language parser | DirectoryLoader as text | Language-aware chunking |
| Corpus (1000+ docs) | DirectoryLoader + glob filter | One-by-one | Parallel load, progress |
from langchain_community.document_loaders import (
PyMuPDFLoader,
WebBaseLoader,
UnstructuredMarkdownLoader,
DirectoryLoader,
)
pdf_docs = PyMuPDFLoader("10-Q-filing.pdf").load()
web_docs = WebBaseLoader(
"https://example.com/article",
header_template={
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"
},
).load()
md_docs = UnstructuredMarkdownLoader("docs/guide.md").load()
corpus = DirectoryLoader(
"./docs", glob="**/*.md",
loader_cls=UnstructuredMarkdownLoader,
show_progress=True,
).load()
Hard limit: keep single-PDF ingestion under 5 MB per call. Larger files
should be pre-split with pdftk / qpdf to avoid OOM on PyMuPDFLoader's
full-document parse.
See Loader Selection Matrix for the
full per-format table with cost and accuracy notes.
Step 2 — Pick a splitter by content type
| Content | Splitter | chunk_size | chunk_overlap | Why |
|---|
| Prose (docs, articles) | RecursiveCharacterTextSplitter.from_language(Language.MARKDOWN) | 1000 | 100 | Preserves code fences (P13) |
| Python source | RecursiveCharacterTextSplitter.from_language(Language.PYTHON) | 1500 | 150 | Splits at def/class |
| FAQ / Q&A | RecursiveCharacterTextSplitter with separators=["\n\n"] | 500 | 50 | One chunk per Q-A pair |
| HTML long-form | HTMLHeaderTextSplitter | — | — | Headers become metadata |
| Generic text | RecursiveCharacterTextSplitter | 1000 | 100 | Safe default |
from langchain_text_splitters import (
RecursiveCharacterTextSplitter,
Language,
HTMLHeaderTextSplitter,
)
md_splitter = RecursiveCharacterTextSplitter.from_language(
Language.MARKDOWN, chunk_size=1000, chunk_overlap=100,
)
py_splitter = RecursiveCharacterTextSplitter.from_language(
Language.PYTHON, chunk_size=1500, chunk_overlap=150,
)
html_splitter = HTMLHeaderTextSplitter(
headers_to_split_on=[("h1", "Header 1"), ("h2", "Header 2")],
)
bad = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
See Language-Aware Splitters for the
full list of Language.* enum values, custom separator patterns, and the
code-fence-detection regex for when you need a custom splitter.
Step 3 — Tune chunk_size and overlap
Defaults from the table work for most corpora. Tune when:
- Retrieval misses context: increase
chunk_size (1000 → 1500) or
chunk_overlap (100 → 200). Overlap is what bridges a concept that crosses
chunk boundaries.
- Retrieval too broad, answers wander: decrease
chunk_size (1000 → 500).
Smaller chunks = more precise retrieval but more chunks to index.
- Tables / structured data: do NOT tune — index them separately (step 4).
A 1% overlap-to-size ratio is too low (200/20000); 20% is the sweet spot for
most prose. Code needs less overlap (10%) because function boundaries are
natural splits.
Step 4 — Detect and index tables as structured records
Tables are not text. If your corpus has financial filings, product specs,
or any tabular data, index tables as separate records with column metadata:
import fitz
def extract_tables_as_records(pdf_path: str) -> list[dict]:
"""Extract tables as one record per row."""
doc = fitz.open(pdf_path)
records = []
for page_num, page in enumerate(doc):
tables = page.find_tables()
for table in tables:
rows = table.extract()
if not rows:
continue
headers = rows[0]
for row_idx, row in enumerate(rows[1:], start=1):
record = {
"page": page_num,
"table_idx": tables.tables.index(table),
"row_idx": row_idx,
"content": " | ".join(f"{h}: {v}" for h, v in zip(headers, row)),
"metadata": dict(zip(headers, row)),
}
records.append(record)
return records
Now a question like "what was Q3 revenue?" retrieves a single row with its
column headers attached, not half a table missing the column meanings. See
Table Preservation for the full pattern
including hybrid retrieval (prose + table records).
Step 5 — Preserve metadata through the pipeline
The loader attaches metadata (source, page, heading); the splitter propagates
it. Front-matter in Markdown, PDF page numbers, and web URLs should all end
up in doc.metadata so retrieval results are citable:
for doc in md_docs:
print(doc.metadata.get("title"), doc.metadata.get("date"))
chunks = md_splitter.split_documents(md_docs)
assert chunks[0].metadata == md_docs[0].metadata
Custom metadata (tenant_id, version, confidence) should be added before
splitting so every chunk inherits it.
Step 6 — Deduplicate noisy corpora
Web crawls and scraped docs often contain near-duplicate pages (nav chrome,
footer boilerplate, syndicated posts). MinHash-based dedup at the chunk level
keeps the index clean:
from datasketch import MinHash, MinHashLSH
lsh = MinHashLSH(threshold=0.9, num_perm=128)
kept = []
for i, chunk in enumerate(all_chunks):
mh = MinHash(num_perm=128)
for tok in chunk.page_content.lower().split():
mh.update(tok.encode())
if not list(lsh.query(mh)):
lsh.insert(str(i), mh)
kept.append(chunk)
A threshold of 0.9 catches near-duplicates (minor wording differences) without
eating legitimate paraphrases.
Step 7 — Compose the pipeline
def build_rag_index(source_dir: str, store):
docs = DirectoryLoader(
source_dir, glob="**/*.md",
loader_cls=UnstructuredMarkdownLoader,
).load()
docs = [d for d in docs if d.page_content.strip()]
splitter = RecursiveCharacterTextSplitter.from_language(
Language.MARKDOWN, chunk_size=1000, chunk_overlap=100,
)
chunks = splitter.split_documents(docs)
store.add_documents(chunks)
return store
For the embedding + indexing + retrieval steps, see langchain-embeddings-search.
Output
- Loader chosen from the selection matrix matching source format and table needs
- Splitter chosen from the decision tree matching content type
- Chunk size + overlap tuned from the defaults (1000/100 prose, 1500/150 code, 500/50 FAQ)
- Tables extracted as structured records with column metadata (not text chunks)
- Web loaders configured with realistic User-Agent and
robots.txt respect
- Metadata preserved through loader → splitter → index
- Optional MinHash dedup (threshold 0.9) for noisy corpora
Error Handling
| Error / symptom | Cause | Fix |
|---|
| RAG retrieves function signature without body | RecursiveCharacterTextSplitter broke inside code fence (P13) | Use from_language(Language.MARKDOWN) or add "```" as first separator |
| Table rows misquoted in RAG answer | PyPDFLoader tore table by page (P49) | Switch to PyMuPDFLoader; index tables as structured records |
WebBaseLoader returns 403 / blank content | Default UA flagged by Cloudflare (P50) | Set header_template={"User-Agent": "Mozilla/5.0 ..."}; respect robots.txt |
ValueError: expected str, NoneType found during split | Empty page_content | Filter [d for d in docs if d.page_content.strip()] before splitting |
MemoryError loading PDF | PDF > 5 MB ingested in one call | Pre-split with pdftk / qpdf; process chunks separately |
| Chunks missing metadata after split | Custom metadata added after loading but before splitting was lost | Add metadata before split_documents(); verify chunks[0].metadata preserved |
| Retrieval quality low on FAQ corpus | Chunks too large, one chunk holds multiple Q-A pairs | Drop to chunk_size=500, chunk_overlap=50 with separators=["\n\n"] |
| Web crawl indexes Cloudflare challenge page | No check for HTTP status / response length | Assert len(doc.page_content) > 500 and reject pages containing "Checking your browser" |
| Duplicate chunks eat retrieval slots | Syndicated content, nav chrome not stripped | MinHash dedup at threshold 0.9 before indexing |
| Reranker scores inconsistent across chunks | Chunks of wildly different size change score distribution (P15) | Normalize chunk size within a corpus; target ±20% of chunk_size |
Examples
Ingesting a Markdown docs site with code examples
Markdown docs with Python code fences require Language.MARKDOWN to keep
fence boundaries intact. Chunk size 1000 with 100 overlap preserves one
function-sized example per chunk. Front-matter fields (title, date, author)
are attached as metadata for citation. See
Language-Aware Splitters.
Ingesting a PDF filing with financial tables
10-Q filings have dozens of multi-row tables. Use PyMuPDFLoader for the prose
and a direct fitz.find_tables() pass to extract tables as structured records.
Index prose with chunk_size=1000 and tables as one-row-per-record with the
header row concatenated. Questions like "what was Q3 revenue?" hit a single row
with column meanings attached. See
Table Preservation.
Crawling a documentation site behind Cloudflare
Set a realistic User-Agent, fetch robots.txt first and respect Disallow
rules, rate-limit to 1 req/sec per host, and prefer the site's sitemap or RSS
feed when available. Assert response length > 500 chars and reject known
interstitial patterns. See Crawler Hygiene.
Ingesting a Python code repo for code RAG
GenericLoader with LanguageParser(language=Language.PYTHON) preserves
function and class boundaries. Chunk size 1500 with 150 overlap gives enough
context for typical function-level queries. Imports and module docstrings
end up in their own chunks — tag them with metadata for higher precision
retrieval on "where is X imported from" queries.
Resources
