Build a fast, deterministic local test loop for LangChain 1.0 / LangGraph 1.0
— FakeListChatModel fixtures, pytest config, VCR cassettes with key redaction,
warning-filter policy. Use when adding tests to a new chain, fixing a flaky
test, or making integration tests reproducible.
Trigger with "langchain pytest", "FakeListChatModel", "VCR langchain",
"langchain test fixtures", "langchain integration test".
Build a fast, deterministic local test loop for LangChain 1.0 / LangGraph 1.0
— FakeListChatModel fixtures, pytest config, VCR cassettes with key redaction,
warning-filter policy. Use when adding tests to a new chain, fixing a flaky
test, or making integration tests reproducible.
Trigger with "langchain pytest", "FakeListChatModel", "VCR langchain",
"langchain test fixtures", "langchain integration test".
Designed for Claude Code, also compatible with Codex
LangChain Local Dev Loop (Python)
Overview
An engineer writes the most natural assertion possible:
deftest_summarize():
out = chain.invoke({"text": "..."})
assert out.content == "expected summary"
It passes locally against Claude at temperature=0. It fails in CI on the third
run with a one-token delta in the output. That is P05: Anthropic's temperature=0
is not greedy — it still samples. Tests against live Claude are not deterministic,
period.
So the engineer swaps in FakeListChatModel(responses=["expected summary"]) and
the assertion passes. Then the downstream callback that logs cost blows up in CI
with KeyError: 'token_usage' — because FakeListChatModel does not emit
response_metadata["token_usage"] (P43). Production code reads that key, so
either the fake has to synthesize it or the test has to skip the callback.
Meanwhile, the first integration test under VCR records a cassette that ships
Authorization: Bearer sk-ant-api03-... in the repo (P44). PR review catches it;
the reviewer revokes the key; the dev loop is hosed for an afternoon.
And none of this matters if pytest cannot even collect the suite because
import langchain_community emits a DeprecationWarning that -W error promotes
to failure (P45).
This skill installs the four layers that make the whole loop fast and safe:
FakeListChatModel / FakeListLLM with a metadata-emitting subclass (fixes P43);
VCR with filter_headers plus a pre-commit hook (fixes P44); pytest
filterwarnings policy in pyproject.toml (fixes P45); and an env-var-gated
integration marker so the default pytest run never touches live APIs.
Speed targets: unit tests with FakeListChatModel run in < 100ms per
test; VCR-replayed integration tests run in 500ms – 2s per test; live
integration tests (the RUN_INTEGRATION=1 gate) run only in nightly or
manual workflows.
For integration tests: at least one provider key (ANTHROPIC_API_KEY, etc.)
Project uses pyproject.toml (PEP 621) for pytest config
Instructions
Step 1 — Deterministic unit tests with FakeListChatModel
Use FakeListChatModel from langchain_core.language_models.fake for chat
chains and FakeListLLM for legacy completion LLMs. Responses cycle through
the list.
from langchain_core.language_models.fake import FakeListChatModel
from langchain_core.prompts import ChatPromptTemplate
deftest_classifier_picks_positive():
fake = FakeListChatModel(responses=["positive"])
prompt = ChatPromptTemplate.from_messages([("user", "Classify: {text}")])
chain = prompt | fake
out = chain.invoke({"text": "I love it"})
assert out.content == "positive"
This is deterministic, runs in single-digit milliseconds, and has zero provider
dependency. Use it for every chain assertion that does not specifically require
real model behavior.
Step 2 — Subclass FakeListChatModel to emit response_metadata (P43 fix)
The stock fake emits no response_metadata["token_usage"]. If your chain has a
callback that records cost, the callback crashes under the fake. Subclass and
synthesize the metadata instead of mocking around the callback:
from langchain_core.language_models.fake import FakeListChatModel
from langchain_core.outputs import ChatGeneration, ChatResult
from langchain_core.messages import AIMessage
classFakeChatWithUsage(FakeListChatModel):
"""FakeListChatModel that emits response_metadata['token_usage'] so
downstream callbacks reading token usage do not crash under test."""def_generate(self, messages, stop=None, run_manager=None, **kwargs):
response = self.responses[self.i % len(self.responses)]
self.i += 1
message = AIMessage(
content=response,
response_metadata={
"token_usage": {
"input_tokens": 10,
"output_tokens": len(response.split()),
"total_tokens": 10 + len(response.split()),
},
"model_name": "fake-chat",
},
usage_metadata={
"input_tokens": 10,
"output_tokens": len(response.split()),
"total_tokens": 10 + len(response.split()),
},
)
return ChatResult(generations=[ChatGeneration(message=message)])
Use FakeChatWithUsage whenever a chain's observability / cost path is in the
assertion surface. See Fake Model Fixtures
for agent, retriever, and embedder fakes.
Step 3 — pytest fixtures that wire the fake into chains
Put fixtures in tests/conftest.py so they are shared across the suite:
# tests/conftest.pyimport pytest
from langchain_core.prompts import ChatPromptTemplate
from tests.fakes import FakeChatWithUsage
@pytest.fixturedeffake_chat():
"""Reusable fake chat model. Override responses per-test via
monkeypatch.setattr(fake_chat, 'responses', [...])."""return FakeChatWithUsage(responses=["ok"])
@pytest.fixturedefsummarize_chain(fake_chat):
prompt = ChatPromptTemplate.from_messages([
("system", "Summarize the user's text in one line."),
("user", "{text}"),
])
return prompt | fake_chat
Step 4 — VCR cassettes for integration tests with key redaction (P44 fix)
Unit tests should never touch the network. Integration tests do, exactly once —
to record a cassette — and every subsequent run replays from the cassette file.
vcrpy records headers by default, which means Authorization: Bearer sk-...
lands in the fixture unless you filter it.
import pytest
@pytest.mark.vcr # cassette at tests/cassettes/<test_name>.yaml@pytest.mark.integrationdeftest_live_claude_short_answer():
from langchain_anthropic import ChatAnthropic
chat = ChatAnthropic(model="claude-sonnet-4-6", temperature=0, timeout=30)
out = chat.invoke("Say 'ok' and nothing else.")
assert"ok"in out.content.lower()
To record (once, locally, with a real key): pytest --record-mode=once tests/.
Every other run replays — cassettes are committed, real API is never hit again.
Pre-commit hook to block key leaks:
# .git/hooks/pre-commit or .pre-commit-config.yaml entry#!/usr/bin/env bashset -e
if git diff --cached --name-only | grep -q '^tests/cassettes/'; thenif git diff --cached -U0 -- 'tests/cassettes/' | \
grep -E '(sk-ant-[a-zA-Z0-9_-]+|sk-[a-zA-Z0-9]{20,}|Bearer\s+[a-zA-Z0-9_-]{20,})'; thenecho"ERROR: API key pattern found in staged cassette." >&2
exit 1
fifi
See VCR Cassette Hygiene for the full
pre-commit config, record-new-episodes flow, shared-cassette patterns, and the
PR review checklist.
langchain_community and some provider SDKs emit DeprecationWarning at import
time. If the suite runs -W error, collection fails before any test does. Set
the policy once in pyproject.toml:
[tool.pytest.ini_options]minversion = "8.0"testpaths = ["tests"]
addopts = [
"-ra",
"--strict-markers",
"--strict-config",
"-W", "error",
]
markers = [
"integration: hits real APIs or replays VCR cassettes (set RUN_INTEGRATION=1)",
"slow: takes > 1s per test",
"smoke: minimal healthcheck run in CI",
]
filterwarnings = [
"error",
"ignore::DeprecationWarning:langchain_community.*",
"ignore::DeprecationWarning:pydantic.*",
"ignore::PendingDeprecationWarning:langchain_core.*",
]
See Pytest Config for the full skeleton
including coverage config and parallel execution notes.
Step 6 — Integration-test gating via env var
Default pytest must never hit real APIs. Gate on RUN_INTEGRATION=1:
# tests/conftest.py (continued)import os
import pytest
defpytest_collection_modifyitems(config, items):
if os.getenv("RUN_INTEGRATION") == "1":
return
skip_integration = pytest.mark.skip(reason="set RUN_INTEGRATION=1 to run")
for item in items:
if"integration"in item.keywords:
item.add_marker(skip_integration)
Step 7 — LangGraph tests: per-test thread_id + state assertions
LangGraph state is scoped to a thread_id. Tests that share a thread_id leak
state between each other. Give every test a fresh thread_id and a fresh
MemorySaver:
from langgraph.checkpoint.memory import MemorySaver
import uuid, pytest
@pytest.fixturedefgraph_config():
return {"configurable": {"thread_id": str(uuid.uuid4())}}
@pytest.fixturedefcheckpointed_graph(fake_chat):
from my_app.graphs import build_graph
return build_graph(fake_chat).compile(checkpointer=MemorySaver())
deftest_node_emits_plan(checkpointed_graph, graph_config, fake_chat):
fake_chat.responses = ["step 1\nstep 2\nstep 3"]
result = checkpointed_graph.invoke({"goal": "deploy"}, graph_config)
# Assert state shape per node, not just the final output:assert result["plan"] == ["step 1", "step 2", "step 3"]
# Time-travel: inspect every checkpoint for debugging
history = list(checkpointed_graph.get_state_history(graph_config))
assert history[-1].values == {"goal": "deploy"} # initial state
Subgraph isolation testing cross-references langchain-langgraph-subgraphs
(pain P21 — parent cannot read child state unless the key is in the parent
schema). See LangGraph Test Patterns
for the subgraph-shared-state test recipe.
Output
tests/fakes.py with FakeChatWithUsage subclass that emits response_metadata
tests/conftest.py with fake-model fixtures, VCR config, and RUN_INTEGRATION gate
pyproject.toml[tool.pytest.ini_options] block with markers and filterwarnings
tests/cassettes/ committed with filtered headers (no Authorization / x-api-key)
Commit 1 — failing test uses real ChatAnthropic, passes locally, fails
1-in-5 in CI at temperature=0 (P05).
Commit 2 — swap to fake model uses FakeListChatModel, passes
deterministically, but the cost-logging callback crashes (P43).
Commit 3 — fake with metadata uses FakeChatWithUsage, the callback
reads response_metadata["token_usage"] cleanly, the test is green and
runs in 40ms.
See Fake Model Fixtures for the full
worked example including agent and retriever fakes.
Recording a cassette without leaking a key
# 1. Ensure conftest.py has filter_headers configured FIRST# 2. Record with real key present in the environment
ANTHROPIC_API_KEY=sk-ant-... pytest --record-mode=once tests/integration/test_summarize.py
# 3. Verify no leak
grep -E 'sk-|Bearer' tests/cassettes/*.yaml && echo"LEAK" || echo"clean"# 4. Commit cassettes/ — pre-commit hook runs the same grep as a hard gate
git add tests/cassettes/ && git commit -m "test: record summarize cassette"
See VCR Cassette Hygiene for
record-new-episodes mode, rerecord-on-mismatch, and the PR review checklist.
LangGraph time-travel debugging on a failing test
When a graph test fails mid-graph, get_state_history(config) returns every
checkpoint — you can replay from any point by passing its config.checkpoint_id
back into graph.invoke. See
LangGraph Test Patterns for the full
time-travel debugging recipe and the subgraph-shared-state test pattern
(cross-ref langchain-langgraph-subgraphs / pain L30).
