langchain-agent-integration-testing

name	langchain-agent-integration-testing
description	Organize and run LangChain agent integration tests against real LLM APIs using pytest markers, env-based secrets, structural assertions, cost controls, and optional HTTP record/replay. Use when tests need live models, CI secrets, flaky LLM output handling, or the user mentions integration tests, real API agents, or pytest -m integration.
disable-model-invocation	true

Documentation Index

Fetch the complete documentation index at: https://docs.langchain.com/llms.txt Use this file to discover all available pages before exploring further.

---

Integration testing LangChain agents

Integration tests call real model APIs (and sometimes other network services). They validate wiring, credentials, latency, and end-to-end behavior. They are nondeterministic compared to unit tests that use fakes (see langchain-agent-unit-testing).

Separate unit and integration tests

Keep integration tests out of the default fast path. Tag them and exclude by default.

import pytest
from langchain.agents import create_agent
from langchain_core.messages import HumanMessage

@pytest.mark.integration
def test_agent_with_real_model():
    # get_weather: your @tool implementation
    agent = create_agent("claude-sonnet-4-6", tools=[get_weather])
    result = agent.invoke(
        {"messages": [HumanMessage(content="What's the weather in SF?")]}
    )
    assert len(result["messages"]) > 1

pytest.ini

[pytest]
markers =
    integration: tests that call real LLM APIs
addopts = -m "not integration"

pyproject.toml (equivalent)

[tool.pytest.ini_options]
markers = [
  "integration: tests that call real LLM APIs"
]
addopts = "-m 'not integration'"

Run only integration tests:

pytest -m integration

Manage API keys

Load credentials from environment variables only; never commit keys.

import os
import pytest

@pytest.fixture(autouse=True)
def check_api_keys():
    if not os.environ.get("OPENAI_API_KEY"):
        pytest.skip("OPENAI_API_KEY not set")

Optional local .env (must be gitignored):

# .env
OPENAI_API_KEY=sk-...

# conftest.py (local dev)
from dotenv import load_dotenv

load_dotenv()

In CI, inject secrets via the platform (e.g. GitHub Actions encrypted secrets). For AWS Bedrock, skip unless AWS_REGION (or profile) and usable credentials are present, plus model access in the account.

Assert on structure, not content

Prefer checks on message types, tool call names, argument keys, and message counts — not exact assistant prose.

from langchain.agents import create_agent
from langchain_core.messages import AIMessage

def test_agent_calls_weather_tool():
    # get_weather: your @tool implementation
    agent = create_agent("claude-sonnet-4-6", tools=[get_weather])
    result = agent.invoke(
        {"messages": [{"role": "user", "content": "What's the weather in SF?"}]}
    )
    messages = result["messages"]
    tool_calls = [
        tc
        for msg in messages
        if hasattr(msg, "tool_calls")
        for tc in (msg.tool_calls or [])
    ]
    assert any(tc["name"] == "get_weather" for tc in tool_calls)
    assert isinstance(messages[-1], AIMessage)
    assert len(str(messages[-1].content)) > 0

For trajectory-level checks with fuzzy matching, use LangChain AgentEvals (see Evals in the docs index).

Reduce cost and latency

• Prefer a small / fast model when the test only needs tool routing or shape validation.
• Cap max_tokens (or provider equivalent) on the model config.
• One behavior per test; avoid long multi-hop chains unless that is what you are testing.
• Run pytest -m integration only in CI or pre-deploy, not on every save.

agent = create_agent(
    "gemini-2.0-flash",
    tools=[get_weather],
    model_kwargs={"max_tokens": 256},
)

(Adjust model id strings to what your stack supports.)

Record and replay HTTP (optional)

For expensive or flaky HTTP, vcrpy + pytest-recording can record cassettes on first run and replay later.

import pytest

@pytest.fixture(scope="session")
def vcr_config():
    return {
        "filter_headers": [
            ("authorization", "XXXX"),
            ("x-api-key", "XXXX"),
        ],
        "filter_query_parameters": [
            ("api_key", "XXXX"),
            ("key", "XXXX"),
        ],
    }

Register a vcr marker and default record mode in pytest config; decorate tests with @pytest.mark.vcr(). Regenerate cassettes when prompts, tools, or expected trajectories change (delete stale YAML cassettes and re-record).

Key rules

1. Default pytest run should exclude integration tests (addopts).
2. Skip with a clear reason when required env vars are missing.
3. Structure over verbatim text for LLM assertions.
4. Scrub secrets in VCR config if using HTTP recording.
5. Treat failures as signal — flakiness may mean assertions are too strict or the model policy shifted.

Related

• Fake models / no-network agent tests: langchain-agent-unit-testing in this repo.
• LangGraph graph tests (checkpointer, nodes): langgraph-testing in this repo.
• Trajectory evals: LangChain docs — Evals (/oss/python/langchain/test/evals).