Wire LangSmith tracing and custom metric callbacks into a LangChain 1.0 chain
or LangGraph 1.0 agent correctly — env-var spelling, subgraph propagation,
per-tenant dimensions, cost and latency counters. Use when setting up
observability on a new service, debugging blank traces in LangSmith, or adding
per-tenant cost breakdowns. Trigger with "langchain observability",
"langsmith tracing", "langchain callbacks", "langchain metrics".
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Instruções da origem · Visualização somente leitura
name
langchain-observability
description
Wire LangSmith tracing and custom metric callbacks into a LangChain 1.0 chain
or LangGraph 1.0 agent correctly — env-var spelling, subgraph propagation,
per-tenant dimensions, cost and latency counters. Use when setting up
observability on a new service, debugging blank traces in LangSmith, or adding
per-tenant cost breakdowns. Trigger with "langchain observability",
"langsmith tracing", "langchain callbacks", "langchain metrics".
Designed for Claude Code, also compatible with Codex
LangChain Observability (Python)
Overview
Engineer sets LANGCHAIN_TRACING_V2=true and LANGCHAIN_API_KEY=... from the
0.2 docs, restarts the service, and sees zero traces in LangSmith — no errors,
no warnings. That is P26: in LangChain 1.0 the canonical env vars are
LANGSMITH_TRACING and LANGSMITH_API_KEY. The LANGCHAIN_* names are
soft-deprecated and fail silently on any chain that goes through 1.0 middleware
or create_react_agent. One-line fix:
Next failure mode: a custom BaseCallbackHandler attached via
chain.with_config(callbacks=[meter]) fires on the parent but is silent on
LangGraph subgraphs and create_react_agent tool calls — token counts
under-report by 30-70% vs the provider dashboard. That is P28: LangGraph
creates a child runtime per subgraph, and bound callbacks do not propagate.
Pass callbacks at invocation time instead:
Optional metric sinks: prometheus_client, statsd, or datadog Python packages
Instructions
Step 1 — Enable LangSmith with the canonical 1.0 env vars
LANGSMITH_TRACING=true is the switch. LANGSMITH_API_KEY authenticates.
LANGSMITH_PROJECT groups traces by environment — use one project per
service-env pair (myapp-prod, myapp-staging), not one per service.
# .env (loaded via python-dotenv or secret manager)
LANGSMITH_TRACING=true
LANGSMITH_API_KEY=lsv2_pt_...
LANGSMITH_PROJECT=my-service-prod
# Legacy fallback names (still work, soft-deprecated — do not use in new code):# LANGCHAIN_TRACING_V2=true# LANGCHAIN_API_KEY=lsv2_pt_...# LANGCHAIN_PROJECT=my-service-prod
Verify in a REPL that the client sees the key before relying on it in
production:
from langsmith import Client
c = Client() # reads LANGSMITH_API_KEY and LANGSMITH_ENDPOINTprint(c.list_projects(limit=1)) # raises LangSmithAuthError if key is wrong
Do NOT set both LANGCHAIN_TRACING_V2 and LANGSMITH_TRACING — mixed settings
have caused stale project routing in 1.0.x. See P26.
For selective sampling in high-traffic services, set
LANGSMITH_SAMPLING_RATE=0.1 (10% of runs). Full detail in
LangSmith Setup.
Step 2 — Write a metric callback for per-request observability
Subclass BaseCallbackHandler. Record token_in, token_out, latency_ms,
tool_calls, and error, tagged with a tenant_id dimension for downstream
grouping.
A thin sink protocol (incr, hist) swaps between Prometheus, StatsD, or
Datadog. Alternative sinks (LangSmith-only, OTEL) do not need this callback
at all — see Step 5. Full sink adapters and P25 retry dedupe in
Custom Metrics Callback.
Step 3 — Pass callbacks via config["callbacks"] at invocation (P28)
This is the single most common observability bug in LangGraph 1.0 services.
Binding callbacks at definition time does not propagate into subgraphs or
create_react_agent tool nodes — those create child runtimes with their own
callback scope.
# WRONG — fires on parent runnable only; silent on subgraphs (P28)
agent_bound = agent.with_config(callbacks=[MetricCallback(tenant_id, sink)])
result = await agent_bound.ainvoke(inputs)
# RIGHT — propagates to every runnable, subgraph, and tool call
meter = MetricCallback(tenant_id, sink)
result = await agent.ainvoke(
inputs,
config={
"callbacks": [meter],
"configurable": {"thread_id": session_id, "tenant_id": tenant_id},
"tags": ["prod", f"tenant:{tenant_id}"],
"metadata": {"request_id": req_id, "tier": "enterprise"},
},
)
Construct the callback inside the request handler so it captures a fresh
tenant_id per request — and in that pattern, invocation-time config is the
only way callbacks reach subgraphs. See Trace Metadata and Tagging
for the full RunnableConfig shape.
Step 4 — Tag and annotate traces via RunnableConfig
LangSmith indexes two per-request fields: tags (flat list, filterable) and
metadata (key-value, searchable). Fix conventions early — LangSmith has no
rename tool.
Hierarchical tag conventions (env:prod, tenant:acme, tier:enterprise)
make LangSmith filters work. Free-form tags ("important", "check-me") do
not. See Trace Metadata and Tagging.
Step 5 — Pick a sink and the stack shape
The callback handler is the integration point. Options, in decreasing order of
fit:
LangSmith only — zero additional overhead; tracing already covers latency
and token accounting. Fine for solo dev, small teams, and LLM-native ops.
Prometheus (pull) — best fit for Kubernetes + existing Prom stack. Export
via prometheus_client HTTP endpoint. Watch tenant label cardinality.
StatsD / Datadog (push) — UDP fire-and-forget; sub-1ms overhead. Safe on
high-throughput async services. Use datadog.dogstatsd for tag support.
OTEL native — multi-service distributed tracing. Defer to
langchain-otel-observability (L33); do not reimplement here.
Decision tree:
Existing OTEL stack (Collector, Tempo, Jaeger)?
├── YES → OTEL-native (L33). LangSmith optional for prompt inspection.
└── NO → LLM-specific features (prompt inspection, evals, queues) enough?
├── YES → LangSmith only. Add MetricCallback only for tenant cost.
└── NO → Hybrid: LangSmith for prompts + Prometheus/Datadog for SLOs.
See references/hybrid-langsmith-otel.md for split-point rules.
Mixing paths without a plan creates double-emission and conflicting trace IDs.
See Custom Metrics Callback for
Prometheus / StatsD / Datadog sink implementations, plus dedupe for P25 retry
double-counts; see Hybrid LangSmith + OTEL
for the split-point contract.
Step 6 — Feed runs back into evals
Real traffic is the best eval set. Route a sampled subset of production runs
into a LangSmith annotation queue for human review; the queue feeds Dataset
objects replayable against candidate models.
from langsmith import Client
Client().create_annotation_queue(
name="prod-regressions",
description="1% sample, weekly review",
)
# Add metadata={"eval_candidate": "true"} on 1% of runs — LangSmith UI has# a rule to route into the queue by metadata filter.
Keep annotation queues under 500 runs/week (reviewers saturate past that).
See LangSmith Setup for the queue and
dataset flow.
Output
LangSmith tracing on via LANGSMITH_TRACING / LANGSMITH_API_KEY /
LANGSMITH_PROJECT with a langsmith.Client() smoke-check
One metric sink wired (Prometheus, StatsD, Datadog, or LangSmith-only)
Explicit choice recorded for LangSmith / OTEL / hybrid / custom
Error Handling
Error
Cause
Fix
No traces in LangSmith, no errors
Used LANGCHAIN_TRACING_V2 spelling on 1.0 middleware path (P26)
Switch to LANGSMITH_TRACING=true and LANGSMITH_API_KEY
langsmith.utils.LangSmithAuthError: Unauthorized
Key is valid but points to a deleted workspace, or copied with trailing whitespace
Regenerate at smith.langchain.com, check repr(os.environ['LANGSMITH_API_KEY']) for \n
Callback fires on parent only, silent on subgraphs
Bound via .with_config(callbacks=[...]) — does not propagate (P28)
Pass via config["callbacks"] at invoke() / ainvoke()
Token counts under by 30-70% vs provider dashboard
Combination of P28 (subgraph silence) and P25 (retry double-count not deduped)
Fix P28 first; for P25 add request_id dedupe key in sink
Trace duration shows 0ms on streamed calls
on_llm_end fires after stream closes but handler records before — timing race
Use time.perf_counter() captured in on_llm_start, not on_chat_model_start
Prometheus cardinality explosion
tenant_id label has high cardinality (>10k tenants)
Bucket tenants into tiers for metrics; keep full tenant_id in LangSmith metadata only
LangSmith UI shows runs under default project, not the configured one
LANGSMITH_PROJECT env var not set at process start
Set before import; LANGSMITH_PROJECT is read once at Client() init
AttributeError: 'NoneType' object has no attribute 'get' in on_llm_end
usage_metadata is None on intermediate streaming chunks
Guard with if meta := getattr(g.message, 'usage_metadata', None):
Examples
Multi-tenant SaaS: per-tenant cost dashboard
A production SaaS has 200 tenants on a shared LangGraph agent. Finance wants
weekly cost reports per tenant. The MetricCallback records token_in,
token_out, and cache_read tagged with tenant_id; Prometheus scrapes the
/metrics endpoint; Grafana aggregates sum by (tenant_id) (rate(llm_token_out_total[1w])) * 0.0000015
for Sonnet output cost. The invocation-time config["callbacks"] propagation
is load-bearing here — without it, subgraph tool calls (the bulk of token
spend) go uncounted. See Custom Metrics Callback
for the full Prometheus integration.
Debugging missing traces in staging
A team deploys a new LangGraph service to staging. No traces show up in
LangSmith. Checking: (1) LANGSMITH_TRACING spelled correctly — yes; (2) API
key valid — langsmith.Client().list_projects(limit=1) returns ok; (3) project
name matches — LANGSMITH_PROJECT=myservice-staging. Traces appear in the
default project, not myservice-staging. Root cause: the env var was set in
the runtime env-file but the process was started before the env-file was
sourced. Client() read LANGSMITH_PROJECT at import time. Fix: restart the
process cleanly. See LangSmith Setup for the
process-order checklist.
Feeding prod traffic to an eval dataset
A team wants to validate a Claude 4.6 → Claude 4.7 upgrade against recent prod
runs. They add metadata={"eval_candidate": "pre-upgrade"} to 1% of runs for
one week, create a LangSmith dataset from the tagged runs, then replay against
the new model and diff outputs. The sampling rule lives in LangSmith UI,
filtered by metadata.eval_candidate. See LangSmith Setup
for the annotation-queue and dataset-creation flow.
