langsmith-trace

name	langsmith-trace
description	LangSmith tracing and trace debugging for AgentSeek templates. Covers CLI installation, adding tracing to LangGraph apps, querying traces, and inspecting run IO. Use when debugging agent backends, investigating slow traces, or adding observability to a template.

LangSmith Trace

Add tracing to your agent and query traces for debugging. Supports Python and TypeScript.

[!IMPORTANT] This skill is tuned for AgentSeek template backends (LangGraph + middleware stacks). For general LangSmith concepts, see the upstream langsmith-skills repo.

[!CAUTION] Never pass --api-key as a CLI flag or expose API keys in shell commands / tool calls. The CLI reads LANGSMITH_API_KEY from the environment automatically. Using --api-key <value> leaks secrets into shell history, process listings, and agent tool-call logs. Always rely on the environment variable set in your shell profile or .env file.

Installation & Setup

1. Install the CLI

curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh

The binary installs to ~/.local/bin/langsmith. If langsmith is not found after install, add to your shell profile:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

2. Set your API key (environment variable only)

Add to ~/.zshrc (or the template's .env file) — the user must fill in their own key:

export LANGSMITH_API_KEY=<your-key-here>  # starts with lsv2_pt_

Key format must start with lsv2_pt_. Get yours at https://smith.langchain.com/settings.

The CLI reads this env var automatically — never use --api-key flags. Do not echo, print, or pass the key value in any shell command or tool call argument.

3. Verify

langsmith project list

If you see a JSON array of projects, you're set. Common failures:

• command not found — ~/.local/bin not in PATH (see step 1)
• 401 Unauthorized — key is wrong or expired; regenerate at LangSmith settings
• Empty array [] — valid auth but no projects yet; create one in the UI or run a traced app

Adding Tracing

LangGraph / LangChain apps (automatic)

Just set environment variables — no code changes needed:

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=<your-key-here>  # must be set; do NOT pass via --api-key flag
export LANGSMITH_PROJECT=my-project  # optional, defaults to "default"

For serverless (Python): also set LANGCHAIN_CALLBACKS_BACKGROUND=false to ensure traces flush before function exit.

Non-LangChain apps

Use the @traceable decorator (Python) or traceable() wrapper (TypeScript) and wrap your LLM client:

from langsmith import traceable
from langsmith.wrappers import wrap_openai
from openai import OpenAI

client = wrap_openai(OpenAI())

@traceable
def my_pipeline(question: str) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": question}],
    )
    return resp.choices[0].message.content

import { traceable } from "langsmith/traceable";
import { wrapOpenAI } from "langsmith/wrappers";
import OpenAI from "openai";

const client = wrapOpenAI(new OpenAI());

const myPipeline = traceable(async (question: string) => {
  const resp = await client.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: question }],
  });
  return resp.choices[0].message.content || "";
}, { name: "my_pipeline" });

Debugging a Trace (Step-by-Step)

This is the recommended workflow for investigating agent behavior:

Step 1: Find the right project

langsmith project list

Look at last_run_start_time to find which project has recent activity. Don't assume — LangGraph apps default to the "default" project, not a named one.

Step 2: List recent traces

langsmith trace list --project default --limit 5

# Or with full hierarchy inline (combines steps 2+3):
langsmith trace list --project default --limit 5 --show-hierarchy

Step 3: Get the trace hierarchy

langsmith trace get <trace-id> --project <name>

This returns the full run tree — use it to understand the agent's execution flow.

Step 4: Get IO for all runs in a trace

langsmith run list --trace-ids <trace-id> --project <name> --include-io

This gives you inputs and outputs for every run (LLM calls, tool calls, middleware).

Step 5: Drill into a specific run

langsmith run get <run-id> --include-io

AgentSeek Trace Structure

Our templates produce traces with this typical hierarchy:

<agent_name> (root chain)
├── SkillsMiddleware.before_agent
├── PatchToolCallsMiddleware.before_agent
├── MemoryMiddleware.before_agent
├── model (chain) ← LLM turn
│   ├── TodoListMiddleware.awrap_model_call
│   ├── SkillsMiddleware.awrap_model_call
│   ├── FilesystemMiddleware.awrap_model_call
│   ├── SubAgentMiddleware.awrap_model_call
│   ├── SummarizationMiddleware.awrap_model_call
│   ├── AnthropicPromptCachingMiddleware.awrap_model_call
│   ├── MemoryMiddleware.awrap_model_call
│   └── ChatOpenAI (llm) ← actual LLM call (inputs/outputs here)
├── TodoListMiddleware.after_model
├── tools (chain) ← tool execution
│   ├── FilesystemMiddleware.awrap_tool_call
│   └── <tool_name> (tool) ← actual tool (inputs/outputs here)
├── model (chain) ← next LLM turn
│   └── ... (same middleware stack)
└── TodoListMiddleware.after_model

Key points:

• The actual LLM call is always the innermost ChatOpenAI run
• Tool results are in the <tool_name> run (e.g., generate_cover, execute)
• Middleware wrappers are transparent — they add latency but the IO you care about is at the leaf nodes

Gotchas

--full vs --include-io on individual runs

Problem: langsmith run get <id> --full can return null for inputs/outputs, even though the data exists. Despite --full being documented as equivalent to --include-metadata --include-io --include-feedback, the underlying API behavior differs for individual run fetches in some CLI versions.

Solution: Always use --include-io explicitly when inspecting specific runs:

# DO THIS
langsmith run get <run-id> --include-io

# NOT THIS (may return null IO despite docs saying it includes --include-io)
langsmith run get <run-id> --full

--full works reliably on trace export and run list, but has inconsistent behavior on individual run get calls. If this is fixed in a future CLI version, --include-io still works correctly — it's always safe.

Null IO even with --include-io

If inputs/outputs come back null even with --include-io, the project has IO logging disabled. Check for these environment variables in the template's .env:

LANGCHAIN_HIDE_INPUTS=true    # hides inputs from traces
LANGCHAIN_HIDE_OUTPUTS=true   # hides outputs from traces

Remove or set to false to enable IO capture for debugging.

Project confusion

LangGraph apps trace to the "default" project unless LANGSMITH_PROJECT is explicitly set. Always run langsmith project list first and check last_run_start_time to find where your traces actually landed.

Tips

• Add --format pretty for human-readable output during interactive debugging
• Use LANGSMITH_ENDPOINT env var if connecting to a self-hosted LangSmith instance
• The middleware ordering in the trace tree is configurable — your template may differ slightly from the diagram above

CLI Reference

Full command reference: reference/cli-commands.md