メニュー
Agent observability, evals, feedback, and experiments. Use when adding observability dashboards, configuring trace capture, setting up evals, creating A/B experiments, or collecting user feedback on agent responses.
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/BuilderIO/agent-native --skill observabilityコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
SKILL.md を表示中
| name | observability |
| description | Agent observability, evals, feedback, and experiments. Use when adding observability dashboards, configuring trace capture, setting up evals, creating A/B experiments, or collecting user feedback on agent responses. |
| scope | dev |
| metadata | {"internal":true} |
The observability system auto-instruments every agent run with zero configuration. Traces, automated evals, and feedback collection work out of the box. All data lives in the app's own SQL database — no external services required. Templates can optionally export to Langfuse, Datadog, or any OTel-compatible platform.
Every runAgentLoop() call is automatically instrumented via instrumentAgentLoop() in packages/core/src/observability/traces.ts. It captures:
Content (prompts, tool args, tool results) is redacted by default. Opt in via the observability-config settings key:
await putSetting(, {
: ,
: ,
: ,
: ,
: ,
: ,
: ,
: ,
});
Self-hosted apps default to no inferred sentiment. First-party apps hosted on
agent-native.com automatically classify 100% of eligible user replies with
gpt-5-6-luna; an explicit stored inferredSentimentEnabled: false remains an
opt-out. Deployment overrides are AGENT_NATIVE_INFERRED_SENTIMENT=on|off,
AGENT_NATIVE_INFERRED_SENTIMENT_SAMPLE_RATE=0..1, and
AGENT_NATIVE_INFERRED_SENTIMENT_MODEL=<model>; off is always the emergency
kill switch.
Classification uses only the original visible user text, capped at 2,000 characters, with no tools, temperature 0, an eight-token output, and a five second timeout. It skips attachment-only turns, internal continuations, chained background chunks, and first turns that have no preceding response to attribute. The managed Builder engine runs the classifier after the main response has streamed, so it does not contend with the user's response.
Successful classifications emit a content-free $ai_sentiment tracking event:
sentiment: positive, negative, or neutralmethod: llmmodel / $ai_model: model that generated the preceding assistant responserun_id / $ai_trace_id: preceding response runthread_id / $ai_session_id: conversationclassification_trigger_run_id: run started by the classified user replyclassifier_model and classifier_engine: classifier attributionNo raw message, prompt, or response text is persisted or tracked.
Explicit — ThumbsFeedback component renders inline thumbs up/down on every agent message in the chat UI. Thumbs down opens a category popover (Inaccurate, Not helpful, Wrong tool, Too slow). Already wired into AssistantChat.tsx via React.lazy.
Implicit — computeSatisfactionScore(threadId) computes a Frustration Index (0-100) from conversation signals:
Score interpretation: 0-20 healthy, 20-40 friction, 40-60 dissatisfied, 60+ broken.
Satisfaction scoring fires automatically after each feedback POST with a threadId.
Three layers, configured via evalSampleRate in the observability config:
Automated (every run): Deterministic scorers that run after every traced run:
tool_success_rate — % of tool calls without errorsstep_efficiency — 1.0 for no-tool runs; penalizes excessive LLM iterations for tool-using runslatency_score — normalized against 10s/tool baselinecost_efficiency — normalized against 50 centicents/tool baselineerror_recovery — 1.0 if the run recovered from tool errors or had noneLLM-as-judge (sampled): Runs on evalSampleRate fraction of runs. Calls the configured engine with a judge prompt that scores against custom criteria.
Dataset evaluation: runDatasetEval(datasetId) runs a golden dataset through the agent and scores each case.
Custom criteria use natural language rubrics:
const criteria: EvalCriteria = {
name: "helpfulness",
description: "Was the response helpful and complete?",
rubric: "0.0 = completely unhelpful, 0.5 = partially helpful, 1.0 = fully resolved the user's need",
};
The three layers above score real production runs after the fact. For an active, deterministic gate, use the first-class *.eval.ts primitive from @agent-native/core/eval (source: packages/core/src/eval/*). It runs the actual agent loop against fixed inputs and exits non-zero below threshold, so it gates CI/deploys.
// evals/faq.eval.ts
import { defineEval, contains, llmJudge } from "@agent-native/core/eval";
export default defineEval({
name: "answers the FAQ",
input: { prompt: "What is your return policy?" },
threshold: 0.7,
scorers: [contains("30 days"), llmJudge({ criteria: "accuracy" })],
});
exactMatch / contains / usesTool (pure JS) and llmJudge (provider-agnostic judge).createScorer with the 4-step preprocess → analyze → generateScore → generateReason pipeline (only generateScore is required).agent-native eval [pattern] [--json] [--threshold N] — discovers **/*.eval.ts and evals/*.ts, runs the agent, and exits non-zero if any eval is below its threshold. An app with no eval files exits 0. Complements (does not replace) the post-hoc scoring in evals.ts. See the Evals doc.A/B testing with sticky user-level assignment:
import { insertExperiment, updateExperiment } from "@agent-native/core/observability";
const exp = {
id: crypto.randomUUID(),
name: "sonnet-vs-haiku",
status: "draft" as const,
variants: [
{ id: "control", weight: 50, config: { model: "claude-sonnet-4-6" } },
{ id: "treatment", weight: 50, config: { model: "claude-haiku-4-5-20251001" } },
],
metrics: ["cost", "latency", "satisfaction"],
assignmentLevel: "user" as const,
startedAt: null,
endedAt: null,
createdAt: Date.now(),
};
await insertExperiment(exp);
// Move it to "running" when ready to start collecting assignments.
await updateExperiment(exp.id, { status: "running" });
The agent loop reads active experiments via resolveActiveExperimentConfig() and applies the variant's model override automatically. Assignment uses consistent hashing — same user always gets the same variant.
Compute results with POST /_agent-native/observability/experiments/:id/results.
In production, experiment management routes require the caller's email in the
comma-separated AGENT_NATIVE_EXPERIMENT_ADMIN_EMAILS allowlist. This gate is
separate from normal app/org admin roles because an experiment affects every
user in that deployment.
ObservabilityDashboard is a React component with 5 tabs:
Add a dashboard route to any template:
// app/routes/observability.tsx
import { ObservabilityDashboard } from "@agent-native/core/client/observability";
export default function ObservabilityPage() {
return (
<div className="min-h-screen bg-background p-6">
<ObservabilityDashboard />
</div>
);
}
All auto-mounted at /_agent-native/observability/*:
| Method | Path | Purpose |
|---|---|---|
| GET | / | Overview stats |
| GET | /traces | List trace summaries |
| GET | /traces/:runId | Trace detail (summary + spans) |
| GET | /traces/:runId/evals | Evals for a run |
| POST | /feedback | Submit feedback |
| GET | /feedback | List feedback entries |
| GET | /feedback/stats | Feedback aggregation |
| GET | /satisfaction | Satisfaction scores |
| GET | /evals/stats | Eval statistics |
| POST | /experiments | Create experiment |
| GET | /experiments | List experiments |
| GET | /experiments/:id | Experiment detail |
| PUT | /experiments/:id | Update experiment status |
| POST | /experiments/:id/results | Compute experiment results |
| GET | /experiments/:id/results | Get experiment results |
All endpoints support ?since=N (ms timestamp) and ?limit=N query params.
9 tables created automatically via ensureObservabilityTables():
agent_trace_spans — individual trace spansagent_trace_summaries — aggregated run summariesagent_feedback — explicit user feedbackagent_satisfaction_scores — computed frustration indexagent_evals — evaluation resultsagent_eval_datasets — golden test datasetsagent_experiments — experiment definitionsagent_experiment_assignments — user → variant assignmentsagent_experiment_results — computed metric resultsAll tables are dialect-agnostic (SQLite + Postgres) and strictly additive.
| File | Purpose |
|---|---|
packages/core/src/observability/types.ts | Shared type definitions |
packages/core/src/observability/store.ts | SQL tables + CRUD |
packages/core/src/observability/traces.ts | Auto-instrumentation |
packages/core/src/observability/posthog-ai.ts | $ai_trace / $ai_span / survey sent emission, content bounding, $ai_error |
packages/core/src/observability/feedback.ts | Feedback + Frustration Index |
packages/core/src/observability/evals.ts | Eval engine (3 layers) |
packages/core/src/observability/experiments.ts | A/B testing system |
packages/core/src/observability/routes.ts | HTTP API handlers |
packages/core/src/client/observability/ObservabilityDashboard.tsx | Admin dashboard |
packages/core/src/client/observability/ThumbsFeedback.tsx | Inline feedback buttons |
packages/core/src/client/observability/useObservability.ts | React Query hooks |
Configure OTLP export in the observability settings:
await putSetting("observability-config", {
enabled: true,
exporters: [
{
type: "otlp",
endpoint: "https://cloud.langfuse.com/api/public/otel",
headers: { Authorization: "Bearer ..." },
},
],
});
The framework emits gen_ai.* semantic convention spans compatible with Langfuse, Datadog, Grafana, New Relic, and any OTel-compatible backend.
Separate from the exporters config above (which ships the in-house traces to an OTLP endpoint), the agent loop can also emit live OpenTelemetry spans for every run, model call, and tool call, so a host that already runs an OTel collector sees agent activity alongside its other distributed traces.
This layer is optional and no-op by default:
@opentelemetry/api is an optional dependency. If it isn't installed, the span helpers degrade to silent no-ops — they never throw into the agent loop.TracerProvider (via @opentelemetry/sdk-node or similar). The framework deliberately does not depend on the heavy SDK/exporter packages and never registers a provider itself — instrumentation is opt-in by the embedding app.The loop emits agent.run (with agent.run_id, agent.thread_id, agent.user_id, agent.model), tool.call (tool.name + status), and llm.call spans, each finished with OK/ERROR status. This is purely additive to the in-house agent_trace_spans / agent_trace_summaries tables. Source: packages/core/src/observability/tracing.ts + traces.ts. See the Observability doc for the full table.
Instrumented agent loops emit server-side tracking events for every run through
track() from @agent-native/core/tracking, so configured PostHog, Agent Native
Analytics, Mixpanel, Amplitude, and webhook providers receive them through the
same best-effort fan-out as other tracking events.
$ai_trace per run, $ai_span per tool call, and $ai_generation
per model call. Every node carries the run id as $ai_trace_id and links
upward through $ai_parent_id so a backend can rebuild the tree.
$ai_session_id is the thread; the browser session is separate and ships as
$session_id, read from X-Agent-Native-Session-Id via
RequestContext.browserSessionId. The agent chat and the action client both
send that header, so a UI action call and the agent's own call during one
visit share a session. setAnalyticsSessionId() from
@agent-native/core/client/analytics pins a custom id and opts it out of the
30-minute idle rotation. Emission lives in posthog-ai.ts.analytics_events with
mirrored query-friendly properties such as run_id, thread_id,
cost_cents_x100, duration_ms, tool_calls, successful_tools,
failed_tools, and status. A content-free tools array includes at most
50 tool names, start offsets, durations, statuses, and coarse error classes;
interrupted calls are finalized as errors, and failed runs still emit with
zero or known usage. tools_truncated marks longer runs while the rollup counts remain complete.
Delegated runs add delegation_protocol, caller_app, a2a_task_id, and
parent_run_id when available. parent_turn_id is separate because one
logical turn may span multiple concrete runs.Constraints that are not visible from the emit site:
onUsage and exposes no per-step hook, so a multi-step
run collapses into a single generation carrying the whole message list.
Per-round-trip latency and intermediate turns are unavailable without a new
seam in ai-sdk-engine.ts / builder-engine.ts. Do not describe the current
output as per-step.$ai_spans_dropped — a truncated run must not read as a complete one.tools array stays because the dashboards
read it; that duplication is deliberate, not cleanup.sentiment. All four feedback types are reported, but a
category follow-up to a thumbs-down is detail about the same vote — counting
it again inflates the metric.Do not build a separate LLM-observability ingestion API unless there is a clear reason the tracking provider registry cannot express the use case. Keep prompt, tool input, and model output content out of tracking by default; use the existing observability config flags for local trace content capture.