| name | otel-instrument |
| description | Add OpenTelemetry observability to applications using auto-instrumentation and optional custom spans/metrics, write a separate instrumentation report, and run verification unless explicitly skipped or blocked. Use when the user types $otel-instrument, asks to "add OTel", "add tracing", "add metrics", "implement observability", "wire up telemetry", "instrument this service", asks to add a specific custom signal like "add a metric to track queue depth", "add a span for payment processing", "track error rate for X", or asks to add signals that make incidents faster to detect or localize, or asks to instrument GenAI/LLM workflows with OpenTelemetry semantic conventions. |
Instrument
Add OpenTelemetry observability to applications using auto-instrumentation and optional custom spans/metrics.
Prefer the application's current runtime shape. If the project already uses Docker/Compose or Kubernetes, fit instrumentation into that path. If the user does not have Docker or does not want Docker, do not introduce containers just for observability; use the host/native runtime patterns.
Before editing application code, read ../references/report-flow-contract.md
and follow the Instrumentation Contract plus Reader-First Report Order.
Workflow
1. Preflight
Before editing anything, ground the plan with repo evidence:
- Confirm the language and framework from actual dependency or source files
- Read
./references/project-runtime-validation.md, inventory the repository's
configured runtime and build/test commands, and select the locally available
project runtime before editing. Do not use the shell's default runtime when
wrappers, toolchain files, manifests, CI, or existing project environments
select another one.
- Confirm the target process from the repo's real start surface:
docker-compose.yml, Kubernetes manifests, package.json scripts, Makefile, Procfile, PM2 configs, Supervisor configs, systemd units, launchd plists, PowerShell scripts, or a plain shell command
- Confirm existing telemetry indicators or record
none found
- Build a provider/exporter topology per signal before choosing SDK or preload
wiring. Find explicit and lazy provider construction, global registration,
exporters, resources, no-op branches, and shutdown paths, then prove
reachability from the selected process. For Python, run
../otel-audit/scripts/scan_python_otel_topology.py <service-root> when that
bundled scanner is available and reconcile its candidates with source.
Existing ownership of any one signal makes this an incremental integration;
it does not prove that the other signals are configured.
- If
.observe/otel.md contains ## Verification Plan, use its
Test Environments and Acceptance Scenarios rows as the initial
implementation and validation plan. Resolve each scenario's Environment
IDs to the reusable runtime and prerequisite rows before running it.
Reconcile them with current config/source; preserve stable IDs and update
stale paths or prerequisites. Keep each Proof Level; do not downgrade a
full runtime row to focused call-site proof.
- If
.observe/otel.md contains the prioritized ## Gaps table, parse every
row by Priority, Area, Required fix, Instrument mode, and
Verification scenarios. Reconcile each row with current source and the
verification contract before editing. If the table is malformed or missing,
stop and regenerate the audit rather than inferring an implementation queue.
- Detect incident-readiness surfaces. Search source and configuration for
user-visible workflows, dependency clients, background jobs, queues/streams,
data freshness, input complexity, synthetic/canary checks, auth/edge paths,
capacity limits, and release/config context. When any are present or when the
user asks for faster incident detection/localization, load
../references/incident-readiness.md.
- When incidents, postmortems, tickets, alerts, or failure examples are supplied,
use Incident-Evidence Mode from
../references/incident-readiness.md: map
each failure mechanism to the owning code or platform surface and classify
the proposed signal as MTTD-improving, localization-only, or still uncovered
before editing.
- Detect GenAI/LLM ownership. Search dependencies, config, and source for
provider clients, model gateways, agent/workflow orchestration, tool/function
dispatch, MCP when present, retrieval/RAG, model/deployment config, fallback,
token usage, prompt/response assembly, AI-derived data jobs, AI-path
synthetic/canary checks, and usage logging. When present, load
../references/genai-readiness.md.
Follow its GenAI Semconv Source Contract before editing: reconcile detected
AI surfaces with official semconv docs when available, record
live-or-snapshot provenance, and build a semconv closure matrix.
- When GenAI incidents, postmortems, tickets, alerts, or failure examples are
part of the request, use GenAI incident-evidence mode from
../references/genai-readiness.md: map each AI pathway failure mechanism to
the owning provider/model gateway, workflow, tool/function execution or
AI-owned session/stream including MCP when present, retrieval/RAG, streaming,
token/context, prompt/response parser, safety/policy, AI-derived data,
model/config rollout, or AI-owned state surface before editing.
- For Java projects, build a trace wiring inventory per
./references/languages/java.md (Preflight section) and classify as auto-only, custom-with-provider, custom-provider-external, or missing before editing.
- Confirm the planned
service.name, service.version, and environment source.
Also record available low-cardinality region, platform, image/artifact,
config, and rollout sources. Prefer existing OTel semantic-convention or
platform resource attribute names, including deployment.environment.name,
cloud.region, cloud.platform, container.image.name, and
container.image.tags. Treat deployment.environment,
deployment.region, deployment.platform, and container.image.tag as
legacy or custom input aliases only, not names to newly emit or reasons to
duplicate the standard attributes.
- Distinguish between application repos and tooling repos such as CLIs, MCP servers, workers, libraries, installers, and build tools. Instrument the executable path users or operators actually run today. Do not invent a web app, Docker path, or entrypoint that is not present.
- If the repo has multiple runnable surfaces, instrument the one the user actually cares about; otherwise ask which one matters
- If the repo is primarily tooling or library code and no runnable surface is obvious, stop and ask instead of inventing an app shell
- Ask one focused clarifying question only if the target process or runtime shape is still ambiguous after checking the repo
Do not proceed until you can state all of these clearly:
- target process
- runtime shape
service.name- environment dimension
- incremental addition vs new scaffold
- selected project runtime, probe command, and affected-module validation
command, or the exact prerequisite that makes validation unavailable
- audit gap closure plan: rows in scope now, rows deferred by mode or explicit
user scope, and the scenario IDs that will prove each in-scope row
- for Java, trace source of truth (see
./references/languages/java.md Preflight section)
- incident-readiness surfaces and the workflow, dependency, input complexity,
freshness, backpressure, synthetic/canary, auth/edge, capacity, and
release/config signals to add or prove when the repo owns those surfaces
- incident-evidence coverage when failure examples are supplied: failure
mechanism, owner, code surface, signal to add or prove, expected MTTD or
localization impact, and remaining external owner
- GenAI workflow surfaces and the GenAI semantic-convention plus service-owned
readiness signals to add, when the repo owns LLM, agent, tool/function, MCP,
retrieval, streaming, model/config, token/context, prompt/response,
safety/policy, AI-derived data, memory/context, evaluation quality, content
capture, framework bridge, app-computed cost, or AI-owned state code
- GenAI incident coverage when GenAI incidents or AI pathway failures are
supplied: failure mechanism, provider/model/tool/retrieval/config/prompt/
AI-derived-data owner, signal to add or prove, expected MTTD/localization
impact, and remaining non-code or dependency owner
State these preflight values together in one explicit progress note before the
first file edit. Do not defer service.name or environment ownership to the
final report.
Fast Path: Targeted Custom Signal
If the user is asking for a specific signal ("add a metric for queue depth",
"track error rate on payments", "add a span for the indexing job") AND the
preflight scan finds OTel SDK already initialized:
- Skip Steps 2-3 (dependencies and auto-instrumentation are already present).
- Go directly to Step 4 (Custom Instrumentation) with the user's request as context.
- Add only the requested signal — do not re-scaffold or re-wire existing setup.
- Proceed to Step 5 (project-runtime validation gate).
If the preflight scan finds no OTel SDK, tell the user auto-instrumentation
needs to be set up first and continue with the full workflow (Steps 2-3).
Audit-Driven Incident Readiness
When the source audit contains ### Incident Readiness under
## Current Instrumentation, treat each partial or missing readiness row and
its matching prioritized ## Gaps row as one implementation contract. The
readiness row names the surface and detection/localization impact; the gap row
names the complete required fix and instrument mode; its acceptance scenarios
name the code path, expected telemetry, proof level, and acceptance criteria.
Do not create a second gap ledger or silently synthesize missing fields. If an
older audit lacks the current prioritized gaps or verification plan, regenerate
the audit before claiming one-to-one closure.
If the user broadly asks to improve incident readiness or MTTD, address every
safe app-owned incident gap allowed by the audit modes: all required /
default rows and all recommended / fix all rows. Do not choose one
representative gap unless the user explicitly narrows scope. Never silently
implement a manual decision row; record the exact external owner, unsafe
boundary, missing source, or choice required.
Use ../references/incident-readiness.md to turn each owned gap into concrete,
low-cardinality signals. In particular, add or prove the applicable surfaces:
- API/workflow outcome, errors, latency, and detector-ready request/job counts;
- dependency timeout, retry, rate-limit, circuit-breaker, endpoint/target
health, availability, and operation outcome;
- input complexity, freshness/age, queue depth/lag/oldest age, dropped or
rejected work, worker/pool saturation, and scheduled-job last success;
- stream/long-lived connection open, auth, active count, duration, close reason,
timeout, cancellation, and send/write failure;
- auth/identity/token/secret/certificate/edge failure reason, expiry/rotation,
route/config mismatch, and synthetic/canary result when owned;
- CPU, memory, disk, inflight/concurrency, desired-vs-healthy,
startup/readiness/healthcheck, target-health, and autoscaling saturation when
observable by the app or its checked-in runtime configuration; and
- low-cardinality service/artifact/config/schema/feature-flag/rollout context,
expected-vs-running state, compatibility failure, and rollout outcome.
Do not treat a bare time-since-last-update or time-since-last-success gauge as
detector-ready staleness when healthy idle periods are possible. Require a
source-backed expected cadence, pending/backlogged work, or accepted input that
should have produced the update. Without that evidence, classify the age gauge
as context or localization-only and use backlog, queue delay, or missed
schedule as the MTTD-improving detector input.
For repositories with both web/API and background-worker processes, apply the
Multi-Process Web And Worker Services contract in
../references/incident-readiness.md. Each process needs a distinct,
operator-overridable service.name default. Initialize its provider and
framework instrumentations only from that process's actual entrypoint or
startup hook; importing a worker/task module from the API must not configure
worker telemetry. Instrument enqueue and task success/failure at their owning
call sites, and explicitly record worker exception status before rethrowing.
Framework hooks or assumed auto-instrumentation are not a substitute for an
app-owned failure path when the readiness gap requires one.
Focused incident-readiness tests must execute the success and failure call
sites and assert emitted telemetry through an in-memory exporter or equivalent
app-code seam. AST, grep, source-string, or compile-only checks are not telemetry
proof. If dependencies cannot be restored or imported, keep the executable
tests, report the exact blocker, and mark dynamic verification Blocked or
Not proven; do not replace the tests with static assertions.
For Go changes involving goroutines, channels, queues, asynchronous persistence
or indexing, eviction, or observable callbacks, run go test -race for every
changed package. If that cannot run, record the exact toolchain/platform blocker;
a normal go test pass does not satisfy this concurrency gate.
For incident evidence, target the failure mechanism rather than its endpoint
symptom. Mark a signal MTTD-improving only when it can support a detector
before or at first customer impact; mark it localization-only when it mainly
narrows an already-detected fault. Endpoint RED metrics alone do not close an
auth handshake, secret expiry, stale output, rollout skew, dependency target
loss, stream lifecycle, or pool-saturation gap.
Extend the internal Audit-Driven Gap Closure matrix with the readiness surface,
required signals, MTTD/localization classification, implemented or proven
signals, tests, remaining signals, and owner. A row cannot be Working while
any required signal is absent, merely listed as a follow-up, or supported only
by an unexecuted test. If no app-owned candidate can be patched accurately and
safely, add no placeholder instrument; owner-map the exact prerequisite and
keep the row Deferred, Not configured, or Not proven as appropriate.
Audit-Driven GenAI Readiness
GenAI Readiness Contract
If .observe/otel.md contains ## GenAI Readiness, use that table as the
source of truth. The audit output is a contract, not background context.
Parse each row by human-readable surface plus required_signals,
owner/source_files, and acceptance_criteria. Use the surface name as the
human-facing identifier throughout implementation and reporting.
Reconcile every GenAI audit gap to a required instrumentation result:
| Audit Gap | Required Instrumentation Result |
|---|
| App-owned + patchable | Code added + tests |
| App-owned but unsafe/too large | Explicitly split into named follow-up batch |
| Provider/platform-owned | Owner mapped with exact missing source |
| Already covered | Proven with source path and signal name |
For each surface row, produce and maintain a closure matrix:
surface -> required_signals -> implemented_signals -> tests -> remaining_signals -> status. The final gate is strict: the instrumentation
pass cannot say covered, fixed, closed, or complete unless every
required GenAI signal is either implemented with tests, proven existing with
source path and signal name, or explicitly owner-mapped with the exact missing
source. Optimize for honesty over broad progress. Partial closure is acceptable;
silent partial closure is the bug.
For every GenAI instrumentation run, include a concise closure summary in the
final response. It must name remaining GenAI signals, or say Remaining signals: none
only when the closure matrix has no partial rows. Do not use unqualified
phrases such as expected coverage, covered, or complete for a GenAI surface
unless the related row is fully closed by the matrix.
When the source audit declares **GenAI ownership detected:** Yes, write
## GenAI Readiness Closure in .observe/otel-instrumentation.md after
## Audit Gap Closure. Copy every Surface and its complete Required Signals
cell from the audit readiness table, then record Implemented / proven,
Tests, Remaining signals, and Result. Use one row per audit surface and
do not merge or omit partial, deferred, or owner-mapped surfaces. Use Working,
Partial, Not working, Not proven, Not configured, Deferred, or
Owner-mapped. Working requires Remaining signals to be exactly None;
all other results must name the remaining signal, blocker, or external owner.
Omit the section only when the source audit explicitly declares No. If the
ownership declaration and readiness table disagree, regenerate the source
audit before instrumentation.
When no source audit exists, do not create ## GenAI Readiness Closure or
invent closure rows from the implementation pass. Record source-derived GenAI
readiness observations and remaining signals under ## Remaining Gaps, and
state that a source audit is required before one-to-one readiness closure can
be claimed.
If .observe/otel.md contains ## GenAI Readiness rows with partial or
missing status and the user asked to instrument, treat those rows as an
approved request for custom GenAI readiness instrumentation. Do not stop after
auto-instrumentation and do not ask the Step 4 custom-instrumentation question
for GenAI gaps that the repo clearly owns.
Audit-Driven Gap Closure
Treat the prioritized ## Gaps table as the implementation queue, not report
background:
- A normal
$otel-instrument run addresses every safe app-owned required
row whose instrument mode is default.
- A request to fix all gaps also addresses safe app-owned
recommended rows
whose instrument mode is fix all.
manual decision rows are never silently implemented. Record the owner,
prerequisite, or explicit choice needed.- An explicit narrower user scope takes precedence. Keep all untouched rows in
Audit Gap Closure and Remaining Gaps so scope reduction is visible.
- A row may require only verification rather than code. Run the mapped
scenarios and do not invent a source change.
- Reconcile GenAI gap rows with
## GenAI Readiness; the readiness row remains
the detailed required-signal contract and the prioritized gap remains the
user-facing work item.
Build an internal closure matrix before editing:
area -> priority -> required fix -> instrument mode -> planned action -> verification scenarios. Update it after validation and verification. Do not
mark a row Working merely because code changed or a shared helper test
passed. Closure requires the source change or proven existing implementation,
the applicable project-runtime validation gate, and $otel-verify proof at the
audit scenario's proof level. In particular, execute every named route, span
call site, metric path, log pipeline, and duplicate-prevention scenario the row
references.
Implementation Report Contract
For every instrumentation run, create or update
.observe/otel-instrumentation.md. Do not update .observe/otel.md as a
change log. Treat .observe/otel.md as the source audit input only unless the
user explicitly asks for a fresh audit.
When no audit report exists, still write .observe/otel-instrumentation.md
with service/runtime evidence, scoped implementation changes, validation gates,
verification results or handoff, and explicit remaining gaps. Do not fabricate
a full audit.
The instrumentation report must be reader-first:
# OTel Instrumentation Report: <service>
**Result:** Pass | Partial | Fail | Blocked
**Source audit:** `.observe/otel.md` | not found
**Verification report:** `.observe/otel-verify.md` | not run | blocked
**Detector report:** `.observe/detectors.md` | not requested | blocked
## Executive Summary
## Flow
## Files Changed
## Signals Changed
## Audit Gap Closure
<!-- Include the next section only for a GenAI source audit. -->
## GenAI Readiness Closure
## Validation Gates
## Verification Handoff / Results
## Detector Handoff / Results
## Remaining Gaps
## Next Steps
Signals Changed is the implementation-change inventory. Include a
signal-level table:
| Signal type | Added | Modified | Removed | Evidence | Verification status |
|---|
| Traces/spans | exact span names or None | exact changes or None | exact removals or None | source paths + tests/harnesses | verified/partial/not run/blocked |
| Metrics | exact metric names or None | exact changes or None | exact removals or None | source paths + tests/harnesses | verified/partial/not run/blocked |
| Logs/events | bridge/event names or None | exact changes or None | exact removals or None | source paths + tests/harnesses | verified/partial/not run/blocked |
| Runtime/config | service/exporter/env/startup settings or None | exact changes or None | exact removals or None | startup/config paths | verified/partial/not run/blocked |
| Dependencies | OTel packages or None | version/package changes or None | removed packages or None | manifest/lockfile paths | verified/partial/not run/blocked |
For incident-readiness work, add this nested table inside ## Signals Changed
even when no source audit exists:
### Incident Readiness Signal Roles
| Surface | Exact signal | Role | Detector use / reason | Proof | Remaining owner / prerequisite |
|---|---|---|---|---|---|
Use exactly MTTD-improving, localization-only,
provider/platform-owned, or uncovered in Role. Write one row per exact
added or proven signal; do not group multiple metric names. Use None for
Exact signal only when owner-mapping an unavailable prerequisite, and name
that owner or prerequisite in the final column. The table is a signal-role
inventory, not another gap ledger; reconcile audited surfaces through the
existing Audit Gap Closure rows.
Do not claim a removal unless the previous report or git diff proves the signal
or config existed and the current source proves it was removed. Use None for
empty cells. The final response must summarize Signals Changed by signal
type, distinguish added, modified, removed, and unchanged signals, and point to
.observe/otel-instrumentation.md.
Audit Gap Closure is the reader-facing reconciliation with the source audit:
| Priority | Gap | What changed | Tested | Result | Evidence / reason |
|---|
| required | exact audit Area value | concrete code/config change or No code change | scenario IDs and test mode | Working / Not working / Not proven / Not configured / Deferred | direct evidence or exact blocker |
Use one row per prioritized audit gap. Keep Not working distinct from Not proven: the former requires an executed failed check, while the latter means
the required scenario did not run or lacked a prerequisite. Use Not configured when requested implementation is absent. Use Deferred only for
an explicit scope choice, owner, prerequisite, or manual decision row. When
there is no source audit, write No source audit gap table was available.
For a GenAI audit, GenAI Readiness Closure is the detailed signal-level
reconciliation and Audit Gap Closure remains the prioritized user-facing work
queue. Do not treat one as a substitute for the other.
Derive the report-level **Result:** from both closure tables. Do not use
Pass when any audit-gap row is Not working, Not proven, or
Not configured, or when any GenAI readiness row is Partial, Not working,
Not proven, or Not configured. Use Partial when meaningful proof passed
but any such row remains. Deferred and Owner-mapped may coexist with Pass
only when the exact external owner or explicit scope decision is recorded.
When a source audit exists, run the dependency-free closure validator bundled
with this skill after writing the instrumentation report:
python3 scripts/validate_gap_closure.py \
.observe/otel.md .observe/otel-instrumentation.md
Resolve scripts/validate_gap_closure.py relative to this skill directory. If
validation fails, repair the report or expose the missing audit row before
finalizing.
Also maintain ## Verification Handoff / Results using the schema in
./references/project-runtime-validation.md. Record the selected runtime,
exact local-safe commands and outcomes, changed source-to-scenario mappings,
the $otel-verify result/report path when run, and any blocked prerequisites.
This section is not proof of emitted telemetry unless a test, harness, or
collector actually observed it.
Use the current audit's prioritized ## Gaps and ## Verification Plan as the
handoff contract. Do not copy unrelated report sections into the audit or
instrumentation report.
When the user asks broadly to apply GenAI readiness skills, improve GenAI MTTD,
or fix found GenAI gaps, treat the scope as all discovered app-owned GenAI
gaps. Do not select one representative or highest-value gap unless the user
explicitly narrows the scope to that gap.
Close code-evidenced AI pathway surfaces from ../references/genai-readiness.md:
provider/model gateway, agent/workflow orchestration, tool/function execution
or AI-owned session/stream lifecycle including MCP when present, retrieval/RAG,
streaming response lifecycle, token/context pressure, safety/policy outcome,
prompt/response assembly, AI-derived data freshness, memory/context operations,
evaluation quality, content governance, framework bridge configuration,
app-computed cost, model/config rollout, and AI-owned cache/session state.
For evaluation quality surfaces, code evidence such as evaluator classes,
scoring functions, LLM-as-judge calls, feedback processors, EvalScore models,
faithfulness/similarity/expectation metrics, pass/fail labels, or quality
report exporters is enough to require eval instrumentation.
Metrics-only coverage does not satisfy selected-trace eval visibility. Add or prove
gen_ai.evaluation.result on the relevant workflow/evaluation span with
gen_ai.evaluation.name, gen_ai.evaluation.score.value when numeric scores
exist, gen_ai.evaluation.score.label when labels/verdicts exist, and safe
parent linkage. Also add or prove detector-ready score distribution, pass/fail
or violation count, sample count/rate, evaluator duration/error/no-data, and
freshness by low-cardinality workflow/model/evaluation name. If the service
only emits eval counters, histograms, logs, or report files, keep the evaluation quality surface partial
and name the missing span-level eval event and remaining detector metrics.
For MCP, JSON-RPC, and tool dispatch, normalize request metadata before adding
attributes or metric dimensions. Never record JSON-RPC request IDs, raw request
IDs, session IDs, trace IDs, user/account/tenant IDs, raw tool arguments, raw
payloads, prompts, completions, or retrieved content as metric dimensions.
Use stable method names only from an allowlist or from known route/tool registration; otherwise record a low-cardinality method family such as
known_tool, unknown_method, invalid_request, or unsupported_method.
Prefer detector-ready metrics and span attributes for outcome, duration,
timeout, retry, rate-limit, fallback, active AI-owned sessions/streams, close
reason family, send/write failure, freshness, empty/low-confidence retrieval,
token budget, prompt build failure, response parse failure, AI-derived data
freshness, prompt/tool schema version, model/config readiness,
model/config compatibility, expected-vs-running model/config state, truncation,
rejection, LLM-call fanout, tool-call fanout, evaluation score/outcome,
memory hit/miss or staleness, content capture mode/redaction, app-owned cost
source, and version dimensions when the service can observe them accurately.
Use OTel GenAI semconv names when possible: gen_ai.evaluation.result,
gen_ai.evaluation.name, gen_ai.evaluation.score.value,
gen_ai.evaluation.score.label, safe gen_ai.evaluation.explanation, memory
operation names such as search_memory, create_memory, update_memory,
upsert_memory, and delete_memory, and opt-in content attributes such as
gen_ai.input.messages, gen_ai.output.messages,
gen_ai.system_instructions, gen_ai.retrieval.documents,
gen_ai.retrieval.query.text, gen_ai.tool.definitions, and
gen_ai.tool.call.arguments. Treat framework bridges as covered only when
OTel-compatible GenAI semconv output and privacy settings are proven. Treat cost
as custom app-owned instrumentation only when the app owns an accurate pricing map; otherwise owner-map the billing or provider source. Generic non-AI runtime,
platform, or job surfaces are out of scope for this GenAI section unless source
evidence shows they carry or block the AI pathway.
If GenAI incident evidence depends on missed, flapping, auto-resolved, or no-data alerts, record detector reliability evidence as a $splunk-configure
handoff instead of adding app metrics for alert lifecycle behavior.
For AI-owned streaming generators, WebSocket/SSE handlers, callback bridges,
or protocol send loops, do not call stream lifecycle coverage complete without
a send/write failure signal. Add a counter, span event, or low-cardinality
outcome attribute for send/write failure when app code can observe it; otherwise
owner-map the missing source explicitly to the framework/server/platform.
For token/context pressure gaps, gen_ai.client.token.usage plus a
context-window usage gauge does not close a broader token-pressure gap unless
the audit contract only requires those two signals. If required_signals
include context budget percent, truncation rate, token-limit errors,
prompt/tool schema size, LLM call count per turn, or tool call count per turn,
mark the row partial until each signal is implemented, proven, or owner-mapped.
When the user broadly asks for GenAI readiness without a preexisting audit
contract, treat token/context pressure as requiring the same concrete signal
check: token usage, context budget percent, truncation, token-limit errors,
prompt/tool schema size or safe proxy, LLM-call count, and tool-call count when
the app can observe them.
Use this exact style when only token and context-window usage were added:
Partial: token usage and context window added; truncation, token-limit error, prompt/tool schema size, and LLM-call fanout remain missing.
If prompt/tool schema size cannot be measured safely, add a low-cardinality
detector-ready proxy metric such as schema JSON length bucket, schema field count,
prompt template length bucket, or tool count when the app can observe it.
Span attributes like prompt template version, schema version, or one-off schema
metadata help traces but do not close prompt/tool schema size pressure by themselves. If no safe metric or detector-ready existing signal exists, the
closure matrix and final response must keep prompt/tool schema size in
remaining_signals with the owner and missing source. Do the same for LLM-call
and tool-call fanout: implement per-workflow counts when observable, prove an
existing signal, or keep them explicitly partial.
Final summaries, PR descriptions, and audit updates must not omit residual
truncation, token-limit, prompt/tool schema size, LLM-call fanout, or
tool-call fanout gaps when they were in the audit contract or when broad GenAI
readiness instrumentation discovered the token/context surface.
For tool/function execution, GenAI spans alone do not satisfy detector-ready
tool coverage. When app code observes tool execution, add or prove a
tool-specific duration histogram and tool error/timeout counter using a stable
tool name and low-cardinality failure class. If only spans are safe, keep tool
latency/error metrics in remaining_signals and do not claim detector-ready
tool coverage.
Do not call GenAI instrumentation complete when an app-owned provider/model,
workflow, tool/function execution or AI-owned session/stream including MCP when
present, retrieval, streaming, token/context, safety/policy, prompt/response,
AI-derived data, memory/context, evaluation quality, content governance,
framework bridge, app-computed cost, model/config rollout, or AI-owned
cache/session surface remains only listed as a follow-up, unless the user
explicitly narrowed scope.
2. Dependencies
Add the OpenTelemetry SDK and auto-instrumentation packages for the detected language. Load the appropriate reference file:
| Language | Reference | Key packages |
|---|
| Python | ./references/languages/python.md | opentelemetry-api, opentelemetry-sdk, opentelemetry-exporter-otlp, framework instrumentation packages |
| Node.js | ./references/languages/node.md | @opentelemetry/sdk-node, @opentelemetry/instrumentation-http, @opentelemetry/exporter-metrics-otlp-http, @opentelemetry/sdk-metrics, detected framework instrumentation packages |
| Java | ./references/languages/java.md | OTel Java agent (javaagent JAR) |
| Go | ./references/languages/go.md | go.opentelemetry.io/otel, go.opentelemetry.io/contrib |
3. Instrument
Apply auto-instrumentation first, then add manual spans for key business operations. Read the language-specific reference for exact patterns.
Critical for APM error tracking:
- Set
otel.status_code to ERROR on failures -- this is how APM backends identify errors
- For HTTP server spans, 5xx responses set ERROR automatically per OTel semantic conventions
- For custom spans wrapping business logic, explicitly set error status on exceptions
- Reuse the app's current startup entrypoint instead of replacing it with a new Docker-only path
- For Python, Node.js, and Java, prefer preload or agent wrappers plus env vars over large code refactors when auto-instrumentation already covers the framework
- For host/native runtimes, default OTLP endpoints to loopback (
http://localhost:4318) unless the existing platform already provides a collector address
- For Python web services, do not satisfy implementation by only changing a Makefile, Docker command, or shell wrapper. Add an explicit setup module such as
otel_setup.py and wire the app entry point to call it before framework instrumentation is activated.
- For Java/Spring Boot, prefer the OpenTelemetry Java agent. The final response must state the service-name setting (
OTEL_SERVICE_NAME or otel.service.name), OTLP endpoint setting (OTEL_EXPORTER_OTLP_ENDPOINT or otel.exporter.otlp.endpoint), and that the agent provides HTTP server spans plus request duration metrics.
Implementation Rules
-
Use only official OpenTelemetry packages (go.opentelemetry.io/otel, go.opentelemetry.io/contrib, @opentelemetry/*, opentelemetry-*). Do not use community or third-party OTel wrappers. The only exceptions are library-maintained integrations where no official package exists (e.g. go-redis/redisotel, XSAM/otelsql).
-
Do not initialize the SDK more than once per process.
-
Find any existing OTel setup before adding new code. Extend it. Treat lazy
provider helpers and providers initialized on first instrument creation as
existing setup even when the startup wrapper itself contains no OTel call.
-
Keep one global provider per signal. When adding traces/logs around an
existing custom metrics provider, consolidate resource/exporter ownership or
adapt the helper to the shared provider; never let auto-instrumentation and
app code race to call set_meter_provider.
-
Preserve operator resource values. Merge app defaults only for absent keys;
do not overwrite OTEL_SERVICE_NAME, deployment.environment,
deployment.environment.name, or service.version from
OTEL_RESOURCE_ATTRIBUTES. Add a focused resource-merge test and assert the
effective resource in live OTLP evidence.
-
Resolve the exporter per signal as an endpoint/protocol/path tuple. Pair gRPC
with the gRPC receiver (commonly 4317) and http/protobuf with an HTTP
signal path such as 4318/v1/metrics. Do not assume one generic endpoint
configures every exporter. A successful trace export does not prove metrics
or logs; exercise each configured signal and repair protocol errors before
reporting it as working.
-
For Java trace wiring, DI binding, and provider rules, follow ./references/languages/java.md (Implementation Rules section).
-
Place OTel initialization code in a separate file.
-
Minimize changes to existing code. Do not move functions between files.
-
Do not create spans for trivial helpers. Only span real diagnostic boundaries.
-
Set span status to ERROR and call recordException on failed operations.
-
When a framework-specific auto-instrumentation package only provides spans (not HTTP server metrics), wrap the outermost handler with otelhttp.NewHandler (Go) or equivalent to ensure http.server.request.duration and http.server.active_requests are emitted. Consult the Framework Selection Guide in the language reference for the correct wrapping pattern.
-
HTTP server instrumentation must produce request-duration metrics as well as
spans. Prefer the current stable metric http.server.request.duration. When
the installed SDK requires a semantic-convention stability opt-in, set it in
the launch environment before importing or constructing the instrumentor.
Accept http.server.duration only when the installed SDK truly emits the
alternate name and record that exact runtime evidence; never claim both names.
-
For local, Docker, and eval-style runtime checks, configure metric export to flush quickly. When constructing a metric reader manually, use the language equivalent of OTEL_METRIC_EXPORT_INTERVAL with a safe local default of 1000 ms and OTEL_METRIC_EXPORT_TIMEOUT with a safe local default of 500 ms instead of relying on SDK defaults.
-
Strictly adhere to OTel semantic conventions for span and metric naming and attributes for domains where such semantic conventions are defined.
-
For domains where OTel semantic conventions exist, use semantic-convention
names and attributes. Start with required spans, metrics, and attributes; add
recommended optional signals only when an approved readiness or verification
requirement depends on them, the service can observe the value accurately,
and privacy/cardinality rules permit it. Do not invent custom spans, metrics,
or attributes where a semantic-convention signal satisfies the requirement.
-
Before adding any custom counter or histogram for an outcome that occurs
inside a call already covered by an auto-instrumented RED metric, check
whether that outcome can instead be recorded as an attribute on the existing
metric via the language's per-call metric-attribute hook (see
#### Language-Specific Musts below), rather than as a new standalone
metric. Each convention below defines its own error/status attributes for
exactly this purpose:
- HTTP server/client metrics:
http.server.request.duration / http.client.request.duration carry
error.type and http.response.status_code.
- RPC server/client metrics:
rpc.server.call.duration / rpc.client.call.duration carry error.type
and rpc.response.status_code. Only fall back to the legacy
rpc.server.duration / rpc.client.duration names when runtime evidence
shows the installed SDK still emits those instead.
- Database client metrics:
db.client.operation.duration carries error.type and
db.response.status_code.
- Messaging metrics:
messaging.client.operation.duration and messaging.process.duration
carry error.type when the operation fails.
Prefer the attribute over a dedicated custom metric whenever the outcome
can be faithfully represented on the RED metric for that call and the
language has a way to set it there: either auto-instrumentation already
emits a relevant error.type/*.response.status_code attribute that
actually distinguishes this outcome, or the language has a supported
per-call metric-attribute hook that can set a standard or custom
attribute on the metric (see #### Language-Specific Musts below -- for
example Go's otelhttp.Labeler, which can attach a custom attribute like
outcome.reason even when the outcome is not expressible via the standard
status/error attributes alone). A dedicated custom metric is correct
whenever that's not the case: a queue-depth gauge or a background job
outcome with no inbound request has no call to attach to; or the outcome
is not expressible via the standard status/error attributes alone (for
example a business outcome that occurs once per request but is not
distinguishable from the standard attributes -- a logical failure returned
as HTTP 200, or two distinct failure causes sharing one HTTP status) and
the language's auto-instrumentation also has no per-call metric-attribute
hook to carry a custom attribute instead (Python's ASGI/WSGI hooks and
Node's @opentelemetry/instrumentation-http only add span attributes, not
metric attributes). In each of those cases, add the dedicated metric
instead of suppressing the only detector-ready signal for the outcome.
-
For incident-readiness work, follow
../references/incident-readiness.md. Instrument only source-evidenced
workflow, dependency, input-complexity, freshness, backpressure,
synthetic/canary, auth/edge, capacity, health/readiness, and release/config
surfaces. Prefer semantic-convention HTTP, RPC, database, messaging, process,
and runtime signals; add custom outcome, lag, freshness, queue, retry,
timeout, rate-limit, endpoint/target-health, drop/reject, circuit-breaker,
saturation, desired-vs-healthy, startup/readiness/healthcheck, compatibility,
and rollout signals only when the service owns and can observe them
accurately.
-
For GenAI/LLM code, follow ../references/genai-readiness.md: create
baseline distributed tracing plus OTel GenAI spans for inference,
invoke_agent, invoke_workflow, plan, execute_tool, retrieval, and
memory operations where code evidence exists; emit
gen_ai.client.operation.duration and
gen_ai.client.token.usage when the data is available; use stable tool names;
add low-cardinality error.type; and avoid raw prompt, completion, retrieved
content, memory record, tool argument, evaluation explanation, user, tenant,
session, task, request, trace, or raw URL values in metric dimensions.
-
Prove every custom metric's exact name, unit, instrument type, and complete
emitted dimension sets. Lifecycle-specific counters must retain their
specific error class; generic terminal errors must not overwrite earlier
token-limit, truncation, timeout, provider, or tool classifications. Do not
attach transient terminal outcome/error dimensions to intermediate gauges or
size measurements unless the metric contract explicitly requires them.
-
For GenAI/LLM code, apply the Single-Source GenAI Span Contract before
adding manual spans. Inventory framework/vendor bridges, provider SDK hooks,
callbacks, middleware, auto-instrumentors, and existing app spans that can
emit GenAI telemetry. Choose one canonical GenAI span source per logical
operation: workflow, agent, model call, tool call, retrieval, memory operation, or
evaluation result. If a framework/vendor bridge already emits correct GenAI
semconv spans with lifecycle, privacy, model/tool attributes, and parent
context, keep that bridge and add only missing workflow/agent context,
aggregates, metrics, or owner mappings; do not create duplicate app-owned
chat or execute_tool spans for those same operations. If app-owned spans
are the canonical source, emit the complete app-owned span set and disable,
opt out of, or suppress overlapping framework/vendor GenAI instrumentation
using the discovered runtime mechanism, such as instrumentor names, bridge
settings, callback configuration, or provider-hook opt-out flags. Do not
hard-code this decision to one framework. Keep HTTP/database/runtime
auto-instrumentation when it does not create duplicate GenAI nodes.
When the process uses preload, agent, opentelemetry-instrument,
NODE_OPTIONS --require, or another auto-instrumentation bootstrap, apply
the suppression in the launch environment before the bootstrap runs. Update
the actual startup surfaces the repo uses, such as Makefile targets, service
runner scripts, Docker or Helm env, VS Code launch configs, procfiles,
systemd units, shell env generators, or generated env scripts. App module
code that mutates environment variables after import is only defense in depth
and is not sufficient proof because framework hooks may already be
registered.
Verification or static proof must show one GenAI node per logical operation,
no wrapper-only spans counted as tools, expected LLM/tool counts, stable
model/tool names, and workflow/agent parent shape such as
invoke_workflow -> invoke_agent -> chat/execute_tool. Required proof should
name stable model/tool names explicitly.
-
Preserve existing application stable business workflow identity when setting
gen_ai.workflow.name and workflow span names. Prefer constants, function or
handler names, workflow registrations, telemetry event names, docs, or prior
trace names. Do not derive GenAI workflow names from HTTP routes, request
resources, session/storage tables, or transport labels. If source evidence
shows a workflow is named assistant_v3_turn, keep assistant_v3_turn;
never rename it to assistant_v3_session_turn, POST /v2/assistant/sessions,
or another route/session-derived value. If the app has no stable workflow
identity, keep the HTTP route as the HTTP span and mark GenAI workflow
coverage partial instead of inventing a low-cardinality workflow name.
Do not invent names from HTTP routes or session-derived labels.
-
Preserve existing application stable agent identity when setting
gen_ai.agent.name and agent span names. Prefer framework agent names, agent
factory names, class names, registration names, callback owner names, docs, or
prior trace names. If source evidence shows a DeepAgents-backed agent, use the
discovered stable agent identity such as deepagents; never rename it to
assistant_v3_agent, assistant, agent, or another generic service-derived
wrapper name. If the app has no stable agent identity, keep agent coverage
partial instead of inventing one.
-
For app-owned LLM/model calls, apply the LLM Inference Lifecycle Contract.
Do not satisfy inference coverage with workflow-level token accounting,
final usage events, token usage, or model names only on the workflow span.
Add or prove a real model-call lifecycle span at the provider request
boundary: start before the call or stream starts, end after the
response/terminal stream event, and end with error status for exceptions,
cancellations, provider timeouts, or stream-close failures. In LangChain,
LangGraph, DeepAgents, callback, or event-stream based systems, hook
on_chat_model_start, on_chat_model_end, and on_chat_model_error or the
equivalent lifecycle callbacks. In direct SDK/model-gateway code, wrap the
provider call or streaming generator. The span must carry
gen_ai.operation.name such as chat, generate_content, or
text_completion, gen_ai.provider.name, gen_ai.request.model when known,
gen_ai.response.model when known, and token usage on that inference span
when provider usage is available.
-
Preserve the owning workflow/agent context for event-derived GenAI spans. In
callback, stream, LangChain, LangGraph, or DeepAgents integrations, capture
the workflow/agent context and use it when starting chat/model and tool
spans. Do not let callback-created chat or execute_tool spans attach to a
generic HTTP root span as siblings of the workflow. A representative trace
should prove a shape such as workflow -> chat, workflow -> execute_tool,
and follow-up workflow -> chat or agent -> chat edges.
-
Capture that workflow/agent context before opening long-lived helper/setup
spans such as memory store, checkpointer, database session, stream-writer,
resource setup, or lifecycle wrappers. These helper spans must not become the
parent for model/tool lifecycle spans. Event-derived chat and execute_tool
spans must use the captured workflow/agent context, not whichever helper span
happens to be current when a callback is translated. Store the workflow/agent
span object or context explicitly so stream cleanup writes aggregates to that
owner even if the current span has changed.
Use this rule for memory store, checkpointer, database session, stream-writer,
or resource setup paths: helper spans must not become the parent; capture the
workflow/agent context before opening helper spans, start event-derived chat
and execute_tool spans with that captured context, and write aggregate
counters to the workflow span, not to whichever current span is active.
-
In async generator, SSE, WebSocket, ping-loop, timeout-wrapper, or task handoff
paths that advance a stream with create_task, wait, anext, or equivalent
scheduling, do not keep an OpenTelemetry current-span context manager open
across yield/task boundaries. Start the workflow/agent span with an explicit
parent context, store a workflow span/context handle, pass that handle into the
callback/event translator, and end the workflow span manually after stream
cleanup.
-
When the captured workflow/agent context must travel through a request, turn
input, event payload, callback state, or config object that may be
immutable/frozen or owned by a framework, do not mutate that carrier in place.
Treat a carrier as immutable/frozen when source evidence shows frozen or
readonly declarations, record/value types, no mutation API, framework request
immutability patterns, or existing code constructs new copies instead of
mutating.
Use the app's idiomatic copy/replacement API: Python dataclasses.replace,
attrs.evolve, pydantic model_copy(update=...) or v1 copy(update=...);
Java records, builders, or copy constructors; TypeScript object spread,
explicit Readonly<T> replacements, or structuredClone only for plain-data
carriers and never for live OTel Context or Span handles; Go value copies
with explicit field replacement; or the framework's request clone/with-context
API. If no safe copy path exists, use a separate invocation-scoped sidecar
context: a local object, context variable, request-scoped map, or callback
state keyed to the invocation lifecycle and cleared after cleanup. Do not key
sidecar context by raw user, tenant, session, request, or trace IDs. Add a
focused test, or explicit static proof when the repo has no test harness, that
proves the parent context is passed downstream and the original immutable
input remains unchanged; for Python, guard against FrozenInstanceError
regressions when a frozen dataclass or model is present.
-
Put aggregate selected-trace GenAI attributes on the workflow span or
most specific owning GenAI span, not on a generic HTTP root span or generic server span.
This includes gen_ai.usage.input_tokens,
gen_ai.usage.output_tokens, gen_ai.usage.total_tokens,
assistant.llm.calls, and assistant.tool.calls. The HTTP root can remain
the server entrypoint, but it should not become the evidence for a GenAI flow
card unless it is explicitly instrumented as the workflow span.
-
If runtime telemetry verification is unavailable, perform a static
instrumentation proof before claiming GenAI trace coverage: identify the
model-call source file, the lifecycle hook or client wrapper, the created
inference span name, required gen_ai.* attributes, parent-context handoff,
aggregate-attribute placement, end/error path, and a focused test or compile
check. If the proof shows only workflow-level usage attributes and no
inference span, or chat/tool spans that are siblings of the workflow under a
generic HTTP root span, keep the surface partial in remaining_signals.
-
For local span-first trace explorers such as Obstudio, metrics alone are not enough
for a selected-trace summary. When provider usage, model, tool,
memory, evaluation, or fanout data is available, also set safe span attributes
on the most specific GenAI span and aggregate to the workflow span when
useful: gen_ai.usage.input_tokens, gen_ai.usage.output_tokens,
gen_ai.usage.total_tokens, gen_ai.request.model,
gen_ai.response.model, service-owned LLM/tool call counts, stable
gen_ai.tool.name, memory operation names, evaluation names/labels, and
low-cardinality error.type.
-
For assistant, agent, or streaming workflows, instrument first non-ping event
or first chunk latency, timeout, cancellation, disconnect, close reason
family, and send/write failure. Normalize timeout classes such as
first_event_timeout instead of relying only on framework exception class
names, while still recording the exception on the span.
-
For custom attribute names use {domain}.{noun}.{adjective} format.
-
Span names must be low-cardinality (no IDs, no variable path segments).
-
Metric attributes must avoid high cardinality.
-
Preserve existing env-var patterns for telemetry config instead of hardcoding endpoints.
-
If the app is a library, provide an opt-in setup path rather than forcing SDK initialization on import.
-
Keep the codebase idiomatic. Match the repo's dependency manager, config style, and lifecycle patterns.
-
Obtain OTel Tracer, Meter once during startup and reuse it. Do not call getTracer or getMeter in hot paths.
-
Create metric instruments once during startup and reuse them. Do not create instruments in hot paths.
-
Metric instruments must be created with appropriate unit and description parameters.
Log Export Scope
- Classify application logs as
correlation-only, otlp, or not requested
during preflight.
- Do not treat MDC/trace-context fields in stdout as OTLP log export.
- Do not silently add an OTLP log bridge when the user or audit contract does
not require explorer-visible logs; log export can affect cost, privacy, and
duplicate ingestion.
- When OTLP logs are required, configure the official bridge/exporter for the
detected logging stack and add proof for body/category, severity,
trace/span correlation, redaction, resource identity, and OTLP visibility.
- Report absent requested export as
Not configured, not Not proven.
- Apply privacy checks to the final logging pipeline: formatter fields,
adapters, MDC/context variables, framework access logs, and exception
rendering. Removing IDs from one
logger.* call is insufficient when a
formatter or access logger adds them back. Keep raw request/user/tenant/
session IDs, raw dynamic URLs, exception text, and tracebacks out of the
approved application-log surface unless the policy explicitly permits them.
Language-Specific Musts
Python:
- Add explicit dependency entries for
opentelemetry-api, opentelemetry-sdk, opentelemetry-exporter-otlp, and each detected framework/client instrumentation package.
- Create a separate setup file such as
otel_setup.py, telemetry.py, or instrumentation.py.
- Configure a shared
Resource.create({"service.name": ...}), trace provider,
meter provider, and requested log provider/exporters in that setup file only
for signals the process does not already own. If a source-owned provider
exists, move or adapt its construction into the shared setup while preserving
existing wrappers, views, file-export modes, and tests; do not create a
second provider.
- Import and call the setup function from the app entry point before creating or instrumenting the app.
- For Flask, call
FlaskInstrumentor().instrument_app(app).
- For FastAPI, call
FastAPIInstrumentor.instrument_app(app) immediately
after app construction and before lifespan/startup begins. Do not first call
it inside lifespan: Starlette/FastAPI middleware installation is too late
after the application has started serving.
- For Celery, call
CeleryInstrumentor().instrument() in the worker path.
- Keep existing Docker/Compose/Makefile commands, but update them only as the startup surface for the explicit setup, not as a replacement for app wiring.
- The ASGI/WSGI instrumentation underlying Flask/FastAPI already sets
http.response.status_code on http.server.request.duration for every
request, and error.type for a 5xx (or otherwise invalid) status, with no
extra code -- a plain 4xx client-error response does not set error.type
on a server span. server_request_hook/response_hook only set span
attributes, not metric attributes; they are not a route to a new dimension
on the duration metric itself. Do not add a standalone counter for a
request outcome the duration metric already attributes correctly.
Node.js:
- Add
@opentelemetry/instrumentation-http explicitly for HTTP server spans.
- Add the detected framework instrumentation explicitly, for example
@opentelemetry/instrumentation-express for Express.
- Add
@opentelemetry/exporter-metrics-otlp-http and @opentelemetry/sdk-metrics when wiring SDK-based metrics.
- Configure
PeriodicExportingMetricReader with exportIntervalMillis: Number(process.env.OTEL_METRIC_EXPORT_INTERVAL || 1000) and exportTimeoutMillis: Number(process.env.OTEL_METRIC_EXPORT_TIMEOUT || 500) so HTTP duration metrics export during short runtime checks.
- Use the current
NodeSDK metric reader option exactly as shown in the Node reference. Do not substitute metricReaders for metricReader unless the installed SDK version documents that option.
- Do not rely on
@opentelemetry/auto-instrumentations-node alone when specific framework packages are expected.
- In the final response, name the updated preload command (
--require or --import), the packages added, and that HTTP server spans plus request-duration metrics are expected.
@opentelemetry/instrumentation-http already sets http.response.status_code
on http.server.request.duration from the response for every request with no
extra code. It does not set error.type from a failing status code:
error.type there is reserved for a lower-level request/response transport
error (for example a socket error before a status was ever sent), not an
ordinary 4xx/5xx completion. Its hooks (requestHook, responseHook,
startIncomingSpanHook) only add span attributes, not metric attributes.
Do not add a standalone counter for a request outcome that http.response.status_code
already distinguishes; @opentelemetry/instrumentation-http has no per-call
metric-attribute hook, so a finer-grained reason dimension that the status
code alone cannot express does need its own custom metric here.
Go:
Java:
- Use the Java agent for Spring Boot unless custom business spans are explicitly requested.
- Avoid adding SDK dependencies to
pom.xml for basic Spring Boot coverage.
- Follow
./references/languages/java.md Implementation Rules for DI binding, provider reuse, and dependency checks.
- Wire the agent through the existing startup surface,
JAVA_TOOL_OPTIONS, or a documented run command.
- In the final response, explicitly mention the agent setup or path,
OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, HTTP server spans, and
http.server.request.duration.
4. Custom Instrumentation
After auto-instrumentation is wired up, prompt the user:
Auto-instrumentation is configured. Would you like me to add custom spans or metrics for your business logic?
Then wait for the user's answer.
Skip this prompt when the user already asked for incident-readiness or GenAI/LLM
workflow instrumentation, a specific custom signal, when an Audit-Driven
Readiness path applies, or when prioritized audit rows are already in scope by
the Audit-Driven Gap Closure rules. In those cases, the user's request and
audit gaps are the approval context; implement the safe scoped signals and
clearly list any unpatched prerequisites.
- If no: proceed to the project-runtime validation gate (Step 5).
- If yes: for each candidate point below, first check whether it occurs
inside a call an auto-instrumented RED metric already measures (HTTP, RPC,
DB, or messaging — see
#### Implementation Rules); if so, prefer adding
an attribute to that existing metric via the language's per-call
metric-attribute hook over defining a new counter/histogram. Analyze the
codebase for high-value custom instrumentation points:
- Error handling paths that catch and handle exceptions
- Key business operations (payments, orders, user registration, etc.)
- External calls not covered by auto-instrumentation libraries
- Background workers and scheduled jobs
- Cache interactions without auto-instrumentation support
- Incident-readiness boundaries: customer-impact workflow outcome,
dependency timeout/retry/rate-limit/error and endpoint/target health, input
complexity, freshness, queue/backpressure, synthetic/canary result,
auth/edge failure, capacity saturation, desired-vs-healthy,
startup/readiness/healthcheck, traffic target health, and release/config
context when code evidence exists
- Incident-evidence boundaries: every supplied failure mechanism must map to
an added or proven code-owned signal or an explicit external owner; generic
endpoint metrics are insufficient when the mechanism is auth handshake,
secret expiry, stale output, rollout skew, dependency target health, stream
lifecycle, or pool saturation
- GenAI/LLM workflow boundaries: workflow span, agent/workflow invocation,
LLM inference, tool/function execution, MCP method dispatch when present,
retrieval, fallback, token usage, model/config readiness, prompt/response
parse outcome, safety/policy outcome, AI-derived data freshness, and
AI-owned session/stream lifecycle when code evidence exists
- Suggest specific spans and metrics with names, attributes, and rationale
- Apply after user approval
5. Validate The Implementation
Read and follow ./references/project-runtime-validation.md. Local,
deterministic validation is the default completion gate; do not ask for
permission to run a project-configured syntax, compile, typecheck, import, or
focused test command. Do not require full app startup, Docker, credentials,
live providers, or an OTLP collector for this gate.
At minimum:
- Probe the selected project runtime and record the version actually used.
- Run static/config checks for changed scripts and manifests, including
git diff --check when Git is available.
- Compile, typecheck, or import every affected application module with the
selected project runtime.
- Run the smallest existing focused tests that exercise changed code. For
custom spans, metrics, or logs, add or update a focused repo-native test
when the existing test framework provides a practical in-memory OTel seam.
Build an exact signal closure matrix and execute every added or modified
span name and metric call site. Do not infer coverage for create, batch,
update, delete, route, tool, or workflow names solely from a shared helper's
test. Parameterize tests when those call sites share setup. For
detector-critical counters, histograms, and observable gauges, drive each
incident state to a non-default value and assert its emitted datapoint and
bounded dimensions; metric registration, name presence, or a zero-value
observation alone is not incident-readiness proof.
Map these executions back to every in-scope
Audit Gap Closure row and its
declared verification scenarios.
- Confirm filtered tests actually ran by checking test output or generated
reports; a no-match guard is reactor plumbing, not test evidence.
- If a validation failure is caused by instrumentation changes, repair it and
repeat the affected gate until it passes. Do not finalize with a compile,
type, syntax, or import error on a modified line.
Skip command execution only when the user explicitly forbids verification or
states that an external eval owns all checks. In that case, still perform
source-level review, record Not run in the handoff, and do not describe the
instrumentation as verified or complete. If the configured runtime or declared
dependencies are unavailable, record Blocked with the exact prerequisite;
do not fall back to an incompatible global runtime.
After the implementation gate, invoke or apply the $otel-verify workflow
unless the user explicitly opts out or a concrete prerequisite blocks it. The
instrumentation goal is not done until code viability is known and verification
has run, been explicitly skipped by the user, or is documented as blocked.
Record the verification result and .observe/otel-verify.md path in
.observe/otel-instrumentation.md. If verification cannot run, record the exact
blocking prerequisite and do not describe the instrumentation as verified.
When verified metric evidence exists and the user requested detectors,
alerting, monitors, Splunk configuration, or $splunk-configure, invoke or
apply that workflow and include the detector/configure verification result in
the instrumentation report.
When any claim depends on auto-instrumentation startup, framework route
resolution, automatic metrics, duplicate automatic-span prevention,
startup/exporter wiring, or runtime-installed OTLP logs, read
../references/full-runtime-acceptance.md and require its conditional full
runtime gate in the verification handoff. Attempt the gate without asking when
the repository provides a safe local profile or fixtures. Otherwise record the
exact prerequisite and keep those rows Partial, Blocked, or Not proven.
Not run or no collector was run alone is not an acceptable blocker. For
each required full-runtime row, record either the executed command and direct
result or the concrete unavailable runtime, listener, dependency, credential,
or fixture that prevented the attempt. Do not finalize while a safe local
profile exists and the required gate has not been attempted.
6. Enable Debugging in VS Code
This step is REQUIRED whenever .vscode/launch.json exists.
- Check whether
.vscode/launch.json exists.
- If it exists, update at least one debug configuration for this service to include:
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318OTEL_METRIC_EXPORT_INTERVAL=1000OTEL_BSP_SCHEDULE_DELAY=100
- After editing, report which configuration was updated, the file path, and whether the env vars were added or already present.
- If
.vscode/launch.json exists and you do not update it, stop and explain why.
- If
.vscode/launch.json does not exist, explicitly report: No .vscode/launch.json found; Step 6 skipped.
7. Finalize
- In the final response, separate file changes from verified outcomes
- State the selected project runtime, affected-module compile/type/import
result, focused tests that actually ran, and the verification result or
blocking prerequisite.
- Write
.observe/otel-instrumentation.md and include a Signals Changed
summary with added, modified, and removed traces/spans, metrics, logs/events,
runtime/config, and dependencies. If no prior audit existed, state that the
report establishes the implementation baseline.
- Include
Audit Gap Closure counts by Working, Not working, Not proven,
Not configured, and Deferred. Keep every source-audit gap visible even
when the user narrowed scope.
- For incident-readiness work, summarize each in-scope workflow, dependency,
input-complexity, freshness, backpressure, synthetic/canary, auth/edge,
capacity, health/readiness, and release/config surface as MTTD-improving,
localization-only, provider/platform-owned, or uncovered. Name every
remaining detector prerequisite and its owner; do not call the pass complete
while an app-owned required signal is only a follow-up unless the user
explicitly narrowed scope.
- When the source audit declares GenAI ownership, include the complete
GenAI Readiness Closure matrix and list every non-Working remaining signal
in the final response.
- For GenAI work without a source audit, explicitly list each unimplemented
token/context-pressure signal in the final response: context budget,
truncation, token-limit errors, prompt/tool schema size, LLM-call fanout, and
tool-call fanout. Do not collapse absent items into a generic readiness claim
and do not create a
GenAI Readiness Closure table without source-audit rows.
- Include
$otel-verify results and .observe/otel-verify.md path when run.
If detectors/configuration were requested, include $splunk-configure
outputs and .observe/splunk-configure-verify.md status when run.
- If verification is partial, say exactly what is working and what is still missing instead of reporting full success
- Never say
complete, working, or verified when the mandatory
compile/type/import gate failed, was blocked, or was not run. Use
implemented; verification blocked/not run and name the prerequisite.
- Always include the service-name configuration, OTLP endpoint configuration, and which automatic spans/metrics are expected from the instrumentation.
- State the selected log scope and the full-runtime acceptance result whenever
either is applicable.
- For GenAI work, state which OTel GenAI operations were instrumented, which
GenAI metrics are expected, whether trace continuity should produce a nested
workflow/agent/tool/chat/retrieval shape, and which privacy/cardinality limits
were enforced.
- For GenAI incident-evidence work, include a concise coverage summary by
incident class or repo surface: MTTD-improving, localization-only, or
uncovered. Name any remaining provider/model, workflow, tool/function
execution or AI-owned session/stream including MCP when present, retrieval,
streaming, token/context, prompt/response, safety/policy, AI-derived data,
model/config rollout, or AI-owned cache/session owner that still blocks
detector-ready coverage.
Credential Safety
When the project uses or introduces env files:
- Ensure
.env is gitignored before writing secrets: Check .gitignore for .env. If it is missing, add it. Never allow a .env with access tokens to be committed.
- Create or update
.env.example with safe placeholders: Do this whenever the instrumentation introduces env vars. .env.example must never contain real tokens.
- Verify no tokens in tracked files: Search tracked config files for access tokens and confirm no real token values appear in files that would be committed.
Scope
- New apps: Full scaffold matching the current runtime shape: instrumentation, SDK init, env var config
- Existing apps: Incremental -- detect what's already present, add only what's missing
