otel-instrument-feature

name	otel-instrument-feature
description	Instrument any new feature, endpoint, route, handler, worker, job, CLI command, or user-facing flow with OpenTelemetry spans, metrics, and logs before it lands. Triggers on requests like 'add a new endpoint', 'implement X', 'create a handler for Y', 'build a job to Z', 'add this feature', 'wire up a new route', or any code change that introduces a new business operation.

Instrument a new feature with OpenTelemetry

Use this skill whenever you are about to write — or have just finished writing — a new business operation in the user's project. A "new feature" is anything a human operator would want to find later when something looks wrong: a new HTTP route, RPC handler, GraphQL resolver, background job, queue consumer, cron task, CLI subcommand, agent tool, scheduled workflow, or any module-level function that represents a meaningful step in a user-facing flow.

Adding code that is just plumbing — a pure helper, a type, a config — does not trigger this skill. Adding code that takes a request in, does work, and produces a side effect or response does.

Step 0 — Confirm OTel is already wired

Before instrumenting, look for an existing bootstrap. Common signs: a telemetry.ts / observability.ts / instrumentation.ts / init_observability() module, @vercel/otel registerOTel(...), NodeSDK/sdk-node setup, Python LoggerProvider + OTLPLogExporter, superlogHeaders(...), or an inlined https://intake.superlog.sh endpoint with an sl_public_ token.

If you cannot find a bootstrap, do not invent one inside the new feature file. Tell the user the project still needs OTel wiring and that this is the job of the superlog-onboard skill — then continue with the feature implementation, leaving a single brief TODO at the bootstrap site (not on every span/metric/log call site). Do not block on this.

If a bootstrap exists, read the applicable companion skill before writing code:

• otel-onboarding-style — general OTel taste and attribute conventions.
• otel-python-style — Python.
• otel-fastapi-style — FastAPI.
• otel-livekit-style — LiveKit agents.
• otel-nextjs-style — Next.js / Vercel.
• otel-expo-style — Expo / React Native.
• otel-supabase-edge-style — Supabase Edge Functions.
• otel-generic-style — anything else (Go, Java, Ruby, Rust, .NET, PHP, Elixir, plain Node, …).

Match the local convention. If the file you are editing already uses withSpan from @superlog/otel-helpers, keep using it; if it uses tracer.startActiveSpan directly, keep using that. Do not introduce a second style alongside the existing one.

Step 1 — Decide what the new feature needs

For each new operation, walk this checklist explicitly before you write the instrumentation. The answer to each question may be "no, and here's why" — that is fine, but answer it.

Span — almost always yes

Wrap the operation in a single span whose name is domain.verb (order.submit, interview.create, email.send, job.reconcile, agent.run, voice.session, llm.generate_copy). The span lives at the entry point of the feature, not inside helpers.

Skip a custom span only when the operation is entirely covered by an existing auto-instrumentation (HTTP server entry, DB query, framework lifecycle) and adds no business-meaningful work on top. If in doubt, add the span — auto-instrumentation almost never has the right name or attributes for a domain operation.

Attributes to set on the span (only what is available, no PII):

• Entity IDs the operator searches by: tenant.id, workspace.id, org.id, user.id, order.id, job.id, request.id. Use bounded IDs only — never raw email/phone/full request bodies.
• Bounded categorical dimensions: outcome (success/failure/skipped), channel, kind, route, action.
• Counts and sizes where they help explain latency: batch.size, items.count, payload.bytes.
• For LLM call sites that provider instrumentation does not already cover: gen_ai.provider.name, gen_ai.request.model, app.gen_ai.use_case, app.gen_ai.call_site. Do not duplicate attributes OpenInference/provider instrumentation already captures.

On the error path, record the exception and set the span status to error. Then re-raise — do not swallow.

Metric — yes if this is a meaningful state transition or has latency that matters

Three categories. Decide which apply to the new feature:

• Business counter — every meaningful outcome: orders.submitted, jobs.completed, emails.delivered, interviews.started, payments.failed. Tag with low-cardinality dimensions (tenant.id, outcome, channel). Never tag with raw user/order IDs.
• Performance histogram — duration of the operation if a human would care about its tail. Reuse the span's existing timing rather than measuring twice when the SDK exposes it; otherwise record from time.perf_counter / performance.now() around the same boundary the span wraps.
• Cost — for LLM call sites only, and only when provider instrumentation does not already capture token usage. Use llm.tokens.input / llm.tokens.output counters with unit="tokens". Do not add llm.cost_usd or any app-side pricing metric — Superlog computes cost centrally from provider/model/token data.

Acquire the meter and create instruments at module scope. Never create a meter or instrument inside the hot path.

Log — yes if a human would want a narrative trail

Use logs for narrative ("starting batch reconcile", "retry 2 of 3 after 5xx") and for exceptional events that need attention. Use the project's existing logger — do not introduce a new logging library and do not wrap log calls in a helper.

An error-level log must mean the operation cannot recover without manual intervention. Recoverable conditions go at warn or below.

Logs emitted inside the span automatically pick up trace_id / span_id if the bootstrap's log bridge is wired correctly. Do not start a throwaway span around a log call to force correlation.

Attributes vs metric dimensions

Span attributes can be high-cardinality (specific IDs, counts). Metric dimensions cannot — keep them bounded. If you find yourself wanting to tag a metric with order.id or request.id, that data belongs on the span, not the counter.

Step 2 — Write the instrumentation alongside the feature

Add the span at the entry point of the new operation. Do not sprinkle micro-spans across every helper inside it — one well-named span with rich attributes beats five generic ones. Sub-spans are useful only when there is a real sub-operation a human would want to isolate (an external API call, a DB transaction with non-obvious latency, a distinct phase like parse / validate / dispatch).

Add counters at the success and failure boundaries of the operation, with outcome tagged. Add a histogram where latency tail matters. Add logs at the start/end of long-running narratives and on exceptional branches.

Do not add backwards-compatibility shims, feature flags, or "telemetry on/off" toggles around individual call sites. The SDK already has a no-op path when no provider is registered; that is the only switch needed.

Do not invent helper APIs like recordCounter, sendSuperlogSpan, withTelemetry, trackEvent. Use the native OTel APIs (or the published withSpan from @superlog/otel-helpers in TS/JS, where the local style already uses it).

Step 3 — Verify the new feature emits what you claimed

Run the feature end-to-end through the project's normal entry point — hit the route with curl, invoke the CLI subcommand, enqueue the job, trigger the agent tool. A unit test that mocks the tracer does not count; the goal is to see the real OTLP payload leave the process.

If the dev server logs OTLP exporter activity (or the project has a debug exporter), confirm:

• The span you added shows up with the name and attributes you claimed.
• Counters you added incremented at least once.
• A log line emitted inside the operation arrived with trace_id / span_id populated.

If the bootstrap uses a generated sl_public_ token and signup is not finished yet, ingest will return 401/403 — that is expected. The signal you care about here is "the SDK produced a payload and tried to send it", not "ingest accepted it."

If only spans appear and metrics/logs do not, the bootstrap has a wiring gap. That is a superlog-onboard problem, not this skill's — flag it to the user and continue with the feature work.

Hard rules

• Never put PII in span attributes, metric dimensions, or log fields: no emails, phone numbers, passwords, tokens, full request bodies, plaintext message content.
• Never tag metrics with raw user/order/request IDs. Those go on spans.
• Never invent a wrapper API for telemetry; use the native OTel SDK and the local style's conventions.
• Never start a throwaway span solely to make a log line carry trace_id. Fix the bridge instead, or accept the uncorrelated log.
• Never add LLM cost math or pricing tables at call sites. Superlog computes cost centrally.
• Do not block the feature on a missing bootstrap — leave a single TODO and tell the user to run superlog-onboard.