agento11y-instrument

name	agento11y-instrument
description	Sets up and instruments a developer's own LLM app or agent to send generations and agentic workflow to Grafana Agent Observability (the Agent Observability SDKs) — greenfield setup, fixing broken instrumentation, or filling gaps in existing instrumentation. Uses gcx for the parts a static prompt can't do: `gcx login` / `gcx cloud stacks` to find the stack, and `gcx agento11y agents|conversations|generations` to VERIFY that data actually lands — so it iterates (instrument → run → verify → fix) until generations arrive, not blindly. Reads the app's code, detects language/framework, classifies instrumentation state (none / partial / broken), then runs a fixed gap checklist whose #1 item is the silent failure no other prompt catches: the SDK emits OTel spans/metrics but never creates a TracerProvider/MeterProvider, so without them all metrics go to a no-op and are lost. Also checks agent_version (required for per-version Performance charts), set_result completeness, SYNC vs STREAM, parent_generation_ids DAG links, and workflow-step coverage. Recommends changes citing file:line and, only with explicit confirmation, applies minimal diffs that don't change app behavior. Pulls SDK reference from agento11y's llms.txt rather than restating it, and hands off to `agento11y-test-starter` once data flows. It does NOT write test suites or set up tenant evaluations, rules, or guards — offline test suites are `agento11y-test-starter`, tenant eval rules + guards are `agento11y-prod-setup`; does NOT install coding-agent telemetry plugins (that is llms.txt "Path A"); does NOT mint or store credentials or invent endpoints. Trigger on phrases like "instrument my app", "send my agent's traces to Grafana", "set up AI observability for my app", "my generations aren't showing up", "why is Performance empty", "add Agent Observability to my code", "fix my instrumentation".
allowed-tools	Bash, Read, Grep, Glob, Edit, Write, WebFetch

Agent Observability — instrument an LLM app

Help a developer wire their own LLM app or agent to Grafana Agent Observability, from zero or from a broken/partial state, and keep going until data actually lands in the stack. The value this skill adds over the static instrumentation prompt is two things a prompt can't do:

1. A mechanical gap checklist run against the real code — headed by the one failure that is completely silent (missing OTel providers → every metric lost, no error).
2. A verification loop through gcx: after each change, run the app and check the gcx agento11y agents / conversations / generations commands to confirm generations arrived. Diagnose the next gap from what's missing, not from guesswork.

The SDK reference (env vars, provider snippets, field lists, framework adapters, workflow steps) lives in agento11y's llms.txt "Path B". Fetch it rather than restating it here; this file holds the flow and the decision logic. A minimal fallback lives in references/instrumentation.md for when the fetch is unavailable.

Rules

• Reference, don't restate. Fetch SDK detail from https://raw.githubusercontent.com/grafana/agento11y/main/llms.txt (Path B). Only inline decision logic here. If the fetch fails, fall back to references/instrumentation.md.
• Never invent an endpoint or a token. Read them from the environment (AGENTO11Y_ENDPOINT, AGENTO11Y_PROTOCOL, AGENTO11Y_AUTH_MODE, AGENTO11Y_AUTH_TENANT_ID, AGENTO11Y_AUTH_TOKEN, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS) or ask the developer. Never fabricate a URL or mint a token.
• Target is Grafana Cloud. The developer supplies the endpoint + token (Step 0), and the gcx verification loop (Step 5) confirms data landing against the Cloud tenant. Never fabricate the endpoint or token — read them from the environment or ask.
• Write AGENTO11Y_* env vars, never SIGIL_*. SIGIL_* is a deprecated legacy fallback. Do this even if sibling apps or existing .env files in the repo use SIGIL_* — matching a stale local convention perpetuates it. If the app already reads SIGIL_*, add the AGENTO11Y_* names (the SDK still honors both) and note the old ones are deprecated. Do not "match the siblings."
• Gate every code WRITE on explicit confirmation. Report first (Step 4), apply only after the developer says yes (Step 5). Read-only gcx verification and re-running the app happen freely inside the loop; editing files does not.
• Keep diffs small; do not change app behavior. Instrumentation is additive. No refactors, no prompt rewrites, no dependency upgrades beyond the SDK/adapter packages actually needed.
• Never change the model, provider, or the app's LLM config — not even with permission, not even "just to run the verify loop." Instrument whatever model the app already uses. This is absolute: changing the model is out of scope for instrumentation, full stop. If a run fails because a provider API key is missing, the only allowed responses are: (a) ask the developer to provide the key for the model the app already uses, or (b) skip the live run and report the wiring as verified-by-construction, telling the developer to run it themselves. Do not offer to switch the provider, do not ask "which provider should I use?", and do not add a new provider dependency (e.g. ) to make the run succeed. If the developer separately says they a different model, that is an app change they own — tell them to make it and re-invoke this skill; do not fold it into the instrumentation diff. Swapping the model silently changes what the app does and what gets observed, which defeats the point. — it authenticates the LLM call, not the telemetry export, and the app already has it if it runs at all. So don't ask for it, configure it, or rewire it; if the live verify-run fails on a missing provider key, skip the run and report verified-by-construction (see Step 5). Just don't conflate the two 401s: a 401 on is observability auth and yours to fix (usually a missing /); an auth error from the is not — surface it and let the developer handle their own key.

Step 0 — Credentials and endpoint

The app needs, in its environment before the SDK starts, seven vars — not five; the two mode vars are the ones most often forgotten, and their absence is a silent 401:

• generation ingest: AGENTO11Y_ENDPOINT, AGENTO11Y_PROTOCOL=http, AGENTO11Y_AUTH_MODE=basic, AGENTO11Y_AUTH_TENANT_ID, AGENTO11Y_AUTH_TOKEN.
• traces/metrics: OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS.

AGENTO11Y_PROTOCOL=http and AGENTO11Y_AUTH_MODE=basic are required for Cloud, not optional — the SDK defaults (grpc / no-auth) return a silent 401 against the Cloud HTTP ingest endpoint. (They scope the ingest channel only; the OTel channel's transport/auth is set entirely by the OTEL_* vars — see references/instrumentation.md.) The only var that is sometimes omittable is OTEL_EXPORTER_OTLP_HEADERS: required when sending directly to the Cloud OTLP gateway (the common case, gateway enforces Basic auth), omittable only when OTEL_EXPORTER_OTLP_ENDPOINT points at a local Alloy / OTel Collector that already holds the Cloud credentials. First check what's already set (including any existing .env) — if all are present, skip to Step 1. Watch for the near-miss where the endpoint is set under the wrong name (e.g. AGENTO11Y_API_ENDPOINT — the SDK reads AGENTO11Y_ENDPOINT, so the wrong name is silently ignored and ingest falls back to a default host).

When any value is missing, do NOT just list the variable names and ask — hand the developer the exact place to get each one (link + clicks), every time. The concrete sources are in point 3 below; surface them proactively. The most common failure of this skill is naming OTEL_EXPORTER_OTLP_ENDPOINT/OTEL_EXPORTER_OTLP_HEADERS and leaving the developer to guess — the answer is the stack OTLP tile + "Generate now", which precomputes both. Give that first.

What gcx does for you (run these):

1. gcx config current-context — is there a working context? If not, just ask the developer to log in to the stack they want the instrumentation to connect to — e.g. "run gcx login against your stack." Do not fabricate the login command yourself (don't guess the host or flags); let them run their own login (the Agent Observability setup screen gives them the exact command, or they use gcx login). Instrumentation itself needs no gcx login — only Step 5 verification does, so this never blocks writing the code.
2. gcx cloud stacks list, then gcx cloud stacks get <stack-slug> — identify the target stack and its URLs. This gives you the stack to point the developer at, and confirms which tenant the Step 5 verification will read from.

What still needs the Connection page (gcx cannot do these today):

3. gcx does not generate the Agent Observability OTLP gateway URL, and does not mint the ingest / access-policy token. When you ask the developer for a value, always tell them exactly where to get it — a link and the clicks — never just name the variable and wait. The two channels come from two different places:

   OTEL_* (traces/metrics) — easiest, let Cloud build them. Send the developer to the stack's OTLP tile: https://grafana.com/orgs/<org-slug>/stacks/<stack-id>/otlp-info. It already shows OTEL_EXPORTER_OTLP_ENDPOINT and the Instance ID; under Password / API Token → "Generate now" it mints a token and then fills an Environment Variables block with all OTEL_* vars — the base64 OTEL_EXPORTER_OTLP_HEADERS is precomputed, ready to copy. No manual base64. (In Python, the value uses Basic%20… — keep it as given.)

   AGENTO11Y_* (generation ingest) — the plugin Connection page. AGENTO11Y_ENDPOINT and the token come from https://<stack>.grafana.net/plugins/grafana-agento11y-app → Connection tab. When the developer creates the token via "Create a token in Cloud Access Policies", tell them the scopes: sigil:write, metrics:write, traces:write, logs:write. UI heads-up: sigil is not in the default resource list — add it via "Add scope" (then tick Write); the scope is still sigil:* (the Cloud resource keeps the old name). The same glc_… token works for both channels if it has all four scopes. Also set AGENTO11Y_PROTOCOL=http and AGENTO11Y_AUTH_MODE=basic (see references/instrumentation.md — the SDK defaults grpc/none give a 401).

   Ask the developer to put the values in a gitignored .env (or export them) themselves — do not ask them to paste a secret token into the chat (it is captured in the transcript). Instrument the code to read from the environment and have them supply the values out-of-band. Never invent a URL or mint a token.

   Two different tokens — don't confuse them. gcx logs in with its own OAuth token (gat_) and refreshes it automatically; that is what authenticates the commands here. It is the app's ingest token. The app needs a separate access-policy token () in / , and gcx does not create that one.

Once gcx has a working context, the Step 5 verification commands (under the gcx agento11y group) work against the developer's tenant even before the app's own credentials are fully wired.

The Connection page is the only manual step. If a future gcx release can create the access-policy token and surface the OTLP endpoint, this step collapses to gcx-only — but do not assume it can today; check gcx cloud --help rather than promising it.

Step 1 — Read the app and detect language / framework / shape

Find and read, recording file:line for each:

1. The generation entrypoint(s) — where the model is invoked.
2. How the LLM client is constructed (which provider: OpenAI / Anthropic / Gemini / other).
3. The app bootstrap / init — where an OTel TracerProvider / MeterProvider would be created.
4. Any existing Agent Observability SDK imports (agento11y / @grafana/agento11y / the Go agento11y package — or the legacy sigil_sdk / @grafana/sigil-sdk-js / Go sigil names in older code) or AGENTO11Y_* usage.

Detect:

• Language — from the manifest / extensions (pyproject.toml/.py, package.json/.ts, go.mod/.go, gradle/.java, .csproj/.cs).
• Framework — grep for langgraph, langchain, openai-agents, llamaindex, google-adk, strands, pydantic-ai, litellm, claude-agent-sdk, vercel-ai-sdk, crewai, or a custom orchestrator.
• Shape — single generation vs agentic pipeline (multiple nodes / a graph / sub-agents). This decides whether workflow steps (checklist #8) and parent links (#7) are in play.

Before recommending a provider wrapper or framework adapter, confirm it exists for this language (fetch llms.txt "SDK API surface" — the matrix is not symmetric across languages).

Step 2 — Classify instrumentation state

State the classification explicitly, with the file:line evidence that led to it:

• none (greenfield) — no SDK import anywhere. Full setup from scratch.
• partial — an SDK client is constructed and some generations are wrapped, but coverage is incomplete (no OTel providers, no agent_version, no workflow steps for an agentic pipeline, no parent links). Run the full checklist; recommend + apply only the gaps.
• broken — the SDK is present but wrong: metrics silently lost (no MeterProvider), export misconfigured, legacy SIGIL_* vars, stream/non-stream mode mismatch, set_result/SetResult never called, or rec.err()/Err() never checked. Fix first, then gap-check.

All three paths converge on the same checklist (Step 3); they differ only in how much is already done.

Step 3 — Run the instrumentation gap checklist

Walk each item against the code. Record PRESENT / MISSING / WRONG with file:line. This mechanical audit is the skill's core value. Items 0, 1, 2, 5, 6 fail silently (no error) — always check them. Items 3, 7, 8 mean data lands but analysis is degraded. For the fix, read the named section of the fetched llms.txt (locate it by its heading — do not trust line numbers, they drift).

#	Check	Silent-failure symptom	llms.txt section
0	The .env actually takes effect. Confirm the app loads its own .env by an explicit path (not a bare load_dotenv() resolved by CWD) and that it wins over vars already in the environment. Verify by printing os.environ["AGENTO11Y_ENDPOINT"] / OTEL_EXPORTER_OTLP_ENDPOINT after all imports, not before	import litellm (and some other libs) inject localhost OTLP/ingest defaults into os.environ at import time (OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4318, AGENTO11Y_ENDPOINT=localhost:8080). A plain load_dotenv() does not override already-set vars → the Cloud endpoints in .env never apply and everything ships to localhost, returning 200 OK if a local stack is up. Zero error signal, and gcx against the Cloud tenant shows nothing. Fix: load_dotenv(<path-relative-to-__file__>, override=True) before constructing providers/client. A bare load_dotenv() also resolves the wrong .env by CWD	"Environment"
1	OTel TracerProvider and MeterProvider created before the SDK client (verify by construction + Performance view / OTLP POSTs — not via gcx, which can't see OTel; see Step 5)	spans/metrics go to no-op → all latency/token/cost metrics lost. The #1 failure.	"OTel setup (required)"
2	Providers shut down after shutdown()	last batch of spans/metrics dropped on exit	"OTel setup (required)"
3	agent_name + agent_version set on generations / handlers	per-version Performance charts break (join on gen_ai.agent.version)	"Agent Observability architecture and ingest model", "Telemetry fields to prioritize"
4	set_result/SetResult includes response_id, response_model, finish/stop reason, full token usage (incl. cache_read_input_tokens, cache_write_input_tokens, ), (system+user prompt in , model reply in )

Step 4 — Recommend (the report)

Emit the report using llms.txt's output contract (its "Output contract" section): top opportunities first, and per opportunity — exact file:line, why it matters, a concrete diff proposal, a test plan, and any risk. Rank by severity: .env not taking effect (#0 — nothing lands at all) first, then missing OTel provider (metrics data loss), then broken export, then missing agent_version, then coverage gaps. Every recommendation cites a file:line. Then stop and ask before applying anything.

Step 5 — Apply, then verify (the loop)

Only after the developer confirms a diff. Bounded to ~3–4 iterations.

1. Apply the confirmed diff (Edit/Write). Order of preference: provider wrapper → framework adapter → hand-instrument the core SDK — only what exists for the language. Add/update a focused test for the changed instrumentation. Preserve flush/shutdown lifecycle. Never touch app logic. Pull exact usage from llms.txt / the per-language README / examples/getting-started/*.

2. Run the app for one turn to generate traffic (ask the developer to run it, or run it if there's a clean entrypoint and they approve). If the run can't happen — missing provider API key, no clean entrypoint, needs a full runtime — do not work around it by changing the model or adding a provider. Stop the loop, report the wiring as verified-by-construction (imports resolve, providers build, client + handler construct), and tell the developer the one thing left is to run one turn themselves with their key. A verified-by-construction result is a fine outcome.

3. Verify — two independent channels, don't conflate them. Instrumentation sends data on two separate paths, and confirming one says nothing about the other.

   First, confirm gcx reads the same tenant the app writes to. A verification against the wrong tenant is worse than no verification — an empty agents list gets misread as "data isn't landing" when it is, just elsewhere. Before drawing any conclusion from a gcx query: read the app's AGENTO11Y_ENDPOINT + AGENTO11Y_AUTH_TENANT_ID from its .env, then check gcx config current-context and that the active context points at that same stack/tenant. If it doesn't (e.g. context is local but the app writes to a Cloud stack), switch context or ask the developer to gcx login to the right stack — do not guess the login command. (If the gcx token is merely expired, that blocks Step 5 verification only, not writing the code — say so and continue.)

   Channel A — generations (the SDK ingest client → /api/v1/generations:export). Carries the prompt, response, tokens, cost, model, finish_reason. This is what gcx can read.

   • Via gcx: gcx agento11y agents list (does the agent appear?); gcx agento11y agents get <agent-name> (is generation_count climbing?). To find the run's conversation, either gcx agento11y conversations list --limit <n> (most-recent first, no filters — the quickest post-run check) or gcx agento11y conversations search --filters 'agent = "<agent-name>"' ( alone is enough; /, RFC3339, are and only needed to narrow a busy tenant). Then or — check tokens, finish reason, cost, and (for a multi-agent pipeline) that reproduce the DAG. This proves generation ingest + are wired.

Step 6 — Hand off

Once generations land and metrics populate, instrumentation is done — that's the prerequisite for everything else. Point the developer at the next step: agento11y-test-starter to build an offline test suite for the agent (useful before shipping and for regression-testing new versions once it's live), and agento11y-prod-setup to set up online eval rules + guards on real traffic once it's deployed. The split is offline test suite vs online rules/guards — not before-traffic vs after-traffic.

Note — keeping this skill in sync

The SDK reference (env vars, provider snippets, field lists, workflow-step schema, adapter matrix) is intentionally not duplicated here — it lives in agento11y's llms.txt "Path B" and the per-language READMEs, which are the shipped source of truth. This skill holds only decision logic (state classification + gap checklist + the gcx verification loop). When a user-facing semantic changes (new SDK field, renamed env var, new framework adapter), update llms.txt (and its onboarding wizard copy); this skill points at llms.txt by section heading, not line number (line numbers drift as llms.txt is edited), so no re-pointing is needed unless a heading itself is renamed. If you find yourself pasting a provider snippet into this file, stop — fetch llms.txt instead. The references/instrumentation.md fallback is deliberately minimal for the same reason.