last9-traces

name	last9-traces
description	Guided trace investigation and tracejson query reference for Last9. Interviews the user with 5 targeted questions (service, environment, time window, symptom, scope), fires the right tool call, and carries the reference card for get_traces tracejson pipelines. Use when investigating a trace issue, debugging a service, building or fixing a tracejson query, or when user says "diagnose" / "investigate" / "what's wrong with".
compatibility	Requires the Last9 MCP server (last9-mcp-server) connected to the session
metadata	{"author":"last9"}

last9-traces — trace investigation + tracejson reference

Extends the get_traces, get_service_traces, and get_trace_attributes tool instructions — read those first. This skill carries the investigation flow (which question, which tool, in what order) and the query reference (operators, field syntax, common mistakes).

Operating principle: progressive narrowing, assisted. At every step, surface the narrowing handles the data offers — services, environments, trace attributes — and let the user pick. Never guess scope; never go broad when a narrowing option exists.

Prerequisites

Requires the Last9 MCP server connected to this session. If get_traces, get_service_traces, or get_trace_attributes are not available as tools, stop — tell the user to install and authenticate the Last9 MCP server first, and link them to the README above. Never fabricate or simulate these tool calls.

Tool parameters at a glance

For agents consuming this skill outside Claude Code (copy-pasted into a custom system prompt), this is the minimum parameter surface; the MCP tool descriptions remain authoritative.

Tool	Purpose	Key params
get_trace_attributes	Discover available trace field names (filter_field)	time params
get_traces	Filter / aggregate via tracejson pipeline	tracejson pipeline, time params
get_service_traces	Service-scoped traces or a specific trace	service, optional trace_id, time params

Time params (every tool): lookback_minutes for relative windows, or start_time_iso + end_time_iso for explicit ranges — see Time discipline below.

Attributes first — strict sequence

Discovery and query are strictly sequential: call get_trace_attributes, wait for its results, then build the query from the filter_field values it returned — never fire discovery and get_traces in the same response. Attribute names vary per service with no error on a wrong guess (attributes['http.status_code'] on one service, attributes['status_code'] on another) — the filter silently returns zero results. Verify the name via discovery before filtering on it.

Time discipline

lookback_minutes or start_time_iso + end_time_iso are top-level tool params.

• Relative requests ("last 15 minutes", "past hour") → lookback_minutes. Never fabricate fixed timestamps for a relative request — a generated "now" is wrong by the time the query runs and silently shifts the window.
• Explicit dates → start_time_iso / end_time_iso in RFC3339/ISO8601 UTC, e.g. 2026-06-08T10:00:00Z. Legacy YYYY-MM-DD HH:MM:SS is compatibility-only — do not generate it.
• Both present (explicit range plus a relative phrase) → explicit timestamps win; drop lookback_minutes.

Guided investigation — 5 questions

Ask one at a time. For each question, provide your recommended answer based on context already in the conversation; if the conversation already answers a question, skip it and state what you inferred.

Q1 — Service "Which service are you investigating?" → If the user named a service loosely, resolve it with did_you_mean (type: service). If they can't name one, don't ask blind — surface candidates: a get_traces aggregate counting traces by ServiceName (symptom-filtered when there is one) and let them pick.

Q2 — Environment "Which environment? (e.g. production, staging) — I'll look up the exact attribute name." → Call get_trace_attributes to find the deployment environment field. Use the returned filter_field verbatim for the environment condition.

Q3 — Time window "When did you first notice the issue? Give a relative window ('last 15 minutes') or absolute start/end times." → Map the answer per Time discipline above: relative → lookback_minutes; absolute → start_time_iso / end_time_iso.

Q4 — Symptom "What are you seeing — errors, high latency, a specific operation failing, or something else?" → Map to: StatusCode = STATUS_CODE_ERROR for errors; Duration > <threshold> for latency; SpanName $contains <name> for a specific operation.

Q5 — Scope "Do you have a specific trace ID to look at, or should we search broadly?" → Specific trace ID → use get_service_traces with trace_id. → Broad search → use get_traces with a tracejson pipeline built from Q1–Q4 answers.

After Q5, immediately execute the tool call — no confirmation step.

tracejson reference

Operator quick reference

Comparison (value must be a string): $eq $neq $gt $gte $lt $lte

String match: $contains $notcontains $icontains $inotcontains $containsWords $notcontainsWords $icontainsWords $inotcontainsWords $regex $notregex $iregex $inotregex $ieq $ineq

Existence: $exists $notnull

Logical: $and $or $not

Form: {"$op": [field, value]} — field is always the first element.

Field mapping priority

Call get_trace_attributes to get the exact filter_field for any resource or span attribute. Use filter_field verbatim — never transform it.

Situation	filter_field to use
resource_service.name	ServiceName
resource_<key>	resources['<key>']
event_<key>	events['<key>']
known top-level field	field name as-is (no brackets)
grpc.status_code	attributes['rpc.grpc.status_code']
anything else	attributes['<raw>']

Top-level fields: TraceId SpanId ServiceName SpanName SpanKind StatusCode StatusMessage Duration Timestamp ParentSpanId TraceState

Common mistakes

Wrong	Correct
"resource_department"	"resources['department']"
"ResourceAttributes.department"	"resources['department']"
"SpanAttributes.http.method"	"attributes['http.method']"
resources["key"] (double quotes)	resources['key'] (single quotes)
{"ServiceName": "checkout"}	{"$eq": ["ServiceName", "checkout"]}
{"function": "count", "alias": "n"}	{"function": {"$count": []}, "as": "n"}
"aggregations": [...]	"aggregates": [...]
"group_by": {...}	"groupby": {...}
{"$eq": ["StatusCode", "ERROR"]}	{"$eq": ["StatusCode", "STATUS_CODE_ERROR"]}
{"$eq": ["SpanKind", "SERVER"]}	{"$eq": ["SpanKind", "SPAN_KIND_SERVER"]}

Failure modes differ by mistake class. Field-ref, quoting, and pipeline-key mistakes are rejected with corrective errors (the API tells you the right form). Enum-value mistakes are NOT: a wrong StatusCode or SpanKind value silently returns zero results with no error. If a StatusCode/SpanKind filter comes back empty, check the value against the STATUS_CODE_* / SPAN_KIND_* forms before trusting the empty result. Numeric values: strings are the guaranteed form; raw numbers are tolerated.

Related skills

This skill owns trace-first investigation and tracejson query craft. For log-first work (log search, log aggregation, logjson pipelines), use last9-logs. On an ambiguous debugging prompt, the logs skill takes the logs path and refers trace work here — symmetrically, refer log questions there; do not attempt logjson queries from this skill.