agent-performance-report

name	agent-performance-report
description	Produces an aggregated markdown performance report from stored trace data — tool call metrics, token usage, cost, bottlenecks, and numbered recommendations. Use when: "session cost report", "what ran this week", "why did this spike", "end-of-sprint performance review", "budget alert fired".

I. Philosophy

A performance report is an accountability artifact. It converts the raw trace data accumulated during agent operation into a human-readable narrative that answers three questions: what ran, how well did it run, and what should change.

The value of this report is not comprehensiveness — it is actionability. A report that surfaces five specific recommendations is more useful than a report that lists every span in the trace store. Structure the output so a non-technical reviewer can understand cost trends and a technical reviewer can locate bottlenecks without scrolling.

II. When to Use

Use this skill:

• At the end of a work session to summarize what the agent did and how much it cost.
• After an incident or unexpected failure to understand which nodes failed, in what order, and why.
• As part of a sprint retrospective to analyze token spend against output quality.
• When tuning agent configuration — use successive reports to measure whether changes to model selection, tool ordering, or plan structure are improving efficiency.
• When a budget alert fires — use this skill to identify which operations triggered the alert and whether the spend was justified.

Do not use this skill for real-time monitoring during an active session. For live tool call visibility, use tool-intercept-logger. This skill operates on stored, completed trace data.

III. Workflow

Step 1 — Accept time range and filters.

Accept the following inputs:

• time_range: required. ISO 8601 start/end pair (e.g., 2026-04-01T00:00:00Z to 2026-04-05T23:59:59Z). If only a start is given, end defaults to now. Convenience shortcuts: "today", "last_7_days", "this_month".
• agent_filter: optional. Restrict to spans generated by a specific agent ID.
• user_filter: optional. Restrict to spans for a specific user ID.
• tool_namespace: optional. Restrict tool-level metrics to a namespace prefix (e.g., mcp_by_dojo:*).

If no filters are provided, report covers all agents and users for the time range.

Step 2 — Query stored spans.

Query TraceStorage for all spans within the time range, applying any filters. Each span contains: SpanID, TraceID, ParentID, Name, StartTime, EndTime, Inputs, Outputs, Metadata, Status.

Also query for orchestration lifecycle events during the same range:

• orchestration_plan_created — provides plan_id, node_count, estimated_cost
• orchestration_node_end — provides duration_ms, state per node
• orchestration_complete — provides total_nodes, success_nodes, failed_nodes per plan

Query BudgetTracker for usage snapshots at the start and end of the time range to calculate net consumption per tier (query, session, monthly).

Step 3 — Aggregate tool call metrics.

For all spans where Name matches "node.<toolName>":

total_calls:         <count>
success_rate:        <success_nodes / total_nodes as %>
durations:           <sorted list of duration_ms values>
  avg_duration_ms:   <mean>
  p50_duration_ms:   <median>
  p95_duration_ms:   <95th percentile>
calls_by_tool:       <map of tool name -> call count>

Compute duration statistics per tool as well as globally.

Step 4 — Aggregate token and cost metrics.

From span Inputs, Outputs, and Metadata (which carries OTEL attributes from the TraceLogger):

token_usage_by_tool:    <map of tool name -> {input_tokens, output_tokens}>
token_usage_by_model:   <map of model ID -> {input_tokens, output_tokens}>
total_input_tokens:     <sum>
total_output_tokens:    <sum>
cost_by_operation_type: <map of {web|compute|file|memory} -> estimated_cost>
total_cost:             <sum of all estimated_cost values>

For cost breakdown, infer operation type from tool name using the same keyword mapping as budget-guard: web/compute/file/memory.

Step 5 — Aggregate DAG execution stats.

From orchestration_plan_created and orchestration_complete events:

plans_created:           <count>
avg_nodes_per_plan:      <mean node_count across all plans>
total_successful_plans:  <plans where failed_nodes = 0>
total_failed_plans:      <plans where failed_nodes > 0>
parallel_ratio:          <nodes with siblings at same DAG depth / total nodes>
sequential_ratio:        <1 - parallel_ratio>
avg_plan_cost_estimate:  <mean estimated_cost from plan_created events>

parallel_ratio requires reconstructing DAG structure from ParentID relationships in span data. If span data is insufficient to reconstruct the DAG, report this metric as "unavailable (insufficient span data)" rather than approximating.

Step 6 — Identify performance issues.

Apply four detection rules:

1. Slow tools: tools where p95 duration > 2x the median p95 duration across all tools. List each with its p95, median, and percent above threshold.

2. Expensive operations: top 5 operations by estimated cost. Include tool name, call count, total cost, and cost per call.

3. Failed nodes: for each plan with failed_nodes > 0, list the failing span names, their Status values, and the error_message from span Metadata. Group repeated errors by pattern rather than listing each instance.

4. Budget utilization trend: compare BudgetTracker usage at start vs. end of the time range for each tier. Flag any tier where utilization increased by more than 20 percentage points within the range.

Step 7 — Generate markdown report.

Structure the report as follows:

# Agent Performance Report
Period: <time_range>
Filters: <agent|user|namespace if applied, else "none">
Generated: <ISO 8601 timestamp>

## Summary
<3-5 sentence narrative: total activity, overall cost, success rate, notable
issues. Written for a non-technical reader.>

## Tool Call Metrics
<table: tool name | calls | success rate | avg ms | p50 ms | p95 ms>

## Token and Cost Breakdown
<table: tool name | input tokens | output tokens | estimated cost>
<table: operation type | tokens | cost | % of total>

## DAG Execution
<table: metric | value>

## Performance Issues

### Slow Tools
<list or "none ">

Step 8 — Output.

Return the formatted markdown report. If the report exceeds 300 lines, offer a condensed version (summary + top issues + recommendations only) alongside the full version.

IV. Best Practices

• The narrative summary in the report header matters as much as the tables. A reviewer scanning under time pressure will read the summary and jump to recommendations. Make both precise and complete.
• Recommendations must be specific. "Optimize slow tools" is not a recommendation. "Consider replacing web_search with a cached lookup for the 3 queries called more than 10 times this week" is a recommendation.
• When span data is incomplete (missing OTEL attributes, partial records), say so explicitly in the affected section rather than computing metrics from partial data without caveat. Partial accuracy is more dangerous than acknowledged gaps.
• Group error patterns before listing individual failures. Five instances of "connection timeout on mcp_by_dojo:github_search" is one pattern, not five findings.
• Token usage by model is the most actionable cost metric. Different models have different cost-per-token rates; the same token count can represent significantly different spend depending on which model was used.
• If the time range spans multiple calendar months, break down cost and budget utilization by month. Monthly budget resets make cross-month aggregates misleading for capacity planning.

V. Quality Checklist

Before completing this skill, verify:

• Time range confirmed with user or defaults applied with explicit note
• All three data sources queried: TraceStorage spans, orchestration events, BudgetTracker snapshots
• Duration statistics computed per-tool (not only globally)
• Cost breakdown includes both by-tool and by-operation-type views
• DAG parallel_ratio reported or noted as unavailable with reason
• All four performance issue types checked (slow tools, expensive ops, failed nodes, budget trend)
• Recommendations are specific and numbered (3-5 items)
• Report includes generated timestamp and applied filters
• Incomplete span data disclosed in affected sections, not silently omitted

Output

• Markdown report returned inline (or offered as file if > 300 lines)
• Sections: Summary, Tool Call Metrics table, Token and Cost Breakdown tables, DAG Execution table, Performance Issues (slow tools, expensive ops, failed nodes, budget trend), Recommendations (numbered 3-5 items)
• Generated timestamp and applied filters included in report header

Examples

Scenario 1: "Give me a performance report for last week" → Full markdown report covering all agents for the prior 7-day window, with tool latency tables, cost breakdown by operation type, and 5 actionable recommendations. Scenario 2: "Why did costs spike on April 3rd?" → Report filtered to 2026-04-03, identifying the top 5 expensive operations, any failed nodes that triggered retries, and the binding budget tier.

Edge Cases

• If TraceStorage has no spans for the requested time range, report this explicitly rather than returning empty tables.
• If DAG parallel_ratio cannot be computed from span data, report it as "unavailable (insufficient span data)" rather than approximating.
• If time range spans multiple months, break down cost and budget utilization by month — monthly budget resets make cross-month aggregates misleading.

Anti-Patterns

• Using this skill for real-time monitoring during an active session — use tool-intercept-logger instead, which captures live per-call data.
• Listing every individual span failure rather than grouping repeated errors by pattern — five instances of the same timeout is one finding, not five.