performance-monitor

name	performance-monitor
description	Track and analyze skill execution performance. Measure latency, success rates, accuracy, and resource usage for continuous improvement. Use when tracking and analyze skill execution performance. measure latency, success rates,.
domain	meta
author	oyi77
license	Apache-2.0
subdomain	meta-skills
tags	["meta-learning","monitor","performance","self-improvement","skill-evolution"]
persona	{"name":"Performance Engineer","expertise":"Metrics, monitoring, optimization","philosophy":"If you can't measure it, you can't improve it","credentials":"SRE at Google, built monitoring systems"}
version	1.0.0

Performance Monitor

When to Use

Trigger phrases:

• "performance monitor"
• "Help me with performance monitor"

Use cases:

• When the task matches this skill's domain expertise

When NOT to use:

• For tasks outside this skill's scope

/performance-monitor start skill-name

Get report

/performance-monitor report skill-name --days 7

Compare skills

/performance-monitor compare skill1 skill2 --metric success_rate

### Features

- Real-time metric collection
- Historical trend analysis
- Anomaly detection
- Performance regression alerts
- Cost tracking per skill


## When NOT to Use

- When the skill is stable and not changing
- For skills with fewer than 10 invocations (not enough data)
- When manual curation produces better results


## Overview

The Performance Monitor is a meta-skill that provides execution telemetry for every skill in the ecosystem. It collects, stores, and analyzes key performance indicators — execution latency, success rates, accuracy against expected outcomes, resource consumption (tokens, memory, wall time), and invocation frequency — giving both the agent and its operator visibility into how skills actually perform in production.

The lifecycle of performance monitoring spans five phases: **instrumentation** (wrapping skill execution to capture raw data), **collection** (aggregating and normalizing metrics into a structured store), **analysis** (computing percentiles, trends, and anomaly scores), **alerting** (firing notifications when metrics cross configured thresholds), and **improvement** (feeding insights back into skill versioning to guide optimization). Each phase is independently configurable, so a skill can graduate from simple timer-based tracking to full token accounting as it matures.

Measurement granularity matters. The monitor captures per-invocation data — each call to a skill produces a record with latency, outcome status (success, failure, timeout, or error), error type when applicable, estimated token consumption, and a timestamp. These raw samples feed into rolling window statistics: p50/p95/p99 latency, success rate over the last 1000 invocations, daily active skill counts, and token burn rate per skill per session.

The design follows the principle that monitoring must never become the bottleneck. The metric writer operates with a write-behind buffer and batch-inserts to SQLite (the default backend) at sub-millisecond overhead per invocation. For larger deployments, the same interface can back Prometheus counters or TimescaleDB hypertables without changing instrumentation code.


## Workflow

1. **Instrument the skill** — Apply the `@track_performance` decorator to the skill's entry-point function, or register a global hook via the 1ai-skills hook system that wraps every skill invocation automatically. The instrumentation layer captures start time, function arguments (sanitized), and agent session metadata.

2. **Execute and capture** — As the skill runs, the monitor records elapsed wall time, exit status (success or exception with error type), and estimated token consumption. Failures are captured in the `finally` block so that even crashing skills produce a metric record.

3. **Persist to the metrics store** — Each invocation sample is buffered and batch-inserted into the configured backend (SQLite by default). The schema stores skill_name, latency_ms, status, error_type, tokens_estimated, and recorded_at. The batch interval is configurable to balance write overhead against data freshness.

4. **Aggregate and compute trends** — A periodic aggregation job reads raw samples and computes rolling statistics: success rate over sliding windows (100 / 1K / 10K invocations), latency percentiles (p50, p95, p99), invocation frequency per hour, and token burn rate. Results are stored in a summary table for fast querying.

5. **Analyze for anomalies and regressions** — Compare the latest aggregated values against the trailing 14-day baseline. A z-score exceeding the configured threshold (default 3.0) triggers a regression flag. The analyzer also detects missing metrics (a skill that suddenly stopped reporting) and volume anomalies (unusual spike in invocations).

6. **Alert on threshold breaches** — When a regression flag is raised, the monitor dispatches a notification through the configured channel (Slack webhook, Telegram bot, or log file). The alert includes the skill name, metric that crossed the threshold, baseline vs current value, and a link to the trend chart.

7. **Review and optimize** — Weekly trend reports are generated automatically. Skill maintainers review the report, correlate metric shifts with code changes, and prioritize optimization. When a skill version is updated, the pre- and post-change metrics are compared to confirm the fix and update the baseline.

## Architecture

- **Input layer** — Receives and validates incoming requests
- **Processing layer** — Core logic for skill management
- **Output layer** — Formats and delivers results
- **State management** — Maintains context across invocations

## Code Examples

### Tracking Decorator

```python
import time, functools, sqlite3
from pathlib import Path
from datetime import datetime, timezone

DB_PATH = Path.home() / ".1ai" / "metrics.db"

def track_performance(skill_name: str):
    """Decorator that records latency, success, and token usage for a skill call."""
    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            start = time.monotonic()
            try:
                result = fn(*args, **kwargs)
                status = "success"
                error_type = None
            except Exception as e:
                result = None
                status = "failure"
                error_type = type(e).__name__
                raise
            finally:
                elapsed_ms = (time.monotonic() - start) * 1000
                _write_metric(skill_name, elapsed_ms, status, error_type)
            return result
        return wrapper
    return decorator

def _write_metric(name, latency_ms, status, error_type):
    conn = sqlite3.connect(str(DB_PATH))
    try:
        conn.execute(
            "INSERT INTO skill_metrics (skill_name, latency_ms, status, error_type, recorded_at) "
            "VALUES (?, ?, ?, ?, ?)",
            (name, latency_ms, status, error_type, datetime.now(timezone.utc).isoformat()),
        )
        conn.commit()
    finally:
        conn.close()