proxy-usage-analysis

name

proxy-usage-analysis

description

Analyze the last 24h of llama-proxy logs (/var/log/llama-proxy/proxy.log*) into per-session fast/cheap CSVs and a Markdown report with data-backed configuration recommendations and an error taxonomy with remediation recommendations. Trigger on user queries such as: 'analyze proxy usage', 'proxy usage report', 'why is the proxy falling back so much', 'proxy fallback analysis', 'local model utilization', 'analyze proxy errors', 'error taxonomy', 'why is the proxy erroring so much', 'generate the daily proxy report'.

Proxy Usage Analysis

Turn the last 24 hours of llama-proxy session and fallback activity into a digestible per-session CSV record (split into fast / cheap buckets per the slot schedule) and an operator-facing Markdown report with highlighted, data-backed recommendations — including an error taxonomy and quantified remediation recommendations for any proxy errors observed in the window (generalized from the Aug 3 error-analysis plan, LP-0MSDFKCK4007CPMY).

When to use

• An operator wants a quick daily read on whether the local models (llama-server via the proxy) are used well, how much traffic fell back to remote providers, why, and what to change. The Local model utilization section answers "how much of the time was the local model busy?" (busy %, idle %, streams, concurrency, hourly/fast-cheap profile).
• Investigating slot counts (fast / cheap, as configured in the slot_schedule of proxy/config.yaml), context limits, or routing thresholds: the report's fallback-reason breakdown and context pressure stats show whether configuration changes are warranted.
• An error spike occurred (e.g. Stream finished: reason=error, slot_save ReadTimeouts, backend_retry timeouts, upstream 429s): the report's Error analysis section categorizes every error event and recommends remediation (recovery-first silent continue, informative-error fallback, ctx-size pressure, upstream 429 cooldown), quantified from the window.

Inputs

• Log source (read-only): /var/log/llama-proxy/proxy.log plus rotated siblings (proxy.log.YYYY-MM-DD_HH, 6-hourly rotation, 90-day retention).
• Config (reference only, for slot schedule + thresholds): proxy/config.yaml in the llm project (auto-discovered by walking up from the current directory, or pass --config). When the persisted operating mode (proxy/.mode) selects a profile, the mode-selected config (config-fast.yaml / config-cheap.yaml) is read instead, so bucketing and recommendations match the config the running proxy actually uses.

Usage

python3 .pi/skills/proxy-usage-analysis/scripts/analyze_proxy_usage.py \
    --log-dir /var/log/llama-proxy \
    --hours 24

Outputs go to ~/proxy-usage-reports/ by default (override with --output-dir).

Options:

Flag	Default	Purpose
--log-dir	/var/log/llama-proxy	Directory containing proxy.log*
--llama-log-dir	/var/log/llama-proxy	Directory containing llama-server.log* for decode/prompt-eval speed stats (falls back to --log-dir when omitted)
--hours	24	Analysis window length
--start / --end	—	Explicit ISO window (YYYY-MM-DD HH:MM:SS); overrides --hours
--output-dir	~/proxy-usage-reports	Where the CSVs and report are written
--config	auto-discovered	Path to proxy/config.yaml
--json	off	Print a machine-readable JSON summary instead of the text summary
--quiet	off	Suppress the stdout summary

Automated daily run

A cron job runs the report automatically every day at 05:00 (output logged to ~/proxy-usage-reports/cron.log so each run's summary and any failures are visible):

0 5 * * * cd /home/rgardler/projects/llm && python3 .pi/skills/proxy-usage-analysis/scripts/analyze_proxy_usage.py >> ~/proxy-usage-reports/cron.log 2>&1

Each run archives the previous day's outputs into a dated subdirectory (see Archival), so a historical daily report accumulates under ~/proxy-usage-reports/YYYY-MM-DD/. cron.log is not an analysis artifact and stays at the root, untouched by archival.

Outputs

Written to --output-dir (default ~/proxy-usage-reports):

• fast_sessions.csv — one row per fast session (the period(s) with the fewest slots per the configured slot_schedule in the active config profile). One row per session, covering ALL sessions in the window (local-only and fallback).
• cheap_sessions.csv — one row per cheap session (the period(s) with the most slots; when all periods have the same slot count there is no cheap bucket and this file is not produced).
• errors.csv — one row per error event in the window (stream finish errors, stream errors, slot_save failures, backend_retry timeouts, upstream HTTP errors), with error type, timestamp, provider/model, session, config entry, error detail, HTTP status, retry attempt/signal, source log file, and the raw evidence line.
• errors.json — aggregated error counts by type plus a provider/model breakdown (nested {error_type: {provider: {model: count}}}; providers or models not derivable from the log line are keyed (unknown)) plus the window bounds.
• report.md — the aggregate report: a single Session summary table (sessions, requests, local/remote split, classifications, fallback events, dispatch denials, context sizes — each with Total / Fast / Cheap columns), fallback-reason and routing-skip breakdowns, per-model breakdown, Error analysis (when the window has error events) with a taxonomy table plus a Provider/model breakdown table (error type × provider × model × count), Local model utilization (busy time %, idle time, streams served, avg stream duration, total compute, avg/peak concurrency, hourly busy profile, fast/cheap split — when the window has local traffic), Decode speed and Prompt eval speed sections (median / p90 / p10 tok/s from llama-server eval-timing lines, split Total / Fast / Cheap), and highlighted recommendations. Every fast/cheap count carries its share of the metric's total (e.g. 285 (74.4%)), and each recommendation's evidence cites the total plus the fast/cheap split.

CSV columns: session id, start/end time, duration, number of messages, start/avg/max context size, avg/max response size, initial model assignment (provider + model), time of move to a remote model (empty if never fell back), fallback reason (empty if never fell back), bucket, slots, local/remote request counts, dispatch denials, decode tok/s (derived from local completion tokens ÷ local active span; empty when not derivable).

Archival

Before writing fresh outputs, the script moves any existing artifacts (report.md, fast_sessions.csv, cheap_sessions.csv, errors.csv, errors.json) into a dated subdirectory named by the run date (YYYY-MM-DD/); when that directory already exists (a same-day repeat, or a manual archive), a _2, _3 … suffix is appended so archives are never overwritten. Only the skill's own artifacts are moved — anything else in the output dir (e.g. cron.log) stays put, and a pristine output dir is left touched-free (no empty archive dirs). The CLI prints the archive path on each run (Previous outputs archived to …).

How it works

1. File discovery — the live proxy.log plus every rotated sibling (proxy.log.YYYY-MM-DD_HH). All rotated files are included regardless of their name-encoded timestamp: in this deployment a rotated file routinely holds data well past its encoded rotation time, so a name-based inclusion test would silently drop in-window data. Per-line timestamp filtering in step 2 is the authoritative window boundary.
2. Streaming parse — files are read line by line (never loaded into memory; the live log can exceed 700 MB). Only structured prefixes are parsed: Stream started, Stream finished, Fallback triggered, routing_skip_local, local_dispatch_denied, plus the error lines (Stream error:, slot_save failed, backend_retry, [remote] upstream error). Unparseable lines are counted and skipped, never fatal.
3. Session grouping — a session is identified by its UUID (session=<uuid>). Per-session context/response sizes use the authoritative tokens=prompt/completion/total from Stream finished lines (payloads in logs are truncated and never used for sizes).
4. Local model utilization (busy time) — local Stream started / Stream finished events are collected across a 1h margin beyond the window (so streams crossing the window boundary pair correctly), paired per session (FIFO), clipped back to the window, and merged. Busy time is the union of active intervals (at least one slot generating), total compute is the sum of clipped stream durations (slot-seconds), and peak concurrency comes from a sweep over interval endpoints. Busy seconds are attributed to hours and to fast/cheap periods (slot schedule) by splitting at hour and period boundaries.
5. Fast/cheap bucketing — derived from the slot_schedule in the active config profile (proxy/config-fast.yaml or proxy/config-cheap.yaml; transition times and slot counts are read from config; the period(s) with the fewest slots are labelled "fast", the period(s) with the most "cheap"; equal counts collapse to a single fast bucket), keyed by session start time; nothing is hardcoded.
6. Recommendations — rule-based heuristics, each citing the data that supports it (see below).
7. Error taxonomy — error events (Stream finished: reason=error, Stream error:, , , ) are parsed in the same streaming pass, collected per window, and rendered into the report's section (taxonomy table plus a table) plus /. Provider/model attribution is best effort: and lines carry / directly; is always the local llama-server (provider , model not in the line); carries only a target URL so the provider is inferred from the endpoint (e.g. → , → , → , → ; unknown endpoints fall back to the bare hostname) and the model is unknown; carries neither. Undetermined values render as in the report and in JSON. Remediation recommendations (recovery-first, informative-error, ctx-size pressure, 429 cooldown) are generated from these events and link to the relevant work items.

Interpreting the report

• Session classification: local-only vs fell back (local → remote) vs remote-only (never used local). A high remote-only share with context_too_large reasons usually means routing thresholds are too low for the context sizes being routed.
• Fallback reasons:
  • local_concurrency_limit / local_lease_active / slot_exhaustion → slot pool contention → raise session_slot_pool_size or the slot_schedule slot counts (keep llama-server --parallel aligned).
  • large_context_bypass → prompts exceed the large-context routing thresholds / per-slot context → raise local ctx-size (models.ini), local_large_context_*_threshold, or session_slot_max_prompt_tokens. Related work item: LP-0MSAOQTJS000FFVM (evaluate increasing the local ctx-size).
  • context_too_large (legacy warm_cache_bypass in rotated logs) → estimated context exceeds the per-slot hard cap → consider raising local ctx-size (models.ini) or local_large_context_warm_cache_threshold.
  • HTTP 4xx/5xx, empty_response, timeouts → remote provider issues (credentials, rate limits) — not slot-related.
• Context pressure: sessions whose max context approaches local_model_ctx_size / slots can force large_context_bypass.
• Local model utilization: busy time is the share of the window with at least one local slot generating. A low busy % with high fallback volume means the router is diverting requests before they reach local (see fallback reasons), not that local is underprovisioned. context_too_large is the largest lever: despite the legacy name (warm_cache_bypass) it fires when the estimated context exceeds the effective warm-cache threshold (the per-slot clamp, local_model_ctx_size // slots - headroom, inflated by token_estimate_multiplier), so large-context sessions never reach local. Concurrency bursts beyond session_slot_pool_size show as local_concurrency_limit / local_lease_active fallbacks. Note the slots-vs-context trade-off: more slots shrink per-slot context and raise bypass volume, so do not add slots without a matching ctx-size increase.

Testing

Run the full suite via the test skill (canonical, cached pipeline):

/skill:test

The suite covers log-line parsing, session aggregation, fallback attribution, fast/cheap bucketing, recommendation rules, llama-server eval-timing parsing (decode + prompt eval, Qwen3 port filtering, fast/cheap speed stats), and an end-to-end run, using fixtures copied from real /var/log/llama-proxy lines (proxy.log and llama-server.log).

Limitations

• Fallback triggered lines carry no session UUID; per-session attribution prefers the session's own routing_skip_local line and otherwise the nearest fallback event within 60s of the first remote stream.
• Sessions spanning a slot-schedule transition may observe a brief restart interruption (llama-server is restarted immediately at the transition time); since LP-0MSF9RUSQ007M346 there is no drain window and no 503 rejection period.
• A session is included when it has at least one Stream started inside the window; the fast/cheap bucket is keyed by its first in-window stream.
• Busy-time pairing reads local Stream started/Stream finished events within a 1h margin of the window (see BUSY_WINDOW_MARGIN); a stream that started more than 1h before the window start is not paired, and streams whose start has no logged finish (aborted/still running) are counted in unfinished_streams and excluded — busy time is a conservative lower bound.
• Log-format drift is tolerated (missing fields default to empty), but a major format change may require updating the regexes in scripts/log_parser.py.
• llama-server.log eval-timing lines carry no timestamps, so the speed section's window filtering and fast/cheap split are approximate: each sample is bucketed by its log file's last-write time. Files whose Qwen3 child port cannot be discovered are counted and skipped (never fatal).
• The per-session CSV decode_tok_s is a session-level average over the local active span (first→last local stream event); it includes inter-request gaps, so it is a conservative lower bound of the true decode rate. It is empty for sessions with no local completions.