Ejecuta cualquier Skill en Manus
con un clic

Ejecuta cualquier Skill en Manus con un clic

metrics-eval

Estrellas4

Forks0

Actualizado23 de junio de 2026 a las 03:02

Run the golden set through the real loop in a network-captured sandbox and compute/interpret the CI metrics — ACR (north star), TCVR, FPVR, MTC, WCR, TCR, ESC — and enforce the hard gates. Use this whenever you measure aegis-cli: a CI run, a model/harness bake-off, investigating a regression, or interpreting a dashboard move. Trigger it before claiming something "works", and any time you swap the model, harness, or prompt.

Instalación

Instalar con Codex o Claude Copia este prompt, pégalo en Codex, Claude u otro asistente, y deja que revise la página de la skill y la instale por ti.

Ejecutar en Manus

Fuente

rtmx-ai

rtmx-ai/aegis-cli

Abrir repositorio de GitHub Ver repositorios del creador

Descarga

Ejecutar en Manus

Ocupaciones relacionadasSOC

Basado en la clasificación ocupacional SOC

Analistas de garantía de calidad de software y probadoresOcupaciones informáticas y matemáticas·SOC 15-1253

SKILL.md

readonly

name

metrics-eval

description

metrics-eval

aegis-cli is measured by running it. Every CI run executes the golden set (eval/golden/) — frozen requirements with known-correct verify outcomes — through the real loop inside a network-captured sandbox, then computes the metrics. This is intent-bench methodology turned on aegis-cli itself: not unit assertions, but "does the whole loop autonomously close known work without egress?" Driver: scripts/ci-metrics.py.

The north star and the dashboard

ACR — Autonomous Completion Rate (closed-by-verify ÷ attempted, no human step) is the number we optimize. Everything else exists to explain a move in ACR, not to be optimized on its own.
Dashboard, trended every run: TCVR (tool-call validity, ↑), FPVR (first-pass verify, ↑), MTC (mean turns-to-close, ↓), WCR (wall-clock per req, ↓ — CPU-bound signal), TCR (token cost per req, ↓), ESC (escalation rate, ↓).

The hard gates (any failure fails the build)

EGRESS = 0. Any non-loopback packet during the run fails CI — the ITAR control as a test, via the same capture as airgap-hygiene. Non-negotiable.
TRACE = 100%. rtmx health passes — no orphaned requirements or tests.
ACR-regression. ACR must not fall more than the configured delta below the rolling baseline (eval/baseline.json). Catches model/harness/prompt regressions.

A green run is not "tests passed" — it is all three gates held and ACR within tolerance.

Reading the numbers, especially in a bake-off

TCVR and MTC are the leading indicators when you swap model or harness. A model change usually shows in TCVR (is it still emitting valid tool calls?) and MTC (is it wandering more?) before it shows in ACR. So in a bake-off, don't wait for ACR to move — watch TCVR and MTC first; a TCVR drop or MTC climb predicts the ACR hit. WCR/TCR tell you the cost of whatever quality you got (see context-discipline for why they're CPU-bound), and they decompose into per-stage timing (serving-calibration) so you can see where the cost is.

Example

Bake-off: swap the harness from opencode to Goose, run scripts/ci-metrics.py --golden eval/golden --baseline eval/baseline.json. EGRESS=0 and TRACE=100% hold. ACR is flat, but TCVR drops 6 points and MTC climbs — Goose is emitting more malformed calls and wandering more, paying it back in extra turns. The leading indicators caught a regression the north star hadn't surfaced yet. Keep opencode; record why in the baseline notes.

Más de este repositorio

mismo repositorio

discovery

rtmx-ai/aegis-cli

Integrate continuous discovery/framing (design thinking) into the rtmx delivery loop. Use this whenever deciding WHAT to build next, before authoring requirements, when reviewing parked work or metrics to reframe, or any time you are tempted to let the loop invent its own work. Pairs with decomposition (the split mechanics) — this owns the framing discipline.

2026-06-234

airgap-hygiene

rtmx-ai/aegis-cli

Write code that makes ZERO external network calls — only loopback to the local model endpoint is permitted — and be able to prove it. Use this whenever you add a dependency, touch config, write anything that could open a socket, or wire up CI's egress gate. Trigger it on any import, any URL, any default that could phone home, and any time you need to demonstrate the build is closed.

2026-06-234

build-to-spec

rtmx-ai/aegis-cli

The core implementation discipline — claim ONE requirement, make the minimal change that satisfies its acceptance test, run the test, verify, release. Use this every time you implement a requirement: it is the inner loop. Trigger it whenever you are about to write code, and especially when you feel the pull to fix a nearby thing, refactor "while you're here", or tweak a test that's in your way.

2026-06-234

context-discipline

rtmx-ai/aegis-cli

Keep context lean for a small, CPU-bound local MoE (~4B active) — prefer LSP/grep/symbol lookup over dumping whole files, scope to one requirement, spend tokens like they're metered. Use this whenever you gather context to work a requirement: reading code, searching, deciding what to put in the prompt. Trigger it any time you're about to read a whole file, paste large output, or widen scope beyond the one requirement in hand.

2026-06-234

decomposition

rtmx-ai/aegis-cli

Break a coarse requirement into atomic children — as PROPOSALS a human approves, never as work the machine authors and then grades itself on. Use this whenever a requirement is too big to close in one scoped pass, when splitting work into atomic units, or when running `aegis propose`. Trigger it any time the agent is tempted to invent sub-requirements, write its own acceptance criteria, or decide on its own what "atomic" means.

2026-06-234

go-conventions

rtmx-ai/aegis-cli

aegis-cli's Go style — one static air-gappable binary, std-lib-first, vendored deps, no CGO unless a serving probe needs it, and no telemetry ever. Use this whenever you write or review Go in this repo: choosing a dependency, structuring a package, deciding whether logic belongs here at all. Trigger it before reaching for a third-party library, before adding a build tag, and any time a task starts to feel like harness logic.

2026-06-234

name

metrics-eval

description

metrics-eval

The north star and the dashboard

ACR — Autonomous Completion Rate (closed-by-verify ÷ attempted, no human step) is the number we optimize. Everything else exists to explain a move in ACR, not to be optimized on its own.
Dashboard, trended every run: TCVR (tool-call validity, ↑), FPVR (first-pass verify, ↑), MTC (mean turns-to-close, ↓), WCR (wall-clock per req, ↓ — CPU-bound signal), TCR (token cost per req, ↓), ESC (escalation rate, ↓).

The hard gates (any failure fails the build)

EGRESS = 0. Any non-loopback packet during the run fails CI — the ITAR control as a test, via the same capture as airgap-hygiene. Non-negotiable.
TRACE = 100%. rtmx health passes — no orphaned requirements or tests.
ACR-regression. ACR must not fall more than the configured delta below the rolling baseline (eval/baseline.json). Catches model/harness/prompt regressions.

A green run is not "tests passed" — it is all three gates held and ACR within tolerance.