원클릭으로 Manus에서 모든 스킬 실행

metrics-eval

스타4

포크0

업데이트2026년 6월 23일 03:02

Run the golden set through the real loop in a network-captured sandbox and compute/interpret the CI metrics — ACR (north star), TCVR, FPVR, MTC, WCR, TCR, ESC — and enforce the hard gates. Use this whenever you measure aegis-cli: a CI run, a model/harness bake-off, investigating a regression, or interpreting a dashboard move. Trigger it before claiming something "works", and any time you swap the model, harness, or prompt.

설치

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

Manus에서 실행

출처

rtmx-ai

rtmx-ai/aegis-cli

GitHub 저장소 열기 Creator 저장소 보기

다운로드

Manus에서 실행

metrics-eval

aegis-cli is measured by running it. Every CI run executes the golden set (eval/golden/) — frozen requirements with known-correct verify outcomes — through the real loop inside a network-captured sandbox, then computes the metrics. This is intent-bench methodology turned on aegis-cli itself: not unit assertions, but "does the whole loop autonomously close known work without egress?" Driver: scripts/ci-metrics.py.

The north star and the dashboard

ACR — Autonomous Completion Rate (closed-by-verify ÷ attempted, no human step) is the number we optimize. Everything else exists to explain a move in ACR, not to be optimized on its own.
Dashboard, trended every run: TCVR (tool-call validity, ↑), FPVR (first-pass verify, ↑), MTC (mean turns-to-close, ↓), WCR (wall-clock per req, ↓ — CPU-bound signal), TCR (token cost per req, ↓), ESC (escalation rate, ↓).

The hard gates (any failure fails the build)

EGRESS = 0. Any non-loopback packet during the run fails CI — the ITAR control as a test, via the same capture as airgap-hygiene. Non-negotiable.
TRACE = 100%. rtmx health passes — no orphaned requirements or tests.
ACR-regression. ACR must not fall more than the configured delta below the rolling baseline (eval/baseline.json). Catches model/harness/prompt regressions.

A green run is not "tests passed" — it is all three gates held and ACR within tolerance.

Reading the numbers, especially in a bake-off

TCVR and MTC are the leading indicators when you swap model or harness. A model change usually shows in TCVR (is it still emitting valid tool calls?) and MTC (is it wandering more?) before it shows in ACR. So in a bake-off, don't wait for ACR to move — watch TCVR and MTC first; a TCVR drop or MTC climb predicts the ACR hit. WCR/TCR tell you the cost of whatever quality you got (see context-discipline for why they're CPU-bound), and they decompose into per-stage timing (serving-calibration) so you can see where the cost is.

Example

Bake-off: swap the harness from opencode to Goose, run scripts/ci-metrics.py --golden eval/golden --baseline eval/baseline.json. EGRESS=0 and TRACE=100% hold. ACR is flat, but TCVR drops 6 points and MTC climbs — Goose is emitting more malformed calls and wandering more, paying it back in extra turns. The leading indicators caught a regression the north star hadn't surfaced yet. Keep opencode; record why in the baseline notes.

이 저장소의 다른 Skills

같은 저장소

discovery

rtmx-ai/aegis-cli

Integrate continuous discovery/framing (design thinking) into the rtmx delivery loop. Use this whenever deciding WHAT to build next, before authoring requirements, when reviewing parked work or metrics to reframe, or any time you are tempted to let the loop invent its own work. Pairs with decomposition (the split mechanics) — this owns the framing discipline.

2026-06-234

airgap-hygiene

rtmx-ai/aegis-cli

Write code that makes ZERO external network calls — only loopback to the local model endpoint is permitted — and be able to prove it. Use this whenever you add a dependency, touch config, write anything that could open a socket, or wire up CI's egress gate. Trigger it on any import, any URL, any default that could phone home, and any time you need to demonstrate the build is closed.

2026-06-234

build-to-spec

rtmx-ai/aegis-cli

The core implementation discipline — claim ONE requirement, make the minimal change that satisfies its acceptance test, run the test, verify, release. Use this every time you implement a requirement: it is the inner loop. Trigger it whenever you are about to write code, and especially when you feel the pull to fix a nearby thing, refactor "while you're here", or tweak a test that's in your way.

2026-06-234

context-discipline

rtmx-ai/aegis-cli

Keep context lean for a small, CPU-bound local MoE (~4B active) — prefer LSP/grep/symbol lookup over dumping whole files, scope to one requirement, spend tokens like they're metered. Use this whenever you gather context to work a requirement: reading code, searching, deciding what to put in the prompt. Trigger it any time you're about to read a whole file, paste large output, or widen scope beyond the one requirement in hand.

2026-06-234

decomposition

rtmx-ai/aegis-cli

Break a coarse requirement into atomic children — as PROPOSALS a human approves, never as work the machine authors and then grades itself on. Use this whenever a requirement is too big to close in one scoped pass, when splitting work into atomic units, or when running `aegis propose`. Trigger it any time the agent is tempted to invent sub-requirements, write its own acceptance criteria, or decide on its own what "atomic" means.

2026-06-234

go-conventions

rtmx-ai/aegis-cli

aegis-cli's Go style — one static air-gappable binary, std-lib-first, vendored deps, no CGO unless a serving probe needs it, and no telemetry ever. Use this whenever you write or review Go in this repo: choosing a dependency, structuring a package, deciding whether logic belongs here at all. Trigger it before reaching for a third-party library, before adding a build tag, and any time a task starts to feel like harness logic.

2026-06-234

name

metrics-eval

description

metrics-eval

The north star and the dashboard

ACR — Autonomous Completion Rate (closed-by-verify ÷ attempted, no human step) is the number we optimize. Everything else exists to explain a move in ACR, not to be optimized on its own.
Dashboard, trended every run: TCVR (tool-call validity, ↑), FPVR (first-pass verify, ↑), MTC (mean turns-to-close, ↓), WCR (wall-clock per req, ↓ — CPU-bound signal), TCR (token cost per req, ↓), ESC (escalation rate, ↓).

The hard gates (any failure fails the build)

EGRESS = 0. Any non-loopback packet during the run fails CI — the ITAR control as a test, via the same capture as airgap-hygiene. Non-negotiable.
TRACE = 100%. rtmx health passes — no orphaned requirements or tests.
ACR-regression. ACR must not fall more than the configured delta below the rolling baseline (eval/baseline.json). Catches model/harness/prompt regressions.

A green run is not "tests passed" — it is all three gates held and ACR within tolerance.