benchmark-optimization-loop

Sterne237.500

Forks36.090

Aktualisiert17. Juli 2026 um 21:13

Use when the user asks to make something faster, try many variants, run recursive optimization, benchmark latency/throughput/cost, or choose the best implementation by repeated measured tests.

Installation

Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.

Quelle

affaan-m

affaan-m/ECC

GitHub-Repository öffnen Creator-Repositorys ansehen

Download

Verwandte BerufeSOC

Basierend auf der SOC-Berufsklassifikation

SoftwareentwicklerInformatik- und Mathematikberufe·SOC 15-1252

SKILL.md

readonly

name	benchmark-optimization-loop
description	Use when the user asks to make something faster, try many variants, run recursive optimization, benchmark latency/throughput/cost, or choose the best implementation by repeated measured tests.
license	MIT
metadata	{"origin":"ECC"}
tools	Read, Write, Edit, Bash, Grep, Glob

Benchmark Optimization Loop

Use this skill to convert "make it 20x faster" or "try 50 recursive optimizations" into a bounded measured loop that can actually improve a system.

Required Baseline

Do not optimize until these exist:

the operation being optimized;
the correctness gate that must stay green;
the metric: wall time, p95 latency, rows/sec, cost/run, memory, error rate;
the current baseline;
the search budget: max variants, max time, max spend, max data impact.

If the user asks for an unrealistic target, keep the ambition but make the loop bounded and measurable.

Loop

Measure the baseline.
Identify bottlenecks from evidence.
Generate variants that test one hypothesis each.
Run variants with the same input shape.
Reject variants that fail correctness, safety, or reproducibility.
Promote the fastest safe variant.
Codify the winning path in a script, command, test, config, or doc.
Rerun the baseline and winner to confirm the delta.

Variant Table

Track variants like this:

Variant | Hypothesis | Command | Time | Correct? | Notes
baseline | current path | npm run job | 120s | yes | stable
batch-500 | fewer round trips | npm run job -- --batch 500 | 42s | yes | winner
parallel-8 | more workers | npm run job -- --workers 8 | 31s | no | rate limited

Recursive Search

For recursive or hyperparameter work:

persist every run to a ledger;
compare against the prior accepted winner, not only the previous run;
keep a holdout or replay check;
stop when improvement is within noise, correctness fails, cost exceeds the budget, or the search starts changing more variables than it can explain.

Use phrases like "best measured safe variant" instead of "global optimum" unless the search space was actually exhaustive.

Promotion Gate

A variant cannot become the new default until:

correctness tests pass;
the performance delta is repeated or explained;
rollback is obvious;
the change is encoded in source control or a durable runbook;
the final summary includes exact commands and measurements.

Mehr aus diesem Repository

gleiches Repository

continuous-learning-v2

affaan-m/ECC

Instinct-based learning system that observes sessions via hooks, creates atomic instincts with confidence scoring, and evolves them into skills/commands/agents. v2.1 adds project-scoped instincts to prevent cross-project contamination.

2026-07-29237.5k

unified-memory

affaan-m/ECC

Share durable, inspectable context and handoffs between Claude, Codex, Hermes, Cursor, OpenCode, and other agents through the local ECC Memory Vault. Use when an agent must save work state, transfer context, resume another agent's task, or search shared project knowledge.

Use when multiple consumers and providers must evolve an API or event schema without field drift, integration surprises, or one side silently redefining the interface.

2026-07-26237.5k

ito-compute

affaan-m/ECC

Query live GPU inventory, submit an authenticated Itô fixed-rate RFQ, inspect RFQ or procurement status, and run explicitly gated node qualification through the separately installed canonical CLI. Use when a user asks to find H100/H200 capacity, request a fixed compute rate, check Itô compute status, or validate GPU nodes.

2026-07-24237.5k

Benchmark Optimization Loop

Use this skill to convert "make it 20x faster" or "try 50 recursive optimizations" into a bounded measured loop that can actually improve a system.

Required Baseline

Do not optimize until these exist:

the operation being optimized;

the correctness gate that must stay green;

the metric: wall time, p95 latency, rows/sec, cost/run, memory, error rate;

the current baseline;

the search budget: max variants, max time, max spend, max data impact.

If the user asks for an unrealistic target, keep the ambition but make the loop bounded and measurable.

Loop

Measure the baseline.

Identify bottlenecks from evidence.

Generate variants that test one hypothesis each.

Run variants with the same input shape.

Reject variants that fail correctness, safety, or reproducibility.

Promote the fastest safe variant.

Codify the winning path in a script, command, test, config, or doc.

Rerun the baseline and winner to confirm the delta.

Variant Table

Track variants like this:

Recursive Search

For recursive or hyperparameter work:

persist every run to a ledger;

compare against the prior accepted winner, not only the previous run;

keep a holdout or replay check;

stop when improvement is within noise, correctness fails, cost exceeds the budget, or the search starts changing more variables than it can explain.

Use phrases like "best measured safe variant" instead of "global optimum" unless the search space was actually exhaustive.

Promotion Gate

A variant cannot become the new default until:

correctness tests pass;

the performance delta is repeated or explained;

rollback is obvious;

the change is encoded in source control or a durable runbook;

the final summary includes exact commands and measurements.