Agent-Loop-Skills

Use when the user has a coding prompt — a feature, bug fix, or refactor — and wants it implemented end to end by a self-checking software loop, not a single pass. It refines the prompt into an executable plan (running the plan-loop internally), then executes the plan task by task: an Engineer subagent implements each PR-sized task editing only source, a separate QA subagent authors the tests that prove the task's acceptance criteria (or grades quality only when tests already cover them), runs the task's tests plus the full accumulated regression suite, and grades the code against a strict conciseness/readability/style-match rubric; the Engineer and QA loop on each task until the tests pass and the quality gate holds, then it commits and opens a pull request for the task — stacked PRs (one per task, each based on the prior) by default, or one combined PR — and moves to the next task. Author and critic have disjoint write scopes (Engineer owns source, QA owns tests) so neither can game the gate. Not for produci

Use when the user has a coding or engineering prompt and wants it refined into a detailed, executable plan before any code is written — the planning stage of a prompt → plan → execute → debug pipeline. It decomposes the prompt from first principles (objective, end state, environment, building blocks, tools, packages), breaks the work into PR-sized tasks each tied to a component with its files, tests, and dependencies, orders them topologically, splits each into atomic subtasks, then a separate principal-engineer agent critiques the plan for alignment, coverage, sizing, and executability; it revises until the critique passes, emitting a plan.md layout and a structured tasks.json that a junior engineer or a smaller model can execute correctly. Not for executing, scaffolding, or debugging the plan (those are downstream loops), and not for research proposals or experiment plans.

Use when the user has concrete failing cases in code or a guardrail/classifier/filter/prompt/API they own — a red-team failure catalogue OR a CI/CD test-failure report (failing pytest/JUnit tests) — and wants the target patched until those failures are closed without breaking what already works. It points straight at the failed cases (normalize any source with tools/ingest.py), fixes one root-cause class per iteration, and re-checks with tools/verify.py — oracle mode against a red-team oracle, or tests mode against the test suite — keeping a patch only if it closes a class while nothing that passed before regresses, else reverting; loops until every class is closed (dry) or the budget runs out, then opens a pull request with the patch set. The defensive fixer half of a find→fix setup. Not for discovering new failures (that is red-team), and not for editing the oracle, tests, or holdout that define ground truth.

Use when the user wants to automatically harden a guardrail, classifier, content filter, prompt, or API they own by running attack and defense together as a closed loop, not just one or the other. It orchestrates the red-team and blue-team loops as independent agents: red finds distinct failure classes against the frozen target, blue patches the target to close them under a regression gate, then a fresh red pass re-verifies — confirming each class is closed and surfacing any new ones the fix introduced. The find→fix→re-verify cycle repeats until a fresh attack pass stays dry (the target is hardened) or a cycle budget is hit, then it opens a pull request with the patch set. Not for attacking a system the user is not authorized to test, and not for a one-shot scan — use red-team alone to only find, or blue-team alone to only fix.

Use when the user has scientific data (or a prompt alluding to scientific data) and wants a publication-quality figure made from it. A generator drafts and renders a figure that lands a frozen communication goal; an adversarial critic critiques it hard and grades it 1-5 per axis against a fixed rubric (message, aesthetic, clarity, integrity, and a conditional domain-completeness axis), aggregates to 0-100, and decides pass; the generator revises against the critic's findings until the grade clears a threshold or the budget is hit. Both roles may consult the literature (Semantic Scholar + arXiv) to verify domain content (e.g. a pathway figure's gene set, or a benchmark's reported numbers), and conform to / grade against a named journal's figure spec fetched via web search. Not for writing a paper or analyzing a dataset, and not for editing an existing finished image — this renders a figure from data/brief and iterates on it.

Use when the user wants an autonomous ML research loop that does more than blindly try changes. After every training run the agent analyses what actually happened inside the model — gradients, activations, embeddings, errors, data — and grounds the next change in that evidence. A `<literature>` on/off dial adds scientific-literature grounding: off behaves as a pure analysis-first loop; on searches papers, grades the evidence, and implements only what prior work supports. One change per run; loops forever until interrupted. Not for one-off training runs or hyperparameter sweeps.

Use when the user wants the LLM to do its own ML research: a fully-autonomous loop that hacks the training code, runs it, and keeps changes that lower a single scalar metric (e.g. val_bpb). One agent proposes one change at a time, runs training in the user's env, keeps it only if the metric improves (advancing a git branch) else reverts, and loops forever until the human interrupts. A faithful adaptation of Karpathy's autoresearch. Not for the analysis-first variant that profiles before editing (that is ml-autoresearch), and not for a budgeted, plateau-stopping refactor.

Use when the user wants to evolve an ML model/program through population-based search rather than a single sequential refine loop — a generational evolution where parallel proposers each apply one small SEARCH/REPLACE diff to a parent, scored by a cascade-evaluated training run, and children are kept in a MAP-Elites archive across islands (with migration + checkpointing) so diverse high performers survive. A finite, bounded-parallelism re-creation of AlphaEvolve/OpenEvolve, bent for ML autoresearch. Runs to a fixed compute budget or until interrupted. Not for the sequential single-thread autoresearch loops (one change → measure → keep/revert), and not for verifying a known bug or external claim — this is parallel, diversity-preserving search over a program.

Use when the user has a known, already-observed anomaly in their data — a metric spike or drop, an outlier, an unexpected number — and wants its root cause diagnosed, not guessed. Forms a slate of candidate causes, tests each against the data, and eliminates the ones the data refutes, narrowing the live candidates until exactly one survives refutation and passes a positive confirming test. The result is an investigation log with the confirmed root cause and the evidence that ruled out the alternatives. Not for open-ended discovery over a dataset with no specific anomaly in hand (that is data-analysis), and not for checking an external claim against sources (that is claim-verify) — this is reactive diagnosis of one anomaly you already know about.

Use when the user has a results draft or a set of data-backed claims and wants each one adversarially verified against the underlying dataset before publishing — a pre-publication red-team of the findings. Extracts the discrete checkable claims from the draft, reproduces each claim's number against the data, stress-tests it against the threats most likely to kill it (outliers, confounds, Simpson's reversals, tiny subgroups, alternative specifications), and marks it verified, fragile, or refuted; fragile and refuted claims are revised — hedged, scoped, or retracted — until every claim is verified or appropriately qualified. The result is a draft where every surviving claim has been reproduced and survived a stress test. Not for open-ended discovery of new findings over a dataset (that is a data-analysis task), and not for diagnosing a single known anomaly or pipeline failure — this is a gate over an existing draft.

Use when the user wants an iterative, self-checking exploratory analysis of a dataset — surfacing findings that are each verified by re-running the computation, not asserted. Proposes one specific hypothesis at a time, writes and runs analysis code to test it, and records the finding only if the numbers support it at a meaningful effect size; loops until no new verified finding appears or the budget is hit. The result is a findings report where every claim is backed by a reproducible number. Not for diagnosing a single known anomaly or pipeline failure, and not for verifying an external claim against sources (that is a claim-verification task) — this is open-ended discovery over a bound dataset.

Use when the user wants two approaches raced head-to-head on a single shared metric — e.g. a classical/algorithmic lane vs an ML/learned lane, or any two strategies for the same task. Each lane runs its own analysis-first research loop confined to its lane, the lanes share a scoreboard and may borrow ideas across the boundary without abandoning their identity, and a shared eval keeps the head-to-head honest; loops until interrupted, reporting the current leader. Not for improving a single approach in isolation (use a single-track research loop), and not for picking between two finished artifacts in one shot (that is a one-time comparison).

Use when the user wants an autonomous ML research loop that explores the space broadly rather than hill-climbing one approach. A temperature scheduler replaces the usual hypothesis step: it forces several wild, diverse swings (full rewrites, different architectures/regimes) early, then enters an adaptive phase that picks swing / merge / exploit per iteration — with a hard stagnation guard that bans further small-step exploits once they run too long, forcing a pivot back to a swing or merge. Tracks an approaches.md registry and a move_type per iteration; analyses every run before the next move. One change per iteration; loops forever until interrupted. Not for the standard analysis-first ml-autoresearch (which lets analysis alone choose each change), one-off training runs, or sweeps.

Use when the user wants to generate and literature-vet a pool of novel, testable research hypotheses for a question or domain. A multi-agent loop: a Generator proposes candidate hypotheses, a LiteratureScout grounds each in real retrieved literature (already known? closest prior work? what gap does it fill?), and a Judge scores them against a fixed rubric and keeps the strong, non-duplicate ones; rounds repeat — mutating toward the open gaps — until fresh rounds stop adding keepers. Not for sharpening or decomposing a research question (no grounding/scoring there), and not for grading an existing written proposal against the literature.

Use when a loop needs scholarly literature — paper discovery, novelty checks, full-text snippet search, citation-graph traversal, single-paper reads, or experimental-result extraction. A shared, stdlib-only CLI (`tools/lit_search.py`) over Semantic Scholar + arXiv (plus optional OpenAlex, Perplexity Sonar, bgpt.pro), all JSON. Not a loop itself: install it alongside the loops and let any of them call it instead of vendoring their own copy. Degrades to the caller's WebSearch/WebFetch.

Use when the user wants a structured, saturating literature survey on a question — not a one-shot summary, but an evidence/contradiction matrix (sources × claims) built by iterative search until coverage stops growing. Each round expands the search (new sub-topic queries plus citation-graph walks), admits new sources, extracts their claims with verbatim snippets, and records where every source stands on each claim (supports / contradicts / qualifies); the feedback signal is how many new matrix-changing sources a round adds, and it stops at saturation, patience, or budget. The output is the matrix plus a synthesis of consensus, disputes, and gaps, every cell backed by a real citation. Not for grading a written proposal against the literature (that is a proposal-evaluation task) and not for generating new hypotheses — this maps what the literature already says.

Use when the user wants to iteratively improve an artifact under a hard correctness bound while minimizing a measured cost — refactoring a code module to cut complexity while its test suite stays green, OR speeding up a SQL query while it returns the same rows. Each iteration applies one focused change, checks a correctness gate that must pass, measures a metric that must drop, and keeps the change only if both hold, else reverts; loops to a plateau or budget. Not for adding features, fixing bugs, or any change that is allowed to alter behaviour or results.

Use when the user is planning a two-arm comparison (an A/B test, a simple RCT, a behavioral study, or a two-model/two-config evaluation) and needs to size it and preregister it before collecting data — finding the per-group sample size that hits target statistical power for the smallest effect worth detecting, auditing the design against a validity checklist, and locking it in a preregistration. Only for a single two-arm comparison with one primary outcome. Not for factorial, repeated-measures, clustered/multilevel, time-series, adaptive, or survival designs; not for analyzing data already collected; not for choosing the outcome or manipulation from domain knowledge.

Use when the user has a prompt that feeds a system they can already score, and wants that prompt automatically improved to raise the score against their own evaluation command. Makes one targeted quality edit per iteration — clarity, context, specificity, structure, examples, decomposition, guardrails — re-runs the user's eval to measure the metric, and keeps the edit only if the metric improves, else reverts; loops to a target, plateau, or budget. The metric is whatever the user's eval command prints (task accuracy, an LLM-judge score, a pass rate, a tool-call success rate); the loop is metric-agnostic and never edits the eval. Not for writing a prompt from scratch, not for tuning model weights or hyperparameters, and not for one-off manual prompt edits without a score.

Use when the user wants to adversarially stress-test a guardrail, classifier, prompt, or API they own or are authorized to test, to surface the distinct ways it fails. Generates adversarial inputs, runs them through the target and a ground-truth oracle, logs every target-vs-oracle disagreement as a failure de-duplicated by technique class, and loops until rounds stop surfacing new classes. Produces a catalogue of distinct, reproducible failures — the attacker half of a find→fix setup. Not for patching the target, and not for attacking systems the user does not own or have permission to test.

Use when the user has a research proposal (problem + proposed methodology + planned experiments) and wants it iteratively strengthened until it clears a passing grade. ScholarEval grades the proposal against the literature (Soundness + Contribution), a Judge scores that feedback 0-100 on a fixed rubric, and a Reviser rewrites the proposal to fix the worst points without diluting the research question; loops until the grade passes or the budget is hit. Not for generating a proposal from scratch, and not for running a literature survey on its own.

Use when the user has a vague topic or area of interest and wants it sharpened into a few strong, novel, feasible research questions. Drafts candidate questions, scores each against a fixed rubric (Specific, Answerable, Novel, Feasible, Significant) with a light literature/web novelty check, and revises the weakest axis until enough questions clear the bar. Not for grading a full written proposal (use the research-proposal loop), and not for turning a question into testable predictions (use the hypothesis-generation loop).

Use when the user has a scientific draft (with its dataset, figures, and optional analysis code) and wants it iteratively revised until it clears a quality bar. Five specialist judges (figures, scientific content, style, formatting, code) critique the draft; a fresh, independent peer_reviewer grades it on those same axes (1-5 each → a percentage) with honesty guardrails; a scientific_writer revises prose, figures, and code — regenerating figures by running the user's plot command — until the peer-review score clears the threshold or the budget is hit. Not for writing a paper from a blank page, and not for a standalone literature survey.

Use when the user has a messy tabular data dump (CSV/TSV/parquet/Excel/JSON) and wants it iteratively cleaned to an inferred data contract — a checklist of deterministic pass/fail checks, not a quality score. A single agent profiles the table, synthesizes a per-column contract compiled into binary checks (types, nulls, duplicates, inconsistent categories, format/range violations, outliers), then applies one targeted transform at a time, keeping it only if it reduces its target check's violations with no regression and no guardrail breach. Stops deterministically when every check passes, every remaining check is an unfixable residual, or a budget is hit; emits a replayable pipeline and an auditable ledger. Not for open-ended analysis of an already-clean dataset, diagnosing one known anomaly, or verifying a claim against sources — those are analytical loops; this rewrites the data to a contract.

Use when the user wants an autonomous ML research loop that pressure-tests competing ideas before spending compute — several research subagents each propose one architecture change, a self-calibrating Judge critiques them against a rubric, the proposers refine, and the Judge picks the single change to run. The Judge learns to pick better over time by scoring its own predictions against realized metric deltas, recording predicted-vs-realized in a calibration ledger and refining its working rubric. The result is an experiment ledger where each iteration's change won a de-biased tournament. Not for running a single pre-decided experiment, and not for analysis-only exploration — for one hypothesis proposed and run per iteration without competition, use the sibling ml-autoresearch loop.