coordinator-perf-multiagent

name

coordinator-perf-multiagent

description

Use when the coordinator runs a PARALLEL multi-agent performance-recovery campaign on decimal-scaled — recovering many benched (bench-branch-compare) regressions at once by partitioning the work across worktree agents, one per policy/function area, with CPU-core-pinned concurrent microbenching. Covers the disjoint partition, int-tier-first holistic ordering, the core-pin scheme to keep concurrent benches under load, the per-agent loadout, and the coordinator's review→gate→merge→push→bbc→refresh-table loop. Builds on subagent-dispatch (per-agent rules), algorithim-optimiser (kernels), architectural-review (compliance), hunters (research lenses).

Coordinator: parallel performance multi-agent wave

How the coordinator runs a fleet of worktree agents to claw back regressions vs the prior release in parallel, without melting the CPU or the architecture. This is the coordinator-side playbook; the agents themselves follow [[subagent-dispatch]] + [[algorithim-optimiser]] + [[architectural-review]] + [[hunters]].

Use it when the bench-branch-compare table has many regressed cells spread across several functions/policies and you want to attack them concurrently rather than one at a time.

Per-op pipeline — INT tier (1→2→3), THEN DECIMAL tier (1→2→3) (owner, 2026-05-30)

Each function (mul, div, …) is optimised tier by tier, and within a tier in three ordered stages. Finish the INT tier's 1→2→3 BEFORE starting the DECIMAL tier; do not interleave tiers, do not skip a stage, do not jump to the next function until both tiers are complete. (Learned the hard way on mul: int-(1) done + a decimal product-bypass fix landed, then I tried to skip to div — wrong; int-(2)/(3) and the full decimal tier come first.)

1. samply-optimise EVERY algorithm — not just the production winner. For EACH algorithm in the policy's Algorithm enum AND each kept alternative (the schoolbook reference, today-losers like the one-level Karatsuba): samply → optimise its hot path → microbench (compare_all, pinned) → iterate to tapped-out or a win.
   • Optimise each algorithm ON ITS OWN MERITS — make IT faster than its CURRENT self. samply ITS weak points → fix them → microbench the SAME algorithm BEFORE vs AFTER (bit-identical). Step 1 is NOT a competition between algorithms — whether one beats the OTHERS is STEP 2's job (the policy-map). "It loses to another algorithm, so don't bother optimising it" is NOT a valid step-1 result — that is missing the point (owner 2026-05-30: subagents kept giving up because the algorithm lost the bench-off instead of optimising it; e.g. abandoning a Karatsuba optimisation because schoolbook is faster — wrong, make Karatsuba the fastest Karatsuba it can be). "Tapped out" is valid ONLY when the algorithm's OWN implementation hits a genuine INTRINSIC limit, proven by ATTEMPTING optimisations and measuring (e.g. a fixed-width schoolbook already at the N² multiply+carry floor) — NEVER merely because a sibling algorithm is faster.
   • Limb size (u64 / u128) is PART of the algorithm in step 1; the CHOICE is step 2 (owner 2026-05-30). Each algorithm SUPPORTS both limb sizes — be Limb-generic (fn k<N, L: Limb>, like mul_full_limb<N,L>); step 1 optimises the algorithm at BOTH u64 and u128, microbenching each at-width before/after. Do NOT pre-pick a limb size in step 1 — that pre-empts the map. The policy-mapper (step 2) picks the limb size per cell (alongside the algorithm).
   • WHY optimise EVERY algorithm × BOTH limb sizes (not just today's winner): the policy-mapper can only choose the best-per-cell if every candidate is fully optimised. Leaving losers un-optimised hides potential wins (a today-loser that wins in SOME region once optimised) and lands the surface in a performance DIP — missed choices and improvements. The cost of optimising a loser is cheap; the cost of a missed map entry is permanent. WHY all of them: step 2 (policy-map) ranks the OPTIMISED algorithms — an un-optimised today-loser can become competitive once optimised and FLIP the map, so optimising must come BEFORE mapping; mapping un-optimised algorithms yields a STALE map. (Failure mode, mul 2026-05-30: optimised only mul_full_limb, tried to jump to the policy-map, left mul_karatsuba un-optimised — caught + corrected.) Tooling: [[algorithim-optimiser]] §4 + §4b (samply: scripts/run_samply.sh fires the elevated SamplyElevated task prompt-free; scripts/samply_symbolize.py → named frames via llvm-symbolizer+PDB). "Tapped out — no conformant win" is a VALID per-algorithm result; record it, don't force a change — but record it for EACH algorithm before moving to step 2.
   • COORDINATOR profiles FIRST, then dispatches with the PROVEN weak points (owner 2026-05-30). The coordinator samply-profiles each algorithm itself and hands the agent the symbolized weak-point frames AT DISPATCH, so the agent STARTS knowing where to optimise — it does not spend its run figuring out (or guessing) the hot spots. ("Saves time if it provably knows where the weak points are when it starts.") The headless pipeline (run_samply.sh + samply_symbolize.py) is coordinator-side; agents microbench + request a profile.
   • Use wait-time productively — pre-figure the NEXT step while an agent runs (owner 2026-05-30). While one agent works, do the "figuring out" for what comes next (the next algorithm's profile, the policy-map verdicts, the next dispatch's scope) so it is ready the moment feedback lands. Don't serialize figure-out-then-act.
   • GOTCHA (mul_full_ab): its custom main() does NOT parse the criterion filter, so a positional filter (Int4096, u128, kara) does NOT restrict — every width/arm runs. For a focused profile, filter the SYMBOLIZED output by function name (samply_symbolize.py … <name-filter>) rather than the bench arg, or add configure_from_args to the bench main.
2. policy-map — [[policy-mapper]]: N-way bench EVERY Algorithm × LimbSize across every width (int = brute-force all widths) / the 5-point scale grid + bisect (decimal) → the verified win-region map. Measure-only; the map is the deliverable.
3. policy-apply — [[policy-applier]]: wire the map into select at the TRUE crossovers, prove select reproduces the map, architectural-review (A–L) + golden-gate, push.

THEN repeat 1→2→3 for the DECIMAL tier of the same op (its own samply-optimise — e.g. a mul's ÷10^SCALE rescale, an exp's range-reduction — then its policy-map + apply). The INT tier goes first because its wins compound under the decimal op (which calls into it). Only when BOTH tiers are complete does the next function begin (mul → div → …).

Priority order — every cycle (STRICT)

Work in this order, always; never drop to a lower tier while a higher one has anything outstanding ([[feedback_coordinator_priority_order]]):

1. Agents first. An agent returns → review diff → merge → push → prune → launch the next agent immediately. Never idle the slot — agent compute is the scarce parallel resource; coordinator overhead (review/merge/docs) is cheap and waits behind it.
2. Table second. Keeping the 8.4 worst-first table current is a STANDING job. Rebuild it from the latest completed bbc that reflects current HEAD — NOT just any completed run (a run predating a merge gives wrong numbers; don't post it). bbc QUEUES; it does NOT cancel-in-progress.
3. Docs last. task_list / tracker / other doc cleanup ONLY when 1 and 2 have nothing outstanding.

And [[feedback_verify_before_asserting]]: verify run status / CI-trigger behaviour / file state before claiming or acting on it — don't invent it.

The model

One agent per DISJOINT file area — partition by policy/function so no two agents edit the same files (merges stay clean; the only coupling is call-boundaries, resolved at merge). The cap is the machine's subagent limit ([[feedback_resource_limits]]: ≤5 subagents + coordinator) — pick the ≤5 highest-value disjoint areas. Fold the research remit (the hunters perf lenses) INTO each fix agent rather than spawning separate research agents, so you stay within the cap.

Every agent TARGETS a regressed DECIMAL function, and its investigation scope is that decimal function's WHOLE call tree — every int function the decimal op uses, all the way down. bbc measures the public decimal ops (D###.exp/ln/mul/div), so an agent's success criterion is always a red decimal cell recovering vs 0.4.4, NEVER "my int microbench improved." The trap (the int-mul agent fell into it): framing the task as "optimise the int mul library" — looking at ONE int op in isolation — instead of starting from "which decimal cell is red, trace its FULL path down through every int fn it calls (mul, sqr, divide, ÷10^w, reductions…), find WHERE the loss is, fix at that layer." Cast each agent as "recover the <decimal fn> regression; trace its entire call tree; your primary lever is the int kernels it uses (fix there first — int wins compound — but the decimal cell must move; if int already matches 0.4.4, the gap is up at the decimal policy/algos and you fix THERE."

Parallel-collision arbitration (coordinator's job): the call-tree scope means several decimal agents legitimately want to touch the SAME shared int kernels (exp, ln, trig all use int mul/divide/sqr). To keep merges clean with concurrent agents, give the shared int kernels a single owner (the int-lever agent) and have the decimal agents FLAG int gaps to it (precise file:line + the 0.4.4 diff) rather than edit them; OR serialise edits to a shared int file. A decimal agent always owns its own decimal policy/algos outright.

Order: int-tier-first (holistic), then decimal; worst-of-table first. The decimal transcendentals (exp/ln/powf/trig) all RIDE on int-tier wide arithmetic (the ÷10^w per-term divide, the wide mul/sqr). So the int divide + int mul levers are the holistic ones — their wins compound under every decimal op — and go first / highest priority. Then the decimal function areas, starting at the top of the regression table. Disjoint-files means an int agent and a decimal agent that calls that int kernel don't collide (the decimal agent edits its policy/algos, not the int kernel); the int win simply lands separately and compounds.

A canonical 5-way split for the wide-transcendental band:

1. int wide divide — int/policy/div_rem,rem + int/algos/div,rem (the dominant per-Taylor-term cost).
2. int wide mul — int/policy/mul,mul_low + int/algos/mul.
3. exp + powf — policy/exp,pow + algos/exp,pow (usually the table's #1 cluster).
4. ln + log — policy/ln,log + algos/ln.
5. trig + inverse-hyperbolic — policy/trig + algos/trig (often partly collapses once 1/3/4 land — tell this agent to re-bench and focus only on the trig-local residual).

CPU: pin each agent's bench to its own core (the concurrency enabler)

Concurrent agents all running cargo bench at once floods the CPU and corrupts each other's measurements. Two ways to keep it sane:

• Preferred — core-pin via scripts/pin_run.ps1: the helper builds the bench UNPINNED (--no-run, across all cores — pinning the cargo proc also pins its child rustc swarm, and a one-core compile crawls: that "locked" feeling is a single-core build, not a deadlock), then runs the cached bench exe PINNED to the target core via Windows ProcessorAffinity so only the measurement is contention-free. It streams output to the console (the agent's Bash captures it) — no cmd, no start /affinity (that path pops a modal Windows error dialog through the bash→cmd boundary and blocks the agent's shell — the failure mode that froze the first wave; never use it). Each agent runs, from its worktree:

  powershell.exe -NoProfile -File scripts/pin_run.ps1 -Core <N> -Bench <name>
  # optional: -Features "wide x-wide xx-wide bench-alt" (default) ; -Extra "--no-run" ; -Mask 0xHEX
  

  and reads the bench's own criterion "A/B verdict" line for success (NOT the exit code). The machine has 24 logical cores, but builds are the real load — five wide-feature compiles at once saturate it even though benches are pinned. Run ≈3 agents concurrently, not 5, pinned to the HIGH end of the core range (e.g. cores 21/22/23 on a 24-core box) — Windows schedules system/foreground work on the LOW cores, so the high indices are the quietest for clean measurements. The int-tier levers (divide, mul) + the #1 regression cluster (exp) make the best first batch, with ln/trig in a second batch after those land (they ride on the int/exp wins and partly collapse — re-bench the residual). Tell agents: never run an unpinned measurement while siblings are active; always go through pin_run.ps1; if it errors in your shell, STOP and report. (-Features must stay one quoted token — the script wraps it; don't hand-split it.) If you push a pin_run.ps1 fix mid-wave, tell live agents to grab just that file with git fetch origin && git checkout origin/release/0.5.0 -- scripts/pin_run.ps1 — safe ONLY because they are in their own isolated worktree (it touches that worktree's copy only, not the shared tree).
• Fallback — serialise: if pinning is unreliable, run the agents one at a time (top-down), merging between each. Slower but unambiguous on CPU.

Keep the coordinator's own cargo (golden gates) off the agents' cores — run them on the free pool, ideally when bench load is low.

Per-agent loadout (put in every launch prompt)

• Dispatch with isolation: "worktree" (commit main first) — [[subagent-dispatch]] Isolation rule. Agents run in their own worktree; they NEVER reset/checkout a shared tree.
• STEP 0 base verification (NON-destructive) (from [[subagent-dispatch]]): verify a known tip-only file exists, STOP and report if absent. Do NOT git fetch/reset --hard/checkout.
• Load skills: architectural-review (compliance — the owner is strict; have them self-check classes A–K), subagent-dispatch (rules + report format), algorithim-optimiser (kernel write/wire/microbench + the algorithm-space rectangle method), hunters (perf-research lens). READ docs/ARCHITECTURE.md + CLAUDE.md (Constitution 1–6).
• Disjoint SCOPE, named explicitly — "touch ONLY these files; you may CALL but not edit siblings' areas."
• Holistic-rectangle rule (CORE RULE — owner-mandated 2026-05-27): map the op's matcher as rectangles over (N [, SCALE, value]); the validity union must TILE with NO GAPS (every cell resolves to a correct algorithm), OVERLAPS are fine, and prioritise the smallest const entries and smallest rectangles first (narrow N, small scale bands). Int-tier first within the area. Never fit a fix to the single benched (width, scale) cell — bbc only SAMPLES the surface ({0,30,S/2,S-1} × some widths), so a fix snapped to one sampled point leaves its neighbours (adjacent scales AND widths) without the boost, or cliffing at the boundary. Every fix is validated/placed across its CONTINUOUS win-region (the bisected true crossover), with straddling + immediate-neighbour cells checked — not the bbc grid points. Coordinator: run architectural-review Class I (single-cell fit) on every agent diff and bounce back any gate snapped to a point.
• Validate with the n-way bench (compare_all, [[algorithim-optimiser]] §4) — pinned to the agent's core, <60s. For each (width, scale) cell you touch, rank EVERY registered algorithm × EVERY LimbSize (u64/u128) THAT ALGORITHM SUPPORTS for that op in ONE compare_all run — not a pairwise A/B. A select arm's winner is the full (algorithm × limb-width) pair: every candidate valid at that cell (the Schoolbook/Series reference + every optimised variant + each table-size M, each at every limb width it supports) raced together at that exact N and SCALE. NOT every algorithm is LimbSize-generic — some kernels are u64-only (or u128-only); rank only the widths each one actually supports, don't fabricate a missing variant. A 2-way bench can crown a local winner that a third candidate — or the same algorithm at the other limb width — beats; the limb-size crossover moves with N and SCALE just like the algorithm crossover. TEST EVERY ASSUMPTION before claiming a win; bit-identity / golden is the correctness wall. No fabricated numbers.
• Rules ([[subagent-dispatch]]): commit ONLY to the worktree branch, NO push, NO touching release/main/tags, NO AI/Co-Authored-By trailer. Agents run micro_golden + their FOCUSED golden subset + cargo check (default AND wide) — not the full ulp_strict_golden (the COORDINATOR runs golden + the full test suite LOCALLY before merge — the gate; agents only cargo check + their focused subset).

The coordinator loop (per agent, as completions arrive)

Driven by completion notifications — do NOT poll. For each agent that finishes:

1. INVOKE the architectural-review skill on the agent's diff — EVERY return, no exceptions (owner-mandated). Run the full A–G review YOURSELF on the actual diff; do NOT trust the agent's self-check (agents misreport). Especially: new trait surface, per-tier pollution, const-work-width / build-max scratch, matcher-bypass (class G), layering inversion. A diff that fails the review does NOT merge — bounce it back or fix it as coordinator.
2. Verify AI-trailer-clean (git log <base>..<branch> --format=%B | grep -iE "co-authored-by|anthropic|claude|🤖" → empty) and the merge-base IS the current release tip (the worktree-leak guard — an agent has historically committed straight onto release; confirm before merging, and watch that a fix didn't already leak onto the branch).
3. Coordinator runs golden + the full cargo test LOCALLY — the correctness gate, BEFORE merge (owner rule 2026-05-29, [[feedback_coordinator_runs_golden_tests_locally]]; SUPERSEDES the old "golden is CI-only"): cargo test --release --features wide,x-wide,xx-wide,macros,golden --test ulp_strict_golden + the full cargo test; require delta==0 and the vector count NOT dropped. The agent's golden is a signal; the coordinator's local run is the proof. If red, bounce the failing cells back — do NOT merge. NEVER run benches locally (bbc/sweeps are GHA-only).
4. Merge (--no-ff) → resolve any call-boundary conflict → push release/0.5.0 (the push fires CI + bench-branch-compare — advisory/data; the local golden was the gate). Then launch the next agent.
5. Monitor bbc but do NOT wait on it (Monitor/notifications). When the LATEST bbc completes, refresh the worst-first table (below) so the priority list stays current as fixes land — this re-ranks remaining work and shows which agents' wins collapsed which rows.

Merge regularly (don't batch all five) so the table reflects reality and conflicts stay small. Serialise YOUR golden runs (one heavy coordinator cargo at a time) to respect the CPU budget.

The worst-first priority table (keep it refreshed)

The 8.4 table in research/0.5.0_task_list.md (git-ignored). Columns: # | cell | prod | branch | Δ | Δ% | ×, worst→best by Δ%, units mixed per ROW (ns if branch < 1000 ns, else µs; one unit per row). Inclusion filter: Δ% > +1% AND absolute Δ > 10 ns — Δ ≤ ~10 ns is below the GHA shared-runner measurement floor (jitter + black_box/loop overhead) and is filtered to a footnote (unassessable). Do NOT mark an included row "noise" — if it cleared the >10 ns floor it is assessable by definition (keep the noise gate and the flag consistent — one rule).

Refresh recipe from a completed bbc run:

gh run download <run-id> -D trace/bbc_<tag>
# python over trace/bbc_<tag>/bench-branch-compare-*/target/criterion/*/{branch,prod}/new/estimates.json
#   median point_estimate (ns); keep if br/pr-1 > 0.01 AND (br-pr) > 10; sort by Δ% desc; per-row unit.

(NOTE: bbc benches each tier at a fixed operand/scale — confirm what compare-common actually runs before reading a cell as representative; an exp/D### cell is one (scale, operand), not the tier's whole range.)

Hazards (learned the hard way)

• Worktree-commit leak: isolation "worktree" does not always isolate the commit — an agent's commit can land on the shared release/0.5.0 and ride out on your next push UN-reviewed. ALWAYS check the merge-base + diff before pushing anything, and verify the agent committed to its own branch.
• Agent misreports green: treat the agent's golden/bench as a signal, re-run the gate yourself on the merged tip.
• Resource cap: ≤5 subagents + coordinator; ≤80% CPU (this skill's core-pin keeps you there). Subagents never run the full golden.
• Bench reality: full sweeps run on GHA (bench-branch-compare), not the owner's machine; local is microbench-only ([[feedback_bench_on_gha_not_local]]). Picosecond change: deltas are noise.

Launch-prompt skeleton (per agent — fill area/files/core/targets)

Perf-recovery agent for decimal_scaled (CWD = your own git worktree). Recover the regression(s) <worst-first targets with Δ%>. STEP 0 (NON-destructive): verify <tip-only file> exists in your worktree, else STOP and report. Do NOT fetch/reset/checkout — you are isolated and must not mutate git state to "fix" a base. LOAD SKILLS: architectural-review, subagent-dispatch, algorithim-optimiser, hunters. READ docs/ARCHITECTURE.md + CLAUDE.md (Constitution 1–6). Conform: matcher is the lever, ONE generic kernel, NO per-tier pollution, NO const-work-width (ComputeInt buffers). SCOPE — touch ONLY: <disjoint files>. May CALL siblings' kernels, not edit them. APPROACH: holistic rectangles over (N[,SCALE,value]) — tile NO GAPS, overlaps fine, smallest const/rectangles first; int-tier first. ALL-WIDTHS / ALL-[axis] RULE (MANDATORY — every optimisation dispatch, no abbreviation, [[feedback_optimisation_all_widths_scales]]): a fix reviews EVERY width × the layer's continuous axis, not just the bbc-target cell. Layer determines the axis: decimal = width × SCALE; int = width × operand-shape (limb-length / magnitude) — int has NO scale axis (policy-mapper skill). Tell the agent which axis applies. COARSE START → ADAPTIVE BISECTION: per affected width, sample the coarse first pass — decimal: 5-point scale set {0, S/4, S/2, 3S/4, S-1}; int: a representative width × operand-shape grid (e.g. N ∈ {1, 8, 16, 24, 32, 48, 64} for an N-axis kernel; representative (a_len, b_len) pairs for a mul) — then BISECT where adjacent samples have different winners until the crossover is localised, down to a single value if needed. The grid is the COARSE STARTING POINT, not the final answer. Carve arms as CONTINUOUS regions over the bisected win-region — NO point-snap to the bbc target. Validate that neighbours of the target are NOT regressed (a gate "just missing" them is the recurring Class I defect). Decimal: narrow tiers (D18/D38/D57/D76) AND every affected wide tier (D115/D153/D230/D307/D462/D616/D924/D1232). Int: every supported N the kernel reaches. A change benched only at the target cell, or a gate snapped to one value, is a single-cell snap — rejected at architectural-review. THE TARGET IS THE FASTEST VALID ALGORITHM PER (width, scale) CELL — NOT merely "beats 0.4.4". Beating 0.4.4 is the REASON the campaign runs (a red bbc cell proves a faster approach exists), it is NOT the goal and NOT a stopping condition. At EVERY cell — INCLUDING cells that already beat 0.4.4 — N-way bench ALL candidate algorithms (× every limb width each supports) and wire the FASTEST one that is bit-identical to the reference (valid). "Already beats prod" is NEVER a reason to skip a cell or leave it on a slower algorithm; only "the currently-selected algorithm IS the fastest valid candidate here, numbers attached" justifies leaving a cell unchanged (this is exactly the [[policy-mapper]] rule: fastest valid candidate wins). REGRESSION = 0.4.4 HAD A FASTER APPROACH — so "do nothing / already optimal" is NEVER an acceptable answer to a red bbc cell without proof. You MUST diff v0.4.4 (git show v0.4.4:<path>) for the algorithm + policy/threshold it used at that cell, find WHY it was faster (different algorithm / limb width / engaged threshold / cheaper reduction), and recover it as a conformant generic 0.5.0 change (the ALGORITHM/IDEA, not 0.4.4's per-tier code — matcher is the lever). Benching only our own arms against each other and calling a tie "optimal" while the bbc cell is still red = you benched the wrong thing. THE bbc CELL IS A DECIMAL OP, NOT THE INT KERNEL: bbc measures public decimal ops (D###.exp/ln/mul/div). Int-tier first because int wins compound — BUT the regression may live at the DECIMAL policy/+algos/ layer (the decimal algorithm/routing/reduction or how it calls into int). An int microbench tying 0.4.4 does NOT clear a red decimal cell. TRACE the regressed decimal op's full call path and diff 0.4.4's DECIMAL algo too; fix int-tier when int is the gap, fix the decimal layer when that's where the loss is. BENCH — PIN TO CORE : powershell.exe -NoProfile -File scripts/pin_run.ps1 -Core <N> -Bench <name> (default features wide x-wide xx-wide bench-alt; add -Extra "--no-run" to pre-compile, -Features "..." to override). Use the n-way compare_all harness: for EVERY (width, scale) cell you touch, rank ALL candidate algorithms × every limb width THAT ALGORITHM SUPPORTS for that op (reference + every variant + each table-size M, each at the limb widths it supports — some kernels are u64- or u128-only) in ONE run — append ("label", run) tuples to the vec!, do NOT settle for a pairwise A/B and do NOT inherit the limb size. Read the criterion "A/B verdict" line + ranking table, NOT the exit code. Test every assumption; NEVER use cmd /c start /affinity (it freezes your shell); never an unpinned bench while siblings run; if pin_run.ps1 errors, STOP + report. RULES: commit to your worktree branch ONLY; NO push/release/main/tags; NO AI trailer; run micro_golden + focused golden for your functions + cargo check (default + wide); keep green; cite real bench numbers. REPORT: base-reset confirmation, branch, changes (file:line), pasted bench verdicts + focused-golden, architectural-review self-check (A–G), deferred. If no benched conformant win, STOP + report the candidates tried with their numbers.