bottleneck-first-scheduling

name	bottleneck-first-scheduling
description	Run BEFORE planning or launching a batch of work in this repo — a train/cert/eval sweep, a multi-PR push, a judge-farm run, or any "let's parallelize this / run these N things" moment, and whenever a queue is backing up, throughput has stalled, GPU/CI/merge is idle-but-blocked, or a measured delta can't be attributed. This cluster has exactly ONE shared Spark GPU, a serialized Mac judge, a human merge gate, and an append-heavy ledger — pushing more work at a saturated constraint only grows queue and mixes variables. Use to find the current constraint, cap work-in-progress at its capacity, subordinate everything else to it, and change ONE variable at a time so deltas stay attributable. The LLM is never the constraint.
metadata	{"short-description":"Theory-of-Constraints operating discipline: find the one constraint, cap WIP to it, subordinate, change one variable"}

Bottleneck-first scheduling (Theory of Constraints, this repo)

See also spark-cluster-ops §1-2 (the bridge poller = the cluster-wide one-GPU lock and SESSION-COORDINATION.md = the live ownership ledger) and git-discipline (the merge gate). This skill is the scheduling layer on top of those mechanics: given the constraint, how much work to admit and in what order.

Throughput is set by the single slowest resource in the chain, not by how many jobs you launch. On this cluster the constraint is almost always one of four shared, serial resources — never the model. Launching more work at a saturated constraint just grows queue depth, hides which change caused which delta, and (on the GPU) can silently kill an in-flight cert/train. The discipline is Goldratt's five focusing steps, applied to the real assets here.

0. The candidate constraints (rank these, pick the ONE that is binding now)

Constraint	What it is here	Saturated when	Capacity signal
Shared Spark GPU	ONE 128 GB GB10; the bridge poller serializes it	any heavy train/cert/judge already resident	nvidia-smi VRAM/util headroom; bridge poller busy
Serialized Mac judge	Mac Studio judge family (mlx_lm.server), one lane per family	judge queue depth > lanes; subject waits on verdicts	judge-lane occupancy; TrainWatch ETA
Human merge gate	main-protection: PR review + required_review_thread_resolution	PRs green but sitting unmerged; branches stacking	open-PR count vs merge rate
Append-heavy ledger	SESSION-COORDINATION.md, agi-proof/failure-ledger.md	concurrent appends collide; state goes stale	last-append recency; git churn on the file

The LLM/subject model is never the constraint — it is fast and infinitely re-runnable; treat it as protective capacity, not the bottleneck. If your plan implies "the model is the slow part," re-diagnose.

1. IDENTIFY — where is the queue actually piling up? (read-only, ~30 s)

Don't assume; measure the four candidates before admitting work.

cat SESSION-COORDINATION.md            # who OWNS what right now; is the GPU/judge already CLAIMed?
nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv   # Spark GPU headroom

• Is the bridge poller mid-job? (spark-cluster-ops §2 — route heavy GPU work through it; it is the cluster-wide one-job lock). If yes, the GPU is the constraint and it is taken — queue, don't launch.
• Is the Mac judge the wait? Check TrainWatch (the trainwatch skill, dashboard at spark-2f2d:8420) — if subject steps finish fast but ETA is judge-bound, the judge is the constraint.
• Are green PRs stacking unmerged? Then the merge gate is the constraint — see git-discipline.

The constraint is whichever queue is growing. There is exactly one binding constraint at a time.

2. EXPLOIT — get maximum throughput from the constraint before adding anything

Make the existing bottleneck lose zero cycles. Do NOT buy more capacity yet.

• GPU: never let it idle and never let a stray second load evict an in-flight job. Keep the bridge poller fed with a queue of ready, decontaminated work so it never waits on you between jobs. Batch small jobs; don't interleave a heavy judge/teacher onto a busy GPU (spark-cluster-ops §2).
• Mac judge: keep every judge lane busy but not over-subscribed; respect the ≥2-family independence gate (see judge-lane-independence-gate memory) — a second non-independent judge adds no real throughput, it fakes it.
• Merge gate: make PRs trivially mergeable — pre-run make claim-check and the artifact-drift checks (ci-artifact-drift) so a human review is never spent on a red gate. Resolve review threads promptly (required_review_thread_resolution is the silent blocker).
• Ledger: short, append-only, one owner per line; read-before-append so you don't clobber.

3. SUBORDINATE — pace everything else to the constraint

Non-constraint resources should run at the constraint's rhythm, not their own max.

• Cap global WIP at the bottleneck's capacity. One heavy GPU job in flight → admit exactly one; don't stage three trains "so they're ready" — a full queue at a saturated GPU buys nothing and invites the eviction footgun. Iterate cheap on the Spark, register once, judge on the Mac (spark-cluster-ops §5): each stage paced to the stage downstream of it.
• CLAIM before you launch in SESSION-COORDINATION.md (session-handover / spark-cluster-ops §1) so peer agents subordinate too — the cluster runs 5+ concurrent sessions on one checkout.
• A non-constraint running flat-out just builds inventory (queued jobs, unmerged branches, stale ledger claims) — that inventory is pure waste until the constraint can consume it.

4. CHANGE ONE VARIABLE AT A TIME — so the delta is attributable

This is the measurement half of the discipline and it is non-negotiable in this repo.

• Between two runs you compare, change exactly one thing (adapter, prompt, gate, seed policy). If you change the constraint and the recipe together, the delta is uninterpretable — you've burned the scarce GPU for a number you can't defend.
• Pin seeds / hold the eval harness fixed across the comparison (see rlvr-harness-traps); a moved harness is a second variable masquerading as a result.
• Record the run and its ONE change where the next session sees it: TrainWatch for live progress, the handover + agi-proof/failure-ledger.md for the verdict. Report elapsed + ETA + progress proactively (the KTA habit).

5. ELEVATE (only after 1-4) — and re-check, because the constraint MOVES

Add capacity to the constraint only once it is fully exploited and subordinated: e.g. a second independent judge family on the Mac to widen judge throughput, or an x86 RunPod pod to offload iteration off the Spark (announce the pod id; mind the cost gate — spark-cluster-ops §3). Then return to step 1: relieving one constraint promotes a new one (elevate the GPU and the Mac judge or the merge gate becomes binding). Never optimize a resource that is no longer the constraint.

Anti-patterns (each has burned real cycles here)

• Launching a batch "to parallelize" onto a GPU the bridge poller already owns → eviction / killed train.
• Staging N trains ahead of a one-job GPU → deep queue, zero extra throughput, stale claims.
• Changing adapter and gate and seed between two runs → unattributable delta, wasted GPU.
• Optimizing the subject model (prompt tweaks, bigger model) while the judge is the real wait.
• Treating the human merge gate as free — stacking green PRs no one is being asked to review.

Guardrail

This is scheduling/measurement discipline only. It makes no capability or AGI claim; finding a constraint is an operations fact, not an uplift. canClaimAGI stays false. If a run's result would change what is claimed (a number's status, "validated"), that is the no-overclaim contract — stop and follow ci-artifact-drift / sophia-agi.