| name | bottleneck-first-scheduling |
| description | Run BEFORE planning or launching a batch of work in this repo — a train/cert/eval sweep, a multi-PR push, a judge-farm run, or any "let's parallelize this / run these N things" moment, and whenever a queue is backing up, throughput has stalled, GPU/CI/merge is idle-but-blocked, or a measured delta can't be attributed. This cluster has exactly ONE shared Spark GPU, a serialized Mac judge, a human merge gate, and an append-heavy ledger — pushing more work at a saturated constraint only grows queue and mixes variables. Use to find the current constraint, cap work-in-progress at its capacity, subordinate everything else to it, and change ONE variable at a time so deltas stay attributable. The LLM is never the constraint.
|
| metadata | {"short-description":"Theory-of-Constraints operating discipline: find the one constraint, cap WIP to it, subordinate, change one variable"} |
Bottleneck-first scheduling (Theory of Constraints, this repo)
See also spark-cluster-ops §1-2 (the bridge poller = the cluster-wide one-GPU lock and
SESSION-COORDINATION.md = the live ownership ledger) and git-discipline (the merge
gate). This skill is the scheduling layer on top of those mechanics: given the constraint,
how much work to admit and in what order.
Throughput is set by the single slowest resource in the chain, not by how many jobs you launch.
On this cluster the constraint is almost always one of four shared, serial resources — never the
model. Launching more work at a saturated constraint just grows queue depth, hides which change
caused which delta, and (on the GPU) can silently kill an in-flight cert/train. The discipline
is Goldratt's five focusing steps, applied to the real assets here.
0. The candidate constraints (rank these, pick the ONE that is binding now)
| Constraint | What it is here | Saturated when | Capacity signal |
|---|
| Shared Spark GPU | ONE 128 GB GB10; the bridge poller serializes it | any heavy train/cert/judge already resident | nvidia-smi VRAM/util headroom; bridge poller busy |
| Serialized Mac judge | Mac Studio judge family (mlx_lm.server), one lane per family | judge queue depth > lanes; subject waits on verdicts | judge-lane occupancy; TrainWatch ETA |
| Human merge gate | main-protection: PR review + required_review_thread_resolution | PRs green but sitting unmerged; branches stacking | open-PR count vs merge rate |
| Append-heavy ledger | SESSION-COORDINATION.md, agi-proof/failure-ledger.md | concurrent appends collide; state goes stale | last-append recency; git churn on the file |
The LLM/subject model is never the constraint — it is fast and infinitely re-runnable; treat it
as protective capacity, not the bottleneck. If your plan implies "the model is the slow part,"
re-diagnose.
1. IDENTIFY — where is the queue actually piling up? (read-only, ~30 s)
Don't assume; measure the four candidates before admitting work.
cat SESSION-COORDINATION.md
nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv
- Is the bridge poller mid-job? (spark-cluster-ops §2 — route heavy GPU work through it; it is
the cluster-wide one-job lock). If yes, the GPU is the constraint and it is taken — queue, don't launch.
- Is the Mac judge the wait? Check TrainWatch (the
trainwatch skill, dashboard at
spark-2f2d:8420) — if subject steps finish fast but ETA is judge-bound, the judge is the constraint.
- Are green PRs stacking unmerged? Then the merge gate is the constraint — see
git-discipline.
The constraint is whichever queue is growing. There is exactly one binding constraint at a time.
2. EXPLOIT — get maximum throughput from the constraint before adding anything
Make the existing bottleneck lose zero cycles. Do NOT buy more capacity yet.
- GPU: never let it idle and never let a stray second load evict an in-flight job. Keep the
bridge poller fed with a queue of ready, decontaminated work so it never waits on you between jobs.
Batch small jobs; don't interleave a heavy judge/teacher onto a busy GPU (spark-cluster-ops §2).
- Mac judge: keep every judge lane busy but not over-subscribed; respect the ≥2-family
independence gate (see
judge-lane-independence-gate memory) — a second non-independent judge
adds no real throughput, it fakes it.
- Merge gate: make PRs trivially mergeable — pre-run
make claim-check and the artifact-drift
checks (ci-artifact-drift) so a human review is never spent on a red gate. Resolve review
threads promptly (required_review_thread_resolution is the silent blocker).
- Ledger: short, append-only, one owner per line; read-before-append so you don't clobber.
3. SUBORDINATE — pace everything else to the constraint
Non-constraint resources should run at the constraint's rhythm, not their own max.
- Cap global WIP at the bottleneck's capacity. One heavy GPU job in flight → admit exactly one;
don't stage three trains "so they're ready" — a full queue at a saturated GPU buys nothing and
invites the eviction footgun. Iterate cheap on the Spark, register once, judge on the Mac
(spark-cluster-ops §5): each stage paced to the stage downstream of it.
- CLAIM before you launch in
SESSION-COORDINATION.md (session-handover / spark-cluster-ops §1)
so peer agents subordinate too — the cluster runs 5+ concurrent sessions on one checkout.
- A non-constraint running flat-out just builds inventory (queued jobs, unmerged branches, stale
ledger claims) — that inventory is pure waste until the constraint can consume it.
4. CHANGE ONE VARIABLE AT A TIME — so the delta is attributable
This is the measurement half of the discipline and it is non-negotiable in this repo.
- Between two runs you compare, change exactly one thing (adapter, prompt, gate, seed policy).
If you change the constraint and the recipe together, the delta is uninterpretable — you've
burned the scarce GPU for a number you can't defend.
- Pin seeds / hold the eval harness fixed across the comparison (see
rlvr-harness-traps); a moved
harness is a second variable masquerading as a result.
- Record the run and its ONE change where the next session sees it: TrainWatch for live progress,
the handover +
agi-proof/failure-ledger.md for the verdict. Report elapsed + ETA + progress
proactively (the KTA habit).
5. ELEVATE (only after 1-4) — and re-check, because the constraint MOVES
Add capacity to the constraint only once it is fully exploited and subordinated: e.g. a second
independent judge family on the Mac to widen judge throughput, or an x86 RunPod pod to offload
iteration off the Spark (announce the pod id; mind the cost gate — spark-cluster-ops §3).
Then return to step 1: relieving one constraint promotes a new one (elevate the GPU and the Mac
judge or the merge gate becomes binding). Never optimize a resource that is no longer the constraint.
Anti-patterns (each has burned real cycles here)
- Launching a batch "to parallelize" onto a GPU the bridge poller already owns → eviction / killed train.
- Staging N trains ahead of a one-job GPU → deep queue, zero extra throughput, stale claims.
- Changing adapter and gate and seed between two runs → unattributable delta, wasted GPU.
- Optimizing the subject model (prompt tweaks, bigger model) while the judge is the real wait.
- Treating the human merge gate as free — stacking green PRs no one is being asked to review.
Guardrail
This is scheduling/measurement discipline only. It makes no capability or AGI claim; finding a
constraint is an operations fact, not an uplift. canClaimAGI stays false. If a run's result
would change what is claimed (a number's status, "validated"), that is the no-overclaim contract —
stop and follow ci-artifact-drift / sophia-agi.
