measure-ab

Estrellas1

Forks1

Actualizado3 de julio de 2026 a las 12:42

Run and interpret a bidding A/B experiment in pons — harness choice, seed hygiene, polite scheduling, dual scoring, and the ship/reject decision. Use whenever the user asks to measure, A/B, benchmark, re-measure, or validate a bidding change, or to interpret A/B results.

Instalación

Instalar con Codex o Claude Copia este prompt, pégalo en Codex, Claude u otro asistente, y deja que revise la página de la skill y la instale por ti.

Ejecutar en Manus

Fuente

jdh8

jdh8/pons

Abrir repositorio de GitHub Ver repositorios del creador

Descarga

Ejecutar en Manus

Ocupaciones relacionadasSOC

Basado en la clasificación ocupacional SOC

Científicos de datosOcupaciones informáticas y matemáticas·SOC 15-2051

SKILL.md

readonly

name	measure-ab
description	Run and interpret a bidding A/B experiment in pons — harness choice, seed hygiene, polite scheduling, dual scoring, and the ship/reject decision. Use whenever the user asks to measure, A/B, benchmark, re-measure, or validate a bidding change, or to interpret A/B results.

Running a bidding A/B

docs/measurement.md is the source of truth — read it now if this session hasn't. This skill is the run procedure.

Procedure

Baseline sanity. The control arm must be the real routing of the same hands (not an analytic pass/invite stand-in), and the treatment must be complete (both sides' continuations) — else the number is meaningless.
Pick the harness. Prefer an existing examples/ab-* (self-play seat-swap; check --help — most questions are a flag away). Use examples/bba-gen via scripts/bba-gen-parallel.sh + ab-dump-diff when you need the reference opponent — mandatory for tuning our own conventions' strength/ranges (self-play can't punish them; pass the matching --advertise-* so BBA reads the bids).
Seeds. export SEED_BASE=$(date +%s) once; all arms of this experiment share it (paired diffs need identical deals). Next experiment → new base. Record the base + git SHA in the results note.
Run. Build first, then hands off the binaries until the run finishes — no rebuilds mid-run. Wrap in scripts/idle-run.sh (shared box), arms strictly sequential; one run already saturates every core. tmux or setsid nohup for long runs. Typical scale: 200k filtered boards/cell contested, ~205k/arm (6400 × nproc) for bba-gen; 40k filtered ≈ 1 min/cell.
Score both ways — plain DD and PD — from the same solved tables, at both vulnerabilities (none/both). Report IMPs/board, IMPs/fired (or IMPs/divergent on filtered harnesses — never IMPs/board there), fired rate, and a CI (two seeds minimum, or bootstrap).
Verdict from the decision table in docs/measurement.md. Remember the two big artifacts: PD-only win = doubling artifact; DD loss on an obstructive/concealment idea = harness blindness, park it opt-in for a single-dummy re-measure (single_dummy_leads prices the blind-lead seam).
Losing? Trace the worst divergent boards before declaring dead. Look for: an unauthored continuation (passed-out cue, unanswered doubled transfer), an over-broad trigger (fires outside its hand class — dilution of the per-fired win is the tell), or a scope leak via a shared node. Fix and re-measure; an unchanged arm's dump can be reused when the change can't affect it (same seeds align the pairing).
Write it down: numbers, board counts, seeds, SHA, verdict, and the knob's shipped default — in the CHANGELOG (and ledger doc if tracked).

Más de este repositorio

mismo repositorio

ai-bidder

jdh8/pons

Advance or resume the "AI instinct bidder" effort in pons — replacing the deterministic bidding floor with a learned model (a hand→call policy net and a description language model). Use when the user asks to work on the neural/AI bidder, the policy net, distillation, the constrained sampler, the description compiler/corpus, or references docs/ai-bidder/. Loads the design, respects the invariants, and teaches the ML grounded in math/bridge/low-level.

2026-07-201

author-convention

jdh8/pons

End-to-end checklist for adding or changing a bidding convention/treatment in pons — authoring the rules, alerts, inference reading, knob, tests, and the A/B measurement that decides whether it ships. Use whenever the user asks to add, author, implement, tune, or fix a convention, treatment, response structure, defense, or any behavior of the bidding system.

2026-07-031

Running a bidding A/B

docs/measurement.md is the source of truth — read it now if this session hasn't. This skill is the run procedure.

Procedure

Baseline sanity. The control arm must be the real routing of the same hands (not an analytic pass/invite stand-in), and the treatment must be complete (both sides' continuations) — else the number is meaningless.

Pick the harness. Prefer an existing examples/ab-* (self-play seat-swap; check --help — most questions are a flag away). Use examples/bba-gen via scripts/bba-gen-parallel.sh + ab-dump-diff when you need the reference opponent — mandatory for tuning our own conventions' strength/ranges (self-play can't punish them; pass the matching --advertise-* so BBA reads the bids).

Seeds. export SEED_BASE=$(date +%s) once; all arms of this experiment share it (paired diffs need identical deals). Next experiment → new base. Record the base + git SHA in the results note.

Run. Build first, then hands off the binaries until the run finishes — no rebuilds mid-run. Wrap in scripts/idle-run.sh (shared box), arms strictly sequential; one run already saturates every core. tmux or setsid nohup for long runs. Typical scale: 200k filtered boards/cell contested, ~205k/arm (6400 × nproc) for bba-gen; 40k filtered ≈ 1 min/cell.

Score both ways — plain DD and PD — from the same solved tables, at both vulnerabilities (none/both). Report IMPs/board, IMPs/fired (or IMPs/divergent on filtered harnesses — never IMPs/board there), fired rate, and a CI (two seeds minimum, or bootstrap).

Verdict from the decision table in docs/measurement.md. Remember the two big artifacts: PD-only win = doubling artifact; DD loss on an obstructive/concealment idea = harness blindness, park it opt-in for a single-dummy re-measure (single_dummy_leads prices the blind-lead seam).

Losing? Trace the worst divergent boards before declaring dead. Look for: an unauthored continuation (passed-out cue, unanswered doubled transfer), an over-broad trigger (fires outside its hand class — dilution of the per-fired win is the tell), or a scope leak via a shared node. Fix and re-measure; an unchanged arm's dump can be reused when the change can't affect it (same seeds align the pairing).

Write it down: numbers, board counts, seeds, SHA, verdict, and the knob's shipped default — in the CHANGELOG (and ledger doc if tracked).