원클릭으로 Manus에서 모든 스킬 실행

measure-ab

스타1

포크1

업데이트2026년 7월 3일 12:42

Run and interpret a bidding A/B experiment in pons — harness choice, seed hygiene, polite scheduling, dual scoring, and the ship/reject decision. Use whenever the user asks to measure, A/B, benchmark, re-measure, or validate a bidding change, or to interpret A/B results.

설치

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

Manus에서 실행

출처

jdh8

jdh8/pons

GitHub 저장소 열기 Creator 저장소 보기

다운로드

Manus에서 실행

Running a bidding A/B

docs/measurement.md is the source of truth — read it now if this session hasn't. This skill is the run procedure.

Procedure

Baseline sanity. The control arm must be the real routing of the same hands (not an analytic pass/invite stand-in), and the treatment must be complete (both sides' continuations) — else the number is meaningless.
Pick the harness. Prefer an existing examples/ab-* (self-play seat-swap; check --help — most questions are a flag away). Use examples/bba-gen via scripts/bba-gen-parallel.sh + ab-dump-diff when you need the reference opponent — mandatory for tuning our own conventions' strength/ranges (self-play can't punish them; pass the matching --advertise-* so BBA reads the bids).
Seeds. export SEED_BASE=$(date +%s) once; all arms of this experiment share it (paired diffs need identical deals). Next experiment → new base. Record the base + git SHA in the results note.
Run. Build first, then hands off the binaries until the run finishes — no rebuilds mid-run. Wrap in scripts/idle-run.sh (shared box), arms strictly sequential; one run already saturates every core. tmux or setsid nohup for long runs. Typical scale: 200k filtered boards/cell contested, ~205k/arm (6400 × nproc) for bba-gen; 40k filtered ≈ 1 min/cell.
Score both ways — plain DD and PD — from the same solved tables, at both vulnerabilities (none/both). Report IMPs/board, IMPs/fired (or IMPs/divergent on filtered harnesses — never IMPs/board there), fired rate, and a CI (two seeds minimum, or bootstrap).
Verdict from the decision table in docs/measurement.md. Remember the two big artifacts: PD-only win = doubling artifact; DD loss on an obstructive/concealment idea = harness blindness, park it opt-in for a single-dummy re-measure (single_dummy_leads prices the blind-lead seam).
Losing? Trace the worst divergent boards before declaring dead. Look for: an unauthored continuation (passed-out cue, unanswered doubled transfer), an over-broad trigger (fires outside its hand class — dilution of the per-fired win is the tell), or a scope leak via a shared node. Fix and re-measure; an unchanged arm's dump can be reused when the change can't affect it (same seeds align the pairing).
Write it down: numbers, board counts, seeds, SHA, verdict, and the knob's shipped default — in the CHANGELOG (and ledger doc if tracked).

이 저장소의 다른 Skills

같은 저장소

ai-bidder

jdh8/pons

Advance or resume the "AI instinct bidder" effort in pons — replacing the deterministic bidding floor with a learned model (a hand→call policy net and a description language model). Use when the user asks to work on the neural/AI bidder, the policy net, distillation, the constrained sampler, the description compiler/corpus, or references docs/ai-bidder/. Loads the design, respects the invariants, and teaches the ML grounded in math/bridge/low-level.

2026-07-201

author-convention

jdh8/pons

End-to-end checklist for adding or changing a bidding convention/treatment in pons — authoring the rules, alerts, inference reading, knob, tests, and the A/B measurement that decides whether it ships. Use whenever the user asks to add, author, implement, tune, or fix a convention, treatment, response structure, defense, or any behavior of the bidding system.

2026-07-031

name	measure-ab
description	Run and interpret a bidding A/B experiment in pons — harness choice, seed hygiene, polite scheduling, dual scoring, and the ship/reject decision. Use whenever the user asks to measure, A/B, benchmark, re-measure, or validate a bidding change, or to interpret A/B results.

Running a bidding A/B

docs/measurement.md is the source of truth — read it now if this session hasn't. This skill is the run procedure.

Procedure

Baseline sanity. The control arm must be the real routing of the same hands (not an analytic pass/invite stand-in), and the treatment must be complete (both sides' continuations) — else the number is meaningless.
Pick the harness. Prefer an existing examples/ab-* (self-play seat-swap; check --help — most questions are a flag away). Use examples/bba-gen via scripts/bba-gen-parallel.sh + ab-dump-diff when you need the reference opponent — mandatory for tuning our own conventions' strength/ranges (self-play can't punish them; pass the matching --advertise-* so BBA reads the bids).
Seeds. export SEED_BASE=$(date +%s) once; all arms of this experiment share it (paired diffs need identical deals). Next experiment → new base. Record the base + git SHA in the results note.
Run. Build first, then hands off the binaries until the run finishes — no rebuilds mid-run. Wrap in scripts/idle-run.sh (shared box), arms strictly sequential; one run already saturates every core. tmux or setsid nohup for long runs. Typical scale: 200k filtered boards/cell contested, ~205k/arm (6400 × nproc) for bba-gen; 40k filtered ≈ 1 min/cell.
Score both ways — plain DD and PD — from the same solved tables, at both vulnerabilities (none/both). Report IMPs/board, IMPs/fired (or IMPs/divergent on filtered harnesses — never IMPs/board there), fired rate, and a CI (two seeds minimum, or bootstrap).
Verdict from the decision table in docs/measurement.md. Remember the two big artifacts: PD-only win = doubling artifact; DD loss on an obstructive/concealment idea = harness blindness, park it opt-in for a single-dummy re-measure (single_dummy_leads prices the blind-lead seam).
Losing? Trace the worst divergent boards before declaring dead. Look for: an unauthored continuation (passed-out cue, unanswered doubled transfer), an over-broad trigger (fires outside its hand class — dilution of the per-fired win is the tell), or a scope leak via a shared node. Fix and re-measure; an unchanged arm's dump can be reused when the change can't affect it (same seeds align the pairing).
Write it down: numbers, board counts, seeds, SHA, verdict, and the knob's shipped default — in the CHANGELOG (and ledger doc if tracked).