| name | w8-biayn-framework |
| description | Operate and extend agent-rein, the AutoHarness post-training pipeline that RL-trains a small code model (via SkyRL GRPO on Google Cloud) to write browser action-verifier harnesses, measured by a before/after search-budget trace. Use when working in this repo on the w8-biayn CLI, the autoharness package (Harness interface, Thompson-sampling search, rewards, Daytona sandbox), SkyPilot/GCP launches, the harness-legality benchmark, the trace-shrinkage experiment, or the SkyRL/vLLM training config. Trigger terms include run the experiment, launch on GCP, trace shrinkage, steps_to_gold, harness reward, w8-biayn, autoharness, GRPO, SkyRL, Daytona sandbox, and gold-threshold. |
agent-rein / w8-biayn framework
This repo post-trains the model that writes web-agent harnesses. AutoHarness uses a frozen LLM as
a code mutation operator inside a Thompson-sampling tree search to synthesize an
is_legal_action(state, action) -> bool harness; we RL-train that mutation operator (CaP-RL-style
GRPO on a dense Qwen2.5-Coder) so it writes a working harness in fewer search steps. The headline
result is a before/after steps_to_gold trace: frozen model → GRPO → trained model, run on one
GCP node via SkyPilot + SkyRL.
Read NEW-SPEC.md for the full spec and README.md for
the operational summary before making changes. This skill is the AI operating guide; the references
below hold the detail.
Mental model (two packages)
src/autoharness/ — the AutoHarness system. Self-contained; does NOT import w8_biayn.
harness/interface.py — typed Harness contract + extraction/validation of model-emitted code.search/tree_search.py — Thompson-sampling search; to_trace() / steps_to_first_gold() are the
search-budget metric. search/mutation.py — vLLM-backed + deterministic mutation operators.rewards/composite.py — legal-action accuracy + DOMDiff composite, safe code execution.
rewards/daytona_executor.py — runs emitted harness code in a Daytona sandbox.integrations/legality_env.py (HarnessLegalityEnv, SkyRL-Gym) + skyrl_harness_main.py.data/legality_dataset.py (cold-start harvest), data/harvest.py, data/sft_dataset.py.experiments/trace_once.py / trace_compare.py — generate and compare before/after traces.
src/w8_biayn/ — thin command/control: cli.py (Typer), sky_config.py (SkyPilot render),
gcp_auth.py + secrets.py (service-account auth), monitor.py (watch), upstreams.py
(pinned SkyRL clone), benchmarks.py, constants.py, shell.py. This is the trimmed remnant of
the browser-is-all-you-need substrate.
The single benchmark is harness-legality; the single pipeline is r3. Everything runs on GCP.
Core workflows
Validate / test locally (no GPU, no spend):
uv run --extra dev --extra browser pytest
uv run python -m compileall src
uv run w8-biayn --help
uv run w8-biayn launch r3 --benchmark harness-legality --dry-run
Run the trace-shrinkage experiment on GCP (renders SkyPilot YAML, sky launch -y --down):
uv run w8-biayn doctor --cloud
uv run w8-biayn upstreams clone
uv run w8-biayn launch r3 --benchmark harness-legality \
--harness-model Qwen/Qwen2.5-Coder-1.5B-Instruct \
--accelerators A100:8 --gold-threshold 0.90 \
--train-count 512 --train-examples-per-obs 24 \
--trace-task-seed 7 --trace-examples-per-obs 24 \
--trace-max-iterations 16 --grpo-epochs 2 --sft-steps 20
uv run w8-biayn watch
The run does five steps on one node: frozen trace → optional oracle SFT warm-start → GRPO
post-training → trained HF export → trained trace + strict trace_compare. Difficulty knobs need no
code change: --harness-model, --gold-threshold, --train-count,
--train-examples-per-obs, --trace-task-seed, --trace-examples-per-obs, --grpo-epochs, and
--sft-steps (also env W8_HARNESS_MODEL, W8_HARNESS_GOLD_THRESHOLD,
W8_HARNESS_SFT_STEPS).
For trace shrinkage, equal steps_to_gold is not success. The after-trace must be strictly shorter;
steps_to_gold = 0 is the ideal zero-shot trained-model result. Keep trace tasks balanced so both
legal and illegal examples are present; all-illegal traces make always-reject predicates look good.
The tracked June 1, 2026 proof run is in docs/results/r3-a100-2026-06-01/: frozen did not reach
gold in 17 nodes, trained reached gold at node 0, and trace_compare.json has shrunk=true.
Hard rules (do not violate)
- GCP auth is service-account JSON only. Never run
gcloud auth or mutate global gcloud config;
the CLI mints tokens from .gcp-service-account.json via scoped env vars (gcp_auth.py).
- If direct SSH to GCE is blocked, use Google Cloud SDK IAP TCP forwarding (
start-iap-tunnel) with
roles/iap.tunnelResourceAccessor and firewall source 35.235.240.0/20 for tcp:22; do not open
broad SSH ingress.
- Never commit
.env, .gcp-service-account.json, .cache/, .w8-biayn/, .venv/, *.parquet.
Output dirs are ignored at repo root only (/data/ etc.) so src/autoharness/data stays tracked.
- Small proof traces and compare files can be tracked under
docs/results/<run-id>/; keep bulky
generated datasets/checkpoints and .w8-biayn/ outputs ignored.
- Do not vendor SkyRL. Clone the pinned SHA via
w8-biayn upstreams clone.
- All GPU work runs on GCP through
w8-biayn launch (dry-run capable, sky launch -y --down).
No local-GPU dependency.
- Verify on GCP after a logical change: dry-run the render, then a minimal real launch (cheap
model via
--harness-model), and confirm it succeeds.
- Tear down clusters when pausing:
uv run w8-biayn down w8-biayn-r3, confirm with sky status.
- Keep secret forwarding out of shell tracing;
DAYTONA_API_KEY must not appear in SkyPilot logs.
- SkyRL passes parquet row metadata to envs as nested
extra_info; HarnessLegalityEnv must unwrap
it before scoring. If GRPO logs all-zero reward, check n_examples before tuning the trainer.
- SkyRL SFT reads the oracle warm-start data from a local dataset directory
(
$HOME/data/w8-biayn/harness-legality/sft), and HF exports load from global_step_*/policy.
When extending the training config (sky_config.py)
The GRPO render in harness_run_script hit a stack of GPU/SkyRL pitfalls; the fixes are subtle and
must be preserved. Before editing the trainer config, read
references/training-pitfalls.md — it documents the OOM,
CUDA-IPC weight-sync, placement-group, and config.json-export issues and the exact fix for each.
When adding/operating skills or symlinks
Keep the real skill in .agents/skills/<name>/ and symlink the Claude/Gemini paths to it (relative
targets). See references/skill-maintenance.md and run
python3 .agents/skills/agent-skills-framework/scripts/validate_skill.py .agents/skills/w8-biayn-framework.
Documentation rule
When code changes any setup step, CLI command, rendered config, cloud behavior, or the pipeline,
update in the same change: README.md (and its Mermaid diagram), AGENTS.md, and this skill.
Pre-handoff checklist
uv run --extra dev --extra browser pytest
uv run python -m compileall src
uv run w8-biayn launch r3 --benchmark harness-legality --dry-run
python3 .agents/skills/agent-skills-framework/scripts/validate_skill.py .agents/skills/w8-biayn-framework
