w8-biayn-framework

name

w8-biayn-framework

description

Operate and extend agent-rein, the AutoHarness post-training pipeline that RL-trains a small code model (via SkyRL GRPO on Google Cloud) to write browser action-verifier harnesses, measured by a before/after search-budget trace. Use when working in this repo on the w8-biayn CLI, the autoharness package (Harness interface, Thompson-sampling search, rewards, Daytona sandbox), SkyPilot/GCP launches, the harness-legality benchmark, the trace-shrinkage experiment, or the SkyRL/vLLM training config. Trigger terms include run the experiment, launch on GCP, trace shrinkage, steps_to_gold, harness reward, w8-biayn, autoharness, GRPO, SkyRL, Daytona sandbox, and gold-threshold.

agent-rein / w8-biayn framework

This repo post-trains the model that writes web-agent harnesses. AutoHarness uses a frozen LLM as a code mutation operator inside a Thompson-sampling tree search to synthesize an is_legal_action(state, action) -> bool harness; we RL-train that mutation operator (CaP-RL-style GRPO on a dense Qwen2.5-Coder) so it writes a working harness in fewer search steps. The headline result is a before/after steps_to_gold trace: frozen model → GRPO → trained model, run on one GCP node via SkyPilot + SkyRL.

Read NEW-SPEC.md for the full spec and README.md for the operational summary before making changes. This skill is the AI operating guide; the references below hold the detail.

Mental model (two packages)

• src/autoharness/ — the AutoHarness system. Self-contained; does NOT import w8_biayn.
  • harness/interface.py — typed Harness contract + extraction/validation of model-emitted code.
  • search/tree_search.py — Thompson-sampling search; to_trace() / steps_to_first_gold() are the search-budget metric. search/mutation.py — vLLM-backed + deterministic mutation operators.
  • rewards/composite.py — legal-action accuracy + DOMDiff composite, safe code execution. rewards/daytona_executor.py — runs emitted harness code in a Daytona sandbox.
  • integrations/legality_env.py (HarnessLegalityEnv, SkyRL-Gym) + skyrl_harness_main.py.
  • data/legality_dataset.py (cold-start harvest), data/harvest.py, data/sft_dataset.py.
  • experiments/trace_once.py / trace_compare.py — generate and compare before/after traces.
• src/w8_biayn/ — thin command/control: cli.py (Typer), sky_config.py (SkyPilot render), gcp_auth.py + secrets.py (service-account auth), monitor.py (watch), upstreams.py (pinned SkyRL clone), benchmarks.py, constants.py, shell.py. This is the trimmed remnant of the browser-is-all-you-need substrate.

The single benchmark is harness-legality; the single pipeline is r3. Everything runs on GCP.

Core workflows

Validate / test locally (no GPU, no spend):

uv run --extra dev --extra browser pytest      # full suite
uv run python -m compileall src
uv run w8-biayn --help
uv run w8-biayn launch r3 --benchmark harness-legality --dry-run

Run the trace-shrinkage experiment on GCP (renders SkyPilot YAML, sky launch -y --down):

uv run w8-biayn doctor --cloud           # GCP enabled + daytona ok/fallback
uv run w8-biayn upstreams clone          # pinned SkyRL into .cache/upstreams
uv run w8-biayn launch r3 --benchmark harness-legality \
  --harness-model Qwen/Qwen2.5-Coder-1.5B-Instruct \
  --accelerators A100:8 --gold-threshold 0.90 \
  --train-count 512 --train-examples-per-obs 24 \
  --trace-task-seed 7 --trace-examples-per-obs 24 \
  --trace-max-iterations 16 --grpo-epochs 2 --sft-steps 20
uv run w8-biayn watch                    # status + streamed logs + live steps_to_gold/RESULT

The run does five steps on one node: frozen trace → optional oracle SFT warm-start → GRPO post-training → trained HF export → trained trace + strict trace_compare. Difficulty knobs need no code change: --harness-model, --gold-threshold, --train-count, --train-examples-per-obs, --trace-task-seed, --trace-examples-per-obs, --grpo-epochs, and --sft-steps (also env W8_HARNESS_MODEL, W8_HARNESS_GOLD_THRESHOLD, W8_HARNESS_SFT_STEPS).

For trace shrinkage, equal steps_to_gold is not success. The after-trace must be strictly shorter; steps_to_gold = 0 is the ideal zero-shot trained-model result. Keep trace tasks balanced so both legal and illegal examples are present; all-illegal traces make always-reject predicates look good. The tracked June 1, 2026 proof run is in docs/results/r3-a100-2026-06-01/: frozen did not reach gold in 17 nodes, trained reached gold at node 0, and trace_compare.json has shrunk=true.

Hard rules (do not violate)

• GCP auth is service-account JSON only. Never run gcloud auth or mutate global gcloud config; the CLI mints tokens from .gcp-service-account.json via scoped env vars (gcp_auth.py).
• If direct SSH to GCE is blocked, use Google Cloud SDK IAP TCP forwarding (start-iap-tunnel) with roles/iap.tunnelResourceAccessor and firewall source 35.235.240.0/20 for tcp:22; do not open broad SSH ingress.
• Never commit .env, .gcp-service-account.json, .cache/, .w8-biayn/, .venv/, *.parquet. Output dirs are ignored at repo root only (/data/ etc.) so src/autoharness/data stays tracked.
• Small proof traces and compare files can be tracked under docs/results/<run-id>/; keep bulky generated datasets/checkpoints and .w8-biayn/ outputs ignored.
• Do not vendor SkyRL. Clone the pinned SHA via w8-biayn upstreams clone.
• All GPU work runs on GCP through w8-biayn launch (dry-run capable, sky launch -y --down). No local-GPU dependency.
• Verify on GCP after a logical change: dry-run the render, then a minimal real launch (cheap model via --harness-model), and confirm it succeeds.
• Tear down clusters when pausing: uv run w8-biayn down w8-biayn-r3, confirm with sky status.
• Keep secret forwarding out of shell tracing; DAYTONA_API_KEY must not appear in SkyPilot logs.
• SkyRL passes parquet row metadata to envs as nested extra_info; HarnessLegalityEnv must unwrap it before scoring. If GRPO logs all-zero reward, check n_examples before tuning the trainer.
• SkyRL SFT reads the oracle warm-start data from a local dataset directory ($HOME/data/w8-biayn/harness-legality/sft), and HF exports load from global_step_*/policy.

When extending the training config (sky_config.py)

The GRPO render in harness_run_script hit a stack of GPU/SkyRL pitfalls; the fixes are subtle and must be preserved. Before editing the trainer config, read references/training-pitfalls.md — it documents the OOM, CUDA-IPC weight-sync, placement-group, and config.json-export issues and the exact fix for each.

When adding/operating skills or symlinks

Keep the real skill in .agents/skills/<name>/ and symlink the Claude/Gemini paths to it (relative targets). See references/skill-maintenance.md and run python3 .agents/skills/agent-skills-framework/scripts/validate_skill.py .agents/skills/w8-biayn-framework.

Documentation rule

When code changes any setup step, CLI command, rendered config, cloud behavior, or the pipeline, update in the same change: README.md (and its Mermaid diagram), AGENTS.md, and this skill.

Pre-handoff checklist

uv run --extra dev --extra browser pytest
uv run python -m compileall src
uv run w8-biayn launch r3 --benchmark harness-legality --dry-run
python3 .agents/skills/agent-skills-framework/scripts/validate_skill.py .agents/skills/w8-biayn-framework