level3-ppo-loop

name	level3-ppo-loop
description	Use when training, evaluating, or tuning the Level3 PPO drone-racing loop for final evaluation on config/level3.toml with the target of at most 7.0s mean successful time and at least 60 percent success.

Level3 PPO Loop

Use this workflow for Level3 PPO train/evaluate/tune work.

For Level3 MPPI oracle/controller work, MPPI hard evals, MPPI teacher trajectories, or MPPI-generated PPO imitation data, use .agents/skills/level3-mppi-loop/SKILL.md instead.

Contract

• Target config: config/level3.toml.
• Target gate: success rate >= 0.60 and mean successful time <= 7.0s.
• State file: experiments/level3_ppo_loop/state.json.
• Orchestrator: scripts/level3_ppo_loop.py.
• Default controller for level3 DR simulation: ppo_level3_inference.py.
• Default W&B project: ADR-PPO-Racing-Level3.
• Use --from-scratch only when intentionally creating a random-init baseline; otherwise continue from the best checkpoint in state, or the latest level3_DR_initial checkpoint if no best exists yet.
• Reward-only mode is no longer the default. The user has approved structural search for Level3, including observation layout, controller, reward structure, PPO/training structure, and hyperparameter changes.
• Hard boundary: do not edit Level3 track geometry/randomization to make the task easier, and do not accept any result unless it is hard-evaluated on config/level3.toml. The final target is the real Level3 track in level3.toml; level3_dr.toml is only a domain-randomized sim-to-real robustness/training config.
• Training curricula or alternate train configs may be explored only as named structural lanes. Final acceptance and state best must always come from hard eval on config/level3.toml.
• Each structural lane must have a clear hypothesis, source or local evidence, a unique proposal/run name, W&B logging, checkpoint milestone evaluation, and a post-run analysis packet before the next training chunk.
• Observation-layout experiments are open. The active deployed observation baseline is the remote-inspired local-obstacle v5 layout level3_target_gate_nearest_gate_2obs_local_history_v5; do not restore the rejected all-gates/v4 lane unless the user explicitly asks.
• v51 explicitly opens planner-guidance observation as a deployed observation layout. The planner may compute deterministic route-intent features at inference, but it must not output actions, override PPO actions, act as MPC, provide a safety shield, replay static seed-specific routes, or modify the Level3 track. The PPO Actor remains the only action-producing controller.
• MPPI oracle/controller work is handled by the separate level3-mppi-loop skill. Do not implement MPPI action output, MPPI hard eval, or MPPI teacher-data generation under this PPO skill.
• The current structural roadmap is experiments/level3_ppo_loop/research/2026-06-22_level3_framework_structural_training_plan.md. The latest pasted-framework synthesis is experiments/level3_ppo_loop/research/2026-06-22_level3_framework_pasted_structural_update.md. Its priority order is PPO correctness, clean longer-rollout baseline, observation/return normalization, asymmetric privileged critic, gate-phase reset curriculum, prioritized level replay, GRU, reward numbers, then speed.
• v32 asymmetric privileged-Critic support and zero-update Actor parity have passed. The first v32 training screen, loop099, reached 19% success / 1.66 mean gates / 81% crash at its 3M checkpoint on config/level3.toml, close to but not better than the loop097 global best of 20% / 1.66 / 80%.
• loop100 matured v32 from loop099 3M to about 18M and still did not beat loop097: best loop100 was 19% success / 1.65 mean gates / 81% crash on unchanged config/level3.toml. Do not continue v32 privileged-Critic maturation without a new explicit decision packet.
• loop101 tested v33 gate-phase reset curriculum for 10M and tied, but did not beat, the old frontier: best loop101 was 20% success / 1.69 mean gates / 80% crash with 6.873s mean successful time. The 8M checkpoint reached 1.81 mean gates but only 19% success. Do not continue v33 as-is without a new explicit decision packet.
• loop102 tested v34 offline train-pool PLR for 10M and regressed: best loop102 was only 17% success / 1.59 mean gates / 83% crash, and final fell to 10% success / 1.43 mean gates / 90% crash. Do not continue v34 as-is and do not start future training from loop102 checkpoints.
• loop106 tested v36 online competence-gated level replay for 10M and did not beat the loop101 frontier. Its best checkpoint was 1M with 20% success / 1.63 mean gates / 80% crash / 7.744s, and final collapsed to 14% success / 1.41 mean gates / 86% crash. Do not continue v36, tune replay probability, or start future training from loop106 checkpoints.
• loop107 tested v37_gru_transfer_memory_structure_from_loop101 for 5M. Its best checkpoint was the 1M checkpoint with 21% success / 1.66 mean gates / 79% crash / 7.578s; later checkpoints drifted down to 15%, 12%, 12%, and 17% success. Do not continue v37 from loop107 final.
• loop112 tested v40_sequence_memory_gru_phase_corridor_from_scratch and regressed hard: every 1M/2M/3M/4M/final milestone had 0% success and 0.0 mean gates, with all failures at gate 0. Do not continue v40 as-is, do not mature it to a long run, and do not start future training from loop112 checkpoints.
• v41 audited GRU/v10 wiring after loop112 and passed: recurrent Actor zero-update parity, train/eval action-scale parity, hidden-state reset/carry parity, v10 observation parity/sanity, and recurrent PPO gradient/update sanity were all clean. This rules out an obvious wiring bug as the cause of v40's zero-gate failure.
• loop113 tested v42_gru_v10_gate_phase_reset_curriculum_from_scratch and also failed to acquire normal-start gate progress: best was the 4M checkpoint with 0% success, 0.01 mean gates, 54% crash, and 46% timeout. Do not continue v42 as-is and do not start future training from loop113 checkpoints.
• loop114 tested v43_success_trajectory_imitation_warmstart_gru_v10 and failed to preserve the BC first-gate signal: best was the 5M checkpoint with 0% success, 0.01 mean gates, 51% crash, and 49% timeout. The highest mean-gates milestone was only 0.06, below the BC-only diagnostic's 0.15. Do not continue v43 as-is and do not start future training from loop114 checkpoints.
• loop115 tested v44_sequence_success_retention_failure_correction_gru_v10 and proved recurrent sequence retention was active, but did not convert it into hard-eval progress. All milestones had 0% success; the best analyzer checkpoint was 2M with 0.06 mean gates, and the final checkpoint reached only 0.17 mean gates with 95% crash. Do not continue v44 as-is and do not start future training from loop115 checkpoints.
• loop116 tested v45_v5_frontier_seed_union_retention_mlp_from_loop110_3m and proved flat v5 MLP retention was active, but did not beat the frontier. Best was the 4M checkpoint with 20% success, 1.60 mean gates, 80% crash, and 6.941s mean successful time; final was 19% success and 1.59 mean gates. Do not continue v45 as-is and do not start future training from loop116 final.
• loop117 tested v47_v5_residual_frontier_union_retention_mlp_from_loop110_3m and proved residual-frontier union retention was active, but did not beat the frontier. Best was the 3M checkpoint with 20% success, 1.58 mean gates, 80% crash, and 7.064s mean successful time. Teacher KL/MSE/agreement improved, so do not continue v47 as-is and do not start future training from loop117 final.
• loop118 tested v48_v5_contact_conversion_reward_structure_from_loop110_3m and regressed: best was the 1M checkpoint with 16% success, 1.50 mean gates, 84% crash, and 6.516s mean successful time. Do not continue v48, do not mature it, and do not start future work from loop118 checkpoints.
• loop119/loop120 tested the v49 hidden512 long-horizon family. loop119 reached the 45M checkpoint before interruption; loop120 recovered from that checkpoint for the effective 45M-to-60M read. Best loop120 was only 15% success, 1.50 mean gates, 85% crash, and 6.741s, with later milestones falling to 14% and 13% success. W&B showed near-zero approx_kl, zero clipfrac, near-zero policy loss, rising entropy, and no gate/finish conversion. Do not continue v49 as-is toward 90M/120M and do not start future work from loop120 final.
• loop121 tested v50_hidden512_update_pressure_conversion_from_loop110_3m for 30M. It fixed the v49 near-zero PPO update-pressure symptom, but hard-eval conversion was weak: best checkpoint was 25M with 18% success, 1.56 mean gates, 80% crash, and 6.283s mean successful time; final fell to 15% success and 1.45 mean gates. It did not meet the target or beat the global best. Do not continue v50 as the next immediate move.
• loop122 tested v51_planner_guidance_obs_ppo256_from_loop110_3m, approved by experiments/level3_ppo_loop/decisions/2026-06-24_loop121_reject_v50_launch_v51_planner_guidance_obs_ppo256.md. It appends deterministic planner-guidance features to the v5 observation, uses planner computation at inference only as observation, keeps a 2x256 Tanh PPO Actor as the only action source, warm-starts from loop110/v39 3M by zero-padding appended planner-channel weights, keeps v39/v50 reward and PPO update-pressure settings, and hard-evaluates on unchanged config/level3.toml. Its best checkpoint was 10M with 18% success, 1.42 mean gates, 81% crash, and 6.991s mean successful time. It did not beat the frontier.
• v52/v53 non-PPO controller exploration did not produce a promoted controller. The v53 virtual Level2-gate adapter was action-finite on seeds 101-105, but scored 0% success and 0.00 mean gates. Do not continue virtual-gate adapter tuning as the primary route.
• The immediate next PPO structural lane is v54_reference_trajectory_tracker_ppo, approved by experiments/level3_ppo_loop/decisions/2026-06-25_launch_v54_reference_tracker_ppo.md. This lane trains a native low-level PPO tracker for hover, point tracking, local reference trajectory tracking, heading alignment, gate-aperture centering, and obstacle-aware low-speed control. The upper planner owns route choice, slowdown near 0.7m-1.0m, visible-geometry replanning, and reference generation. Do not launch long training until builder/checker support and hover/point/gate-aperture smoke checks pass.
• Early checkpoints are diagnostic health checks, not growth exams. If a lane creates a 1M checkpoint, use it only for NaNs, action/observation mismatch, checkpoint metadata, W&B logging, PPO health, or catastrophic zero-gate failure. For v50, use 5M/10M/15M/20M as health and trend checks; do not accept any result without hard eval on unchanged config/level3.toml.
• loop103 tested v35 competence-gated gate-phase reset for 10M and did not beat the loop101 frontier: best loop103 was 19% success / 1.68 mean gates / 81% crash with 7.245s mean successful time, and final fell to 17% success / 1.58 mean gates / 83% crash. Do not continue v35 as-is and do not start future training from loop103 checkpoints.
• The rejected previous lane was v31d_longer_rollout_maturation_from_loop097_12m_to_30m. It reached only 19% success / 1.63 mean gates / 81% crash at its best loop098 checkpoint, worse than the loop097 global best of 20% success / 1.66 mean gates / 80% crash. Do not continue v31d clean-PPO maturation without a new explicit decision packet.
• The stateirving reference packet experiments/level3_ppo_loop/research/2026-06-19_stateirving_level3_remote_reference.md is the current source-backed packet for v5 and remote reward-scale evidence.
• The orchestrator still retains historical structural hypotheses such as v5_localobs_remote_reward, but prefer the current framework/decision packet unless the user explicitly asks to revisit an older lane.
• Plateau guard is active by default. If consecutive evaluated trials do not improve success rate or mean gates, the loop should hold unless the user explicitly provides a new hypothesis or enables automatic hypothesis search.
• A state-level hold guard may be active after a post-run audit. If state.json records stage2_after_loop014_escalation_audit or another hold requiring user approval, scripts/level3_ppo_loop.py must hold before training unless the command explicitly passes --override-state-hold plus a named structural command and/or explicit parameter values with an --approved-hypothesis-packet.
• Use --keep-latest-params after analysis says the current hypothesis only needs more training evidence. It reuses the latest reward numbers and avoids an automatic reward proposal for that chunk.
• Use --auto-hypothesis-search to let the loop rotate through predefined reward-number hypotheses after plateau. Use --relaxed-reward-bounds only when wider reward-number search is intended.
• Use --codex-autonomous-loop for the user's unattended Codex-supervised mode. It records standing authorization for Codex to spawn analysis/research subagents and choose the next structural or reward hypothesis without per-run confirmation. It enables automatic structural search and does not bypass analysis, hard-eval, or long-run guards.
• Use the Level2 checkpoint step-curve packet experiments/level3_ppo_loop/analysis/2026-06-18_level2_checkpoint_step_curve.md to calibrate Level3 training horizons: 30M is screening/debug evidence, while the first serious success-rate decision for a promising branch is 60M-90M. Promising means non-zero hard-eval success or meaningful gate progress.
• Use milestone-aware checkpoint evaluation for longer chunks. Evaluate intermediate 30M/60M/90M-style checkpoints plus recent checkpoints and never assume final.ckpt is best.
• Use --allow-step-curve-maturation only for a long continuation whose selected initial checkpoint already has promising hard-eval evidence recorded in state.json. It does not bypass the post-run analysis requirement.

Builder/Checker Gate

Use this gate before launching training for any structural lane that changes code or config semantics. It imports the user's loop pattern: builder writes, checker verifies, and the main agent arbitrates.

• Builder role: implement only the approved task. It may edit files, run local checks, and fix checker-reported root causes. It must read AGENTS.md, this skill, and relevant project config before editing, then report the goal, files changed, completion standard, and local check result.
• Checker role: read-only verification. It must not edit files. It must rediscover the relevant checks from repo config and the Level3 loop context, run/inspect them, and report either ALL GREEN with proof or FAILED with concrete file:line evidence and key error lines.
• Main-agent role: decide whether the checker evidence is sufficient to launch a train/evaluate chunk. The main agent must not let builder self-approval replace checker approval.
• Required scope: observation layout, planner-guidance observation, inference action path, PPO/training semantics, reward structure, evaluator/parity scripts, and loop orchestration changes.
• Optional scope: pure analysis packets, decision packets, and reader notes do not need the builder/checker gate unless they also change code semantics.
• Minimum checker expectations in this repo: git diff --check, Python compile checks for touched scripts/modules, the relevant loop --dry-run, and any parity/smoke command implied by the changed surface. Checker must explicitly verify that config/level3.toml track geometry/randomization was not modified.
• Templates for delegated prompts live under .agents/skills/level3-ppo-loop/agents/.

Steps

1. Read experiments/level3_ppo_loop/state.json if it exists.
2. Inspect the latest best checkpoint and recent trial summaries.
3. Run a dry run before changing the loop command: pixi run -e gpu python scripts/level3_ppo_loop.py --dry-run.
4. If the next structural lane requires code or config-semantics changes, run the builder/checker gate first. Do not launch training until checker reports approval or the main agent writes an explicit hold.
5. For real work, run one bounded train/evaluate iteration: pixi run -e gpu python scripts/level3_ppo_loop.py --max-iterations 1 --wandb-enabled. In the user's unattended Codex-supervised loop, still use one script iteration at a time with --codex-autonomous-loop; do not use --allow-unanalysed-multi-iteration.
6. After every completed train/evaluate iteration, build a post-run analysis packet before choosing another training command: pixi run -e gpu python scripts/analyze_level3_ppo_trial.py --wandb-entity aojili77-technical-university-of-munich --require-wandb-online.
7. When the analysis is nontrivial, spawn exactly three separate subagents for evaluator metrics, W&B/PPO diagnostics, and structure/research synthesis using the generated briefs under experiments/level3_ppo_loop/analysis/.
8. The analyzer writes pending_post_run_decision into state. If its status is awaiting_main_agent_decision, do not launch the next train/evaluate chunk until the main agent has written a markdown decision packet under experiments/level3_ppo_loop/decisions/.
9. The main agent combines the analysis packet and subagent findings, then chooses exactly one of: stop_target_met, hold_for_more_analysis, continue_same_hypothesis, change_reward_or_training_numbers, or launch_named_structural_lane. The user's current loop is repeated structural search, so a rejected structural lane should normally lead to a newly named structural lane or an explicit hold, not to silently repeating the same failed lane.
10. After the main-agent decision packet, spawn one additional reader-note subagent to explain the completed loop in plain Chinese for the user. This reader-note subagent is separate from, and must not replace, the three required decision-review subagents. Write the final note under drone_notes/level3_loops/. Use pixi run -e gpu python scripts/write_level3_loop_reader_note.py --update-state as the metric/path scaffold when useful; pass a reader-summary file if the reader-note subagent produced extra prose.
11. The next training command after analysis must attach both provenance files: --analysis-packet <analysis.md> and --approved-hypothesis-packet <decision.md>.
12. If the user wants automatic tuning after plateau, dry-run a bounded screening command with --auto-structural-search or --auto-hypothesis-search; prefer 20M-30M before any 80M extension.
13. If a 30M branch shows non-zero hard-eval success or meaningful gate progress, prefer continuing the same hypothesis toward 60M and then 90M with --allow-step-curve-maturation, rather than immediately changing reward numbers.
14. For nontrivial parameter changes, request or create a research synthesis in experiments/level3_ppo_loop/research/ and attach it with --research-packet.
15. Stop when the hard gate is met. Report the checkpoint path and metrics.
16. Keep structural lanes explicit: proposal name, observation layout, controller/reward/training changes, source packet, and hard-eval summary must be recorded in state or a markdown packet.
17. Never modify config/level3.toml track geometry/randomization as a way to improve the metric. Any alternate training config, including level3_dr.toml, must be clearly labeled as training-only and still evaluated on config/level3.toml.

Tuning Rules

• If success is low and mean gates is low, emphasize gate shaping and finish reward before optimizing speed.
• If multiple gate+safe trials plateau, do not keep scaling the same parameters. Move to a named structural hypothesis or hold for an evidence-backed review.
• --relaxed-reward-bounds may be used for exploratory screening, but only with evaluator gates and W&B conversion checks.
• For random-init experiments, use shorter checkpoint intervals so the loop can evaluate early candidates before committing to very long continuation.
• If crash rate or tilt is high, increase safety and smoothness penalties before pushing time.
• If success meets the target but time is too slow, increase time pressure and relax smoothness slightly.
• Treat PPO instability metrics as diagnostics only. Do not tune learning_rate, ent_coef, target_kl, num_minibatches, update_epochs, hidden_dim, or n_obs silently. They are allowed only as named structural/training-structure lanes with hard-eval follow-up.
• Do not treat W&B reward curves as the acceptance criterion. The evaluator CSV is the gate.
• Use W&B curves to diagnose why evaluator metrics moved: reward component balance, gate progress, finish/crash/timeout rates, tilt/smoothness pressure, KL/clip fraction, entropy, explained variance, and SPS.
• W&B analysis must happen after each completed iteration and before launching the next training iteration.
• Use W&B for live train curves and post-train evaluator metrics. If W&B auth is missing, ask the user to run pixi run -e gpu wandb login or set WANDB_API_KEY.
• After completing code, loop-state, analysis, decision, or loop-instruction changes, commit them and push to aojili-test/main by default. Before committing, inspect git status and avoid adding checkpoints, W&B run directories, CSV/NPZ datasets, logs, caches, or other bulky generated training artifacts unless the user explicitly asks.
• Commit small reader notes under drone_notes/level3_loops/ after each loop so the user can review what happened without opening state JSON or W&B. Keep drone_notes/.obsidian/ ignored.
• External evidence should influence the next experiment through a written synthesis packet. Do not let papers or GitHub examples override local metrics; use them to choose structural or reward hypotheses, then let hard eval on config/level3.toml decide.
• Do not edit notebooks unless the user asks. Keep loop changes in scripts, state, and small docs.

Good Prompts

• "Use $level3-ppo-loop and run one dry-run."
• "Use $level3-ppo-loop, run one W&B-tracked 20M-step iteration, then summarize the best checkpoint and next tuning move."
• "Use $level3-ppo-loop to inspect state and propose the next parameter change without launching training."
• "Use $level3-ppo-loop to analyze the latest trial with W&B curves, ask subagents to review the analysis, then propose the next structural or reward move."
• "Use $level3-ppo-loop to dry-run a v5 local-obstacle observation experiment, but do not start training yet."
• "Use $level3-ppo-loop to dry-run --structural-hypothesis v5_localobs_remote_reward."