ppo-credit

name	ppo-credit
description	Change Ola PPO credit assignment, reward scale, greedy feedback, or log_prob. Use when the user mentions AI策略, PPO, evolution_pref, Hamming, log_prob, reward, or greedy feedback.

PPO Credit Assignment

Online writes go through learner.step → decode_action → SafetyGuard.filter_action → apply_params_with_rollback. Do not expand ACTION_DIM.

Constraints (do not revert)

• encode_params_to_pref is an offset in [-0.2, 0.2] relative to 0.5. fuse_evolution_pref / apply_evolution_pref adds 0.1 * pref. Never mix pref as a [0,1] action (0.8 must not be pulled toward 0.2).
• apply_params_with_rollback returns (applied, rollback, ...). If applied is false (interval or unchanged params), do not push_transition and do not overwrite last_action.
• After filter_action, re-encode with encode_params_to_action so buffer / last_action match the landed params.
• After a greedy accept, call learner.sync_last_action and tuning::sync_applied_params so Hamming and LAST_PARAMS follow the verified set.
• cpu_usage is [0,1] (global_cpu_usage()/100). Idle gate is < 0.15. Power penalty is > 0.80. GPU busy is 0-100.
• compute_log_prob_scalar and compute_log_prob_batch both clamp (z-μ)/σ to ±10.
• PPO early-stop KL is E_{a~π_old}[log π_old - log π_new], not the reversed difference.
• Exploration EMA is 0.9*old + 0.1*new (ema_alpha=0.1).

After a stage

Follow session-memory: /home/wlq/wlq/record.sh with WLQ_DIR=/home/wlq/wlq.