configuring-hyperparameter-tuning

name	Configuring Hyperparameter Tuning
description	This skill should be used when the user asks to "configure hyperparameter tuning", "set up Optuna for RL", "implement hyperparameter search", "use Ray Tune PBT", "implement population based training", "prune bad RL trials", "tune learning rate for RL", or "automate hyperparameter optimization". Do NOT hallucinate parameters outside the boundaries of Configure Hyperparameter Tuning.
version	0.1.0

Configuring Hyperparameter Tuning

RL hyperparameter tuning requires domain-specific sampling strategies. Learning rate and discount factor are not interchangeable with classification tasks: gamma defines the horizon length via 1 / (1 - gamma), making uniform sampling catastrophic. Two toolchains cover the full scale range: Optuna for single-machine studies and Ray Tune PBT for distributed co-evolution.

Optuna: Objective Function

Construct the objective(trial) function. Optuna samples from log-uniform distributions for multiplicative hyperparameters:

import optuna
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy

def objective(trial):
    # Log-uniform for learning rate — it spans orders of magnitude
    lr = trial.suggest_float("lr", 1e-5, 1e-2, log=True)

    # Categorical for gamma — horizon is 1/(1-gamma), profoundly non-linear
    # NEVER sample gamma uniformly; the effective horizon jumps from 10 to 1000
    gamma = trial.suggest_categorical("gamma", [0.9, 0.95, 0.99, 0.999])

    # Other hyperparameters
    n_steps = trial.suggest_categorical("n_steps", [512, 1024, 2048, 4096])
    ent_coef = trial.suggest_float("ent_coef", 1e-4, 0.1, log=True)

    model = PPO(
        "MlpPolicy",
        env,
        learning_rate=lr,
        gamma=gamma,
        n_steps=n_steps,
        ent_coef=ent_coef,
        verbose=0,
    )
    model.learn(total_timesteps=500_000)
    mean_reward, _ = evaluate_policy(model, eval_env, n_eval_episodes=10)
    return mean_reward

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=50)

Gamma sampling rule: gamma=0.99 gives a horizon of 100 steps; gamma=0.999 gives 1000 steps. These are qualitatively different regimes. Always use categorical values, never suggest_float("gamma", 0.9, 0.999).

Pruning Bad Trials

RL runs take days. Kill underperforming trials early with the MedianPruner:

pruner = optuna.pruners.MedianPruner(
    n_startup_trials=5,    # don't prune until 5 trials have completed
    n_warmup_steps=100_000 # don't prune in the first 100k steps of any trial
)
study = optuna.create_study(direction="maximize", pruner=pruner)

Inside objective, report intermediate values and check for pruning:

def objective(trial):
    ...
    for step in range(0, 1_000_000, 50_000):
        model.learn(total_timesteps=50_000)
        mean_reward, _ = evaluate_policy(model, eval_env, n_eval_episodes=5)
        trial.report(mean_reward, step)
        if trial.should_prune():
            raise optuna.TrialPruned()
    return mean_reward

Median Pruner kills any trial that has not reached at least the median reward of all completed trials by the same step count.

Population Based Training (PBT)

For hyper-complex environments where optimal hyperparameters shift during training, use Ray Tune PBT. PBT co-evolves a population of agents: the top 20% survive and perturb their hyperparameters; the bottom 20% are replaced with copies of top performers.

from ray import tune
from ray.tune.schedulers import PopulationBasedTraining

pbt = PopulationBasedTraining(
    time_attr="training_iteration",
    perturbation_interval=10,
    hyperparam_mutations={
        "lr": tune.loguniform(1e-5, 1e-2),
        "gamma": [0.9, 0.95, 0.99, 0.999],
    },
)

tuner = tune.Tuner(
    "PPO",
    run_config=tune.RunConfig(stop={"training_iteration": 200}),
    tune_config=tune.TuneConfig(
        scheduler=pbt,
        num_samples=16,   # population size
    ),
    param_space={
        "env": "HalfCheetah-v4",
        "lr": 3e-4,
        "gamma": 0.99,
    },
)
results = tuner.fit()

PBT is strictly superior to grid/random search for long training runs where early hyperparameter choices become suboptimal later.

See scripts/optuna_rl_study.py for a complete Optuna study with pruning, persistent storage, and parallel workers.

Decision Checklist

Scenario	Toolchain
Single machine, < 48h per trial	Optuna + MedianPruner
Multi-node, very long training	Ray Tune PBT
Gamma / discount tuning	Categorical only, never uniform
Learning rate, entropy	Log-uniform (log=True)

Additional Resources

Scripts

• scripts/optuna_rl_study.py — Complete Optuna study with pruner, SQLite storage, parallel workers, and best-trial export

References

• references/hyperparameter_ranges.md — Validated hyperparameter ranges per algorithm (PPO, SAC, TD3, DQN), gamma horizon table, PBT mutation schedules