hyperparameter-optimization

name	hyperparameter-optimization
description	This skill should be used when the user asks about "hyperparameter tuning", "HPO", "Bayesian optimization", "Gaussian process", "acquisition function", "Expected Improvement", "Upper Confidence Bound", "Hyperband", "ASHA", "Optuna", "Ray Tune", "learning rate search", "grid search", "random search", "hyperparameter importance", "fANOVA", or when a model's validation loss is suboptimal and systematic tuning is needed.
version	1.0.0

Hyperparameter Optimization — Bayesian Search and Efficient Tuning

Provides methodology for finding optimal hyperparameter configurations λ* in high-dimensional spaces using Gaussian Process surrogate modeling, acquisition function optimization, and compute-efficient pruning strategies.

Why Bayesian Optimization Over Grid/Random Search

• Grid search: exponential cost in number of hyperparameters (O(k^d) for k points per dimension)
• Random search: sample-efficient over grid but ignores past observations
• Bayesian optimization: builds a probabilistic surrogate model of the objective — uses past evaluations to guide next trial intelligently

For d ≥ 3 hyperparameters and a budget of < 200 trials, Bayesian optimization typically requires 5–10× fewer trials than random search to reach the same optimum.

Gaussian Process Surrogate

The surrogate model f̂(λ) ~ GP(μ(λ), k(λ, λ')) encodes:

• Prior belief about the objective surface (encoded in the kernel k)
• Posterior updates as trial results arrive (Bayesian updating)

Kernel selection:

• Matérn 5/2: smooth but not infinitely differentiable — best for most HPO tasks
• RBF / Squared Exponential: infinitely smooth — can oversmooth rough objective surfaces
• Warped GP: apply input transformation for log-uniform parameters (learning rate, weight decay)

After n trials, the GP posterior provides: expected value μ_n(λ) and uncertainty σ_n(λ) at any new point λ.

Acquisition Functions

See references/acquisition-functions.md for mathematical derivations.

Summary:

• EI (Expected Improvement): E[max(0, f(λ) − f(λ*))]. Exploration-exploitation balance. Default choice.
• UCB (Upper Confidence Bound): μ(λ) + κ·σ(λ). κ controls exploration. Use when variance is high early.
• PI (Probability of Improvement): P(f(λ) > f(λ*)). Conservative — only cares about beating best.
• Thompson Sampling: sample a function from GP posterior; pick its argmax. Naturally diversifies search.

Hyperband Early Stopping

Hyperband extends SHA (Successive Halving Algorithm) to handle unknown budget:

1. Divide budget B into brackets with different η (halving rate).
2. In each bracket, sample n configurations and run to r_min resources.
3. Keep top 1/η fraction; run to next rung (r_min × η).
4. Repeat until budget exhausted.

ASHA (Asynchronous SHA): runs brackets in parallel without synchronization barriers — better GPU utilization.

Hyperband reduces wasted compute by 3–10× vs. running all trials to completion.

Hyperparameter Importance

After HPO, estimate which hyperparameters most influence the objective using fANOVA:

• Fit a random forest on (trial hyperparameters → trial L_val).
• Compute variance explained by each single hyperparameter and interaction pairs.
• High importance (> 10%): tune carefully in follow-up runs.
• Low importance (< 5%): can be fixed to a reasonable default.

See references/bayesian-optimization.md for full GP formulation, implementation details, and Optuna/Ray Tune integration patterns.

Search Space Design Rules

1. Use log-uniform for scale-invariant parameters (learning rate, weight decay, regularization).
2. Use categorical for optimizer type — treat as discrete.
3. Constrain batch size to powers of 2 for GPU memory alignment.
4. Always warm up with 5–10 random trials before fitting the GP.
5. Expand if best config is at a boundary — the true optimum may be outside the current range.