gradient-methods

name	gradient-methods
description	Problem-solving strategies for gradient methods in optimization
allowed-tools	["Bash","Read"]

Gradient Methods

When to Use

Use this skill when working on gradient-methods problems in optimization.

Decision Tree

1. Basic Gradient Descent

   • Update: x_{k+1} = x_k - alpha * grad f(x_k)
   • Step size alpha: fixed, diminishing, or line search
   • Convergence: O(1/k) for convex, linear for strongly convex

2. Step Size Selection

   Method	Approach
   Fixed	alpha constant (requires tuning)
   Backtracking	Armijo condition: f(x - alphagrad) <= f(x) - calpha*
   Exact line search	minimize f(x - alpha*grad) over alpha
   Adaptive	Adam, RMSprop (ML applications)

3. Accelerated Methods

   • Momentum: add velocity term
   • Nesterov: look-ahead gradient
   • Conjugate gradient: for quadratic functions
   • scipy.optimize.minimize(f, x0, method='CG') - conjugate gradient

4. Newton's Method

   • Update: x_{k+1} = x_k - H^{-1} * grad f
   • Requires Hessian (expensive but quadratic convergence)
   • Quasi-Newton (BFGS): approximate Hessian