optimization

name	optimization
description	Mathematical optimization including linear programming, convex optimization, gradient descent, and constrained optimization for machine learning and engineering.
category	mathematics
tags	["mathematics","optimization","linear-programming","convex-optimization","gradient-descent","constrained-optimization","machine-learning"]
difficulty	intermediate
author	neuralblitz

Mathematical Optimization

What I do

I provide comprehensive expertise in mathematical optimization, the field concerned with finding the best solution from feasible alternatives. I enable you to formulate and solve optimization problems including linear programming, convex optimization, unconstrained and constrained minimization, and integer programming. My knowledge spans from classical optimization techniques to modern machine learning optimization methods essential for operations research, engineering design, machine learning model training, and resource allocation problems.

When to use me

Use optimization when you need to: train machine learning models by minimizing loss functions, allocate resources efficiently in business operations, design systems subject to constraints (weight, cost, performance), find optimal parameters through grid or random search, solve linear programming problems for logistics, perform hyperparameter tuning for ML models, minimize energy or cost functions in physics simulations, or optimize portfolios in financial applications.

Core Concepts

• Objective Functions: Functions to be minimized or maximized representing the goal of the optimization problem.
• Feasibility and Constraints: Conditions that feasible solutions must satisfy including equality, inequality, and bound constraints.
• Convexity: A property ensuring any local minimum is a global minimum, simplifying optimization significantly.
• Gradient-Based Methods: Optimization algorithms using gradient information to guide search directions toward minima.
• Linear Programming: Optimization of linear objective functions subject to linear equality and inequality constraints.
• KKT Conditions: Necessary and sufficient conditions for optimality in constrained optimization problems.
• Dual Problems: Reformulated optimization problems providing bounds and insights into primal solutions.
• Convergence Analysis: Understanding how quickly optimization algorithms approach optimal solutions.
• Stochastic Optimization: Methods using random sampling to handle noisy objectives and escape local minima.

Code Examples

Gradient-Based Optimization

import numpy as np

def gradient_descent(f, gradient, x0, learning_rate=0.01, max_iter=1000, tol=1e-8):
    """Standard gradient descent with momentum."""
    x = np.array(x0, dtype=float)
    velocity = np.zeros_like(x)
    momentum = 0.9
    
    for i in range(max_iter):
        grad = gradient(x)
        velocity = momentum * velocity - learning_rate * grad
        x = x + velocity
        
        if np.linalg.norm(grad) < tol:
            print(f"Converged at iteration {i}")
            break
    
    return x

def adam(f, gradient, x0, lr=0.001, beta1=0.9, beta2=0.999, 
         epsilon=1e-8, max_iter=1000, tol=1e-8):
    """Adam optimizer combining momentum and RMSprop."""
    x = np.array(x0, dtype=float)
    m = np.zeros_like(x)  # First moment
    v = np.zeros_like(x)  # Second moment
    
    for i in range(max_iter):
        grad = gradient(x)
        
        m = beta1 * m + (1 - beta1) * grad
        v = beta2 * v + (1 - beta2) * (grad ** 2)
        
        m_hat = m / (1 - beta1 ** (i + 1))
        v_hat = v / (1 - beta2 ** (i + 1))
        
        x = x - lr * m_hat / (np.sqrt(v_hat) + epsilon)
        
         np.linalg.norm(grad) < tol:
            ()
            
    
     x


 ():
     ( - x[])** +  * (x[] - x[]**)**

 ():
     np.array([
        - * ( - x[]) -  * x[] * (x[] - x[]**),
         * (x[] - x[]**)
    ])

x0 = [-, ]
x_opt = gradient_descent(rosenbrock, rosenbrock_grad, x0, learning_rate=)
()
()
()


x_adam = adam(rosenbrock, rosenbrock_grad, x0, lr=)
()
()

Linear Programming with Scipy

import numpy as np
from scipy.optimize import linprog, milp, Bounds

# Linear programming: minimize c^T x subject to A_ub x <= b_ub, A_eq x = b_eq

# Example: Maximize profit in production planning
# Variables: x1 = units of product A, x2 = units of product B
# Maximize: 3*x1 + 5*x2
# Subject to:
#   x1 + 2*x2 <= 8 (labor hours)
#   4*x1 + 3*x2 <= 24 (materials)
#   x1 >= 0, x2 >= 0

c = [-3, -5]  # Minimize negative for maximization
A_ub = [[1, 2], [4, 3]]
b_ub = [8, 24]
bounds = [(0, None), (0, None)]

result = linprog(c, A_ub=A_ub, b_ub=b_ub, bounds=bounds, method='highs')

print("Linear Programming Result:")
print(f"  Optimal x: {result.x}")
print(f"  Optimal value (profit): {-result.fun:.2f}")

# Sensitivity analysis (shadow prices)
print(f"\nShadow prices (reduced costs):")
print(f"  Labor: {result.slack[0]:.2f}")
print(f"  Materials: {result.slack[1]:.2f}")

# Transportation problem

cost_matrix = np.array([[, , ], [, , ]])
supply = [, ]  
demand = [, , ]  


c_transport = cost_matrix.flatten()
A_eq = [[, , , , , ], [, , , , , ], [, , , , , ], [, , , , , ]]
b_eq = supply + demand

result_transport = linprog(c_transport, A_eq=A_eq, b_eq=b_eq, bounds=[(, )]*, method=)
()
()
()

Convex Optimization

import numpy as np
from scipy.optimize import minimize

def convex_quadratic(x, Q, c):
    """Convex quadratic function: f(x) = 0.5*x^T Q x + c^T x."""
    return 0.5 * x @ Q @ x + c @ x

def convex_grad(x, Q, c):
    """Gradient of convex quadratic."""
    return Q @ x + c

# Define convex quadratic: f(x) = x^2 + 2y^2 + 3z^2 + 2xy + 3yz + 4zx
Q = np.array([
    [2, 2, 4],
    [2, 4, 3],
    [4, 3, 6]
])
c = np.array([0, 0, 0])

# Check convexity (Q should be positive semi-definite)
eigenvalues = np.linalg.eigvalsh(Q)
print(f"Q eigenvalues: {eigenvalues}")
print(f"Is convex: {all(e >= -1e-10 for e in eigenvalues)}")

# Find minimum using Newton's method
from scipy.optimize import minimize

result = minimize(
    convex_quadratic,
    np.array([1.0, 1.0, 1.0]),
    args=(Q, c),
    method='Newton-CG',
    jac=convex_grad,
    hess=lambda x: Q
)

print(f"\nConvex optimization result:")
()
()


 ():
    
     scipy.optimize  NonlinearConstraint
    
     ():
         x[]** + x[]** + x[]**
    
     ():
          * x
    
    constraint = NonlinearConstraint(
         x: np.linalg.norm(x),
        , 
    )
    
    result = minimize(objective, np.array([, , ]), 
                     method=, jac=gradient, 
                     constraints=[constraint])
     result

result_tr = trust_region_example()
()

Constrained Optimization

import numpy as np
from scipy.optimize import minimize, LinearConstraint, NonlinearConstraint

# Constrained optimization using SLSQP
def constrained_optimization():
    """Minimize f(x,y) = (x-1)^2 + (y-2)^2 subject to:
    x + y >= 3 (inequality constraint)
    x^2 + y^2 <= 25 (inequality constraint)
    x >= 0, y >= 0 (bounds)
    """
    def objective(x):
        return (x[0] - 1)**2 + (x[1] - 2)**2
    
    def gradient(x):
        return np.array([2 * (x[0] - 1), 2 * (x[1] - 2)])
    
    # Constraints
    constraints = [
        {'type': 'ineq', 'fun': lambda x: x[0] + x[1] - 3},  # x + y >= 3
        {'type': 'ineq', 'fun': lambda x: 25 - x[0]**2 - x[1]**2}  # x^2 + y^2 <= 25
    ]
    
    bounds = [(0, None), (0, None)]
    
    result = minimize(objective, np.array([, ]), 
                     method=, jac=gradient,
                     bounds=bounds, constraints=constraints)
    
     result

result_constrained = constrained_optimization()
()
()
()
()
()
()


 ():
    
    x = np.array(x0, dtype=)
    
     i  (max_iter):
        
         ():
             f(x_aug) + penalty * constraint_violation(x_aug)**
        
         ():
             gradient(x_aug) +  * penalty * constraint_violation(x_aug) * constraint_grad(x_aug)
        
        x = gradient_descent(aug_obj, aug_grad, x, learning_rate=)
        
         constraint_violation(x) < tol:
            
    
     x


 ():
    
     ():
         (x[] - )** + (x[] - )**
    
     ():
         np.array([ * (x[] - ),  * (x[] - )])
    
    constraints = [
        {: , :  x: x[] + x[] - },  
        {: , :  x: x[] - }  
    ]
    
    result = minimize(objective, np.array([, ]), 
                     method=, jac=gradient,
                     constraints=constraints,
                     options={: , : })
     result

result_sqp = sqp_example()
()

Hyperparameter Optimization

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

# Grid Search
def grid_search(param_grid, X, y, cv=5):
    """Exhaustive grid search for hyperparameters."""
    best_score = -np.inf
    best_params = {}
    
    for params in param_grid:
        model = RandomForestClassifier(**params, random_state=42)
        scores = cross_val_score(model, X, y, cv=cv)
        mean_score = scores.mean()
        
        if mean_score > best_score:
            best_score = mean_score
            best_params = params
    
    return best_params, best_score

# Random Search
def random_search(param_distributions, X, y, n_iter=50, cv=5):
    """Random search over hyperparameter distributions."""
    best_score = -np.inf
    best_params = {}
    
    for _ in range(n_iter):
        params = {k: np.random.choice(v) for k, v in param_distributions.items()}
        model = RandomForestClassifier(**params, random_state=42)
        scores = cross_val_score(model, X, y, cv=cv)
        mean_score = scores.mean()
        
        if mean_score > best_score:
            best_score = mean_score
            best_params = params
    
    return best_params, best_score

# Example parameter grids
param_grid = [
    {'n_estimators': 100, 'max_depth': , : },
    {: , : , : },
    {: , : , : },
]

param_distributions = {
    : [, , , ],
    : [, , , , , ],
    : [, , ],
    : [, , ]
}


X, y = make_classification(n_samples=, n_features=, random_state=)


best_grid, score_grid = grid_search(param_grid, X, y)
()
()

best_random, score_random = random_search(param_distributions, X, y, n_iter=)
()
()


 :
    
     ():
        .param_space = param_space
        .acquisition = acquisition
        .X_observed = []
        .y_observed = []
    
     ():
        
        
        
    
     ():
        
        
        
    
     ():
        
         _  (n_iter):
            
            X_next = ._choose_next_point()
            y_next = objective_func(X_next)
            
            .X_observed.append(X_next)
            .y_observed.append(y_next)
        
         (.y_observed), .X_observed[np.argmin(.y_observed)]

()

Best Practices

• Start with simpler optimization methods before moving to sophisticated algorithms; gradient descent often suffices for convex problems.
• Scale features to similar ranges when using gradient-based methods to ensure faster convergence.
• Use appropriate learning rate schedules (decay, warm-up) to balance exploration and exploitation.
• For non-convex problems, try multiple random initializations to escape poor local minima.
• When formulating linear programs, verify that the problem is feasible and bounded before solving.
• Use warm-start capabilities in iterative solvers when solving sequences of related problems.
• For constrained optimization, prefer specialized methods (SQP, interior-point) over penalty methods for better accuracy.
• Monitor optimization progress using multiple metrics (objective value, gradient norm, constraint violation).
• Consider computational complexity; some theoretically optimal methods may be impractical for large-scale problems.
• Validate optimization results by checking KKT conditions and testing against analytical solutions when available.