• Home
    Tables – Mathematics – Optimization Algorithms

    No.AlgorithmDescription/Key Equation
    1Gradient DescentUpdates parameters in the negative direction of the gradient
    2Stochastic Gradient Descent(SGD)Updates parameters using the gradient of a single randomly-selected instance
    3Mini-Batch Gradient DescentCompromise between Batch Gradient Descent and SGD, it uses a mini-batch of m instances
    4MomentumGradually build up speed if gradient keeps pointing in the same direction
    5Nesterov Accelerated GradientMeasures gradient of the cost function not at the local position but slightly ahead in the direction of the momentum
    6AdagradAdapts the learning rates to the parameters, performing smaller updates for parameters associated with frequently occurring features
    7RMSPropFixes the diminishing learning rates problem of Adagrad by accumulating only the most recent iterations
    8Adam OptimizationCombines the advantages of RMSProp and momentum
    9AdaMaxA variant of Adam Optimization with more stable behavior in terms of large gradient steps
    10NadamStands for Nesterov-accelerated Adaptive Moment Estimation, combines Nesterov and Adam.
    11FtrlFollow the Regularized Leader – it combines L1 and L2 regularization
    12Newton’s MethodUses second-order information to define a quadratic approximation of the loss function and then optimizes the quadratic approximation
    13Broyden–Fletcher–Goldfarb–Shanno (BFGS)Quasi-Newton method for optimizing
    14Conjugate GradientAn algorithm for the numerical solution of particular systems of linear equations
    15Covariance Matrix Adaptation Evolution Strategy (CMA-ES)An evolutionary algorithm for difficult non-linear non-convex optimization problems in continuous domain
    16Particle Swarm Optimization (PSO)A method that optimizes a problem by iteratively trying to improve a candidate solution with regard to a given measure of quality
    17Ant Colony Optimization (ACO)An algorithm for finding optimal paths that is based on behavior of ants searching for food
    18Genetic Algorithm (GA)A heuristic that mimics the process of natural selection, using methods such as mutation and crossover to generate new genotype in the hope of finding good solutions to a given problem
    19Simulated Annealing (SA)A probabilistic technique for approximating the global optimum of a given function
    20Tabu SearchA metaheuristic that guides a local heuristic search procedure to explore the solution space beyond local optimality