AdaGrad
ADAGRAD: adaptive gradient algorithm; Adaptive subgradient methods for online learning and stochastic optimization. We present a new family of subgradient methods that dynamically incorporate knowledge of the geometry of the data observed in earlier iterations to perform more informative gradient-based learning. Metaphorically, the adaptation allows us to find needles in haystacks in the form of very predictive but rarely seen features. Our paradigm stems from recent advances in stochastic optimization and online learning which employ proximal functions to control the gradient steps of the algorithm. We describe and analyze an apparatus for adaptively modifying the proximal function, which significantly simplifies setting a learning rate and results in regret guarantees that are provably as good as the best proximal function that can be chosen in hindsight. We give several efficient algorithms for empirical risk minimization problems with common and important regularization functions and domain constraints. We experimentally study our theoretical analysis and show that adaptive subgradient methods outperform state-of-the-art, yet non-adaptive, subgradient algorithms.
Keywords for this software
References in zbMATH (referenced in 97 articles , 1 standard article )
Showing results 1 to 20 of 97.
Sorted by year (- Aggarwal, Charu C.: Linear algebra and optimization for machine learning. A textbook (2020)
- Akyildiz, Ömer Deniz; Crisan, Dan; Míguez, Joaquín: Parallel sequential Monte Carlo for stochastic gradient-free nonconvex optimization (2020)
- Boffi, Nicholas M.; Slotine, Jean-Jacques E.: A continuous-time analysis of distributed stochastic gradient (2020)
- Burkhart, Michael C.; Brandman, David M.; Franco, Brian; Hochberg, Leigh R.; Harrison, Matthew T.: The discriminative Kalman filter for Bayesian filtering with nonlinear and Nongaussian observation models (2020)
- Erway, Jennifer B.; Griffin, Joshua; Marcia, Roummel F.; Omheni, Riadh: Trust-region algorithms for training responses: machine learning methods using indefinite Hessian approximations (2020)
- Jiang, Bo; Lin, Tianyi; Zhang, Shuzhong: A unified adaptive tensor approximation scheme to accelerate composite convex optimization (2020)
- Joulani, Pooria; György, András; Szepesvári, Csaba: A modular analysis of adaptive (non-)convex optimization: optimism, composite objectives, variance reduction, and variational bounds (2020)
- Kissas, Georgios; Yang, Yibo; Hwuang, Eileen; Witschey, Walter R.; Detre, John A.; Perdikaris, Paris: Machine learning in cardiovascular flows modeling: predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks (2020)
- Kotłowski, Wojciech: Scale-invariant unconstrained online learning (2020)
- Kylasa, Sudhir; Fang, Chih-Hao; Roosta, Fred; Grama, Ananth: Parallel optimization techniques for machine learning (2020)
- Lei, Lihua; Jordan, Michael I.: On the adaptivity of stochastic gradient-based optimization (2020)
- Liu, Hailiang; Markowich, Peter: Selection dynamics for deep neural networks (2020)
- Ning, Hanwen; Zhang, Jiaming; Feng, Ting-Ting; Chu, Eric King-wah; Tian, Tianhai: Control-based algorithms for high dimensional online learning (2020)
- Ogal’tsov, A. V.; Tyurin, A. I.: A heuristic adaptive fast gradient method in stochastic optimization problems (2020)
- Palagi, Laura; Seccia, Ruggiero: Block layer decomposition schemes for training deep neural networks (2020)
- Paquette, Courtney; Scheinberg, Katya: A stochastic line search method with expected complexity analysis (2020)
- Rius-Sorolla, G.; Maheut, J.; Coronado-Hernandez, Jairo R.; Garcia-Sabater, J. P.: Lagrangian relaxation of the generic materials and operations planning model (2020)
- Ruehle, Fabian: Data science applications to string theory (2020)
- Wang, Yi; Zhang, Hao; Chae, Kum Ju; Choi, Younhee; Jin, Gong Yong; Ko, Seok-Bum: Novel convolutional neural network architecture for improved pulmonary nodule classification on computed tomography (2020)
- Ye, Lifeng; Beskos, Alexandros; De Iorio, Maria; Hao, Jie: Monte Carlo co-ordinate ascent variational inference (2020)