Authors: Akhil Kumar
In this paper, I introduce the Gradient Reservoir Optimizer (GRO), a novel optimizationalgorithm for neural network training that combines short-term gradient updates with long-term gradient trends. GRO maintains a dynamic "reservoir" of recent gradient directions and utilizes their aggregated trends to influence parameter updates. By blending current gradients with a history-aware reservoir, GRO aims to stabilize convergence and improve robustness to noisygradients. This novel approach provides an additional mechanism to mitigate common issueslike gradient noise and plateaus in training loss. I demonstrate the theoretical underpinnings of GRO, provide its algorithmic structure, and evaluate its performance on benchmark datasets. The results show promise for GRO as a viable alternative to existing optimizers like SGD, Adam, and RMSProp. Additionally, GRO offers flexibility for tuning the influence of historical gradients, making it adaptable across a variety of tasks and architectures.
Comments: 4 Pages.
Download: PDF
[v1] 2025-01-07 21:52:23
Unique-IP document downloads: 147 times
Vixra.org is a pre-print repository rather than a journal. Articles hosted may not yet have been verified by peer-review and should be treated as preliminary. In particular, anything that appears to include financial or legal advice or proposed medical treatments should be treated with due caution. Vixra.org will not be responsible for any consequences of actions that result from any form of use of any documents on this website.
Add your own feedback and questions here:
You are equally welcome to be positive or negative about any paper but please be polite. If you are being critical you must mention at least one specific error, otherwise your comment will be deleted as unhelpful.