Researchers introduced Row-wise Matrix AdaGrad and Column-wise Matrix AdaGrad, two matrix-aware adaptive optimization methods that improve optimization stability and trainability for neural networks. These methods are derived from an Online Mirror Descent framework and can be used to train larger and deeper networks.