Open Internet by MindsNet

Failure of Optimizers in Regression Losses

Optimizers like Adam and SOAP fail to converge to the minimum in regression losses like MSE, getting stuck in a ball of radius around the solution. This is due to the structural problems with the Empirical Fisher matrix used in these optimizers.

Mathematics & logic, Mathematics, Optimization