Dr. Peter Bartlett – Professor, Department of Statistics at University of California, Berkeley

CANCELLED

Title: Improving optimization efficiency by choosing the step size too large for gradient descent

Abstract: Optimization in deep learning relies on simple gradient descent algorithms. Although these methods are traditionally viewed as a
time discretization of gradient flow, in practice, large stepsizes—large enough to cause oscillation of the loss—exhibit performance
advantages. We study gradient descent in logistic regression with a constant step size that is so large that the loss initially
oscillates. We show the benefits of this initial oscillatory phase, achieving a loss of 1/T^2 in T steps, where a step size small
enough to ensure a monotonic decrease of the loss cannot do better than 1/T. We show similar benefits in a nonlinear setting.

Based on joint work with Jingfeng Wu, Matus Telgarsky and Bin Yu.

Coffee served before lecture at 3pm