CANCELLED Britton Lecture – Peter Bartlett – Topics in Deep Learning Theory: In-context learning linear models with transformers
Apr 24, 2024
3:30PM to 4:30PM
Date/Time
Date(s) - 24/04/2024
3:30 pm - 4:30 pm
Britton Lecture
Dr. Peter Bartlett – Professor, Department of Statistics at University of California, Berkeley
CANCELLED
Title: In-context learning linear models with transformers
Abstract: Transformer networks have demonstrated a remarkable ability at in-context learning (ICL): given a short prompt sequence of
labeled data, they can behave like supervised learning algorithms. We consider ICL in transformers with linear self-attention and
multi-layer perceptron components. We study the optimization dynamics of a single linear self-attention layer trained by gradient
flow on linear regression tasks, focusing on robustness to distribution shifts; we show how in-context learning performance
improves with the number of independent tasks; and we investigate the importance of the MLP component in learning a
prior over regression parameters.
Based on joint work with Ruiqi Zhang, Spencer Frei, Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, and Quanquan Gu.
Coffee served before lecture at 3pm