9.520/6.860: Statistical Learning Theory and Applications Fall 2022: Course Syllabus

Follow the link for each class to find a detailed description, suggested readings, and class slides. Some of the later classes may be subject to reordering or rescheduling.

 

Class

Class Date Title Instructor(s)
Class 01 Thu Sep 08 Course Outline. Statistical Machine Learning LR
Class 02 Tue Sep 13 Empirical Risk Minimization and Regularization for Linear Models LR
Class 03 Thu Sep 15 Kernels and Feature Maps LR
Class 04 Tue Sep 20 Optimization: GD and SGD. Regularization and implicit regularization TG
Class 05 Thu Sep 22 Error Decomposition and Approximation Error AR
Class 06 Tue Sep 27 Estimation Error and Generalization Gap AR
Class 07 Thu Sep 29

Stability of Ridge and Ridgeless Regression

 AR
Class 08 Tue Oct 04

Deep Learning Theory: Approximation

 TP
Class 09 Thu Oct 06 Introduction to Deep Networks  AR
Monday 10th October - Indigenous People's Day, Tuesday 11th October - Student Holiday 
Class 10 Thu Oct 13

Deep Learning: Optimization and Dynamics

 TP
Class 11 Tue Oct 18  Deep Learning: Bias towards Low Rank TG
Class 12 Thu Oct 20 Deep Learning: Neural Collapse  AR + TG
Class 13 Tue Oct 25

Deep Learning: Generalization in Sparse Overparametrized Networks

 TP
Class 14 Thu Oct 27

Group Invariance and Equivariance in Vision and Learning

 Fabio Anselmi
Class 15 Tue Nov 01 Transformers  Brian Cheung
Class 16 Thu Nov 03 Neural Networks and the Ventral Stream Thomas Serre + Gabriel Kreiman
Class 17 Tue Nov 08 Loose Ends

Staff
Class 18 Thu Nov 10 Brain and Neural Networks - Identification Problems Brian Cheung + Yena Han
Class 19 Tue Nov 15 Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit   Benjamin Edelman
Class 20 Thu Nov 17 Optimal tradeoff approximation/generalization in Underparametrized  deep networks  Sophie Langer
Class 21 Tue Nov 22 Adversarial examples Alexander Madry
Thursday 24th November - Thanksgiving
Class 22 Tue Nov 29 Neural Assemblies Christos Papadimitriou +Santosh Vempala
Class 23 Thu Dec 01 Brainstorming on deep learning puzzles+projects and other thoughts TP and AR
Class 24 Tue Dec 06 Sparsity in linear models and deep networks Yuan Yao
Class 25 Thu Dec 08 The loss landscape of overparametrized deep nets Yaim Cooper
Class 26 Tue Dec 13 Transformers Tomer Ullman

Reading List

Notes covering the classes will be provided in the form of independent chapters of a book currently in draft format. Additional information will be given through the slides associated with classes (where applicable). The books/papers listed below are useful general reference reading, especially from the theoretical viewpoint. A list of additional suggested readings will also be provided separately for each class.

Book (draft)

  • L. Rosasco and T. Poggio, Machine Learning: a Regularization Approach, MIT-9.520 Lectures Notes, Manuscript, Dec. 2017 (provided).

Primary References

Resources and links