Mathematics of deep learning
Welcome to the course page for the Mathematics of deep learning course at the Intelligence Artificielle, Systèmes, Données - Informatique (IASD) and the Artificial Intelligence, Systems, Data - Mathematics Track (MASH) M2 programs from PSL University. Here you’ll find all the course information and materials.
Course Information
- Instructor: Dr. Bruno Loureiro
- Email: bruno.loureiro@di.ens.fr
- Semester: Winter 2025
- Class Times: Fridays, 9:00 AM - 12:15 AM
- Location: Check at the ENT.
Evaluation
50% Homework + 50% paper presentation and discussion.
Course Description
In the absence of a well-defined body of mathematics that could be called a bona fide “theory of deep learning”, our goal in these lectures will be instead to introduce the students to some recent mathematical ideas that emerged in the study of deep learning. We don’t aim at being exhaustive, but rather to train the students at reading paper and preparing them to do research in deep learning theory, inasmuch as this can be defined.
- Introduction and challenges.
- Universal approximation theorems
- The lazy limit of large-width networks.
- The double descent phenomena and benign overfitting.
- Implicit bias of GD/SGD.
Requirement
Undergraduate level linear algebra, analysis and probability will be assumed. Please check you are familiar with the maths checklist.
Recommended literature
The course is based on the following lecture notes.
When preparing them, I took inspiration in some excellent ressources which are freely available, and which you might also find useful:
- Francis Bach book Learning theory from first principles.
- Matus Telgarsky Deep learning theory lecture notes.
- Romain Couillet and Zhenyu Liao book on Random Matrix Methods for Machine Learning.
- Theodor Misiakiewicz and Andrea Montanari Six Lectures on Linearized Neural Networks.
- Scott Pesme PhD thesis on linear diagonal neural networks.
Course Schedule
Date | Lecture Topic | Materials |
---|---|---|
Jan 10 | - Motivation - Review of ERM | Notes Homework 1 |
Jan 17 | - Curse of dimensionality - Approximation theory | Sec. 1 to 3.1 of Notes Homework 2 |
Jan 24 | - Approximation theory (continued) | Sec. 3.2 to 4 of Notes Homework 3 |
Jan 31 | No class | |
Feb 07 | - NTK | Notes Homework 4 |
Feb 14 | - Intro to RMT | Notes Homework 5 |
Feb 21 | - Analysis of ridge regression - Double descent and benign overfitting | Notes Homework 6 |
Feb 28 | - Implicit bias of algorithms | Notes |
Mar 14 | - Paper discussion |