Mathematics of deep learning

Welcome to the course page for the Mathematics of deep learning course at the Intelligence Artificielle, Systèmes, Données - Informatique (IASD) and the Artificial Intelligence, Systems, Data - Mathematics Track (MASH) M2 programs from PSL University. Here you’ll find all the course information and materials.

Course Information

  • Instructor: Dr. Bruno Loureiro
  • Email: bruno.loureiro@di.ens.fr
  • Semester: Winter 2025
  • Class Times: Fridays, 9:00 AM - 12:15 AM
  • Location: Check at the ENT.

Evaluation

50% Homework + 50% paper presentation and discussion.

Course Description

In the absence of a well-defined body of mathematics that could be called a bona fide “theory of deep learning”, our goal in these lectures will be instead to introduce the students to some recent mathematical ideas that emerged in the study of deep learning. We don’t aim at being exhaustive, but rather to train the students at reading paper and preparing them to do research in deep learning theory, inasmuch as this can be defined.

  • Introduction and challenges.
  • Universal approximation theorems
  • The lazy limit of large-width networks.
  • The double descent phenomena and benign overfitting.
  • Implicit bias of GD/SGD.

Requirement

Undergraduate level linear algebra, analysis and probability will be assumed. Please check you are familiar with the maths checklist.

The course is based on the following lecture notes.

When preparing them, I took inspiration in some excellent ressources which are freely available, and which you might also find useful:

Course Schedule

Date Lecture Topic Materials
Jan 10 - Motivation
- Review of ERM
Notes
Homework 1
Jan 17 - Curse of dimensionality
- Approximation theory
Sec. 1 to 3.1 of Notes
Homework 2
Jan 24 - Approximation theory (continued) Sec. 3.2 to 4 of Notes
Homework 3
Jan 31 No class  
Feb 07 - NTK Notes
Homework 4
Feb 14 - Intro to RMT Notes
Homework 5
Feb 21 - Analysis of ridge regression
- Double descent and benign overfitting
Notes
Homework 6
Feb 28 - Implicit bias of algorithms Notes
Mar 14 - Paper discussion