Mathematics of deep learning

Welcome to the course page for the Mathematics of deep learning course at the Intelligence Artificielle, Systèmes, Données - Informatique (IASD) and the Artificial Intelligence, Systems, Data - Mathematics Track (MASH) M2 programs from PSL University. Here you’ll find all the course information and materials.

Course Information

• Instructor: Dr. Bruno Loureiro
• Email: bruno.loureiro@di.ens.fr
• Semester: Winter 2025
• Class Times: Fridays, 9:00 AM - 12:15 AM
• Location: Check at the ENT.

Evaluation

50% Homework + 50% paper presentation and discussion.

Course Description

In the absence of a well-defined body of mathematics that could be called a bona fide “theory of deep learning”, our goal in these lectures will be instead to introduce the students to some recent mathematical ideas that emerged in the study of deep learning. We don’t aim at being exhaustive, but rather to train the students at reading paper and preparing them to do research in deep learning theory, inasmuch as this can be defined.

• Introduction and challenges.
• Universal approximation theorems
• The lazy limit of large-width networks.
• The double descent phenomena and benign overfitting.
• Implicit bias of GD/SGD.

Requirement

Undergraduate level linear algebra, analysis and probability will be assumed. Please check you are familiar with the maths checklist.

Recommended literature

The course is based on the following lecture notes.

When preparing them, I took inspiration in some excellent ressources which are freely available, and which you might also find useful:

• Francis Bach book Learning theory from first principles.
• Matus Telgarsky Deep learning theory lecture notes.
• Romain Couillet and Zhenyu Liao book on Random Matrix Methods for Machine Learning.
• Theodor Misiakiewicz and Andrea Montanari Six Lectures on Linearized Neural Networks.
• Scott Pesme PhD thesis on linear diagonal neural networks.

Course Schedule