Mathematics of deep learning

Welcome to the course page for the Mathematics of deep learning course at the Intelligence Artificielle, Systèmes, Données - Informatique (IASD) and the Artificial Intelligence, Systems, Data - Mathematics Track ()MASH) M2 programs from PSL University. Here you’ll find all the course information and materials.

Course Information

• Instructor: Dr. Bruno Loureiro
• Email: bruno.loureiro@di.ens.fr
• Semester: Winter 2025
• Class Times: Fridays, 9:00 AM - 12:15 AM
• Location: Check at the ENT.

Evaluation

Course Description

In the absence of a well-defined body of mathematics that could be called a bona fide “theory of deep learning”, our goal in these lectures will be instead to introduce the students to some recent mathematical ideas that emerged in the study of deep learning. We don’t aim at being exhaustive, but rather to train the students at reading paper and preparing them to do research in deep learning theory, inasmuch as this can be defined.

• Introduction and challenges.
• Universal approximation theorems
• The lazy limit of large-width networks.
• The double descent phenomena and benign overfitting.
• Implicit bias of GD/SGD.
• The speciation transition in diffusion models.

Requirement

Undergraduate level linear algebra, analysis and probability will be assumed. Please check you are familiar with the maths checklist.

Recommended literature

We will cover a variety of research topics which cannot be found in a single textbook. Nevertheless, when preparing this course I took inspiration in some excellent ressources which are freely available, and which you might also find useful:

• Francis Bach book Learning theory from first principles.
• Matus Telgarsky Deep learning theory lecture notes.
• Romain Couillet and Zhenyu Liao book on Random Matrix Methods for Machine Learning.
• Theodor Misiakiewicz and Andrea Montanari Six Lectures on Linearized Neural Networks.

Course Schedule