2024/2025 PSL Week on Statistical Physics and Machine Learning

The objective of the intensive week on machine learning and statistical physics is to present methods, ideas, and connections between these two fields. Indeed, the methods and ideas developed in the statistical physics of disordered systems can provide new tools for analyzing high-dimensional non-convex problems that arise in machine learning.

During the first part of the week, after a general introduction, we will focus on simple machine learning problems and analyze them using a statistical physics method that is rigorous and exact (though mathematically non-rigorous). This will allow us to concretely present some of the connections between machine learning and statistical physics. In particular, we will introduce the replica method, which has proven extremely useful in physics and other branches of science, as illustrated by the 2021 Nobel Prize in Physics awarded to Giorgio Parisi.

The second part will be devoted to more realistic models and current research questions, such as the Double Descent phenomenon and convex and non-convex optimization.

Course Information

• Instructors: Francis Bach (INRIA & DI-ENS), Giulio Biroli (LPENS), Bruno Loureiro (CNRS & DI-ENS).
• Dates: 25-29 November, 2024
• Location: See PSL ENT or contact us.

Evaluation

Assiduity + Paper review.

Course Description

• Introduction to Machine Learning.
• Introduction to Statistical Physics.
• Spiked GEO matrix: replica solution, rigorous derivation, BBP transition.
• Classical SGD theory
• Scaling limits of SGD in high-dimensions for single-index and two-layer neural networks: overparametrisation, information exponent.
• Diffusion models: introduction to stochastic processes and diffusion, speciation transition.

Requirements

Basic probability theory, linear algebra and analysis. No background in statistical physics will be assumed.

Recommended literature

• Classical SGD theory, Chapter 5 in Francis’ book.
• For one-pass SGD, Bruno’s lecture notes.

Course Schedule