Explore the intricacies of deep network approximation in this 50-minute conference talk by Remi Gribonval from Inria, presented at the Alan Turing Institute. Delve into the mathematics of data science, bridging computational statistics, machine learning, optimization, information theory, and learning theory. Begin with an introduction to feedforward neural networks and study the expressivity of deep neural networks (DNNs). Examine the ReLU activation function and its universal approximation properties. Investigate the benefits of sparsely connected networks and various network shapes. Compare direct and inverse estimates in approximation with sparse networks. Understand the role of skip-connections, neuron count vs. connections, and activation functions. Analyze spline activation functions and guidelines for choosing appropriate activations. Explore the benefits of network depth and its impact on approximation capabilities. Conclude with a comprehensive summary of approximation with DNNs and future perspectives in this cutting-edge field.
Overview
Syllabus
Introduction
Feedforward neural networks
Studying the expressivity of DNNS
Example: the ReLU activation function
ReLU networks
Universal approximation property
Why sparsely connected networks?
Same sparsity - various network shapes
Approximation with sparse networks
Direct vs inverse estimate
Notion of approximation space
Role of skip-connections
Counting neurons vs connections
Role of activation function 0
The case of spline activation functions Theorem 2
Guidelines to choose an activation ?
Rescaling equivalence with the ReLU
Benefits of depth ?
Role of depth
Set theoretic picture
Summary: Approximation with DNNS
Overall summary & perspectives
Taught by
Alan Turing Institute
