Splines and Imaging - From Compressed Sensing to Deep Neural Networks

Overview

Explore a comprehensive lecture on splines and imaging, covering the journey from compressed sensing to deep neural networks. Delve into the optimality of splines for solving inverse problems in imaging and designing deep neural networks. Examine a representer theorem that demonstrates how extremal points of linear inverse problems with generalized total-variation regularization are adaptive splines. Discover the connection between continuous-domain solutions and compressed sensing algorithms. Investigate the application of the theorem to optimize activation shapes in deep neural networks, leading to the concept of "optimal" deep-spline networks with piecewise-linear spline activations. Gain insights into the variational justification of ReLU architecture and explore new computational challenges in determining optimal activations. Learn about the structure of iterative reconstruction algorithms, the algebra of CPWL functions, and the implications for deep ReLU neural networks.

Syllabus

Intro
Variational formulation of inverse problem
Linear inverse problems (20th century theory)
Learning as a (linear) Inverse problem
Splines are analog, but intrinsically sparse
Spline synthesis example
Spline synthesis: generalization
Representer theorem for TV regularization
Other spline-admissible operators
Recovery with sparsity constraints: discretization
Structure of iterative reconstruction algorithm
Connection with deep neural networks
Deep neural networks and splines
Feedforward deep neural network
CPWL functions in high dimensions
Algebra of CPWL functions
Implication for deep ReLU neural networks
CPWL functions: further properties
Constraining activation functions
Representer theorem for deep neural networks
Outcome of representer theorem
Optimality results
Deep spline networks: Discussion
Deep spline networks (Cont'd)