Deep Learning Meets Nonparametric Regression: Are Weight-decayed DNNs Locally Adaptive?

Explore the intersection of deep learning and nonparametric regression in this 56-minute conference talk presented by Yu-Xiang Wang at USC Information Sciences Institute. Delve into the capabilities of deep neural networks (DNNs) in curve fitting compared to classical tools like splines and wavelets. Gain insights on why DNNs outperform kernels, the advantages of deep networks over shallow ones, the significance of ReLU activation, generalization under overparameterization, the role of sparsity in deep learning, and the validity of the lottery ticket hypothesis. Examine the statistical perspective on DNNs' success, explore the "adaptivity" conjecture, and understand how weight-decayed DNNs relate to total variation regularization. Learn about the equivalence between weight-decayed L-Layer PNNs and sparse linear regression with learned basis functions. Discover how parallel ReLU DNNs approach minimax rates as they deepen, and compare their performance to classical nonparametric regression methods. Investigate examples of functions with heterogeneous smoothness and approximation error bounds.