Geometric Representations of Far Future Predictions in Deep Neural Networks - Next-Token Training

Explore a fascinating lecture on the geometric representations of far future predictions in deep neural networks trained on next-token prediction. Delve into Paul Riechers' presentation from the Institute for Pure & Applied Mathematics' Theory and Practice of Deep Learning Workshop. Discover how modern AI models develop implicit world models and perform Bayesian inference through next-token prediction training. Examine the universal, sometimes fractal, geometric representations of beliefs linearly embedded in neural network activations. Investigate the similarities between recurrent neural networks (RNNs) and transformers in this context. Uncover the intriguing discovery of 'quantum' and 'post-quantum' low-dimensional models within classical stochastic processes, as represented by neural networks. Gain insights into the complex world of deep learning and its implications for understanding AI's predictive capabilities.