Mathematical Structure Computed by Large Language Models - A First Approximation

Explore the mathematical structure behind Large Language Models in this comprehensive lecture. Delve into the conditional probability distributions of text extensions and their representation as a directed metric structure on the space of texts. Discover how this structure is encoded in a directed metric polyhedron, with texts isometrically embedded as generators of special extremal rays. Learn about the tropical generation of the polyhedron and its relation to a duality theorem connecting text extensions and restrictions. Examine the approximation of text generators using Boltzmann weighted linear combinations of word generators. Gain insights into the categorical interpretations of these constructions, including the Yoneda embedding and generalizations of language as a monoid or poset. This joint work with Stéphane Gaubert offers a deep dive into the mathematical foundations of LLMs, presented by Yiannis Vlassopoulos from the Athena Research Center.