Synthetic Data - Friend or Foe in the Age of Scaling?

Explore the impact of synthetic data on AI and large language model scaling in this 56-minute lecture by Julia Kempe from the Institut des Hautes Etudes Scientifiques (IHES). Delve into the theoretical framework of model collapse through scaling laws, examining how the increasing presence of synthesized data in training corpora affects model improvement and performance. Discover various decay phenomena, including loss of scaling, shifted scaling across generations, skill "un-learning," and grokking when combining human and synthetic data. Learn about the validation of this theory through large-scale experiments using a transformer for arithmetic tasks and the LLM Llama2 for text generation. Gain insights into the potential future challenges and implications for AI development as synthetic data becomes more prevalent in training datasets.