Training LLMs with Diffusion Techniques - From Theory to Implementation

Learn to implement Discrete Diffusion Modeling for text generation through a detailed 45-minute technical video that breaks down the paper "Estimating the Ratios of the Data Distribution." Explore a competitive alternative to GPT-2 using diffusion techniques, starting with an introduction to the GitHub repository and the fundamental concepts of applying diffusion to text generation. Progress through hands-on code demonstrations, including a pre-trained SEDD demo, noise and transition graph analysis, perturbation scripts, and sample transition functions. Master the implementation details of the sampler and experience running actual training scripts, concluding with an interactive Q&A session. Access additional resources including the original paper, detailed notes, and community support through Oxen AI's platform, which specializes in dataset versioning for AI development.