Scala and Spark in the New Deep Learning Era

Explore the emerging era of deep learning and its impact on natural language processing and other industries in this 40-minute conference talk. Delve into the transformative effects of new neural network architectures, including self-attention models, on the machine learning landscape. Learn how high-capacity models and transfer learning methods enable knowledge extraction from supercomputing centers for adaptation to specific industry needs, surpassing traditional machine learning algorithms in accuracy. Examine the challenges of incorporating these neural network architectures into existing systems and discover how to integrate them securely and efficiently into data centers and computing infrastructures. Gain insights into leveraging tools from the open ecosystem, such as Apache Spark, Scala, and BigDL, to incorporate these models into big data infrastructures. Explore a novel implementation of a distributed fine-tuning process for public pre-trained large language models, allowing for their extraction from public hubs and representation as new Spark estimators. Understand how to integrate these powerful new tools into existing systems and capitalize on the opportunities they present in the evolving landscape of artificial intelligence and machine learning.