Streaming Data Pipelines From Supernovas to LLMs

Embark on a hands-on journey through streaming data pipelines in this 50-minute tutorial from Databricks. Dive into a comprehensive use case utilizing the Databricks Intelligence Platform, focusing on data engineering techniques. Learn to analyze real-time data from collapsing supernovas emitting gamma-ray bursts, provided by NASA's GCN project. Discover how to ingest data from message buses and choose between Delta Live Tables, DBSQL, or Databricks Workflows for stream processing. Master ETL pipeline coding in SQL, including Kafka ingestion. Explore Databricks Data Rooms for natural language analytics and compare it to notebook streaming data into a Vector Database for open-source LLMs with RAG. Ideal for data engineers, code-savvy data architects, genAI enthusiasts, and astronomy lovers, this session teaches when and how to use various Databricks products. Replicate the demo at home and gain valuable insights into streaming data pipelines from supernovas to large language models.