Explore the Fugue framework, an abstraction layer unifying various big data analytics solutions like Apache Spark, TensorFlow, Druid, Dask, and Flink. Learn how this SQL-like language represents end-to-end pipelines, extensible with Python, to create reliable, performant, and maintainable data processing workflows. Discover the benefits of a unified K8S Spark environment for interactive development, batch processing, and near real-time streaming jobs. See demonstrations of instant dependency updates, on-demand Spark K8s cluster management, and Fugue extensions for Kinesis and Kafka. Understand how Fugue provides abstraction for machine learning pipelines, enabling distributed training, hyperparameter tuning, and inference across various ML libraries. Gain insights into extensive testing on Spark 3.0 and the resulting performance improvements in this 22-minute talk from Databricks.
Overview
Syllabus
Intro
Motivation of Fugue
Node Vec: Fugue Code
Fugue Programming Model
A Workflow Example
The Fugue Extensions
Fugue SQL vs Spark SQL
Fugue Programming Interface vs SQL
Fugue ML Components
Model & Parameter Sweeping model
Benchmark Test
An Interactive On-demand Spark Ecosystem
Summary
Taught by
Databricks
Related Courses
-
Scalable Machine Learning on Big Data using Apache Spark
-
Data Engineering, Big Data, and Machine Learning on GCP
-
Scaling Machine Learning Workflows to Big Data with Fugue
-
Introducing Apache Spark 3.0 - A Decade of Progress and Future Outlook
-
Running Apache Spark Jobs Using Kubernetes
-
Lessons Learned from Big Data and AI/ML Collaboration for Giant Hogweed Eradication
