Migrating Airflow-Based Apache Spark Jobs to Kubernetes - The Native Way

Learn how to migrate Apache Spark workloads from AWS EMR to Kubernetes in this 21-minute conference talk by Databricks. Explore the challenges of existing Spark infrastructure and the motivation behind migrating to Kubernetes. Discover aspects of running Spark natively on Kubernetes, including monitoring and logging. Gain insights into best practices for using Airflow as an orchestrator. Follow the journey of Nielsen Identity as they process massive amounts of data using Apache Spark, and understand how they combined the GCP Spark-on-K8s operator with a native Airflow integration to achieve their goals. Dive into topics such as Kubernetes auto-scaling, Spark-On-Kubernetes overview, and making the migration production-ready. This talk provides valuable information for data engineers and architects looking to optimize their Spark workloads and reduce operational costs.