This course is all about big data. It’s for students with SQL experience that want to take the next step on their data journey by learning distributed computing using Apache Spark. Students will gain a thorough understanding of this open-source standard for working with large datasets. Students will gain an understanding of the fundamentals of data analysis using SQL on Spark, setting the foundation for how to combine data with advanced analytics at scale and in production environments. The four modules build on one another and by the end of the course you will understand: the Spark architecture, queries within Spark, common ways to optimize Spark SQL, and how to build reliable data pipelines.
The first module introduces Spark and the Databricks environment including how Spark distributes computation and Spark SQL. Module 2 covers the core concepts of Spark such as storage vs. compute, caching, partitions, and troubleshooting performance issues via the Spark UI. It also covers new features in Apache Spark 3.x such as Adaptive Query Execution. The third module focuses on Engineering Data Pipelines including connecting to databases, schemas and data types, file formats, and writing reliable data. The final module covers data lakes, data warehouses, and lakehouses. By the end of this course, students will hone their SQL and distributed computing skills to become more adept at advanced analysis and to set the stage for transitioning to more advanced analytics as Data Scientists.
Overview
Syllabus
- Introduction to Spark
- In this module, you will be able to discuss the core concepts of distributed computing and be able to recognize when and where to apply them. You'll be able to identify the basic data structure of Apache Spark™, known as a DataFrame. Additionally, you'll review the collaborative Databricks workspace.
- Spark Core Concepts
- In this module, you will be able to explain the core concepts of Spark. We'll discuss common ways to increase query performance by caching data and modifying Spark configurations. We'll also review the Spark UI to analyze performance and identify bottlenecks, as well as optimize queries with Adaptive Query Execution.
- Engineering Data Pipelines
- In this module, you will be able to identify and discuss the general demands of data applications. You'll be able to review data in a variety of formats and compare and contrast the tradeoffs between these formats. You will explore and examine semi-structured JSON data (common in big data environments) as well as schemas and parallel data writes. You will be able to understand an end-to-end pipeline that reads data, transforms it, and how it saves the result.
- Data Lakes, Warehouses and Lakehouses
- In this module, you will identify the key characteristics of data lakes, data warehouses, and lakehouses. Lakehouses combine the scalability and low-cost storage of data lakes with the speed and ACID transactional guarantees of data warehouses. You will review a production grade lakehouse combined with Spark in an open-source project, Delta Lake. Whoever said time travel isn't possible hasn't been to a lakehouse!
Taught by
Brooke Wenig and Conor Murphy
