What you'll learn:
- Learn about Data Lake vs. Data Warehouse
- Key components of a Data Lake Architecture
- Query files directly using SQL
- Hands-on integration using Kinesis Firehose, Lambda, Comprehend AI, Glue, Athena and S3
Hello, my name is Chandra Lingam, and I will be your instructor for the Data Lake in AWS course.
In this course, we will begin by gaining an understanding of the fundamental concepts of a data lake and when it is the appropriate solution as opposed to a data warehouse
We will then delve into the various components that make up a data lake solution, including the ability to query files directly using SQL for rapid ad hoc analysis of datasets
During the course, we will cover the topic of handling changes to the structure of the files in the data lake. We will delve into the various scenarios, such as new fields, new partitions, changes in data types, and missing data, and discuss the techniques on how to handle them effectively. We will also delve into Glue Catalog Management and the evolution of schemas, with a focus on minimizing disruption to downstream systems
We will also look into different data formats, such as CSV, Parquet, Avro, and ORC, and examine their respective strengths and weaknesses. Following that, we will delve into Glue ETL, a robust Apache Spark-based solution for data transformation.
This course is filled with hands-on exercises and projects.
You will analyze a University Rankings dataset, which is easy to understand, useful, and has a mix of data types with many data quality issues.
You will learn to utilize Athena for querying data, tackle data quality problems through SQL, and cleanse the data using Glue - Apache Spark ETL.
Additionally, the course covers techniques for simplifying queries using views and visualizing data using Amazon QuickSight.
To showcase the scalability of Athena, we will query the large Amazon Customer Reviews dataset containing over 130 million reviews. Finally, we will construct a serverless application using Kinesis Firehose, Lambda, Comprehend AI, Glue, Athena, and S3, which can process an unlimited number of customer reviews, perform sentiment analysis, and store the results in the data lake for querying.
I am excited to meet you soon!
Thank you!
Chandra Lingam
Compute With Cloud Inc
