Leverage AWS Kinesis Data Firehose, AWS Glue, S3, Athena and the AWS SDK to build a Data Lake.
What you'll learn:
What you'll learn:
- How to build a Data Lake using Firehose API for .NET, S3, AWS Glue and Athena
The purpose of this class is to demonstrate a proof of concept using a series of lab exercise's (in the AWS Console using AWS Kinesis Data Firehose, AWS Glue, S3, Athena and the AWS SDK, with C# code using the AWS SDK) of building a Data Lake in the AWS ecosystem.
In this class, we will be sending data from a local SQL Server database to AWS RDS securely, and automatically.
I am utilizing the .NET SDK for AWS, however, this could easily be migrated to your language of choice once you understand the concepts that Iam teaching.
I will be providing the full working source code of this proof of concept.
I will also provide a working example of creating Parquet files and sending to S3 (without using Firehose).
