AWS Data Collection and Storage

Overview

This course is designed to guide you through the essential AWS services for data collection and storage. You'll begin with in-depth exploration of data streaming using Amazon Kinesis. Learn to handle data producers and consumers, set up Kinesis Data Firehose, and integrate CloudWatch filters, all while gaining hands-on experience with real-time data ingestion. Practical exercises will help you understand key scaling and security features, ensuring that your data streams are managed efficiently. Next, you will focus on AWS storage solutions with a strong emphasis on Amazon S3. Explore storage classes, replication, versioning, and lifecycle policies, all critical to effective data management. Through hands-on labs, you’ll configure S3 buckets, implement encryption, and set up event notifications, providing you with the skills needed to manage vast amounts of data securely. DynamoDB services will be introduced for scalable data storage solutions, with hands-on practice in setting up databases and optimizing throughput for performance. By the end of this course, you’ll be able to create and maintain complex data pipelines, ensuring both scalability and security. You'll have the confidence to manage data at scale, implement efficient storage solutions, and optimize performance using AWS best practices, positioning yourself to handle any data challenges AWS environments may present. This course is ideal for data engineers, IT professionals, and AWS users who want to specialize in data collection and storage. Basic knowledge of AWS services is recommended, but no prior certification is required.

Syllabus

Introduction

In this module, we will introduce the overall structure of the course, including a detailed overview of the hands-on case study on Cadabra.com. Additionally, we will guide you through the process of setting up an AWS budget to manage costs efficiently while using various AWS services throughout the course.

Domain 1: Collection

In this module, we will explore the different AWS tools for real-time data collection, including Kinesis Data Streams, Firehose, and SQS. You'll engage in hands-on exercises to set up Kinesis Producers and Consumers, work with CloudWatch subscription filters, and build a system for populating an S3 data lake. The module also includes advanced topics like enhanced fan-out, scaling, and security best practices.

Domain 2: Storage

In this module, we will focus on AWS storage solutions, primarily S3 and DynamoDB. Through hands-on activities, you will gain practical experience configuring S3 security policies, versioning, replication, and event notifications. Additionally, you will explore DynamoDB’s advanced features, including read/write capacity units, global/local indexes, and DynamoDB Streams. We will also examine the interplay between S3 and DynamoDB in large-scale data management solutions.