Streaming Data with AWS Kinesis and Lambda

Overview

Learn how to work with streaming data using serverless technologies on AWS.

What powers the systems that we use every day without realizing, like fraud detection to keep our transactions secure or traffic signals that keep traffic flowing smoothly? The answer is streaming data (data that is continuously generated by different sources) and serverless technologies—like Amazon Kinesis and AWS Lambda.

In this course, you’ll learn how to leverage these powerful technologies by helping a fictional data engineer named Cody. Your goal is to help her to collect real-time streaming data from city-owned vehicles, analyze the data, and send relevant alerts like speed warnings to drivers. Using Amazon Kinesis and Firehose, you’ll learn how to ingest data from millions of sources before using Kinesis Analytics to analyze data as it moves through the stream. You’ll also spin up serverless functions in AWS Lambda that will conditionally trigger actions based on the data received. By the end of this training you’ll know how to create live ElasticSearch dashboards with AWS QuickSight and CloudWatch—and hopefully helped Cody complete her ambitious project.

Syllabus

Streaming in the cloud

In this first chapter, you will learn about the differences between batch and stream processing, create your first stream, manage its permissions, write to it and read from it.

Going serverless

The next step in your streaming data journey is learning how to use transformational lambda functions to go serverless. Through hands-on exercises, you will add lambda layers and trigger lambda functions on specific conditions.

Analyzing streaming data

You're now ready to encode and decode streaming data and analyze data directly in the stream. You will even use multiple streams to get daily vehicle top speeds.

Monitoring and visualizing streaming data

In this final chapter, you will discover how to monitor your stream's performance using logs, metrics, alarms and dashboards. You will use Elasticsearch and build your own Kibana dashboard.