Big Data Systems

Overview

There are 2 main roles in the Big Data industry: Big Data Engineer and Big Data Architect. We will focus on the Architect role. We will look at the characteristics of Big Data, its business value, and how companies are using Big Data today. We study the most popular storage and processing frameworks.....aka...Big Data ecosystem components. We will dive deep into NoSQL, how it differs from traditional relational databases, how to model, and what the tool interface looks like. Finally, we will talk about the benefits, challenges, design patterns of Data Lake technology. Your final project is based on a real-world scenario that will require you to think like an architect and build an end-to-end data lake system proposal.

Syllabus

Introduction to Big Data Systems

In this lesson, we will take a 30000 foot view of Big Data and see why it is so important. We will meet the instructor and hear about the components of the course, including the final project.

Characteristics of Big Data

In this lesson you will learn about the main characteristics of Big Data, called the 4Vs. You will also start to explore the Big Data ecosystem.

Ingestion, Storage and Processing Frameworks

In this lesson, you'll take a look at several of the layers that make Big Data possible, We will also look at some of the tools that help implement those layers.

NoSQL Databases

In this lesson, we will look at the differences between NoSQL and SQL. We will also see why and how NoSQL databases provide capabilities that allow Big Data to be possible.

Scalable Data Lake Architecture

In this lesson, we will see what a Data Lake storage implementation of Big Data looks like. In addition to the benefits, we will see what considerations, risks, and challenges organizations face.

Project - Designing an Enterprise Data Lake System

In this lesson, we will lead you through the scenario and instructions for completing the final project, which is a proposal for an actual Data Lake architecture.