NDC Conferences Talk: Analyzing StackExchange with Azure Data Lake from NDC Conferences

Analyzing StackExchange with Azure Data Lake

Overview

Explore big data analysis using Azure Data Lake Store and Analytics in this comprehensive conference talk. Learn how to store and analyze over 150 GB of data from StackExchange websites, gaining valuable insights into user behavior. Discover the power of Azure Data Lake Store, comparing it to traditional data warehouses, and understand its security features and pricing model. Dive into the process of converting XML to CSV and leveraging Azure Data Analytics for efficient data processing. Master the use of U-SQL scripts, metadata models, and security models to extract meaningful information. Gain hands-on experience with Visual Studio Code for script development and optimization. Finally, visualize your findings using Power BI to create impactful, at-a-glance overviews of the analyzed data.

Syllabus

Introduction
Tom Kerkhove
Big Data
Big Data Analysis
Data Like Store
What is a Data Lake
Data Lake vs Warehouse
Security
Pricing
Blob Storage
DataLikeStore
XML to CSV
Azure Data Analytics
Data Stores
Script Overview
Code Behind
Metadata Model
Scripts
Security model
Application limitations
Visual Studio Code
Extract Script
Submit Script
Output Data
Cleaning Users
Job Overview
Optimized Script