Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Scaling Machine Learning Feature Engineering in Apache Spark at Facebook

Databricks via YouTube

Overview

Explore a 21-minute conference talk on scaling machine learning feature engineering in Apache Spark at Facebook. Dive into the implementation of Feature Injection and Feature Reaping techniques, including Spark core/SQL enhancements, indexed/aligned tables, and the new ORC FlatMap encoding. Learn about catalyst optimizations, new ORC physical encodings for feature maps, and the process of writing/committing indexed feature tables. Gain insights into Facebook's approach to improving prediction model quality through efficient data management and processing techniques in Spark.

Syllabus

Intro
Machine Learning at Facebook
Data Layouts (Tables and Physical Encodings)
Background: Apache ORC
How is a Feature Map Stored in ORC?
Introducing: ORC Flattened Map
Feature Reaping
Introducing: Aligned Table
Query Plan for Aligned Table
Reading Aligned Tables
End to End Performance
Summary
Future Work

Taught by

Databricks

Reviews

Start your review of Scaling Machine Learning Feature Engineering in Apache Spark at Facebook

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.