Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Accelerating Apache Spark Shuffle for Data Analytics on Cloud with Remote Persistent Memory Pools

Databricks via YouTube

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Explore a 33-minute conference talk on accelerating Apache Spark shuffle operations for cloud-based data analytics using remote persistent memory pools. Dive into the challenges of serving growing data-driven AI and analytics workloads in disaggregated storage and compute environments. Learn about a proposed fully disaggregated shuffle solution leveraging persistent memory and RDMA technologies, including a new pluggable shuffle manager and distributed storage system. Discover how this innovative approach improves Spark's scalability, performance, and reliability, with experimental results showing up to 10x performance speedup over traditional shuffle solutions. Gain insights into the architecture, optimization features, and workflow of this cutting-edge solution presented by Databricks.

Syllabus

Introduction
Agenda
Motivation
Recap
Original Example
Results
New Challenges
Rtmp Architecture
Optimization Features
Workflow
Summary
Performance Evaluation
Examples
Call to Action
Optima Natives

Taught by

Databricks

Reviews

Start your review of Accelerating Apache Spark Shuffle for Data Analytics on Cloud with Remote Persistent Memory Pools

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.