Completed
What are the main challenges?
Class Central Classrooms beta
YouTube videos curated by Class Central.
Classroom Contents
Best Practices for Building and Deploying Data Pipelines in Apache Spark
Automatically move to the next video in the Classroom when playback concludes
- 1 Intro
- 2 Cox Automotive
- 3 KPMG Lighthouse
- 4 What is this talk about?
- 5 What do we mean by 'Data Pipeline?
- 6 Who is in a data team?
- 7 What do we need to think about when building a pipeline?
- 8 What about the business logic?
- 9 What about deployments?
- 10 What are the main challenges?
- 11 How were we dealing with the main challenges?
- 12 Could we make better use of the skills in the team?
- 13 What tools and frameworks would we need to provide?
- 14 How would we design a Data Engineering framework?
- 15 How would we like to manage deployments?
- 16 Simpler data ingestion
- 17 Simpler business logic development
- 18 Simpler environment management
- 19 Simpler deployments
