Explore high-throughput data processing using Akka Streams in this 41-minute conference talk from Scala Days Copenhagen 2017. Discover how Credit Karma, with over 60 million members, tackled the challenges of large-scale data transfer. Learn about the initial implementation using Akka Actors, the limitations encountered, and the transition to Akka Streams. Gain insights into backpressure, warning signs of system capacity limits, and when to consider Akka Streams for data transfer. Examine best practices, optimizations, and real-world examples of warehouse import and analytics export implementations. Dive into benchmarking results, garbage collection considerations, and key learnings from Credit Karma's experience. Ideal for developers and companies building or optimizing high-throughput Akka Actor systems.
Overview
Syllabus
Intro
Scale of Data at Credit Karma
Data Warehouse Import
Warehouse import problems
Akka Streams for Fun and Profit
Akka Streams. Easy to Unit Test
Akka Streams Built-in Stages
Warehouse Import with Streams
Warehouse Import Stream Code
Analytics Export with Streams
Analytics Export Stream Code
Analytics Export Heap Space Pull
Warehouse Import Garbage Collection
Benchmarking Setup
Benchmarking test Baseline Test, read parse XML
Benchmarking Results
Benchmarking Code - Akka Actors
Benchmarking Code - Akka Streams
Garbage Collection Times
Learnings
Taught by
Scala Days Conferences
