Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Scaling Validation and Quality of Streaming Data Products at Twitter

Strange Loop Conference via YouTube

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Explore the intricacies of scaling validation and quality assurance for streaming data products at Twitter in this 33-minute conference talk from Strange Loop. Dive into the challenges faced by the Data Products Quality Engineering team as they build and maintain services, tools, and processes to support feature teams in providing a robust distributed streaming architecture. Learn about the evolution from simple QA to sophisticated tools for identifying issues in pre-production and live microservices. Discover the team's specialized approaches, including end-to-end stress testing, measuring pipeline data loss against product SLAs, and per-message data quality assessment. Gain insights into the value these tools and processes bring to engineers and Twitter's data customers. Follow along as the speaker covers topics such as staging environments, operating constraints, testing methodologies, and the development of better testing tools to meet the growing demands of Twitter's data business.

Syllabus

Introduction
Good nip at Twitter
Quality engineering
Staging environment
Operating constraints
DDI DNS adage
Testing for data products
PowerTrack
Two problems to solve
Puli as hell to solve
Experimenting with embedding
Testing as a service
Five factor testing
Better testing tools
Overhead
What Were We Missing
Stress Testing
Load Testing
Load Tool
Reinventing Tools

Taught by

Strange Loop Conference

Reviews

Start your review of Scaling Validation and Quality of Streaming Data Products at Twitter

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.