Completed
First bottleneck: disk IO on old Amazon EBS
Class Central Classrooms beta
YouTube playlists curated by Class Central.
Classroom Contents
Storage Systems at a Rapidly Scaling Startup - Instagram's Infrastructure Evolution
Automatically move to the next video in the Classroom when playback concludes
- 1 Intro
- 2 Approach to data scaling problems
- 3 2 total engineers
- 4 First bottleneck: disk IO on old Amazon EBS
- 5 Django DB Routers
- 6 PG Replication to bootstrap nodes
- 7 Scaling up Redis
- 8 fork() and COW
- 9 Vertical partitioning by data type
- 10 No easy migration story; mostly double-writing
- 11 Replicating + deleting often leaves fragmentation
- 12 Why not Redis for kv caching?
- 13 Slab allocator
- 14 Focus on client
- 15 Testing & monitoring kept concurrent fires to a minimum
- 16 Scaling Out
- 17 Database Scale Out
- 18 Double write, shadow reads
- 19 Stressing about Primary Key
- 20 Data loss, segfaults
- 21 train + rapidly approaching cliff
- 22 Logical partitioning, done at application level
- 23 note to self: pick a power of 2 next time
- 24 Postgres "schemas"
- 25 9.2 upgrade: bucardo to move schema by schema
- 26 ID generation
- 27 Snowflake, other options
- 28 41 bits: time in millis (41 years of IDs) 13 bits: logical shard ID 10 bits: auto-incrementing sequence, modulo 1024.
- 29 Lesson learned
- 30 minimize moving parts
- 31 Ending the year
- 32 Launched Android
- 33 Stability, FB
- 34 Scaling cut-overs, ramp- ups, and development
- 35 Dynamic ramp-ups and config
- 36 Python Knobs
- 37 Decouple deploy from feature rollout
- 38 In memory requirement
- 39 Simplest thing was breaking
- 40 Trimming
- 41 C* cluster is 35% of the size of Redis one
- 42 Handling deletes
- 43 Redis way: LREM
- 44 Not so hot for an AP system
- 45 2014 project
- 46 Spam fighting
- 47 Generic features + machine learning
- 48 Hadoop + Hive + Presto
- 49 2010 vintage infra
- 50 #1 impact: recruiting
- 51 Wrap up
