Hyper Speed - When Big Data Blooms

Explore the challenges of processing massive data volumes in this NDC Oslo 2021 conference talk. Dive into Report URI's infrastructure and learn how Bloom Filters, an often-overlooked data structure, enable handling immense amounts of data with exceptional speed and space efficiency. Discover techniques for processing up to 65,000 JSON payloads per second containing global client telemetry. Gain insights into cross-site scripting, Coinhive, Magecart, Content Security Policy, and CSP reports. Understand the implementation of Bloom Filters, including insertion methods, probabilities, and width considerations. Examine real-world applications through case studies from Akamai and Google Safe Browsing. Explore additional data structures like Count-Min Sketch, Top-K, and HyperLogLog for big data analysis.