The Benefit of Hindsight - Tracing Edge-Cases in Distributed Systems

Explore a groundbreaking approach to troubleshooting rare edge-cases in distributed systems through this 16-minute conference talk from NSDI '23. Discover Hindsight, a lightweight and always-on distributed tracing system that implements retroactive sampling, allowing developers to capture detailed traces of problematic requests without the high overheads associated with traditional tracing methods. Learn how this innovative system, analogous to a car dash-cam, can scale to millions of requests per second, add only nanosecond-level overhead, and seamlessly integrate with existing distributed tracing frameworks. Gain insights into how Hindsight overcomes the limitations of current tracing techniques, providing a solution that balances specificity and overhead in capturing edge-case traces for issues such as high tail latency, errors, and bottlenecked queues.