Lessons Learned in Black Box Monitoring 25,000 Endpoints

Discover how the United States Digital Service tackles the challenge of monitoring 25,000 government endpoints without internal access in this 23-minute conference talk from SREcon19 Americas. Learn about black box monitoring techniques, the architecture of their monitoring system, and the implementation of tools like Prometheus Black Box Exporter and Alert Manager. Explore the process of identifying problems across government services, including case studies on Air Now.gov and USPTO. Gain insights into cost savings, results achieved, and valuable lessons learned in alerting and monitoring government systems. Understand the importance of SRE teams in government and how this approach aims to change the culture of service reliability in the public sector.