General Coordinates Network: Harnessing Kafka for Real-Time Open Astronomy at NASA

Explore the innovative use of Kafka in astronomy research through this conference talk on the General Coordinates Network (GCN). Discover how streaming technologies are revolutionizing the study of explosive transients and variability in the "time domain," particularly for black holes and neutron stars. Learn about the architectural choices, challenges, and lessons learned in adapting Kafka for open science and open data. Gain insights into a novel approach to OpenID Connect / OAuth2 in Kafka, designed to securely scale access from within organizations to the general public. Examine the case study of GCN, a NASA-run public collaboration platform that has facilitated seminal advances in astronomy over the past 30 years by disseminating observations, predictions, and observing plans. Understand how GCN connects space- and ground-based observatories, physics experiments, and thousands of astronomers worldwide, enabling real-time data sharing and collaboration in the field of high-energy, multi-messenger, and transient phenomena.