Astronomer's Journey from Cloud Lock-in to Open Source Independence

Explore Astronomer's journey from cloud lock-in to open-source independence in this 24-minute Linux Foundation conference talk. Learn how the data engineering platform transitioned from AWS-managed services to open-source solutions running on Apache Mesos and DC/OS. Discover the challenges faced with proprietary components and the steps taken to adopt Apache Airflow, Apache Kafka, and Marathon. Gain insights into the company's engineering and product obstacles, infrastructure-as-code practices using Terraform, monitoring solutions with Prometheus, and future plans involving Kafka Connect, Amazon S3, and Kubernetes. Understand the benefits and considerations of moving away from cloud provider lock-in and embracing open-source technologies for greater flexibility and independence in data engineering workflows.