Reproducible and Deployable Data Science with Open-Source Python

Discover how to create scalable, reproducible, and deployable data science code in this EuroPython Conference talk. Learn to bridge the gap between rapid iteration and engineering reliability by leveraging three powerful open-source tools: Kedro, Apache Airflow, and Great Expectations. Explore techniques for developing modular and maintainable data science code with Kedro, orchestrating workflows using Apache Airflow with Astronomer, and ensuring data quality through Great Expectations. Gain insights into overcoming challenges associated with Jupyter notebooks, implementing declarative data management, and managing parameters and configurations. Delve into the AI lifecycle for IT production, examine different strategies for taking advantage of Kedro's extensibility, and understand the benefits of developing with Kedro while orchestrating with Airflow. By the end of this 30-minute talk, acquire the knowledge to create consensus among data scientists, data engineers, and machine learning engineers, enabling efficient collaboration in data science projects.