Making Data the Central Feature: From Task-Oriented to Data-Oriented Workflows

Explore a conference talk from Data Science Conference Europe 2023 that delves into the evolution of data workflows from task-oriented to data-oriented approaches. Learn how traditional data workflows relied on task-oriented interfaces and tools like Airflow, Luigi, and Prefect for orchestrating and managing data jobs. Discover the industry's paradigm shift towards prioritizing data sets as end products rather than focusing solely on the processes that create them. Understand how modern tools like Mage.ai, Airbyte, and Dagster exemplify this transformation, enabling data engineers to develop high-quality, accessible, and well-documented data assets. Through practical demonstrations, examine various data-oriented tools and grasp the fundamental importance of placing data at the center of engineering workflows rather than the tasks that manipulate it.