CI/CD for Data - Building Dev/Test Data Environments with Open Source Stacks

Explore industry best practices for data lifecycle management and learn how to implement them using open source tools like Jenkins and lakeFS in this informative conference talk. Discover the challenges faced by data teams in incorporating software development best practices into data products, and understand the importance of ensuring accuracy, resiliency, and observability in production data environments. Gain insights into building high-quality data products by applying techniques such as testing changes in isolation, reproducing data errors, and implementing rollback mechanisms. Compare the evolved field of software development with the emerging field of data product development, and learn how to bridge the gap by extending software development best practices to data products.