Explore key developments in enterprise-level data provenance through this 44-minute conference talk from the Dutch Seminar on Data Systems Design. Dive into Microsoft's Gray Systems Lab research on provenance capture, querying, and applications, featuring detailed discussions of OneProvenance and DSProvenance engines. Learn about OneProvenance's implementation in Microsoft Purview for extracting provenance from Azure SQL, and understand how DSProvenance handles both static and dynamic provenance in data science pipelines. Discover PurviewQL, a SQL-based solution for simplified provenance and metadata management in data catalogs. Gain insights into practical applications of provenance across various domains including query optimization, job scheduling, semantic type inference, code synthesis, and data quality. The presentation covers essential aspects of connecting datasets, generation workflows, and metadata for enterprise governance, auditing, and observability purposes.
Provenance Research in Data Management Systems at Microsoft
DSDSD - Dutch Seminar on Data Systems Design via YouTube
Overview
Syllabus
We hold bi-weekly talks on Fridays from PM to 5 PM CET for and by researchers and practitioners designing and implementing data systems. The objective is to establish a new forum for the Dutch Data Systems community to come together, foster collaborations between its members, and bring in high-quality international speakers. We would like to invite all researchers, especially also Ph.D. students, who are working on related topics to join the events. It is an excellent opportunity to receive feedback early on from researchers in your field.
Taught by
DSDSD - Dutch Seminar on Data Systems Design
Related Courses
-
Leveraging Generative AI for Data Processing and Database Management
-
LingoDB: An Open Compilation and Optimization Framework for Sustainable Data Processing
-
Decoupling Compute and Storage for Stream Processing Systems
-
Efficient Detection of Multivariate Correlations in Static and Streaming Data
-
Lambda Functions and List Operations in DuckDB SQL
-
Towards LLM-augmented Database Systems
