Smart Data Pipelines to Azure - Ingesting and Migrating Data the DataOps Way

Overview

Explore smart data pipelines for ingesting and migrating data to Azure platforms in this 30-minute expert talk. Discover how StreamSets, a Microsoft startup program member, helps customers accelerate Azure cloud adoption using its DataOps platform. Learn about optimizing data for Azure Synapse, ADLS, and HDInsight, and embedding DataOps principles in data pipelines. Gain insights on integrating StreamSets with Azure technologies, leveraging Azure Data Factory alongside StreamSets, implementing DevOps and DataOps practices in data workflows, ensuring GDPR compliance, and deploying data pipelines efficiently. Understand the challenges in modern data applications, the benefits for developers, and the typical target customers for StreamSets solutions.

Syllabus

- Introduction.
- What is StreamSets? What are some of the gaps you see in existing ETL tools?.
- What are some of the biggest trends and issues with modern data applications?.
- What's the breath of Azure technology supported by StreamSets? How to integrate with Azure Synapse?.
- How do you see customers leveraging both these Azure Data Factory pipeline capabilities alongside StreamSets?.
- What other Azure services or products does StreamSets agree with?.
- How do you see DevOps and DataOps practices fitting into these data workflows and data pipelines? Why should developers care about these tools?.
- How does StreamSets remains GDPR compliant?.
- How quickly can StreamSets deploy a data pipeline? How is the deployment process?.
- Does deploying workloads to Azure require rewrites?.
- What are the typical target customers?.
- Closing Notes.