Explore the world of big data workflow scheduling in this 41-minute conference talk from ApacheCon 2022. Dive into Apache DolphinScheduler, a distributed, scalable, and visual cloud-native workflow task scheduling platform. Learn about its decentralized architecture, microkernel plug-in design, and enhanced permission isolation. Discover how to configure inter-workflow dependencies and adjust workflow runtime. Gain insights into the new features of version 3.0, including AWS and Kubernetes support, and the Python API for workflow-as-code implementation. Understand the basic concepts, usage examples, and latest community developments. Master the platform's core features, task scheduling techniques, Python API utilization, and upcoming roadmap. Explore topics such as architecture, typical use cases, logs, task management, multi-cloud support, ML orchestration, data management, ETL job handling, and multi-tenancy.
Big Data Workflow Scheduling - Introducing Apache DolphinScheduler
Overview
Syllabus
Introduction
Agenda
What is Dolphin Scheduler
Architecture of Dolphin Scheduler
Features of Dolphin Scheduler
Dolphin Scheduler History
Typical Use Cases
Features
Basic Functions
Logs
Task Management
Multilevel Monitoring
Tasks
MultiCloud
Service Product Interface
New Features
Python DolphinScheduler
ML Orchestration
Virtual Learning Overflow
Data Management
ML Ops
Kubernetes
MultiCluster Management
MultiMulti tenancy
ETL Job Management
ETL Pipeline
UDF
Kafka Scaling
SQL Task Compensation
Resources
Task
References
Multitenant
Taught by
The ASF
