Explore the challenges and solutions for scaling Python with Dask on distributed hardware in this PyCon US talk. Dive into deployment strategies for Dask on cluster resource managers like Kubernetes, Yarn, and cloud platforms. Learn how the Dask library extends popular Python data science tools to handle 100+TB datasets across multi-core workstations and distributed clusters. Discover approaches to balance load, share resources, control access, and ensure security when deploying Dask within organizations. Examine real-world examples showcasing Dask's positive social impact in large-scale data processing. Gain insights into uniform software environments, resource sharing, credentials management, and cost optimization for IT professionals. Understand the landscape of friendly resource managers, managed solutions, and opinionated approaches for efficient Python deployment at scale.
Overview
Syllabus
Introduction
Why this talk
Data Science Libraries
Task
Desk
Environment Management
Uniform software environments
Data science vs IT
Resource sharing
Access
IT Professional
Credentials
Security
Costs
Avoid Track Optimize
Cost
Conclusion
Friendly Resource Managers
Managed Solutions
opinionated solutions
Coyle Computing
Managed Services
Summary
Taught by
PyCon US
