Inside TensorFlow - tf.data + tf.distribute

Overview

Explore best practices for tf.data and tf.distribute in this 46-minute TensorFlow presentation by Software Engineer Jiri Simsa. Dive into building efficient TensorFlow input pipelines, improving performance with the tf.data API, and implementing distributed training strategies. Learn about software pipelining, parallel transformation, and parallel extraction techniques. Discover the benefits of TensorFlow Datasets (TFDS) and various distributed training approaches, including multi-GPU all-reduce synchronous training and multi-worker setups. Gain insights into performance optimization for ResNet models, parameter servers, and central storage concepts. Understand the programming model and features supported in TensorFlow 2.0 Beta for distributed training.

Syllabus

Intro
ML Building Blocks
TensorFlow APIs
Why input pipeline?
tf.data: TensorFlow Input Pipeline
Input Pipeline Performance
Software Pipelining
Parallel Transformation
Parallel Extraction
tf.data Options
TFDS: TensorFlow Datasets
Why distributed training?
tf.distribute. Strategy API
How to use tf.distribute.Strategy?
Multi-GPU all-reduce sync training
All-Reduce Algorithm
Synchronous Training
Multi-GPU Performance ResNetso v1.5 Performance with
Multi-worker all-reduce sync training
All-reduce sync training for TPUs
Parameter Servers and Workers
Central Storage
Programming Model
What's supported in TF 2.0 Beta