Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Inside TensorFlow - tf.distribute.Strategy

TensorFlow via YouTube

Overview

Dive into an in-depth technical session on TensorFlow's tf.distribute.Strategy, presented by TensorFlow Software Engineer Josh Levenberg. Explore the design principles behind this powerful feature, which aims to simplify distribution across various use cases. Learn about data parallelism, parameter servers, central storage, mirrored variables, and all-reduce algorithms. Understand the differences between strategies, including OneDevice and Default, and how they affect training with Keras and Estimator. Discover key concepts such as mirrored vs. per-replica values, replica vs. variable locality, and the implementation of custom training loops. Gain insights into optimizer implementations, loss averaging, and metric calculations in distributed environments. Perfect for developers and researchers looking to leverage TensorFlow's distributed computing capabilities effectively.

Syllabus

Intro
A class with multiple implementations
Data parallelism
Parameter servers and workers
Central Storage
Mirrored Variables
All-reduce algorithm
Ring all-reduce
Hierarchical all-reduce
OneDevice Strategy
Parallel input preprocessing: coming
What changes when you switch strategies?
# Training with Keras
# Training with Estimator
Concept: Mirrored vs. per-replica values
Support computations following this pattern
setup
loss, optimizer
# Custom training loop, part 3: each replica
Concept: Modes
all replicas
outer loop
Default Strategy
# Average loss using the global batch size
# Optimizer implementation, part 1
merge_call(fn, args) is our secret weapon
# Optimizer implementation, part 2
Concept: Replica vs. variable locality
One standard pattern for updating state
# Example: Mean metric
Questions?

Taught by

TensorFlow

Reviews

Start your review of Inside TensorFlow - tf.distribute.Strategy

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.