Explore a lecture on natural language instruction following for robotic control and collaboration. Dive into two projects that investigate learning to execute natural language commands with robotic agents and in collaborative scenarios. Discover new datasets, interpretable models, and learning methods combining supervised and reinforcement learning. Examine the task of mapping raw observations and language to low-level continuous control of a quadcopter drone, as well as the CerealBar multi-player 3D game for studying instruction following in collaborative environments. Learn about innovative approaches to natural language understanding in situated interactive scenarios, presented by Cornell University Assistant Professor Yoav Artzi. Gain insights into experimental setups, human evaluation, and cascaded evaluation metrics for assessing instruction following performance.
Overview
Syllabus
Intro
Mapping Instructions to Actions
Mapping Instructions to Control • Drone state is determined by a configuration of target velocities
Modular Approach
Single-model Approach
Visitation Distribution for Navigation
Simulation-Reality Joint Learning
Training Data
Learning Architecture Supervised Learning Learning
RL for Control
Reward Goals
SuReAL Supervised and Reinforcement Asynchronous Learning
Experimental Setup
Human Evaluation
Game Environment
Two Players
Game Objective
Action and Dynamic
The CerealBar Scenario
The Task Goal: learn to map leader instructions to follower actions
Instruction Following in Collaborations
Learning with Error Propagation
Generating Error Recovery Examples
Generated Recovery Example
Data Collection
Data Statistics
Data Under-utilization
Cascaded Evaluation Metrics
Results: Held-out Games
Taught by
Paul G. Allen School
