Generalizable Robotic Agents Through Large-Scale Simulation

Explore the groundbreaking research on training robotic agents through large-scale simulation in this insightful talk by Luca Weihs, research scientist and lead of the AI2-THOR team at the Allen Institute for AI. Discover how imitating shortest-path planners in simulation produces Stretch RE-1 robotic agents capable of navigating, exploring, and manipulating objects in both simulated and real-world environments using only RGB sensors. Learn about the end-to-end, transformer-based SPOC architecture and the importance of powerful visual encoders paired with extensive image augmentation. Gain insights into the massive scale and diversity of training data, involving millions of frames collected from approximately 200,000 procedurally generated houses with 40,000 unique 3D assets. Explore how scale and careful caching enable online reinforcement learning with transformers in diverse environments, leading to even more proficient agents. Understand the implications of this research for the future of generalizable robotic agents and the potential applications in real-world scenarios.