New Perspectives on Harnessing Foundation Models for Robot Learning

Explore new perspectives on leveraging foundation models for robot learning in this insightful conference talk by Mandi Zhao. Delve into two recent projects that address the challenge of incorporating high-level pre-trained capabilities into robotic systems for low-level embodied tasks. Discover RoCo, a project utilizing zero-shot Large Language Models (LLMs) as a communication tool for multi-robot collaboration. Learn about Real2Code, an innovative approach to the Real2Sim2Real problem for articulated objects that adapts both LLM and large pre-trained vision models. Gain valuable insights from Zhao's research at Stanford University and her experiences at Meta AI and Nvidia Seattle Robotics Lab. Understand the potential of data-driven approaches in enabling embodied systems to perceive, reason, and make sequential decisions in the real world.