RoboCat - A Self-Improving Generalist for Robotic Manipulation

Explore a groundbreaking conference talk on RoboCat, a multi-embodiment, multi-task generalist agent for robotic manipulation. Delve into the development of this visual goal-conditioned decision transformer capable of consuming action-labelled visual experience across various robotic arms and tasks. Learn how RoboCat demonstrates the ability to generalize to new tasks and robots, both zero-shot and through rapid adaptation. Discover the agent's potential for autonomous improvement through self-generated data for subsequent training iterations. Examine large-scale evaluations conducted in simulation and on three different real robot embodiments, revealing RoboCat's cross-task transfer capabilities and increased efficiency in adapting to new tasks as its training data grows and diversifies. Gain insights from Giulia Vezzani, a Staff Research Engineer at Google DeepMind, as she shares her expertise in generalist agents and quick adaptation for robotic manipulation.