Explore an in-depth interview with the authors of SayCan, a groundbreaking approach combining large language models with robotic affordances. Delve into the intricacies of integrating semantic knowledge from language models with low-level robotic skills to execute complex, real-world tasks. Learn about the system's architecture, data collection challenges, experimental results, and future directions in robotics and AI. Gain insights into the authors' problem-solving strategies, project management techniques, and perspectives on emerging technologies like the Tesla Bot. Discover how SayCan bridges the gap between natural language instructions and physical task execution, paving the way for more capable and versatile robotic systems.
Author Interview - SayCan - Do As I Can, Not As I Say - Grounding Language in Robotic Affordances
Overview
Syllabus
- Introduction & Setup
- Acquiring atomic low-level skills
- How does the language model come in?
- Why are you scoring instead of generating?
- How do you deal with ambiguity in language?
- The whole system is modular
- Going over the full algorithm
- What if an action fails?
- Debunking a marketing video :
- Experimental Results
- The insane scale of data collection
- How do you go about large-scale projects?
- Where did things go wrong?
- Where do we go from here?
- What is the largest unsolved problem in this?
- Thoughts on the Tesla Bot
- Final thoughts
Taught by
Yannic Kilcher
