LLMR: Real-time Prompting of Interactive Worlds Using Large Language Models

Explore a groundbreaking framework for creating and modifying interactive Mixed Reality experiences in real-time using Large Language Models (LLMs). Delve into the LLMR (Large Language Model for Mixed Reality) system, which tackles challenges in scenarios with limited training data or complex design goals requiring synthesis of internal dynamics, intuitive analysis, and advanced interactivity. Learn about novel strategies implemented in LLMR, including scene understanding, task planning, self-debugging, and memory management techniques that significantly outperform standard GPT-4 in error reduction. Discover how this framework leverages text interaction and the Unity game engine to achieve cross-platform interoperability, demonstrated through various example worlds. Gain insights from a usability study revealing positive user experiences and the system's potential for widespread adoption in creating diverse objects, tools, and scenes in Mixed Reality environments.