Explore a groundbreaking development framework for complex multimodal interactions using large language models in this conference talk from CHI 2024. Delve into ReactGenie, which combines voice and touch interactions to create more efficient multimodal interfaces. Learn how this framework simplifies the development process by separating multimodal input from the computational model, enabling easier creation of capable interfaces. Discover the use of NLPL (Natural Language Programming Language) and neural semantic parsing to translate user commands. Examine the framework's ability to interpret parsed NLPL and compose primitives for implementing complex user commands. Gain insights from an evaluation showing developers can learn and build non-trivial ReactGenie applications quickly, and how end-users complete tasks faster with less cognitive load compared to traditional GUIs.
ReactGenie - Development Framework for Complex Multimodal Interactions Using Large Language Models
Association for Computing Machinery (ACM) via YouTube
Overview
Syllabus
ReactGenie: A Development Framework for Complex Multimodal Interactions Using Large Language Models
Taught by
ACM SIGCHI
Related Courses
-
Development with Large Language Models Tutorial – OpenAI, Langchain, Agents, Chroma
-
Memoro: Using Large Language Models for Real-Time Memory Augmentation
-
Discovering Natural Language Commands in Multimodal Interfaces
-
Building Humane User Interfaces with Language Models
-
LLMR: Real-time Prompting of Interactive Worlds Using Large Language Models
-
MUD - Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling
