Explore a groundbreaking development framework for complex multimodal interactions using large language models in this conference talk from CHI 2024. Delve into ReactGenie, which combines voice and touch interactions to create more efficient multimodal interfaces. Learn how this framework simplifies the development process by separating multimodal input from the computational model, enabling easier creation of capable interfaces. Discover the use of NLPL (Natural Language Programming Language) and neural semantic parsing to translate user commands. Examine the framework's ability to interpret parsed NLPL and compose primitives for implementing complex user commands. Gain insights from an evaluation showing developers can learn and build non-trivial ReactGenie applications quickly, and how end-users complete tasks faster with less cognitive load compared to traditional GUIs.
ReactGenie - Development Framework for Complex Multimodal Interactions Using Large Language Models
Association for Computing Machinery (ACM) via YouTube
Overview
Syllabus
ReactGenie: A Development Framework for Complex Multimodal Interactions Using Large Language Models
Taught by
ACM SIGCHI
Related Courses
-
Development with Large Language Models Tutorial – OpenAI, Langchain, Agents, Chroma
-
Redefining User Experience with Large Language Models - From Chat to Voice Interactions
-
Memoro: Using Large Language Models for Real-Time Memory Augmentation
-
Discovering Natural Language Commands in Multimodal Interfaces
-
Building Humane User Interfaces with Language Models
-
DeLLMa: A Framework for Decision Making Under Uncertainty with Large Language Models
