Lip-Interact - Improving Mobile Device Interaction with Silent Speech Commands

Explore a groundbreaking interaction technique for smartphones in this 20-minute conference talk from the ACM User Interface Software and Technology Symposium. Discover Lip-Interact, a system that enables users to control their devices through silent speech commands. Learn how the front camera captures mouth movements and utilizes deep learning to recognize 44 different commands for both system-level and application-level functionalities. Examine the results of three user experiments evaluating recognition accuracy, input efficiency compared to touch, and privacy considerations versus voiced commands. Gain insights into how this innovative approach can enhance one-handed device operation, improve interaction fluency, and provide efficient access to smartphone features in various contexts.