Explore the intersection of emotion and touch in virtual reality through this IEEE VR 2022 tutorial. Delve into introductory research on enhancing social interaction in VR by incorporating emotions based on voice, facial expressions, and touch. Learn about the potential of haptic devices and tactile sensations to bridge the gap in nonphysical interpersonal communication. Discover how machine learning can be applied to emotion recognition using data from various sensory devices. Examine the importance of emotions in VR for creating a sense of connection and presence, and explore current VR offerings such as vibration feedback, haptic gloves, and full-body coverage devices. Investigate the concept of aggregating emotions data to simulate tactile sensations and improve immersion through smart garments. Consider different types of touch and emotion in virtual environments, including classical touch, physical touch, and background emotions. Gain insights into the creation of databases for facial expression and speech recognition, as well as the current setup using Oculus Rift and Muse 2 for various biometric measurements. Explore related applications in museums and retail, and understand the relevance of this technology for long-distance communication, pandemic situations, and individuals with sensory impairments. Conclude by examining future work in wearable devices, object recognition via touch, and the development of cost-effective, accessible solutions for enhancing virtual reality experiences.
Overview
Syllabus
Intro
College campus visits Read bedtime story Virtual showrooms
Problem No technical language to define touch Important for immersive VR experience Bring body back into equation
Importance of emotions in VR Sense of connection Sense of presence -Enhanced experience
Current Offerings in VR - Vibration (forced feedback) Haptic Gloves - Fingertip devices Partial body coverage Full body coverage
The Big Idea - Aggregating emotions data Simulate tactile sensations Smart garments -Better immersion
Things to think about Type of touch and emotion
Virtual Environment Restaurant Two parties sitting at a table Ocean or forest sounds
Classical Touch Initiated Awareness before touch Receive response Will awareness match response
Physical Touch Bumping into world or avatar Awareness of action Feeling object Feeling avatar Will awareness match feeling
Background Emotions Sounds Visual effects Indicator of next action Will awareness match feeling
Steps Deep research of emotions Creation of databases Implementation of Agent
Datasets Creation of databases - facial expression and speech recognition - Collecting data Processing data Aggregate data
Current Setup Oculus Rift (2) Muse 2
Facial expression recognition (eye-tracking) Body gestures (cameras) Skin temperature - Pupil dilation Speech recognition
Related: Museums VR Touch Museum Mixture of real-life & visual props Potential increase in sense of presence and learning Results: better sense of presence
Related: Retail Consumer able to touch products Work best for buying tools vs Clothing Convince self touching tool but not the feel of fabric - Need for touch
Reason before pandemic Reason during pandemic Reason post pandemic Persons dealing with loss of a sense
Future work Wearable devices Object recognition via touch - Touch between avatars Smart textiles Cost-effective and accessible solution
Taught by
IEEE Virtual Reality Conference
