Human-Computer Interaction
Synergistic Multi-Modal Annotation for 3D Objects on Vision Pro
Team: Bo, Danny, Theo
Human-computer Interaction, XR, Gesture Recognition, Annotation.
In VR annotation, minimizing cognitive load is key for an enjoyable experience. While current platforms offer diverse input modalities (controllers, gaze, gestures), multi-modal interactions have been shown to enhance performance and overcome single-modality limitations [1]. We present a spatial annotation framework on Vision Pro that integrates (1) gaze targeting, (2) gesture-voice triggers (open palm to start, fist to stop, "yeah" to post), and (3) AI-driven speech refinement. The design prioritizes user agency by allowing modality selection, aiming to explore low-cognitive-load annotation that balances multi-modal benefits with user comfort and control.
On-Skin Interface
Team: Bo, Bryan, Jaron
Human-computer Interaction, Nano-material, VR, CAVE-like System, Arduino, Wireless Communication, Signal Processing
The idea was inspired by a VR project developed for patients with grasping disabilities and the patients are not able to grasp the traditional hand-held controller. We would like to introduce the innovative wearable interface, on-skin interface, applied in a Cave Automatic Virtual Environment system, as known as CAVE-like system, providing a more immersive and inclusive interaction within virtual environments. The essential feature of the interface is that we take use of nano materials and it can be self-powered and positioned anywhere on the user's body according to individual needs and preferences. This flexibility ensures that the controller can accommodate a wide range of physical abilities and ergonomic design. For example, in our demonstration, users can replicate the distinctive hand gestures of the character Spiderman, sending commands to the system to simulate web launching in a gaming environment.
Development Journal check here






Proposal
The short paper is awarded as The Best Short Paper in IEEE VR 2024 XRIOS.
DOI: 10.1109/vrw62533.2024.00051​
Demo Showcase
Hpatic Glove


