top of page

Human-Computer Interaction

Synergistic Multi-Modal Annotation for 3D Objects on Vision Pro

Team: Bo, Danny, Theo

Human-computer Interaction, XR, Gesture Recognition, Annotation.

In VR annotation, minimizing cognitive load is key for an enjoyable experience. While current platforms offer diverse input modalities (controllers, gaze, gestures), multi-modal interactions have been shown to enhance performance and overcome single-modality limitations [1]. We present a spatial annotation framework on Vision Pro that integrates (1) gaze targeting, (2) gesture-voice triggers (open palm to start, fist to stop, "yeah" to post), and (3) AI-driven speech refinement. The design prioritizes user agency by allowing modality selection, aiming to explore low-cognitive-load annotation that balances multi-modal benefits with user comfort and control.

On-Skin Interface

Team: Bo, Bryan, Jaron

Human-computer Interaction, Nano-material, VR, CAVE-like System, Arduino, Wireless Communication, Signal Processing

The idea was inspired  by a VR project developed for patients with grasping disabilities and the patients are not able to grasp the traditional hand-held controller. We would like to introduce the innovative wearable interface, on-skin interface, applied in a Cave Automatic Virtual Environment system, as known as CAVE-like system, providing a more immersive and inclusive interaction within virtual environments. The essential feature of the interface is that we take use of nano materials and it can be self-powered and positioned anywhere on the user's body according to individual needs and preferences. This flexibility ensures that the controller can accommodate a wide range of physical abilities and ergonomic design. For example, in our demonstration, users can replicate the distinctive hand gestures of the character Spiderman, sending commands to the system to simulate web launching in a gaming environment.

Development Journal check here

Proposal

Demo Showcase

Hpatic Glove

Hpatic Glove with Flex Sensor
00:13
DIY Haptic Glove
00:05
Weart Haptic Glove testing in CAVE
00:25

©2023 by My Site. Proudly created with Wix.com

bottom of page