The Brain Knows What You Prefer: Using EEG to Decode AR Input PreferencesUnderstanding user input preferences is crucial in immersive environments, where input methods such as gestures and controllers are common. Traditional evaluation methods rely on post experience questionnaires, which don't capture real-time preferences. This study used brain signals to classify input preferences during Augmented Reality (AR) interactions. Thirty participants performed three interaction tasks (pointing, manipulation, and rotation) using hands or controllers. Their electroencephalogram (EEG) data were collected at varying task difficulties (low, medium, high) and phases (preparation, task, and completion). Machine learning was used to classify preferred and non-preferred input methods. Results showed that EEG signals effectively classify preferences with accuracies up to 86%, with the completion phase being the best indicator of preference. In addition, different input methods exhibited distinct EEG patterns. These findings highlight the potential of EEG signals for decoding real-time input preference in AR, offering insights for enhancing user experiences.2025KZKaining Zhang et al.University of South Australia, Empathic Computing LabBrain-Computer Interface (BCI) & NeurofeedbackAR Navigation & Context AwarenessCHI
RadarHand: a Wrist-Worn Radar for On-Skin Touch based Proprioceptive GesturesWe introduce RadarHand, a wrist-worn wearable with millimetre wave radar that detects on-skin touch-based proprioceptive hand gestures. Radars are robust, private, small, penetrate materials, and require low computation costs. We first evaluated the proprioceptive and tactile perception nature of the back of the hand and found that tapping on the thumb is the least proprioceptive error of all the finger joints, followed by the index finger, middle finger, ring finger, and pinky finger in the eyes-free and high cognitive load situation. Next, we trained deep-learning models for gesture classification. We introduce two types of gestures based on the locations of the back of the hand: generic gestures and discrete gestures. Discrete gestures are gestures that start at specific locations and end at specific locations at the back of the hand, in contrast to generic gestures, which can start anywhere and end anywhere on the back of the hand. Out of 27 gesture group possibilities, we achieved 92% accuracy for a set of seven gestures and 93% accuracy for the set of eight discrete gestures. Finally, we evaluated RadarHand’s performance in real-time under two interaction modes: Active interaction and Reactive interaction. Active interaction is where the user initiates input to achieve the desired output, and reactive interaction is where the device initiates interaction and requires the user to react. We obtained an accuracy of 87% and 74% for active generic and discrete gestures, respectively, as well as 91% and 81.7% for reactive generic and discrete gestures, respectively. We discuss the implications of RadarHand for gesture recognition and directions for future works.2024MHMr Ryo Hajika et al.Vibrotactile Feedback & Skin StimulationFoot & Wrist InteractionUIST
Modulating Heart Activity and Task Performance using Haptic Heartbeat Feedback: A Study Across Four Body PlacementsThis paper explores the impact of vibrotactile haptic feedback on heart activity when the feedback is provided at four different body locations (chest, wrist, neck, and ankle) and with two feedback rates (50 bpm and 110 bpm). A user study found that the neck placement resulted in higher heart rates and lower heart rate variability, and higher frequencies correlated with increased heart rates and decreased heart rate variability. The chest was preferred in self-reported metrics, and neck placement was perceived as less satisfying, harmonious, and immersive. This research contributes to understanding the interplay between psychological experiences and physiological responses when using haptic biofeedback resembling real body signals.2024AVAndreia Valente et al.Vibrotactile Feedback & Skin StimulationUIST
The Impact of Sharing Gaze Behaviours in Collaborative Mixed RealityIn a remote collaboration involving a physical task, visualising gaze behaviours may compensate for other unavailable communication channels. In this paper, we report on a 360° panoramic Mixed Reality (MR) remote collaboration system that shares gaze behaviour visualisations between a local user in Augmented Reality and a remote collaborator in Virtual Reality. We conducted two user studies to evaluate the design of MR gaze interfaces and the effect of gaze behaviour (on/off) and gaze style (bi-/uni-directional). The results indicate that gaze visualisations amplify meaningful joint attention and improve co-presence compared to a no gaze condition. Gaze behaviour visualisations enable communication to be less verbally complex therefore lowering collaborators’ cognitive load while improving mutual understanding. Users felt that bi-directional behaviour visualisation, showing both collaborator’s gaze state, was the preferred condition since it enabled easy identification of shared interests and task progress.2022AJAllison Jing et al.XR Collaboration; XR CollaborationCSCW
VRhook: A Data Collection Tool for VR Motion Sickness ResearchDespite the increasing popularity of VR games, one factor hindering the industry's rapid growth is motion sickness experienced by the users. Symptoms such as fatigue and nausea severely hamper the user experience. Machine Learning methods could be used to automatically detect motion sickness in VR experiences, but generating the extensive labeled dataset needed is a challenging task. It needs either very time consuming manual labeling by human experts or modification of proprietary VR application source codes for label capturing. To overcome these challenges, we developed a novel data collection tool, VRhook, which can collect data from any VR game without needing access to its source code. This is achieved by dynamic hooking, where we can inject custom code into a game's run-time memory to record each video frame and its associated transformation matrices. Using this, we can automatically extract various useful labels such as rotation, speed, and acceleration. In addition, VRhook can blend a customized screen overlay on top of game contents to collect self-reported comfort scores. In this paper, we describe the technical development of VRhook, demonstrate its utility with an example, and describe directions for future research.2022EWElliott Wen et al.Motion Sickness & Passenger ExperienceImmersion & Presence ResearchUIST
Emotion Recognition in Conversations using Brain and Physiological SignalsEmotions are complicated psycho-physiological processes that are related to numerous external and internal changes in the body. They play an essential role in human-human interaction and can be important for human-machine interfaces. Automatically recognizing emotions in conversation could be applied in many application domains like health-care, education, social interactions, and entertainment. Facial expressions, speech, and body gestures are primary cues that have been widely used for recognizing emotions in conversation. However, these cues can be ineffective as they cannot reveal underlying emotions when a person involuntarily or deliberately conceals their emotions. Researchers have shown that analyzing brain activity and physiological signals can lead to more reliable emotion recognition since they generally cannot be controlled. However, these body responses in emotional situations have been rarely explored in interactive tasks like conversations. This paper explores and discusses the performance and challenges of using brain activity and other physiological signals in recognizing emotions in a face-to-face conversation. We present an experimental setup for stimulating spontaneous emotions during a face-to-face conversation while recording brain and physiological activity. We then describe our analysis strategies for recognizing emotions using Electroencephalography (EEG), Photoplethysmography (PPG), and Galvanic Skin Responses (GSR) signals in a subject-dependent and subject-independent approach. Finally, we describe new directions for future research in conversational emotion recognition, and the limitations and challenges.2022NSNastaran Saffaryazdi et al.Brain-Computer Interface (BCI) & NeurofeedbackBiosensors & Physiological MonitoringIUI
Bringing the Jury to the Scene of the Crime: Memory and Decision-Making in a Simulated Crime SceneThis paper investigates the use of immersive virtual reconstructions as an aid for jurors during a courtroom trial. The findings of a between-participant user study on memory and decision-making are presented in the context of viewing a simulated hit-run-death scenario. Participants listened to the opening statement of a prosecutor and a defence attorney before viewing the crime scene in Virtual Reality (VR) or as still images. We compare the effects on cognition and usability of using VR over images presented on a screen. We found several significant improvements, including that VR led to more consistent decision-making among participants. This shows that VR could provide a promising solution for the court to present crime scenes when site visitations are not possible.2021CRCarolin Reichherzer et al.University of South AustraliaImmersion & Presence ResearchMuseum & Cultural Heritage DigitizationCHI
OmniGlobeVR: A Collaborative 360-Degree Communication System for VRIn this paper, we present a novel collaboration tool, OmniGlobeVR, which is an asymmetric system that supports communication and collaboration between a VR user (occupant) and multiple non-VR users (designers) across the virtual and physical platform. OmniGlobeVR allows designer(s) to explore the VR space from any point of view using two view modes: a 360° first-person mode and a third-person mode. In addition, a shared gaze awareness cue is provided to further enhance communication between the occupant and the designer(s). Finally, the system has a face window feature that allows designer(s) to share their facial expressions and upper body view with the occupant for exchanging and expressing information using nonverbal cues. We conducted a user study to evaluate the OmniGlobeVR, comparing three conditions: (1) first-person mode with the face window, (2) first-person mode with a solid window, and (3) third-person mode with the face window. We found that the first-person mode with the face window required significantly less mental effort, and provided better spatial presence, usability, and understanding of the partner’s focus. We discuss the design implications of these results and directions for future research.2020ZLZhengqing Li et al.Social & Collaborative VRImmersion & Presence ResearchDIS
A User Study on Mixed Reality Remote Collaboration with Eye Gaze and Hand Gesture SharingSupporting natural communication cues is critical for people to work together remotely and face-to-face. In this paper we present a Mixed Reality (MR) remote collaboration system that enables a local worker to share a live 3D panorama of his/her surroundings with a remote expert. The remote expert can also share task instructions back to the local worker using visual cues in addition to verbal communication. We conducted a user study to investigate how sharing augmented gaze and gesture cues from the remote expert to the local worker could affect the overall collaboration performance and user experience. We found that by combing gaze and gesture cues, our remote collaboration system could provide a significantly stronger sense of co-presence for both the local and remote users than using the gaze cue alone. The combined cues were also rated significantly higher than the gaze in terms of ease of conveying spatial actions.2020HBHuidong Bai et al.University of AucklandFull-Body Interaction & Embodied InputEye Tracking & Gaze InteractionMixed Reality WorkspacesCHI
Mixed Reality Remote Collaboration Combining 360 Video and 3D ReconstructionRemote Collaboration using Virtual Reality (VR) and Augmented Reality (AR) has recently become a popular way for people from different places to work together. Local workers can collaborate with remote helpers by sharing 360-degree live video or 3D virtual reconstruction of their surroundings. However, each of these techniques has benefits and drawbacks. In this paper we explore mixing 360 video and 3D reconstruction together for remote collaboration, by preserving benefits of both systems while reducing drawbacks of each. We developed a hybrid prototype and conducted user study to compare benefits and problems of using 360 or 3D alone to clarify the needs for mixing the two, and also to evaluate the prototype system. We found participants performed significantly better on collaborative search tasks in 360 and felt higher social presence, yet 3D also showed potential to complement. Participant feedback collected after trying our hybrid system provided directions for improvement.2019TTTheophilus Teo et al.University of South AustraliaSocial & Collaborative VRImmersion & Presence ResearchCHI
On the Shoulder of the Giant: A Multi-Scale Mixed Reality Collaboration with 360 Video Sharing and Tangible InteractionWe propose a multi-scale Mixed Reality (MR) collaboration between the Giant, a local Augmented Reality user, and the Miniature, a remote Virtual Reality user, in Giant-Miniature Collaboration (GMC). The Miniature is immersed in a 360-video shared by the Giant who can physically manipulate the Miniature through a tangible interface, a combined 360-camera with a 6 DOF tracker. We implemented a prototype system as a proof of concept and conducted a user study (n=24) comprising of four parts comparing: A) two types of virtual representations, B) three levels of Miniature control, C) three levels of 360-video view dependencies, and D) four 360-camera placement positions on the Giant. The results show users prefer a shoulder mounted camera view, while a view frustum with a complimentary avatar is a good visualization for the Miniature virtual representation. From the results, we give design recommendations and demonstrate an example Giant-Miniature Interaction.2019TPThammathip Piumsomboon et al.University of Canterbury & University of South AustraliaSocial & Collaborative VRMixed Reality Workspaces360° Video & Panoramic ContentCHI
Warping Deixis: Distorting Gestures to Enhance CollaborationWhen engaged in communication, people often rely on pointing gestures to refer to out-of-reach content. However, observers frequently misinterpret the target of a pointing gesture. Previous research suggests that to perform a pointing gesture, people place the index finger on or close to a line connecting the eye to the referent, while observers interpret pointing gestures by extrapolating the referent using a vector defined by the arm and index finger. In this paper we present Warping Deixis, a novel approach to improving the perception of pointing gestures and facilitate communication in collaborative Extended Reality environments. By warping the virtual representation of the pointing individual, we are able to match the pointing expression to the observer's perception. We evaluated our approach in a co-located side by side virtual reality scenario. Results suggest that our approach is effective in improving the interpretation of pointing gestures in shared virtual environments.2019MSMaurício Sousa et al.INESC-ID Lisboa & Universidade de LisboaHand Gesture RecognitionSocial & Collaborative VRCHI
Evaluating the Combination of Visual Communication Cues for HMD-based Mixed Reality Remote CollaborationMany researchers have studied various visual communication cues (e.g. pointer, sketching, and hand gesture) in Mixed Reality remote collaboration systems for real-world tasks. However, the effect of combining them has not been so well explored. We studied the effect of these cues in four combinations: hand only, hand + pointer, hand + sketch, and hand + pointer + sketch, with three problem tasks: Lego, Tangram, and Origami. The study results showed that the participants completed the task significantly faster and felt a significantly higher level of usability when the sketch cue is added to the hand gesture cue, but not with adding the pointer cue. Participants also preferred the combinations including hand and sketch cues over the other combinations. However, using additional cues (pointer or sketch) increased the perceived mental effort and did not improve the feeling of co-presence. We discuss the implications of these results and future research directions.2019SKSeungwon Kim et al.University of South AustraliaMixed Reality WorkspacesCHI
Pinpointing: Precise Head- and Eye-Based Target Selection for Augmented RealityHead and eye movement can be leveraged to improve the user’s interaction repertoire for wearable displays. Head movements are deliberate and accurate, and provide the current state-of-the-art pointing technique. Eye gaze can potentially be faster and more ergonomic, but suffers from low accuracy due to calibration errors and drift of wearable eye-tracking sensors. This work investigates precise, multimodal selection techniques using head motion and eye gaze. A comparison of speed and pointing accuracy reveals the relative merits of each method, including the achievable target size for robust selection. We demonstrate and discuss example applications for augmented reality, including compact menus with deep structure, and a proof-of-concept method for on-line correction of calibration drift.2018MKMikko Kytö et al.Aalto University School of Science, University of South AustraliaEye Tracking & Gaze InteractionAR Navigation & Context AwarenessCHI
Mini-Me: An Adaptive Avatar for Mixed Reality Remote CollaborationWe present Mini-Me, an adaptive avatar for enhancing Mixed Reality (MR) remote collaboration between a local Augmented Reality (AR) user and a remote Virtual Reality (VR) user. The Mini-Me avatar represents the VR user’s gaze direction and body gestures while it transforms in size and orientation to stay within the AR user’s field of view. A user study was conducted to evaluate Mini-Me in two collaborative scenarios: an asymmetric remote expert in VR assisting a local worker in AR, and a symmetric collaboration in urban planning. We found that the presence of the Mini-Me significantly improved Social Presence and the overall experience of MR collaboration.2018TPThammathip Piumsomboon et al.University of South AustraliaSocial & Collaborative VRMixed Reality WorkspacesIdentity & Avatars in XRCHI