V-DAT (Virtual Reality Data Analysis Tool): Supporting Self-Awareness for Autistic People from Multimodal VR Sensor Data

Title of the Paper

V-DAT (Virtual Reality Data Analysis Tool): Supporting Self-Awareness for Autistic People from Multimodal VR Sensor Data

Paper Information

• Research Domain: Virtual Reality, Accessibility Systems, Autism Studies
• Keywords: Autism, Virtual Reality, Accessibility, Data Analysis Pipeline, Multimodal Sensor Data, User Interface, Sensory Issues
• Conference/Journal: UIST '23 (The 36th ACM Symposium on User Interface Software and Technology)
• Publication Date: October 29 - November 1, 2023
• DOI: https://doi.org/10.1145/3586183.3606797

Research Background and Issues

• Issues and Challenges:

  • Virtual reality technology offers flexible immersive experiences for autistic individuals, which can be used for social and educational purposes. However, existing studies primarily rely on single-signal analysis and lack integration of multimodal sensor data.
  • Understanding the behavior of autistic individuals requires comprehensive analysis across different signals (e.g., head movements, eye tracking, physiological data), but current research lacks suitable data management pipelines for autism studies.
  • Many studies overlook the perspectives of autistic individuals themselves, failing to assess the acceptability of data analysis results.

• Significance:

  • Comprehensive analysis of multimodal data helps identify behavioral characteristics related to sensor data and facilitates the development of autism training techniques.
  • Enables autistic individuals to reflect on their behavior while assisting experts in making clinical decisions supported by quantitative data evidence.

• Motivation and Related Work:

  • Virtual reality is widely used to support autistic individuals in daily skills, driving, interview techniques, and social interaction training.
  • Sensor data (e.g., eye tracking, audio, physiological signals) has proven useful in understanding stress, engagement, and anxiety levels in autistic individuals.
  • Research needs to address consistency in data collection and analysis while providing more comprehensive data visualization tools for experts and general users.

Solution

• Method and Approach:

  • Introduced V-DAT (Virtual Reality Data Analysis Tool), an integrated multimodal sensor data management pipeline for autism research and training.
  • V-DAT employs four primary sensor data types: head position and rotation, eye movement, audio data, and physiological signals (EDA, BVP, HR, TMP, IBI).
  • Implements data collection, processing, and visualization, including anomaly detection, audio analysis, gaze region definition, and physiological signal processing.

• Innovations:

  • V-DAT not only facilitates data collection but also synchronizes video presentations of training analysis results through comprehensive data visualization, making result interpretation more intuitive.
  • Supports processing and synchronization of multiple sensor modalities, marking a first in existing autism-related VR research.
  • Provides a user-friendly interactive interface, enabling both autistic individuals and experts to easily access and utilize analysis results.

• Implementation Steps and Key Technologies:

  • Data Collection: Utilizes head-mounted display (HMD) and Empatica E4 wristband devices, combined with Unity3D development environment for data acquisition.
  • Data Processing: Introduces anomaly detection APIs and audio data segmentation modules, ensuring timeline alignment of multimodal data through synchronization techniques.
  • Visualization: Develops a web interface integrating video and data visualization, presenting training segments and sensor data synchronously.

Research Outcomes

• Specific Results:

  • V-DAT can display multimodal sensor data (e.g., head movement, eye focus, audio volume, physiological signal anomalies) of autistic individuals during virtual training through a graphical interface.
  • Provides a user-friendly interface to help autistic individuals reflect on their training content, supporting more effective independent analysis and social interaction.

• Advantages and Experimental Results:

  • V-DAT successfully addresses the limitations of previous single-modality data analysis, offering synchronized and intuitive multimodal data visualization.
  • In a case study involving 20 autistic participants, V-DAT enabled them to reflect on their behavior, enhancing their perceived self-efficacy (PSE scores significantly increased, p < 0.05).
  • Experts indicated that V-DAT could be used for clinical diagnosis, bridging communication gaps between home care centers and outpatient services.

• Limitations and Future Directions:

  • Potential Privacy Issues: Participants expressed concerns about the scope of data usage or sharing.
  • Practical Application Challenges: Experts noted that system connectivity might face challenges in hospital environments due to device failures or loading issues.
  • Scalability: More virtual training content or scenarios tailored for autistic individuals need to be developed to cover a broader range of social interaction contexts.

Conclusion

V-DAT integrates a multimodal sensor data management pipeline to provide autistic individuals with opportunities for behavior reflection supported by quantitative data, while assisting experts in making data-driven clinical decisions. This study demonstrates V-DAT's potential in promoting social skills training and diagnosis, while proposing directions for sustainable application through improved data management and content expansion.