Predicting Opportune Moments to Deliver Notifications in Virtual Reality

Document Title

Predicting Opportune Moments to Deliver Notifications in Virtual Reality

Document Information

• Field of Study: Human-Computer Interaction, Virtual Reality, Notification Management
• Keywords: Virtual Reality, Notifications, Interruptibility, Predictive Models, Data-Driven

Research Background and Problem

• Key Issues and Challenges:

  • The high level of immersion in Virtual Reality (VR) can lead to a disconnect between users and real-world information, causing missed important notifications and potential user anxiety.
  • Delivering notifications while users are focused on VR activities may disrupt the experience and reduce task performance.
  • Identifying "opportune moments" to deliver notifications can mitigate the negative effects of interruptions, a research area that remains underexplored, especially in VR contexts.

• Research Significance:

  • With the increasing popularity of commercial VR devices, the demand for real-time notifications is growing, making it crucial to balance their disruptiveness with user experience.
  • Findings from studies on interruptibility management in mobile or desktop environments may not directly apply to the highly immersive VR setting.

• Motivation and Related Work:

  • Previous research has explored the impact of interruptions on workflow tasks and proposed sensor-based interruptibility prediction models.
  • However, predicting notification timing in VR remains largely unstudied, and the immersive nature of VR may uniquely influence users' perception of interruptions.

Solution

• Overview of Approach or Solution:

  • Utilize sensor data from VR devices (e.g., head-mounted displays, controllers, eye trackers) to design deep learning models for predicting opportune moments for notifications.
  • Incorporate additional data on the type of VR activity and user engagement levels alongside real-time sensor data.
  • Investigate the contribution of different sensors to prediction performance in various scenarios and compare the effectiveness of personalized models versus generalized models.

• Innovations:

  • The first study to introduce the concept of predicting opportune moments for notifications in VR environments.
  • Integration of multi-sensor data and activity context information to enhance prediction accuracy.
  • Exploration of the impact of activity type and user engagement levels on improving notification timing predictions.

• Technical Implementation Steps:

  1. Data Collection:

     • Recruit 20 participants for multiple VR experiments, combining sensor data with user-annotated data.
     • Use HTC Vive Pro Eye to record user movements and eye-tracking data across different VR scenarios.

  2. Data Annotation:

     • Retrospective annotation: Participants review their VR sessions and label the appropriateness of notification timing (opportune/inopportune).
     • Additional annotation of engagement data based on task focus (high/low engagement) as supplementary information.

  3. Model Development:

     • Design deep learning time-series models (including 1D-CNN, LSTM, and MLSTM-FCN).
     • Use the previous 5 seconds of sensor data to predict whether the next moment is suitable for delivering notifications.

  4. Model Evaluation:

     • Quantify performance differences between personalized and generalized models.
     • Analyze the contribution of different sensors to model performance and evaluate the performance improvement from incorporating activity and user engagement data.

Research Findings

• Specific Results:

  • For models based solely on sensor data:
    • Personalized model performance: 72% recall, 71% precision, 0.86 AUROC.
    • Generalized model performance: 53%-59% recall, 60% precision, 0.73 AUROC.
  • After incorporating activity type and user engagement data:
    • Personalized model performance improved to 81% recall, 82% precision, 0.93 AUROC.
    • Generalized model performance also improved (up to 70% precision, 0.81 AUROC).

• Advantages Over Existing Solutions:

  • The integration of multi-dimensional motion sensor data from VR devices and contextual information enables the model to achieve higher temporal insight and applicability.
  • A shift toward personalized models significantly enhances prediction accuracy, effectively reducing erroneous notifications that could cause disruptions.

• Experimental Results:

  • Different sensors (headset, controllers, eye trackers) contribute variably to interruption prediction across different types of VR tasks.
  • Using multi-sensor data in complex activities best captures dynamic user state changes, resulting in optimal prediction performance.
  • Activity conditions (e.g., game difficulty) and subjective user engagement are critical for improving model performance.

• Limitations and Future Directions:

  • Sample Limitations:

    • Data sampling skews toward younger demographics, limiting generalizability.
    • Limited diversity in experimental VR activities, failing to fully represent real-world VR scenarios.

  • Environmental Constraints:

    • Data collection was conducted in a laboratory setting, which may not fully reflect users' experiences with notifications in real-world scenarios.

  • Future Research Directions:

    • Incorporate a wider variety of VR activities and notification modalities (visual, auditory, haptic) to study their impact on user experience.
    • Validate model prediction performance in real-world usage scenarios and explore the integration of long-term user data with personalized learning.
    • Consider additional biometric data (e.g., EEG, pulse) and computer vision features to further enhance prediction capabilities.