Predicting the Noticeability of Dynamic Virtual Elements in Virtual Reality

Title of the Paper

Predicting the Noticeability of Dynamic Interface Elements in Virtual Reality

Paper Information

• Field of Study: Prediction of the noticeability of dynamic interface elements in virtual reality
• Keywords: Virtual reality, mixed reality, computational interaction, saliency prediction, user interface, dynamic elements, visual attention, animation design, noticeability, predictive models

Research Background and Issues

• Problems and Challenges:

  • In virtual reality (VR), interface elements (e.g., notifications) can be presented in any location and manner, but designing these elements to be noticeable without being overly intrusive remains a challenge.
  • Whether users can notice a dynamic element depends on their current context, such as task load and environment.
  • Current systems cannot confirm whether users have noticed a notification until they explicitly respond.

• Significance:

  • Many VR applications rely on notification elements, such as message alerts, navigation tasks, and task assistance. Ensuring notifications are noticeable without being overly disruptive is key to enhancing user experience.
  • Future "always-on" XR headsets may widely use notification designs, making alignment with user attention crucial.

• Research Motivation and Related Work:

  • Traditional studies focus on when and how to interrupt users but lack modeling of the saliency of dynamic elements in VR environments.
  • The field of visual attention modeling has developed many methods combining low-level features and context, but their application in VR environments is still in its infancy.
  • This study aims to develop a system to predict whether users will notice dynamic UI elements, reducing subjectivity in design.

Solution

• Methods and Innovations:

  • A method based on visual saliency prediction and LSTM (Long Short-Term Memory) models is proposed to compute in real-time whether users will notice dynamic UI elements.
  • Innovatively combines changes in visual saliency distribution with changes in animated regions to measure shifts in attention distribution.
  • Utilizes Earth Mover's Distance (EMD) to quantify the match between visual saliency distribution and animated regions.

• Implementation Steps:

  1. Collect video frames of the user's field of view (30fps) and generate visual saliency maps and animation masks.
  2. Compute saliency maps using the TASED-Net model, and represent dynamic change regions with animation masks generated using Gaussian blur.
  3. Evaluate differences between saliency maps and animation masks using EMD, and calculate ΔE (color changes) and animation region size as additional features.
  4. Input time-series data into the LSTM model to predict whether users will notice the animation.

Research Results

• Specific Results:

  • The proposed model can predict whether users will notice dynamic changes with an accuracy of AUC (Area Under the ROC Curve) of 0.75 and an average prediction error of 2.56 seconds.
  • Compared to simple threshold-based methods, the LSTM model demonstrated significantly higher performance.
  • The method's generalizability across different environments, tasks, and animation types was validated, including detecting new "transparency transformation" animations.

• Advantages:

  • Does not rely on direct user gaze data (e.g., eye tracking) and achieves high accuracy based solely on visual saliency prediction.
  • Highly scalable, integrating well with various virtual backgrounds, users, and animation types.
  • Supports real-time operation, with potential applications in practical VR and XR design workflows.

• Experiments and Evaluation:

  • Data collection involved 24 participants, with 12 new users for evaluation experiments.
  • Multiple rounds of validation were conducted across different tasks (e.g., video watching and text input), animation types (color, size, position, and transparency), and two models (LSTM and simple threshold-based models).

• Limitations and Future Directions:

  • The current model primarily targets queried animated elements, and further optimization is needed for unexpected animations (e.g., rare event alerts).
  • Future research is recommended to incorporate more animation types (e.g., audio or olfactory cues) and high-level semantic factors (e.g., UI layout relevance) to expand model capabilities.
  • Further refinement is needed to adapt the model to real-time interactive contexts and various psychological states of users.

Conclusion

This study presents an innovative predictive model based on changes in saliency distribution and deep learning, providing theoretical support and practical direction for designing dynamic notifications in VR environments. By predicting user attention distribution, this method opens up more precise possibilities for adaptive UI design, enabling notifications in future VR/XR environments to better balance saliency and non-intrusiveness in a personalized manner.