Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays

Title of the Paper

Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays

Paper Information

• Research Area: Human-Computer Interaction, Computer Science, Visual Perception
• Keywords: Multitasking, Heads-Up Computing, Computational Rationality, Deep Reinforcement Learning, Constrained Optimal Control

Research Background and Problem

• Identified Issues or Challenges: When using Optical Head-Mounted Displays (OHMD) to read content, users need to constantly switch their attention between digital information and the surrounding environment. Head movement may further reduce reading efficiency and pose safety risks while walking.
• Significance: Understanding users' attention-switching strategies is crucial for optimizing OHMD interface design to improve reading experiences and ensure walking safety.
• Motivation and Related Work:
  • Although multitasking behavior modeling has been explored in other domains such as driving, these findings cannot be directly applied to the specific context of OHMDs.
  • The transparent display of OHMDs integrates digital content with the user's physical environment, adding complexity to attention-switching research.
  • Previous studies have primarily relied on experimental data, whereas computational models have the potential to evaluate designs and optimize human-computer interfaces.

Solution

• Method and Solution: A hierarchical reinforcement learning model is proposed to model users' attention-switching behavior as a constrained sequential decision-making problem. It consists of three control levels:
  1. Supervisory Control Layer: Optimizes task priorities and determines attention allocation.
  2. Task Layer: Controls reading and environmental scanning behaviors, using working memory to track task states.
  3. Motor Control Layer: Manages eye movements to acquire information of interest and controls gait.
• Innovations:
  • Utilizes a Partially Observable Markov Decision Process (POMDP) to capture key factors of users' visual attention behavior, such as memory decay and visual perception.
  • Embeds pixel-based visual perception, enhancing simulation realism through the physics engine MuJoCo.
  • Automatically generates strategies that adapt to environmental and task changes using reinforcement learning techniques.
• Implementation Steps and Key Techniques:
  • Employs a hierarchical reinforcement learning structure, setting long and short time steps to distinguish between task-level and motor-level operations.
  • Builds a user model and environment in MuJoCo and trains agents to maximize cumulative rewards.

Research Findings

• Specific Results:
  • The model successfully captured OHMD users' multitasking behavior, including attention switching and reading resumption processes.
  • Simulation results closely matched human data, excelling in reading efficiency, walking speed, as well as reading recovery time and error rate.
• Advantages Compared to Existing Methods:
  • Unlike traditional methods that rely on manually defined behavioral rules, the proposed model generates dynamic solutions through learned strategies, adapting to different interaction scenarios.
  • Predicts realistic behavioral trends without requiring extensive human data.
• Experiments and Evaluation Results:
  • The model's performance was validated through four experiments:
    1. Users' attention-switching strategies under varying task priorities and walking speeds.
    2. The impact of walking-induced head movement on reading speed, compared with human data.
    3. Reading recovery performance across three text layouts, including time cost and error rate.
    4. Overall model behavior in complex task scenarios, including environmental attention allocation and walking speed adjustments.
  • The model accurately predicted optimal text layouts, such as potential optimization between 100px and 125px line spacing.
• Limitations and Future Directions:
  • The simulation simplified gait control and visual data processing capabilities. Future work could incorporate more detailed biomechanical models and environmental recognition capabilities.
  • Current evaluation methods focus on average metrics; future studies could include additional parameters to support fine-grained temporal behavior analysis.
  • The model could be extended to more complex real-world scenarios, such as urban street navigation or rural path walking.

Conclusion

This paper presents an innovative computational rationality model to simulate users' multitasking behavior when using OHMDs. The study demonstrates that the model not only accurately simulates users' attention-switching strategies but also provides theoretical and practical support for interface optimization, paving the way for improving the OHMD user experience.