Multi-modal Multi-scale Attention Guidance in Cyber-Physical Environments

Document Title

Multi-modal Multi-scale Attention Guidance in Cyber-Physical Environments

Document Information

• Subject Area: Information and Computational Sciences, specifically Human-Computer Interaction and Attention Guidance in Cyber-Physical Environments
• Keywords: Attention Guidance, Cyber-Physical Environments, Smart Environments, Multi-modal, Multi-scale

Research Background and Problem

• Problem or Challenges:

  • Cyber-Physical Environments (CPE) require dynamic perception and response to environmental changes, involving how to effectively guide users' attention to critical targets or areas while avoiding distractions.
  • Current CPEs lack direct communication methods with humans, such as "pointing" to an object to indicate a focus of attention.
  • The coordination between different output devices to optimize attention guidance performance remains unresolved.

• Significance:

  • Attention guidance can enhance user interaction experiences in complex and dynamic environments, applicable to scenarios like navigation, object searching, and safety alerts.
  • Effective attention guidance can reduce users' cognitive load and improve task completion efficiency.

• Research Motivation and Related Work:

  • The selective process of attention is influenced by two main factors: top-down goal-directed processing and bottom-up stimulus-driven processing.
  • Existing methods have demonstrated that multi-modal cues can enhance attention guidance effectiveness; however, adaptive methods for dynamic multi-scale coordination across devices are lacking.
  • Current approaches, such as Subtle Gaze Direction, project navigation systems, and visual highlighting systems in virtual environments, have shown initial success but are insufficient for application in complex real-world environments.

Solution

• Method or Solution:

  • Propose a multi-modal (visual and auditory), multi-scale attention guidance method aimed at selecting the most suitable device in real-time based on environmental information to guide user attention.
  • Develop a general adaptive function—“Suitability Function”—to evaluate the effectiveness of output devices by calculating their "salience" and "closeness."

• Innovations:

  • Introduced a comprehensive evaluation function based on salience and closeness, allowing dynamic adjustment of weights to balance their importance in dynamic environments.
  • Supports cross-modal information guidance through visual and auditory channels, considering users' dynamic positions and the multi-device characteristics of the environment.
  • Utilizes a "multi-scale transition" mechanism to address visual disruptions caused by device switching.

• Implementation Steps and Key Techniques:

  1. Salience Function:
     • Predicts a device's ability to attract user attention through visual characteristics such as brightness and size, and auditory characteristics such as volume.
     • Adjusts for users' visual field limitations for visual devices and sound impact for auditory devices, with appropriate range normalization.
  2. Closeness Function:
     • Evaluates a device's guidance performance using accuracy and precision metrics.
     • Dynamically adjusts the function based on the distance between the user and the target, ensuring looser guidance for long distances and higher precision requirements for short distances.
  3. Multi-scale Transition Mechanism:
     • Introduces a transition "threshold" parameter (margin λ) to avoid frequent device switching and enhance system stability.
     • Adds a "momentum weight" for devices, prioritizing the maintenance of the current device's guidance in the short term.

Research Outcomes

• Specific Outcomes:

  • Designed experiments in a virtual reality supermarket environment to validate the effectiveness of the proposed method.
  • Experiments showed that in "visual + auditory" multi-modal guidance, the introduction of a transition threshold significantly reduced the distance users traveled to complete tasks.
  • Without the threshold, the system exhibited greater performance fluctuations, and the advantages of multi-modal guidance were less apparent.

• Advantages Over Existing Solutions:

  • The combination of multi-modal approaches significantly improved attention guidance performance, particularly in terms of adaptability and stability in dynamic environments, compared to traditional single-modal solutions.
  • Highlighted system flexibility: function parameters can be optimized for specific environments, adapting to various CPE scenarios.

• Experimental or Evaluation Results:

  • Under mixed-modal guidance, the average travel distance was significantly reduced (M=33.12, SD=3.38), while visual single-modal guidance performed worse (M=39.02, SD=9.79).
  • Device switching frequency was significantly higher without a transition threshold (M=25.16) compared to the scenario with a threshold (M=14.05).
  • User feedback supported the multi-modal approach, with auditory cues effectively complementing the limitations of visual guidance.

• Limitations and Future Directions:

  • Limitations:
    • Experiments were conducted in virtual environments; real-world applications must account for sensor errors and output device availability.
    • Certain parameters (e.g., interaction weights) did not show significant effects, potentially due to experimental design constraints.
  • Future Work:
    • Extend system testing to real-world scenarios and address hardware adaptation challenges.
    • Optimize function design by incorporating more accurate salience and closeness computation models.
    • Explore additional output devices (e.g., haptic feedback) and their potential to enhance attention guidance performance.
    • Develop multi-user attention guidance methods to support collaborative interactions.