Detection and Differentiation of Obstacles in Repeated Adaptive Human-Computer Interactions from Brain Activity and User Behavior

Document Title

Multimodal Differentiation of Obstacles in Repeated Adaptive Human-Computer Interactions

Document Information

• Subject Area: Adaptive Human-Computer Interaction, Multimodal Data Fusion, User Interface Adaptation
• Keywords: HCI obstacles, multimodal detection, Bayesian fusion, adaptive systems, EEG, memory load, visual impairment, user experience, behavior modeling

Research Background and Problem Statement

• Identified Issues and Challenges:

  1. In Human-Computer Interaction (HCI), users may experience performance and experience degradation due to interaction obstacles such as memory load or visual impairments.
  2. Current methods for detecting interaction obstacles often focus on a single obstacle type and rarely consider changes in obstacles during continuous interactions.
  3. Incorrect user interface (UI) adaptation can have adverse effects on users, especially when obstacle types are misidentified.
  4. Real-time, multimodal detection of obstacles and matching adaptive interfaces remain unresolved challenges.

• Significance:

  1. Enhancing the naturalness of HCI and providing personalized user experiences.
  2. Delivering the most suitable UI for users in various scenarios to mitigate the negative impact of obstacles.
  3. Supporting sequential user interactions to adapt to users' dynamic needs.

• Research Motivation and Related Work:

  1. Existing systems are primarily adaptive to single obstacle types and lack the capability for unified decision-making across different obstacles.
  2. Combining multimodal sensors (e.g., EEG, behavioral interaction data) with machine learning can more accurately identify obstacles and adapt UIs.
  3. Using Bayesian methods to aggregate multimodal data in continuous interactions can stabilize obstacle detection and improve UI adaptability.

Solution

• Proposed Method:

  1. Integrate multiple data modalities (behavioral data and EEG data) to detect memory load and visual obstacles.
  2. Develop a Dynamic Bayesian Network (DBN) to enable real-time detection, fusion, and adaptive decision-making for different obstacles.
  3. Update UI adaptations based on the type of obstacle identified in each interaction session.

• Innovations:

  1. Developed a multimodal detector for user obstacles by combining behavioral and EEG data.
  2. Proposed a continuous Bayesian-based adaptive method to incrementally optimize obstacle detection and corresponding UI adaptation across multiple interactions.
  3. The model can correct errors introduced by initial adaptations, improving long-term user experience.

• Implementation Steps and Key Techniques:

  1. Data Collection:
     • Simulated real-world interaction obstacles using paired memory games (e.g., simulating red-green color blindness through similar colors and simulating memory load through mental calculations).
     • Recorded behavioral data (user click patterns) and EEG (brain activity signals).
  2. Interaction Obstacle Detector:
     • Used behavior modeling (based on LSTM networks) to detect visual obstacles.
     • Used an EEG classifier (based on Support Vector Machines, SVM) to detect memory load obstacles.
  3. Bayesian Fusion:
     • Integrated outputs from multimodal detectors into a Dynamic Bayesian Network to iteratively update obstacle types and optimal UI adaptation decisions.
  4. Experimental Evaluation:
     • Validated the model's multi-round adaptation performance using participants' continuous interaction data.

Research Findings

• Specific Results:

  1. Obstacle classification accuracy was 72.5% during the first interaction, improving to 98% across multiple interactions.
  2. Demonstrated the complementary nature of behavioral data and EEG signals: behavioral data is more suitable for visual obstacles, while EEG is better for memory load obstacles.
  3. The proposed Dynamic Bayesian Network achieved accurate UI adaptation under different obstacle types and corrected erroneous decisions in subsequent interactions.

• Advantages:

  1. Enhanced Adaptability: Significantly improved the stability and efficiency of adaptive interaction systems by integrating multimodal data.
  2. Real-Time Capability: The system design supports real-time obstacle detection and adaptation updates.
  3. High Generalizability: The proposed framework can be extended to more obstacle types and data modalities.

• Experimental Evaluation Results:

  1. Accuracy: Behavioral and EEG standalone detectors achieved accuracy rates of 82.4% and over 84%, respectively, in their corresponding scenarios.
  2. UI adaptation significantly improved subjective user ratings of game experience and task completion efficiency under different obstacle conditions.
  3. Lack of adaptation or incorrect adaptation significantly reduced users' task efficiency and satisfaction.

• Limitations and Future Directions:

  1. This study simulated obstacle conditions; future work should validate the system's applicability in real-world scenarios and with a broader user population.
  2. Consider expanding to more modalities (e.g., speech, haptic feedback) to enhance the robustness of multimodal detection.
  3. Optimize real-time processing capabilities and user experience evaluation methods for more complex cognitive and behavioral states in dynamic environments.