Sensing What Surveys Miss: Understanding and Personalizing Proactive LLM Support by User Modeling

Paper Title

Sensing What Surveys Miss: Understanding and Personalizing Proactive LLM Support by User Modeling

Publication Info

• Topic area: Adaptive support systems for cognitive overload in survey tasks.
• Keywords: Adaptive systems, cognitive load, proactive assistance, LLMs, user modeling, electrodermal activity, mouse tracking, survey methodology, personalized support, real-time intervention.

Background and Problem

• Problem / challenge: Existing survey systems lack the ability to detect and respond to real-time cognitive states, leading to issues like satisficing, disengagement, and reduced data quality. Current approaches rely on static or reactive methods that fail to address moment-to-moment variations in user difficulty.
• Significance: Addressing cognitive overload in surveys can improve data quality, user experience, and task performance, with implications for fields like education, healthcare, and professional workflows.
• Motivation and related work: Prior work has explored static personalization, reactive support, and proactive LLM-based assistance, but these systems fail to adapt dynamically to individual cognitive states. Physiological and behavioral sensing has been used in other domains but remains underexplored in survey contexts.

Solution

• Proposed approach: An adaptive survey assistance system combining electrodermal activity (EDA) and mouse movement data to predict cognitive load and trigger personalized, real-time LLM-based clarifications.
• Novelty:
  1. Integration of multimodal sensing (EDA and mouse dynamics) for real-time cognitive state detection.
  2. Personalized classifiers with dynamic threshold adaptation to align interventions with individual user needs.
  3. Systematic comparison of adaptive timing strategies (Aligned-Adaptive, Misaligned-Adaptive, Random-Adaptive) to evaluate the impact of timing on user experience and performance.
• Procedure and key techniques:
  • Feature selection identified key predictors of cognitive load (e.g., tonic EDA levels, mouse hover time).
  • Personalized classifiers were fine-tuned using calibration data and dynamically updated through rule-based threshold adjustments.
  • The system provided LLM-based assistance when cognitive load exceeded the threshold, allowing users to select specific text for clarification.
  • A within-subjects study (N=32) compared three adaptation strategies across sequential knowledge-based survey tasks.

Results

• Concrete findings:
  • Aligned-Adaptive condition improved response accuracy from 41% (baseline) to 62%, compared to 51% (Misaligned-Adaptive) and 55% (Random-Adaptive).
  • False negative rates were lowest in the Aligned-Adaptive condition (21%) compared to Misaligned-Adaptive (44%) and Random-Adaptive (43%).
  • Participants rated the Aligned-Adaptive system highest in efficiency, dependability, and benevolence.
  • Subjective workload (NASA-TLX effort) was lowest in the Aligned-Adaptive condition.
• Advantage over baselines:
  • The Aligned-Adaptive system achieved higher accuracy, lower false negatives, and better user experience ratings compared to Misaligned-Adaptive and Random-Adaptive systems.
  • Assistance in the Aligned-Adaptive condition was more frequently accepted (67.6% acceptance rate) than in the other conditions.
• Experiments / evaluation:
  • A within-subjects design tested 32 participants on multiple-choice tasks with three adaptation strategies.
  • Metrics included task accuracy, workload (NASA-TLX), user experience (efficiency, dependability, benevolence), and system performance (confusion matrices, acceptance rates).
• Limitations and future work:
  • Limited ecological validity due to controlled lab settings and short task duration.
  • Cold-start issues in personalization during initial calibration.
  • Future work should explore field deployments, generalization across diverse item types, and alternative outcome metrics (e.g., satisficing, response consistency).

Summary

This study presents an adaptive survey assistance system that uses multimodal sensing (EDA and mouse dynamics) to provide personalized, real-time LLM-based support. The Aligned-Adaptive condition significantly improved task accuracy, reduced workload, and enhanced user experience compared to misaligned or random timing strategies. Findings highlight the importance of temporal precision in proactive assistance and suggest design implications for adaptive systems across domains. Future research should extend this approach to real-world settings and explore its application in education, healthcare, and professional workflows.