Chartist: Task-driven Eye Movement Control for Chart Reading

Research Background and Problem

• Identified Issues or Challenges: The authors point out that current scanpath simulation models for information visualization typically only capture low-level attention patterns associated with free browsing and fail to effectively characterize task-driven gaze paths. These task-driven gaze paths reflect how users employ visual attention to reason while performing analytical tasks. Additionally, collecting real user gaze trajectory data (via eye trackers) is both expensive and time-consuming, necessitating the use of simulation methods.
• Significance: Understanding task-driven gaze behavior is crucial for optimizing data visualization design, evaluation, and enhancing the interpretability of human-computer interaction. By predicting users' gaze paths under specific tasks, traditional user study costs can be reduced, and interface design can be improved.
• Research Motivation and Related Work: Previous research has primarily focused on attention models for free browsing but has not sufficiently explored task-driven gaze behavior. Moreover, compared to static attention heatmaps, task-based gaze paths need to capture both temporal information and personalized behavior, providing dynamic gaze sequences.

Solution

• Proposed Method and Solution: This study introduces a computational model named "Chartist" to simulate users' gaze paths when completing specific chart analysis tasks (e.g., value retrieval, filtering, and finding extrema). The core innovation lies in a two-tier hierarchical control architecture: a high-level cognitive controller responsible for task understanding and decision-making, and a low-level gaze controller for precise pixel-level gaze operations.
• Innovations of the Method:
  1. Chartist is the first to extend task-driven gaze prediction to the domain of chart analysis, rather than traditional free browsing or target search tasks.
  2. The high-level controller leverages large language models (LLMs) for semantic task analysis and subtask scheduling based on memory management, while the low-level controller uses deep reinforcement learning (RL) to simulate detailed pixel-level gaze paths.
  3. The model adopts the principle of Computational Rationality, reducing reliance on human eye-tracking data and enabling training directly in synthetic environments.
• Implementation Steps and Key Techniques:
  1. Task and Environment Generation: Utilize real and synthetic chart data, annotate areas of interest (AOIs), and generate task descriptions (value retrieval, filtering, finding extrema) based on rules.
  2. Hierarchical Modeling:
     • The high-level cognitive controller is modeled using a POMDP framework, employing LLMs for task decision-making and working memory management.
     • The low-level gaze controller predicts specific gaze locations using deep learning strategies, optimizing gaze costs based on tasks and visual constraints.
  3. Training and Testing: The high-level controller leverages pre-trained language models (e.g., GPT-4o), while the low-level controller optimizes strategies through reinforcement learning (PPO). After training, the two components jointly predict gaze paths.

Research Outcomes

• Specific Outcomes:
  1. Chartist can generate task-driven gaze paths similar to those of humans based on an input chart and task description.
  2. The model demonstrates high-fidelity predictive capabilities across tasks in task-driven scenarios.
• Advantages Over Existing Solutions:
  1. Chartist significantly outperforms baseline models (e.g., UMSS and Free-Viewing models) in task-based gaze path prediction, particularly excelling in semantic sequence alignment and task sensitivity.
  2. The model's predicted scanpaths not only approximate human users but also generate statistical summaries of human behavior (e.g., gaze dwell time, number of region transitions).
• Experimental or Evaluation Results:
  1. Across multiple quantitative metrics (e.g., Dynamic Time Warping [DTW], Levenshtein Distance [LEV], and Sequence Score), Chartist consistently aligns most closely with human performance in various tasks.
  2. The model's predicted gaze paths fall entirely within one standard deviation of human performance for multiple statistical characteristics, particularly excelling in capturing gaze proportions on task-relevant areas.
  3. Through qualitative analysis, Chartist better captures user gaze strategies for specific tasks (e.g., filtering data points), whereas most competing models (e.g., VQA and DeepGaze iii) frequently focus on task-irrelevant regions.
• Limitations and Future Directions:
  1. Limitations:
     • The model faces technical compatibility issues with complex multimodal information processing (e.g., maps, network diagrams).
     • It struggles to accurately predict gaze paths in situations involving highly dense data point charts or advanced logical reasoning.
  2. Future Directions:
     • Extend the model to a wider range of complex visualization types (e.g., networks, maps) and incorporate necessary domain-specific prior knowledge.
     • Integrate multimodal learning (e.g., visual-language models combining text and images) to enhance question-answering capabilities for complex tasks.
     • Enhance the model's cognitive mimicry of numerical computation and data relationship exploration to support richer and more complex analytical tasks.

Conclusion

Chartist represents a significant step forward in task-driven gaze path simulation. Its innovative hierarchical control architecture and task alignment capabilities provide powerful tools for evaluating and optimizing information visualization as well as explainable artificial intelligence. Future work will need to further expand its applicability, particularly in complex charts and reasoning tasks.