SocialEyes: Scaling Mobile Eye-tracking to Multi-person Social Settings

Research Background and Issues

• Identified Problems or Challenges:

  • Traditional eye-tracking research is mostly confined to single-user laboratory settings, lacking application in real-world, multi-user social environments. Although advancements in mobile eye-tracking technology have improved data collection in natural settings, synchronizing and analyzing data across multiple observers remains challenging.
  • In multi-user scenarios, eye-tracking data needs to be aligned from the wearer's subjective coordinate system to a shared spatial coordinate system to analyze collective gaze dynamics.
  • Existing techniques rely on manual analysis or coarse region mapping, which are inefficient and difficult to scale for large datasets.

• Significance:

  • Eye-tracking data can reveal the dynamics of human attention allocation, social behavior, and non-verbal communication, advancing research in shared gaze scenarios such as music performances and classroom teaching.
  • The ability to conduct synchronized multi-user eye-tracking in natural social settings can provide new insights for fields such as cognitive neuroscience, human-computer interaction, and team collaboration.

• Research Motivation and Related Work:

  • Previous studies have primarily focused on small groups (e.g., dyads) and used static scenes as substitutes for real, complex dynamic environments, resulting in limited ecological validity.
  • Emerging technologies, such as mobile eye-tracking devices, have improved the accuracy of data collection in natural settings, but there is currently no dedicated tool for synchronized analysis in dynamic, multi-user real-world scenarios.

Solution

• Proposed Method or Solution: The authors propose a novel software framework—SocialEyes—designed to enable multi-device data synchronization, non-intrusive data monitoring, semantic gaze mapping, and shared gaze analysis.

  • Enables simultaneous recording and analysis of data from multiple mobile eye-tracking devices.
  • Introduces a gaze mapping method based on homography projection, aligning gaze from the wearer's perspective to a shared scene perspective.
  • Develops various visualization and analysis tools to interpret and present social gaze dynamics among multiple individuals.

• Innovations:

  1. Cross-Perspective Mapping: Utilizes a homography-based projection method to accurately align individual gaze from subjective perspectives to a shared perspective.
  2. Time Synchronization: Ensures microsecond-level synchronization across multiple devices using the Network Time Protocol (NTP) and robust linear drift correction.
  3. Real-Time and Offline Data Modes: Implements flexible operational modes to accommodate offline complex analysis or real-time interaction needs.
  4. Group Gaze Analysis Metrics: Introduces new metrics, such as normalized contour area and gaze heatmap similarity, to quantify collective gaze distribution and social behavior similarity.

• Implementation Steps and Key Technologies:

  • Modular Data Flow: The framework processes gaze coordinates, perspective videos, and central scene video streams from eye-tracking devices.
  • Homography Projection: Employs feature point matching (e.g., SuperGlue neural network) and projective transformation mathematics to achieve cross-perspective projection.
  • Real-Time Monitoring and Storage: Modules enable real-time monitoring of hardware status, transmission errors, and device performance.
  • Analysis and Visualization: Uses multimodal displays such as heatmaps and time-series plots to showcase collective gaze behaviors.

Research Outcomes

• Specific Achievements:

  1. Successfully developed and validated the multi-device eye-tracking framework SocialEyes.
  2. Conducted experiments recording real-time gaze behavior from 30 simultaneous participants and applied the framework to two public events (a concert and a movie screening), significantly expanding the scale of research scenarios.
  3. Demonstrated the accuracy of multi-user gaze point mapping and time synchronization in dynamic scenes.

• Comparison with Existing Solutions and Advantages:

  • Significantly reduces the manual analysis and synchronization workload for researchers, enhancing the automation of data processing.
  • Proposes a novel multi-perspective gaze alignment method, supporting real-time performance in dynamic, complex scenarios.
  • Enables the study of collaborative gaze behavior among multiple individuals, which is difficult to achieve in traditional laboratory studies.

• Experimental and Evaluation Results:

  • Time Synchronization Performance: Achieved millisecond-level time deviations across 30 devices (average 19.58–45.57ms), successfully correcting offsets and drifts.
  • Projection Accuracy: Accurately translated individual gaze distributions to a shared perspective in dynamic scenes using the homography projection method.
  • Group Gaze Behavior Analysis: Analyses revealed that dynamic changes in gaze heatmaps and normalized contour areas effectively captured participants' responses to scene changes and highlighted attention distribution in social interactions.

• Limitations and Future Directions:

  1. Projection Method Limitations: The current homography projection assumption may fail in close-range or non-planar scenarios; future work could explore solutions based on 3D reconstruction or multi-perspective calibration.
  2. Hardware Dependency: Framework performance is closely tied to device hardware capabilities; adopting more open-source solutions is recommended to improve adaptability and scalability.
  3. Real-Time Mode Optimization: Future efforts should optimize streaming data decoding and runtime efficiency in multi-user, dynamic scenarios.
  4. Cross-Environment Expansion: Extend applicability to more social scenarios (e.g., multi-user collaborative work environments) and support integration with various sensors (e.g., physiological sensors) to enhance multimodal analysis capabilities.

Conclusion

The SocialEyes framework achieves ecological, multi-user eye-tracking research through technological innovation, offering groundbreaking advancements in understanding collective attention dynamics. It provides practical applications for fields such as human-computer interaction, education, and cultural studies. Additionally, the authors have made the code publicly available to promote widespread replication and extension within the academic community.