Perceived Asynchrony of Rhythmic Stimuli Affects Pupil Diameter and Smooth Pursuit Eye Movements

Research Background and Issues

• What problems or challenges did the authors identify?
  This study focuses on the issue of audiovisual asynchrony caused by latency in network interactions, particularly the perceived synchrony of rhythmic audiovisual stimuli. Latency can disrupt audiovisual synchrony, such as inconsistencies in the stimulus onset asynchrony (SOA) between audio and video, thereby affecting user experience.

• Why is this issue important?
  Synchrony is a key factor in fostering trust, intimacy, and group cohesion in interpersonal interactions. In many application scenarios that rely on rhythmic interaction, such as remote music collaboration, virtual gaming, and online fitness classes, latency issues negatively impact the experience. Understanding how latency affects perceived synchrony and designing methods to mitigate these negative effects are crucial for improving related human-computer interaction (HCI) systems.

• Research Motivation and Related Work
  Previous studies have shown that sensitivity to perceived latency varies depending on the task and stimulus type. However, existing research has largely focused on non-rhythmic audiovisual stimuli, leaving the impact of rhythmic audiovisual stimuli on perceived synchrony underexplored. Furthermore, while eye movement data (e.g., pupil diameter and smooth pursuit eye movements) are considered potentially related to cognitive load and perceived synchrony, there remains a significant research gap in this area.

Solutions

• What methods or solutions did the authors propose?
  The authors conducted experiments manipulating different rhythmic frequencies and SOA of audiovisual events to study their effects on subjective synchrony judgments (Point of Subjective Synchrony, PSS) and the synchrony window (Window of Subjective Synchrony, WSS). They also recorded and analyzed eye movement data (pupil diameter and smooth pursuit eye movements) as potential physiological indicators.

• What are the innovative aspects of this solution?

  • Systematic exploration of the effects of rhythmic frequency on perceived synchrony and eye movement responses for the first time.
  • Investigation of pupil dilation and smooth pursuit eye movements as potential physiological indicators of perceived synchrony.
  • Introduction of a broader SOA range to simulate various latency scenarios common in network environments.

• What are the implementation steps and key technologies used?

  1. Experimental Design: Based on 11 SOA conditions (-250ms to +250ms), three frequency conditions (0.5 Hz, 1 Hz, and 2 Hz), and two visual composition conditions (constant speed and constant distance).
  2. Participant Tasks: Participants followed visual markers and judged the synchrony of audiovisual events using a four-point rating scale.
  3. Data Collection and Processing: Tobii Pro Nano eye tracker was used to record pupil dilation and eye movement data, followed by baseline correction and outlier processing.
  4. Statistical Analysis: Linear mixed models (LMM) were employed to analyze the effects of SOA, frequency, and other factors on subjective ratings and eye movement data.

Research Findings

• What specific findings were obtained?

  1. Effect of Frequency on Perceived Synchrony:
     • PSS was significantly influenced by frequency, with higher frequencies making audio slightly lagging behind video more likely to be perceived as synchronous.
     • WSS was not significantly affected by frequency, indicating that the time window for synchrony perception remains relatively stable.
  2. Relationship Between Eye Movements and Perceived Synchrony:
     • As SOA increased (greater delay), pupil diameter significantly increased, indicating higher cognitive load.
     • Pupil dilation was sensitive to frequency changes, with the effect diminishing at higher frequencies.
     • Smooth pursuit eye movements (SPEM) exhibited significant lag under higher frequencies and larger SOA conditions.

• What advantages does it have compared to existing solutions?

  1. Proposed the potential use of eye movement data (pupil diameter and SPEM) as indicators of perceived synchrony.
  2. Provided design suggestions for compensating audio lag in asynchronous audiovisual scenarios, particularly optimizing for specific frequency contexts (e.g., remote music collaboration or virtual dance).

• What were the experimental or evaluation results?

  • The data indicated that negative SOA (audio arriving first) is more likely to be perceived as asynchronous compared to positive SOA (video arriving first).
  • The rate of pupil diameter increase was 0.0001724 mm per millisecond, demonstrating a quantifiable effect of SOA on pupil dilation.
  • All quality assessments (e.g., AIC, BIC) showed that the model had good fit in explaining variations in the dependent variables.

• Limitations and Future Directions:

  • Limitations:
    1. High brightness in the laboratory environment may have influenced pupil dilation data.
    2. The experimental design involved monotonous tasks with limited user interaction dimensions, failing to fully simulate real-world usage scenarios.
  • Future Directions:
    1. Validate findings in more complex user interaction environments (e.g., virtual reality or remote collaboration applications).
    2. Investigate differences in synchrony perception across different participant groups.
    3. Explore the effects of more complex rhythm types (e.g., variable beats).

Through this study, the authors conducted an in-depth analysis of the roles of SOA and frequency in rhythmic audiovisual synchrony, validated the potential of eye movement data as a measure of synchrony, and proposed design recommendations for enhancing user experience in specific HCI applications. This lays the groundwork for future research and technological optimization.