Understanding Social Interactions in Reality Versus Virtuality

Research Background and Issues

What problems or challenges did the authors identify?

• Although extended reality (XR) technologies open new possibilities for social interaction, there remain significant differences between social interaction experiences in XR media and face-to-face interactions.
• Social signals in XR (e.g., facial expressions, gestures) may be unstable, semantically altered, or entirely absent.
• There is currently a lack of research methods and data that quantitatively compare social interactions in real and virtual environments.

Why is this issue important?

• Social interaction is an indispensable part of human communication, and achieving the same fluidity and authenticity in virtual environments as in real-world interactions is critical for designing meaningful and satisfying immersive experiences.
• XR is becoming an important medium for work, learning, and entertainment; understanding and addressing its limitations can promote technological advancement and improve user experiences.

Research Motivation and Related Work

• The authors were inspired by research in behavioral science and psychology, which extensively studies face-to-face interaction signals and their social significance.
• While existing work has explored social interactions in XR (e.g., collaboration, motion synchronization), there is a lack of experimental designs and data that directly compare social signals across real and virtual environments.
• Commercial XR products (e.g., Meta Horizon, Bigscreen) are widely adopted, yet these platforms neglect the reconstruction of authentic interaction signals and their impact on user experience.

Solution

What methods or solutions did the authors propose?

• This study designed an experimental method to directly compare face-to-face and XR-mediated social interactions by constructing physical and virtual "twin" environments.
• An open-source dataset containing 1.8 million rows of data was created, covering signals such as spatial positioning, gaze, gesture tracking, and audio.

What are the innovative aspects of this solution?

1. Novel Experimental Design:

   • Conducted identical social tasks in both physical and virtual environments to ensure consistent experimental conditions.
   • Used instrumented devices (e.g., Project Aria and Meta Quest Pro) to capture participants' behavioral data in both contexts.

2. Data Analysis Methods:

   • Developed a set of analysis tools based on social signals to quantify direct comparisons between real and virtual interactions.
   • Tested metrics such as "social synchrony" (e.g., head movement synchronization) and "social signal stability," demonstrating differences in interaction outcomes.

3. Open Data Accessibility:

   • Provided baseline data and an accompanying analysis framework for cross-reality and virtual environment research for the first time.

What are the implementation steps and key technologies used?

1. Experimental Design:

   • 36 participants were divided into 6 groups, each completing four social tasks (e.g., dyadic conversation, arrangement tasks, guessing games) in both physical labs and their virtual twin environments.
   • A Latin square design was used to balance task order, while all signals (including head position, gesture direction, gaze data, and voice data) were collected.

2. Data Collection Platforms:

   • Physical experiments used Project Aria devices for precise spatial and hand tracking.
   • Virtual experiments employed Meta Quest Pro headsets equipped with facial, hand, and gaze tracking capabilities.

3. Data Processing and Analysis:

   • Integrated data from different experimental conditions using a unified framework and timeline.
   • Calculated metrics such as social synchrony, equality index, and conversational balance.
   • Conducted qualitative coding of post-experiment interviews to complement quantitative data.

Research Outcomes

What specific results were achieved?

1. Quantitative Results:

   • "Spatial positioning" in social signals was consistently stable across real and virtual environments, aligning with traditional theoretical assumptions.
   • "Social synchrony" in XR environments was significantly lower than in face-to-face interactions, making it harder to establish social connections and intimacy.
   • The proportion of "silence" during conversations increased significantly under XR conditions, indicating reduced interaction fluidity.

2. Qualitative Observations:

   • Participants reported insufficient facial expressions and subtle social cues (e.g., eye contact) in XR environments, hindering task completion.
   • Despite quantitative results showing consistent spatial usage, participants subjectively felt that spatial perception in XR was limited, revealing an "experience gap."

3. Open Data and Tools:

   • Provided an open, high-quality social interaction dataset and analysis tools, establishing a benchmark for future research.

How does it compare to existing solutions?

• The authors achieved the first direct comparison between real and virtual environments, enabling more precise measurement of social signals.
• Open data and standardization set a precedent for future multi-user XR experiments and foster collaborative research in the XR domain.

What are the experimental or evaluation results?

• Quantitative data demonstrated consistency in certain social signals (e.g., spatial distance) across real and virtual environments while confirming deficiencies in XR social synchrony and conversational fluidity.
• Interaction challenges in XR (e.g., facial tracking, gesture response delays) were identified as primary causes of poor user experience.

Limitations and Future Directions

1. Limitations:

   • Data was primarily collected from participants in a shared physical location; fully distributed virtual interactions may involve additional critical social signals.
   • Current XR technologies (e.g., facial tracking resolution) may limit the applicability of experimental results.

2. Future Directions:

   • Enhance data analysis depth by incorporating more complex social signals, such as conversational rhythm and emotional expression.
   • Explore technological interventions (e.g., improved virtual avatar facial algorithms) to narrow the "experience gap" between XR and real-world interactions.
   • Extend experiments to fully distributed XR interaction scenarios and investigate the potential impact of personalized virtual avatars.

Conclusion

This paper takes a significant step toward understanding the iteration of social interactions in real and virtual environments. Through precise experimental design and data analysis, the authors reveal the stability, shortcomings, and potential improvements of social signals in both contexts. The open-source data and standardized framework establish a new benchmark for theoretical and applied research while providing critical insights for enhancing future social XR experiences.