Glass Chirolytics: Reciprocal Compositing and Shared Gestural Control for Face-to-Face Collaborative Visualization at a Distance

Paper Title

Glass Chirolytics: Reciprocal Compositing and Shared Gestural Control for Face-to-Face Collaborative Visualization at a Distance

Publication Info

• Topic area: Augmented videoconferencing for collaborative data visualization.
• Keywords: Augmented videoconferencing, collaborative visualization, gestural interaction, remote collaboration, data analysis, synchronous communication, nonverbal cues, presence, decision-making, STEM tutoring.

Background and Problem

• Problem / challenge: Conventional videoconferencing tools often fail to incorporate nonverbal communication cues, such as hand gestures and eye contact, which are critical for effective face-to-face collaboration. Screen sharing approaches limit mutual interaction and awareness of analytical intent, particularly in paired analytical decision-making scenarios.
• Significance: Addressing these limitations could enhance remote collaboration by restoring the benefits of nonverbal communication, improving mutual understanding, and fostering more engaging and efficient analytical conversations.
• Motivation and related work: Prior work has explored augmented video for presentations, emphasizing one-way communication. However, these approaches lack support for collaborative analytical tasks, which require richer interactions with complex visualization artifacts. This paper builds on research in collaborative visualization, gesture recognition, and augmented video compositing.

Solution

• Proposed approach: Glass Chirolytics—a system that composites shared visualization artifacts and interface widgets over mirrored webcam video of conversation partners, enabling bimanual mid-air gestural interaction for collaborative analysis.
• Novelty:
  1. Reciprocal compositing of shared visual aids and mirrored video for mutual visibility and analytical intent awareness.
  2. Introduction of a bimanual gestural vocabulary tailored to analytical use cases, including novel gestures for coarse and fine selection.
  3. Demonstration of the approach across diverse application scenarios, including decision-making, exploratory analysis, tutoring, and technical interviewing.
  4. Evaluation of the system's impact on presence, workload, and collaborative dynamics compared to conventional videoconferencing.
• Procedure and key techniques:
  • Visualization artifacts are composited over grayscale mirrored webcam video of conversation partners.
  • Gestural vocabulary includes pointing, spreading, pinching, grabbing, and zooming gestures for selecting, navigating, and manipulating visualization elements.
  • Gesture recognition is achieved using MediaPipe models and custom-trained classifiers.
  • Synchronization of shared state documents and video feeds is handled via Yjs and WebRTC.

Results

• Concrete findings:
  • Glass Chirolytics significantly improves participants' sense of presence (Temple Presence Inventory scores) compared to a baseline videoconferencing application.
  • Participants reported lower temporal demand but higher physical demand with Glass Chirolytics (NASA-TLX scores).
  • Observations revealed more fluid and simultaneous collaboration with reduced verbal coordination effort in the Glass Chirolytics condition.
• Advantage over baselines:
  • Enhanced mutual awareness of analytical intent and provenance through visible gestures and nonverbal cues.
  • Reduced reliance on explicit verbal coordination and territorial strategies compared to mouse-based interfaces.
• Experiments / evaluation:
  • Study with 16 participants (8 pairs) comparing Glass Chirolytics to a baseline application in a collaborative flight itinerary decision-making task.
  • Metrics included performance, presence (TPI), workload (NASA-TLX), and qualitative feedback from interviews.
• Limitations and future work:
  • Limited to face-to-face collaboration between two participants; scalability to larger groups remains unexplored.
  • Increased physical demand and potential fatigue for prolonged use.
  • Gestural vocabulary requires further systematic evaluation for accuracy and memorability.
  • Future research opportunities include longitudinal studies, asymmetrical technology setups, and integration with speech input or 3D visualization.

Summary

Glass Chirolytics introduces an augmented videoconferencing system that combines reciprocal compositing of visualization artifacts with bimanual mid-air gestures, enabling engaging and effective face-to-face analytical conversations. The system demonstrated significant improvements in presence and collaborative dynamics compared to conventional videoconferencing, fostering mutual awareness of analytical intent and reducing temporal demand. While promising for paired analytics and STEM tutoring, scalability to larger groups and extended use scenarios requires further investigation. The approach opens new avenues for enriching remote collaboration around complex data artifacts.