Touch, Gesture, and Conversation: A Case Study in the Choreography of Interaction with a Data Physicalization

Paper Title

Touch, Gesture, and Conversation: A Case Study in the Choreography of Interaction with a Data Physicalization

Publication Info

• Topic area: Interaction with data physicalizations in social and physical environments.
• Keywords: Data physicalization, embodied cognition, touch interaction, gesture studies, sensemaking, interaction analysis, human-computer interaction, embodied interaction, data visualization, social interaction.

Background and Problem

• Problem / challenge: Limited research exists on the detailed, moment-by-moment interactions—touches, gestures, and verbal dialogue—during sensemaking with data physicalizations. Prior work has focused on where people touch and how tasks influence these interactions but lacks fine-grained analysis of their holistic embodied experience.
• Significance: Understanding these interactions can inform the design of data physicalizations to better support communication, problem-solving, and embodied sensemaking in shared social spaces.
• Motivation and related work: Previous studies have explored touch and gesture in data physicalizations and other contexts, but gaps remain in understanding how these behaviors integrate with conversation and cognition. This paper builds on embodied cognition theory and gesture studies to address these gaps.

Solution

• Proposed approach: A detailed case study analyzing one participant’s interactions with a static data physicalization, focusing on touches, gestures, and conversation during three tasks.
• Novelty:
  1. Provides a fine-grained analysis of individual touches and gestures in the context of a sensemaking process.
  2. Introduces a categorization of touches and gestures into communicative and reflective types, with subcategories.
  3. Explores the choreography of interactions, including repetition, timing, and phrasing, to understand their role in sensemaking.
  4. Highlights the interplay between touches and gestures, emphasizing the need to study both in data physicalization research.
• Procedure and key techniques:
  • Selected one participant from a prior study for in-depth analysis.
  • Used video coding to document and categorize 383 touches and 125 gestures across three tasks.
  • Developed a detailed timeline of interactions, aligning touches, gestures, and conversation.
  • Analyzed patterns such as repetition, timing, and spatial distribution to understand the choreography of interactions.

Results

• Concrete findings:
  • The participant made 383 touches and 125 gestures across three tasks, with task-specific variations in rates (e.g., 20.9 touches/min in Task 1 vs. 7.5 touches/min in Task 3).
  • 36% of touches and 16% of gestures were reflective (orientation and thinking), not directly tied to communication or information gathering.
  • Repetition occurred in three forms: repeated touches on the same point, the same type of feature, and the same pattern of touches across features.
  • Touches and gestures often overlapped or transitioned, highlighting a continuum of movement.
• Advantage over baselines:
  • Provides a more detailed and nuanced understanding of touch and gesture behaviors compared to prior studies, which often focused on aggregate or static measures.
  • Identifies reflective interactions as a significant component, expanding the scope of what is considered meaningful interaction with data physicalizations.
• Experiments / evaluation:
  • Tasks included exploring and describing the model, hypothesizing about its data encoding, and interpreting the revealed data.
  • Data collection included video analysis, touch pattern imaging, and conversation transcripts.
  • Coding categorized interactions into communicative (feature-based, expressive) and reflective (information gathering, orientation and thinking) types.
• Limitations and future work:
  • Results are based on a single participant and may not generalize to others.
  • The study did not capture gaze or facial expressions, which could provide additional context.
  • Future work should analyze multiple participants, explore dynamic data physicalizations, and investigate the impact of design choices on interaction behaviors.

Summary

This case study provides a detailed analysis of how one participant interacted with a static data physicalization through touches, gestures, and conversation. It categorizes interactions into communicative and reflective types, highlighting the significant role of reflective behaviors in sensemaking. The study also demonstrates the interplay between touches and gestures, emphasizing the need to consider both in data physicalization research. These findings contribute to understanding the embodied and social aspects of interacting with data physicalizations and offer insights for designing more effective and engaging data representations.