Touch with Meaning: A Contextual Analysis of Social Touch

Paper Title

Touch with Meaning: A Contextual Analysis of Social Touch

Publication Info

• Topic area: Contextual and cultural analysis of social touch in human-computer interaction (HCI).
• Keywords: Social touch, contextual analysis, HCI, computer vision, cultural norms, touch taxonomy, mediated communication, relational dynamics, gesture recognition, touch datasets.

Background and Problem

• Problem / challenge: Existing HCI research often treats touch in isolation, ignoring the relational, cultural, and situational factors that shape its meaning. Current datasets are small, staged, and lack contextual depth.
• Significance: Understanding the nuanced meanings of social touch is critical for designing socially aware technologies, such as avatars, social agents, and mediated communication systems.
• Motivation and related work: Prior studies have explored affective touch and haptic technologies but fail to incorporate the contextual subtleties of real-world interactions. This paper addresses the gap by analyzing how touch acquires meaning through relational, cultural, and environmental contexts.

Solution

• Proposed approach: A large-scale contextual analysis of social touch using a dataset of 5,016 annotated touch events extracted from YouTube videos, analyzed through a semi-automated computer vision pipeline.
• Novelty:
  1. Introduction of the largest publicly accessible contextual dataset of social touch, annotated with 46 features.
  2. Development of a taxonomy linking gestures, body locations, relational dynamics, and situational settings to social touch meanings.
  3. Design implications for socially aware technologies that move beyond gesture recognition to meaning recognition.
• Procedure and key techniques:
  • Retrieval of 96,637 culturally diverse YouTube videos, filtered to 5,016 validated social touch events.
  • Use of computer vision (YOLOv8, OpenPose) and AI-assisted annotation (Gemini) to detect and annotate touch events.
  • Human validation of annotations to ensure reliability.
  • Development of a taxonomy categorizing social touch meanings into ten core categories, such as affection, comfort, and dominance.

Results

• Concrete findings:
  • Identical gestures (e.g., a pat on the shoulder) can convey different meanings (e.g., comfort vs. dominance) depending on context.
  • Similar intentions (e.g., encouragement) can be expressed through different gestures (e.g., back pats, hand clasps).
  • Cultural differences: U.S. contexts favor formal gestures like handshakes, while Korean contexts involve more intimate arm and shoulder contact.
  • Public settings constrain touch to brief, formal gestures, while private settings allow for more intimate expressions.
• Advantage over baselines:
  • Dataset size (5,016 validated events) and contextual richness surpass prior datasets like CoST, SATED, and VTD.
  • Captures naturalistic, in-the-wild interactions rather than staged or lab-controlled scenarios.
• Experiments / evaluation:
  • Chi-square and Kruskal-Wallis tests reveal significant contextual variability in touch meanings, durations, and body locations.
  • Human validation achieved high inter-annotator reliability (Cohen’s κ = 0.75).
• Limitations and future work:
  • Limited cultural scope (U.S. and Korea); broader cultural comparisons are needed.
  • Dataset prioritizes breadth over granularity, potentially missing micro-patterns in specific gestures.
  • Future work includes expanding cultural contexts, computational modeling for real-time touch interpretation, and integrating findings into interactive systems.

Summary

This paper presents a large-scale, contextual analysis of social touch, introducing a dataset of 5,016 annotated touch events and a taxonomy of ten core meanings. Using computer vision and AI-assisted annotation, the study reveals how relational, cultural, and situational contexts shape the interpretation of touch. Findings highlight the variability of touch meanings, cultural differences, and the importance of embedding context in mediated touch systems. The work provides actionable design implications for haptic technologies, avatars, and social robots, emphasizing the need for meaning-driven and context-aware touch rendering.