Understanding How Mobile Interactions Shape Grasp and Contact Patterns Beyond the Touchscreen

Paper Title

Understanding How Mobile Interactions Shape Grasp and Contact Patterns Beyond the Touchscreen

Publication Info

• Topic area: Human-computer interaction focusing on smartphone ergonomics and physical engagement.
• Keywords: smartphone interaction, hand-device contact, thermal imaging, grasp patterns, mobile ergonomics, multimodal interaction, touchscreen, back-of-device, user study, haptic design.

Background and Problem

• Problem / challenge: Limited understanding of how fingers and palms engage with smartphone surfaces beyond the touchscreen, and the methodological challenges of capturing detailed contact points due to visual occlusion and hardware constraints.
• Significance: Insights into hand-device engagement can inform ergonomic smartphone designs, improve user comfort, and enable novel multimodal interactions beyond the touchscreen.
• Motivation and related work: Previous studies have explored general grasp types, back-of-device interactions, and finger gestures but lack detailed, task-specific models of hand-device contact. Existing methods like external sensors or pressure arrays are either insufficient or overly complex, leaving gaps in understanding spatial contact patterns.

Solution

• Proposed approach: A user study combining thermal imaging and video analysis to capture and analyze hand-smartphone contact patterns across nine common interaction tasks.
• Novelty:
  1. Empirical characterization of hand grasps and contact patterns on the smartphone’s back and side edges for nine tasks.
  2. Creation of a dataset of 1023 labeled thermal images depicting hand-smartphone contact regions.
  3. Extension of thermal imaging techniques to electronic devices for capturing residual heat traces.
• Procedure and key techniques:
  • Participants performed nine interaction tasks while their grasps were recorded via multi-view cameras.
  • Heat traces left on the smartphone were captured using a thermal camera.
  • Contact maps were labeled and analyzed using manual refinement and automated segmentation tools.
  • Statistical and machine learning methods were applied to analyze contact patterns and predict tasks based on thermal data.

Results

• Concrete findings:
  • Contact regions varied significantly across tasks, with the back surface accounting for 45.9% of hand contacts on average, followed by side edges (20.9% left, 18.3% right).
  • Thermal imaging revealed distinct spatial-frequency patterns, with portrait tasks showing edge-focused contact and landscape tasks exhibiting widespread back contact.
  • Machine learning achieved modest accuracy (52.5%) in predicting tasks from contact patterns, indicating limited discriminative power of contact data alone.
• Advantage over baselines: Thermal imaging provided higher spatial accuracy and ecological validity compared to pressure or capacitive sensing, capturing detailed handprints without modifying the device surface.
• Experiments / evaluation:
  • Study involved 23 participants performing nine tasks (e.g., typing, gaming, calling) using a Google Pixel 7 smartphone.
  • Data included 1023 thermal images and multi-angle video recordings.
  • Statistical analysis and machine learning were used to evaluate contact patterns and task predictability.
• Limitations and future work:
  • Sample size was modest and skewed toward right-handed participants.
  • Use of a single smartphone model limits generalizability across devices.
  • Contact patterns alone are insufficient for reliable task prediction; future work could integrate real-time sensors and explore foldable smartphones or accessory impacts.

Summary

This study investigated how hands physically engage with smartphones beyond the touchscreen across nine interaction tasks using thermal imaging and video analysis. Findings revealed distinct grasp configurations and spatial contact patterns, highlighting opportunities for ergonomic device design and multimodal interactions. The dataset of 1023 labeled thermal images provides a resource for future research. While thermal imaging proved effective for capturing detailed contact regions, task prediction based on contact patterns alone showed limited accuracy, suggesting the need for complementary sensing methods. Results inform smartphone design, haptic feedback placement, and hardware component positioning.