Somatic Drawing Tool: 3D Body Sheet as Material for Articulating Synaesthetic Experiences

Paper Title

Somatic Drawing Tool: 3D Body Sheet as Material for Articulating Synaesthetic Experiences

Publication Info

• Topic area: Interactive technologies for expressing synaesthetic experiences
• Keywords: Synaesthesia, VR, 3D Body Sheet, sensory articulation, participatory design, soma design, multisensory interaction, cross-modal HCI, immersive technology, embodiment

Background and Problem

• Problem / challenge: Synaesthetes face challenges in articulating and sharing their experiences due to the limitations of language and traditional sensory vocabularies. Existing HCI tools often treat synaesthesia metaphorically rather than addressing the lived experiences of synaesthetes.
• Significance: Understanding and communicating synaesthetic experiences can foster self-awareness, reduce isolation, and enable inclusive design practices for diverse sensory experiences.
• Motivation and related work: While HCI has explored multisensory and cross-modal interaction, few studies explicitly design with and for synaesthetes. Prior work highlights the difficulties in articulating synaesthetic experiences and the potential of VR and soma design for sensory expression. This paper builds on these insights to develop tools that support the nuanced articulation of synaesthesia.

Solution

• Proposed approach: The Somatic Drawing Tool, a VR system incorporating a 3D Body Sheet, was developed to enable synaesthetes to externalise and communicate their sensory experiences through embodied and immersive interaction.
• Novelty:
  1. Introduction of a 3D Body Sheet for mapping cross-sensory experiences in VR.
  2. Iterative co-design process with synaesthetes to personalise the tool for diverse sensory profiles.
  3. Integration of multimodal and embodied methods for articulating synaesthetic experiences.
  4. Expansion of soma design practices to include cross-sensory and multisensory articulation.
• Procedure and key techniques:
  • Conducted participatory workshops with nine synaesthetes using verbal, non-verbal, and VR-based methods.
  • Developed the Somatic Drawing Tool by customising Open Brush, incorporating bespoke brushes, 3D body mapping, and environmental controls.
  • Iteratively refined the tool based on feedback from five case-study participants across two sessions each.

Results

• Concrete findings:
  • VR enabled participants to articulate spatial, dynamic, and embodied aspects of their synaesthetic experiences that were difficult to express in 2D or verbal forms.
  • Participants reported increased self-awareness and satisfaction with the tool’s ability to capture nuances like temperature, movement, and spatial positioning.
  • The tool supported multimodal articulation, but participants noted limitations in tactile feedback, sound integration, and artistic skill requirements.
• Advantage over baselines: Compared to traditional 2D materials and verbal descriptions, the tool allowed for richer, more precise articulation of synaesthetic experiences, including spatial and embodied dimensions.
• Experiments / evaluation:
  • Workshops with nine participants explored verbal and non-verbal articulation methods.
  • Case studies with five participants evaluated the Somatic Drawing Tool, focusing on usability, personalisation, and effectiveness in capturing synaesthetic experiences.
  • Data included VR drawings, 3D body mappings, verbal descriptions, and participant feedback.
• Limitations and future work:
  • Limited to five types of synaesthesia; broader applicability needs exploration.
  • Visual-only articulation insufficient for some multisensory experiences; future iterations should integrate sound, haptics, and temporal dynamics.
  • Artistic skill disparities among participants highlighted the need for more inclusive design features.

Summary

This study developed the Somatic Drawing Tool, a VR-based system with a 3D Body Sheet, to help synaesthetes articulate and share their sensory experiences. Through participatory workshops and iterative co-design, the tool enabled participants to externalise spatial, dynamic, and embodied aspects of their synaesthesia. While effective in capturing nuances beyond verbal and 2D methods, the tool’s limitations in multimodal integration and artistic accessibility suggest areas for future improvement. The findings contribute to HCI by advancing methods for sensory articulation and inclusive design, with potential applications in education, therapy, and creative practice.