Feel the Force, See the Force: Exploring Visual-tactile Associations of Deformable Surfaces with Colours and Shapes

Paper Title

Feel the Force, See the Force: Exploring Visual-tactile Associations of Deformable Surfaces with Colours and Shapes

Paper Information

• Domain: Haptic User Interfaces, Multisensory Interaction, Deformable Interface Design
• Keywords: Multisensory Interaction, Force, Crossmodal Correspondence, Colour, Haptics

Research Background and Problem

• Problem and Challenges: With the development of deformable screen technologies, there is still a lack of consistent visual or tactile design elements to represent the stiffness and interaction methods of deformable interfaces. How deformable interactive interfaces can convey their interaction potential remains an unresolved issue.
• Significance: Addressing these issues can enhance the intuitiveness and user experience of such interfaces while providing scientific foundations for related interface designs.
• Motivation and Related Work: The authors build on the theory of crossmodal correspondence (CC), which explores associations between human senses (e.g., associations between colour and touch). Although existing literature has investigated crossmodal phenomena, research on the correspondence between stiffness and visual elements (such as colours and shapes) in the emerging field of deformable interfaces remains unexplored.

Solution

• Proposed Approach:

  1. Based on the theory of crossmodal correspondence, study how surfaces of different stiffness levels associate with colours, 2D graphics, and 3D shapes.
  2. Identify closely related shape characteristics (e.g., curvature, porosity) and colour properties, and collect data through user experiments.
  3. Provide a preliminary framework mapping colours and shapes to the stiffness of deformable surfaces.

• Innovations:

  • This is the first study in the HCI context to use crossmodal correspondence to investigate the intuitive associations between stiffness and visual stimuli (shapes, colours).
  • Introduces the Morphees framework's shape resolution and innovatively analyzes the relationship between 2D/3D shapes and stiffness.
  • Activates the potential for visual-tactile associations, offering references for future deformable interface and shape-changing designs.

• Implementation Steps:

  1. Experimental Material Design: Provide a set of colours, 2D graphics, 3D shapes, and five deformable silicone cubes with varying stiffness levels.
  2. Experimental Tasks:
     • Task 1: Ask 30 participants to match given stiffness stimuli with colours, 2D graphics, and 3D shapes.
     • Task 2: Reverse matching, linking the visual elements selected in Task 1 back to a specific stiffness level.
  3. Analysis Methods:
     • Use repeated measures ANOVA to test the significance of experimental data on colours, shapes, and stiffness.
     • Apply cluster analysis to categorize and decompose the selection frequencies of colours and shapes.

Research Outcomes

• Specific Findings:

  1. Stiffness levels significantly influence the choice of colour brightness: surfaces with low stiffness tend to be associated with bright colours, while high-stiffness surfaces are more suited to darker colours.
  2. The curvature and characteristics (e.g., Bouba-Kiki) of 2D and 3D shapes are significantly associated with stiffness levels. For instance, rounded shapes are typically linked to softer surfaces, while sharp, angular shapes correspond to harder surfaces.
  3. A preliminary mapping framework was developed between stiffness and colours/shapes, where high brightness and rounded shapes strongly indicate softness or hardness characteristics.

• Strengths and Comparisons:

  • Strengths: Provides a novel sensory association perspective, offering a theoretical foundation for deformable interface design in future HCI; compared to existing solutions, this study introduces an intuitive multisensory design approach.
  • Limitations:
    1. Lacks exploration of broader combinations of shapes and surface textures.
    2. Investigates only the independent correspondence of 2D and 3D shapes without combined analysis.
    3. The deformable interface field is still in its early stages, and technical limitations may affect implementation potential.

• Experimental or Evaluation Results:

  • Data from Task 1 show a significant correlation between colour brightness selection and stiffness, with brightness decreasing as stiffness increases.
  • The angles of 2D and 3D shapes (e.g., Bouba-Kiki) are significantly correlated with stiffness, exhibiting a linear trend.
  • Clustering of colours and shapes further validates the distribution of stiffness: for example, high-brightness light yellow (associated with softness) and dark gray (associated with hardness).

• Design Recommendations and Future Directions:

  1. Use bright colours (high brightness) to represent soft surfaces and dark colours (low brightness) for harder surfaces.
  2. Rounded and less curved shapes should represent soft and hard surfaces, respectively.
  3. In 2D designs, long shadows can indicate softness, while short shadows highlight hardness. Future research could expand the range of shapes and colours and explore the application of multisensory synergies in AR/VR and dynamic interactions.