Marking Material Interactions with Computer Vision

Title of the Paper

Exploring Material Interaction Using Computer Vision Markers

Paper Information

• Subject Area: Physical Computing, Computer Vision-Driven Interaction Design, Practical Interface Development
• Keywords: Physical Computing, Tangible Interaction, Computer Vision, Maker Culture, Materiality

Research Background and Issues

• Identified Problems or Challenges:

  • Traditional electronics-driven physical computing methods (e.g., using microcontrollers or sensors) involve complex circuit construction and debugging, posing challenges, especially for beginners in design.
  • Designers face barriers in hardware infrastructure while striving to expand possibilities in material and interface design for creating practical interactive systems.
  • Characteristics of computer vision, such as ambient lighting and spatial requirements, impose certain limitations on system design.

• Significance:

  • Provides an alternative to traditional electronics-driven methods, simplifying interaction design and enabling designers and educators to create functional tangible interfaces with simple tools.

• Research Motivation and Related Work:

  • Inspired by works such as Reactivision, Sauron, and other computer vision marker-based systems, this study proposes a new computer vision (CV)-driven approach to physical computing.
  • It explores the diverse applications of CV markers from the perspective of "materials potential" and aims to open new directions in physical interaction design through the "annotated portfolio" method to document design examples.

Solution

• Proposed Method:

  • Utilizes CV markers (e.g., Aruco markers) as materials for interaction design, combining cameras, computers, and network platforms to detect real-time interaction events via markers.
  • Introduces a dedicated JavaScript development library, "Beholder," which supports real-time marker detection and integrates seamlessly with user interface design tools like p5.js.
  • Conducts experimental teaching workshops to assist designers in creating practical interfaces and analyzes outcomes using the annotated portfolio method.

• Innovations:

  • Markers simultaneously represent interaction events and constraints, facilitating the design of diverse interactions.
  • Embeds debugging into the design process, allowing human visual verification of interaction logic, reducing debugging burdens.
  • Expands the range of interaction design materials by leveraging "democratic materials" (e.g., cardboard, everyday tools).

• Implementation Steps/Key Techniques:

  1. Develop and introduce the Beholder software library for marker detection and debugging.
  2. Organize three design workshops to promote interactive design and exploration through teaching tasks:
     • DIY CV Interfaces: Develop practical interfaces themed around simple daily activities to explore diverse marker applications.
     • Tinycade: Create a DIY game controller platform based on mobile devices.
     • CV Arcade: Design a new DIY gaming platform, adapting marker interfaces to meet various user needs.
  3. Document design works and reflect on them using the annotated portfolio method.

Research Outcomes

• Specific Results:

  • Developed 15 concrete projects, including educational tools, physical therapy devices, and game controllers, encompassing a wide range of interactions such as sliding, pressing, flipping, light recognition, and wind detection.
  • Proposed a new method of using CV markers as materials for interaction design: markers can record the identity of physical objects, interaction constraints, and define interaction events at the physical level.
  • Summarized the materiality of CV markers, including their adaptability to everyday materials and the constraints posed by ambient lighting on design.

• Advantages Compared to Existing Solutions:

  • Offers a practical interaction design method without requiring circuits or traditional sensors, enabling interactions through simple two-dimensional patterns.
  • The affordability and design flexibility of markers lower the learning barrier, encouraging participation from users across different fields.

• Experimental or Evaluation Results:

  • Projects from the workshops demonstrated how designers could use CV markers for interaction design, validating their broad applicability in education, gaming, and therapy.
  • Designers utilized "visual debugging" to understand and improve their physical interaction mechanisms, boosting confidence and enhancing iteration efficiency.

• Limitations and Future Directions:

  • Limitations:
    • Sensitive to external environments, such as lighting and motion blur, which may affect system detection accuracy.
    • The design method currently supports only input interactions and does not enable haptic feedback or active control.
    • Requires compatibility between the camera's field of view and the markers, imposing spatial design constraints.
  • Future Directions:
    • Develop standardized marker template libraries and more comprehensive development tools.
    • Extend CV marker functionality to output and control domains, enabling integration with other materials.

This research breaks through the limitations of traditional sensors and electronic interfaces, showcasing the potential of computer vision-driven tangible interaction design while paving the way for new innovations in the field of physical computing.