TacNote: Tactile and Audio Note-Taking for Non-Visual Access

Document Title

TacNote: Tactile and Audio Note-Taking for Non-Visual Access

Document Information

• Subject Area: Accessibility Technology and Interaction Design
• Keywords: Assistive Technology, Tactile Graphics, 3D Printing Pen, Mobile Application, Accessible Design, Non-Visual Access

Research Background and Problem

• Identified Challenges:

  • Blind and visually impaired (BVI) individuals primarily rely on non-visual senses (e.g., touch and hearing) to understand content and spatial relationships, which imposes significant cognitive load for identifying and remembering objects in complex environments.
  • Common high-tech assistive tools (e.g., computer vision) provide information access but lack tactile feedback, potentially increasing user burden.
  • Existing methods (e.g., labels, Braille markers) for creating tactile graphics have high learning barriers, limited operability, and insufficient autonomy.

• Significance:

  • Reducing the cognitive and memory load for BVI individuals in daily life and learning scenarios enhances their ability to access information and familiarize themselves with environments, benefiting independent living and improving learning and creative efficiency.

• Research Motivation and Related Work:

  • Recent studies have explored methods combining 3D printing and voice-assisted technologies, but these often require assistance from sighted individuals and have limited flexibility and personalization.
  • This study aims to leverage the portability and low learning curve of 3D printing pens to provide BVI users with a method to independently create and edit tactile graphics, while integrating voice assistance to enhance interaction and memory support.

Solution

• Proposed Solution:

  • TacNote System: A tactile and audio-based note-taking and interaction mobile system designed for BVI users.
  • TacNote consists of two components: a modified 3D printing pen for drawing tactile graphics and a mobile application supporting voice feedback and graphic interaction.

• Innovations:

  • Using a 3D printing pen, users can independently create and define personalized tactile graphics, addressing the dependency on others in traditional methods.
  • Incorporates real-time layered voice feedback and clock-face-based directional descriptions to help users understand spatial relationships and hierarchical information.
  • For the first time, combines tactile object creation, voice assistance, and mobile capture to support instant information retrieval, editing, and recall.

• Implementation Steps and Key Technologies:

  • 3D Printing Pen Modification: Utilizes a low-temperature 3D printing pen to ensure safety and tactile operability; integrates an Arduino microcontroller and buzzer for auditory status feedback (e.g., heating complete, low battery).
  • Tactile Image Processing: Employs Python and OpenCV for color extraction and shape recognition, and MediaPipe for gesture recognition and voice interaction.
  • Input and Browsing Process:
    • Create Note Mode: Users draw tactile graphics and input information such as titles, content, order, and relative positions.
    • Read Note Mode: Users capture images of target objects, and the system identifies tactile graphic symbols and reads the associated content.
  • Filtering and Prompt Mechanisms: Enhances user experience with features like photo quality checks, prompts for retaking photos, and automatic skipping of unnecessary steps.

Research Outcomes

• Specific Outcomes:

  1. Developed the TacNote system, enabling BVI users to independently create, edit, and interact with tactile symbols and understand content through voice feedback.
  2. Conducted a 15-day diary study and user survey to validate the applicability and potential of the 3D printing pen among BVI individuals, collecting their experiences and feedback across various scenarios.
  3. User studies demonstrated that TacNote significantly reduced memory load and provided greater autonomy in customizing and accessing information.

• Advantages Compared to Existing Solutions:

  • Increased Autonomy: BVI users can easily create tactile graphics without relying on assistance from sighted individuals.
  • Rich Information: Supports hierarchical voice descriptions, allowing users to access detailed content as needed.
  • Flexibility: The marking method based on the 3D printing pen is suitable for various scenarios, including categorization, education, and navigation.

• Experimental and Evaluation Results:

  • TacNote achieved an average high score of 80.25 on the SUS scale in user evaluations, demonstrating good user-friendliness and efficiency.
  • In the first task (distinguishing similar objects), participants' memory ability scores increased from 2.4 without the system to 4.9 with system support.
  • In the second task (spatial understanding), TacNote's assistance received high praise compared to traditional methods, with a score of 4.6.

• Limitations and Future Directions:

  • Limitations:
    • The photography skills of BVI users affected some aspects of the user experience.
    • The system's reliance on blue printing material may cause recognition issues with certain background colors.
  • Future Directions:
    • Provide camera enhancement features and comprehensive guidance to improve photo-taking quality.
    • Expand the range of symbol colors to support user-customized color options.
    • Develop features for sharing tactile graphics to facilitate multi-user collaboration or public applications.

Conclusion

TacNote combines tactile graphics and voice technologies to provide an innovative information recording and interaction system for blind and visually impaired individuals, significantly enhancing their information access capabilities and independence. This system has the potential to expand into broader applications such as education, culture, and spatial exploration.