I-Scratch: Independent Slide Creation With Auditory Comment and Haptic Interface for the Blind and Visually Impaired

Research Background and Issues

Issues and Challenges

• Visual-centric User Interface Barriers: Current presentation software (e.g., PowerPoint) primarily caters to sighted users with complex and visually oriented interfaces, posing significant obstacles for blind and visually impaired users (BVIs).
• Challenges in Spatial Understanding and Editing: While screen readers (e.g., JAWS, NVDA) provide text and basic interface element voice feedback, they fail to effectively assist BVIs in complex spatial understanding, such as editing slide layouts and aesthetics.
• Limitations of Existing Solutions: Previous research has attempted to convert visual information into sound, but this approach imposes cognitive burdens when conveying spatial information. Although tactile feedback has enhanced interaction, there remains a significant gap in enabling BVIs to create and edit presentation slides.

Importance

• Presentations are essential tools for conveying information in professional and educational settings. However, BVIs often rely on sighted assistance due to limited access to primarily visual content, restricting their creative expression and independence.

Research Motivation and Related Work

• Motivation: To address the high cognitive load and reliance on others faced by BVIs when using presentation software, this study develops an innovative system that combines tactile and audio feedback with AI technology.
• Related Work: While prior research has provided tools for BVIs to explore 2D content (e.g., charts, maps), digital drawing, and web design, there has been limited focus on supporting them in independently editing, creating, and managing complex presentations. Evaluations have often emphasized replicating predefined slides rather than fostering creation and content optimization.

Solution

Methodology and Innovations

• This study introduces I-Scratch, a multimodal interaction system integrating a tactile graphic display (GTD), voice feedback, and AI capabilities. The system enables BVIs to independently create, explore, and edit presentation slides.
• Innovations:
  1. Multimodal Feedback: Combines tactile touch and voice guidance to reduce cognitive load.
  2. Simplified UI: Restructures PowerPoint’s complex interface into tactile windows that are easy to navigate.
  3. AI-driven Visual Support: Incorporates generative AI (e.g., DALL·E, GPT-4) for image generation and content interpretation, providing users with image descriptions and layout guidance.

Implementation Steps and Key Technologies

• System Architecture: Integrates five devices (PC, GTD, speaker, webcam, and keyboard) with multiple software components (e.g., image generator, voice renderer, tactile window manager).
• User Interaction:
  • Tactile windows support 2D exploration via direct interaction with users’ finger tracking.
  • Users operate functions through keyboard, touch, and voice commands.
• Tactile Feedback Presentation:
  • Uses a dot matrix display (GTD) to allow users to perceive slide layouts and content through touch, such as boundaries, shapes, and proportions.
• AI-powered Features:
  • AI generates images and provides detailed voice descriptions for users.
  • AI analyzes slide layouts and offers suggestions to improve visual presentation and aesthetics.
• Participatory Design: Iterative design improvements through three rounds of feedback from blind users, including the addition of “undo/redo points” and more intuitive zoom and rotation interface designs.

Research Outcomes

Specific Results

1. Functionality Testing: Experiments demonstrated that BVIs could fully or partially complete slide creation and editing tasks using I-Scratch:
   • Across all experiments, 91.25% of tasks achieved completion (fully or partially successful).
   • Creativity Support Index (CSI) scored 85.07 (out of 100), indicating high user satisfaction.
2. Enhanced Independence: The system significantly reduced users’ reliance on sighted assistance.

Advantages

• Independent Creation: I-Scratch greatly improved users’ ability to edit slides, regardless of prior PowerPoint experience.
• Intuitive Feedback: Combines tactile and AI technologies to provide rich and comprehensible feedback.
• Support for Generative Creativity: AI functionalities encouraged users to generate original content and optimize slide presentation.

Experiment and Evaluation Results

• The combination of tactile and voice feedback enabled blind users to efficiently complete complex slide layout, rotation, and scaling tasks.
• In the second experimental phase, users successfully modified slides generated by AI or manually based on predefined text scripts.
• The system received high evaluations in dimensions such as “explorability,” “expressiveness,” and “effort-worthy creative outcomes.”

Limitations and Future Directions

1. Technical Limitations:
   • Efficiency loss due to device switching (e.g., from GTD to keyboard).
   • Limited support for style attributes (e.g., fonts, colors).
2. Future Directions:
   • Enhancing Operability: Introduce STT functionality to reduce text input burden.
   • Expanding Application Scenarios: Develop assistive systems for other document tools, such as Excel.
   • Improving AI Assistance: Create smarter revision and suggestion tools to simplify BVIs’ editing processes.

I-Scratch, as an innovative technology, empowers BVIs with the ability to independently create presentation slides. It not only fills gaps in existing solutions but also provides practical references for future development of accessible technologies.