ImageAssist: Tools for Enhancing Touchscreen-Based Image Exploration Systems for Blind and Low Vision Users

Document Title

ImageAssist: Tools for Enhancing Touchscreen-Based Image Exploration Systems for Blind and Low Vision Users

Document Information

• Subject Area: Design of touchscreen image exploration assistive tools and accessibility technology
• Keywords: Accessibility technology, visual impairment, touchscreen image browsing, image exploration tools, voice feedback, navigation tools, Braille user support, tactile graphics, independence, user experience

Research Background and Problems

• Problems or challenges identified by the authors:

  • Although touchscreen image exploration systems allow visually impaired users to gain a more comprehensive understanding of images, existing technologies suffer from low efficiency, users missing image elements, and poor experiences when handling small and dense areas.
  • Users need to explore images independently, leading to high cognitive load, which remains a major unresolved obstacle.

• Why this problem is important:

  • Digital images are becoming increasingly important in areas such as social media, art exhibitions, and education. However, visually impaired users face numerous challenges in understanding digital images, and current tools fail to effectively meet these needs.
  • Providing smoother and more efficient exploration functionality can enhance user independence, reduce frustration, and help visually impaired users comprehend complex graphics.

• Research motivation and related work:

  • Existing methods, such as manually written alt text or AI-generated descriptions, often suffer from quality issues or misleading information and provide limited support for complex images.
  • This research aims to offer an alternative for visually impaired users by leveraging touchscreen-based image exploration tools on mobile devices, designed to supplement or enhance traditional alt text functionality.

Solution

• Methods or solutions proposed by the authors:

  • Developed a set of tools called ImageAssist to improve touchscreen image exploration systems, incorporating the following three features:
    1. Menu and beacon tools: Provide a voice menu listing areas within the image, assisting users in locating areas of interest via audio beacons.
    2. Guidance tools: Guide users to explore efficiently through recommended areas, volume-based importance indicators, and first-touch prompts.
    3. Quadrant zoom tools: Allow users to zoom into the four quadrants of an image for easier viewing of dense or small areas.

• Innovative aspects of the solution:

  • Added navigation functionality to touchscreen exploration to reduce randomness and disorder.
  • Provided audio beacons and voice reinforcement to significantly lower cognitive load.
  • Dynamic first-touch indication to enhance users' awareness of image coverage.
  • A newly designed quadrant-specific zooming approach simplifies the experience of visually impaired users exploring complex images.

• Implementation steps and key technologies used:

  • Developed an experimental application using the Unity engine.
  • Employed AI models (Smooth Grad-CAM++) to generate visual importance maps for the guidance tools.
  • Conducted user testing via remote deployment and cloud-based backend support.

Research Outcomes

• Specific achievements:

  • ImageAssist significantly improved the efficiency and coverage of image exploration for visually impaired users, with the proportion of accessible areas increasing from 84% to 94% compared to baseline technologies.
  • The menu and beacon tools enhanced users' ability to understand the overall image, while the guidance tools facilitated targeted and efficient exploration.
  • First-touch indication and volume-based prompts improved user experience and reduced cognitive load.
  • Quadrant zoom tools performed well on small-screen devices and in handling dense areas.

• Advantages compared to existing solutions:

  • Provided a new framework that integrates touchscreen exploration with voice feedback.
  • Compared to baseline tools, users demonstrated more consistent exploration times and greater confidence in achieving full coverage.
  • Significantly reduced frustration during the image exploration process for visually impaired users.

• Experimental or evaluation results:

  • Users provided positive feedback through questionnaires and open-ended interviews, with high scores particularly for the menu and beacon tools and the guidance tools.
  • Experiments supported the need for touchscreen image exploration tools to include "overview" features and enhanced guidance information.
  • Feedback on the quadrant zoom tools was polarized, with some users suggesting they should be used in conjunction with other assistive tools.

• Limitations and future directions:

  • Image regions were manually annotated; future work should test integration with automated segmentation technologies.
  • Research focused solely on visually impaired users and has not yet been adapted for other groups, such as those with hearing or motor impairments.
  • Limited sample size (only 9 participants); future studies should expand the user base to validate broader applicability.
  • Future exploration could consider alternative zooming methods, such as dynamic adaptation of content to panoramic screen formats.

Application Scenarios

• STEM Education: Supports visually impaired students in accessing high-level information from scientific charts, advancing accessibility in education.
• Map Exploration: Helps users "preview" complex environment maps before navigation, enhancing independence.
• Social Media and Personal Photos: Provides effective exploration tools for personal or social media images, enabling users to better understand the content of images they share or receive.

These findings offer valuable practical references for the design and application of touchscreen-based accessible image exploration systems, while also encouraging further attention to ensuring the universality and usability of such tools.