Toucha11y: Making Inaccessible Public Touchscreens Accessible

Document Title

Toucha11y: Making Inaccessible Public Touchscreens Accessible

Document Information

• Subject Area: Human-Computer Interaction (HCI), Accessibility Design, Assistive Technology
• Keywords: Accessibility, Touchscreen Devices, Robotics, Visual Impairment, User Study, Reverse Engineering, Privacy Protection, Technology Evaluation, Personal Assistive Devices, Computer Vision

Research Background and Problem

• Identified Issues or Challenges:

  • Touchscreen devices are widely used in public spaces, yet many public touchscreen devices are inaccessible to blind users.
  • Users rely on visual interaction with touchscreens, and the lack of tactile or voice assistance makes these devices extremely unfriendly for blind users.
  • Although laws (e.g., the Americans with Disabilities Act) mandate certain public devices to be accessible, most devices remain unusable, especially older, already-deployed systems.
  • Waiting for manufacturers to update all touchscreen devices is impractical, creating an urgent need for short-term solutions.

• Significance:

  • The inability to access touchscreen devices in public spaces (e.g., hospitals, kiosks, restaurants) limits the independence of blind users and poses risks of privacy breaches.
  • Addressing this issue through technological solutions can significantly enhance the autonomy and quality of life for blind users, reducing social isolation.

• Research Motivation and Related Work:

  • Existing solutions, such as tactile overlays and crowdsourced labeling methods, face challenges like implementation difficulty, lack of support for dynamic content, or complexity of use.
  • Since blind users commonly use smartphones with built-in accessibility features, leveraging existing devices and simplifying interaction processes has become a research focus.

Solution

• Proposed Method or Solution:

  • Development of a prototype system called "Toucha11y," comprising a robotic device, a smartphone application, and a backend server.
  • Toucha11y combines hardware and software to transmit the content of public touchscreen devices to the user's smartphone, enabling interaction via screen readers (e.g., VoiceOver and TalkBack).
  • The robotic device captures the touchscreen content for self-positioning and triggers corresponding actions on the touchscreen based on user input.

• Innovative Features:

  • Accessible Interaction Migration: Transfers complex and inaccessible interactions from public touchscreens to smartphones, enabling blind users to complete tasks using familiar devices.
  • Hardware Positioning and Triggering: Employs a compact robotic device for high-precision screen positioning and touch activation, addressing direct interaction challenges.
  • Privacy Protection: Allows users to input sensitive information via their smartphones without relying on others, reducing privacy risks.

• Implementation Steps and Key Technologies:

  1. The user launches the Toucha11y app and attaches the robot to the touchscreen device.
  2. The robot captures the screen image for positioning and uploads it to the server, where computer vision algorithms determine precise coordinates.
  3. The user browses the screen content on their smartphone and makes a selection.
  4. The robot automatically performs the selected action on the touchscreen device.
  5. The backend server updates the smartphone interface in real-time to reflect the interaction results.

Research Outcomes

• Specific Results:

  • Toucha11y successfully enabled blind users to independently complete simulated tasks, such as ordering drinks on a self-service kiosk.
  • The system achieved an average error distance of 6.50 mm, with touch activation accuracy suitable for most touchscreen interfaces.
  • Experiments demonstrated that the device is easy to learn, comfortable to use, and highly efficient in interaction.

• Advantages Compared to Existing Solutions:

  • Eliminates the need for customized tactile assistive tools, simplifying the interaction process and supporting dynamic content interfaces.
  • Offers broader applicability and greater user autonomy compared to crowdsourced or 3D-printed solutions.
  • Effectively addresses privacy concerns.

• Experimental or Evaluation Results:

  • Blind users were able to independently complete tasks during experiments, with an average task completion time of 87.2 seconds.
  • Users reported that the system reduced cognitive load, protected privacy, and significantly enhanced independence.
  • Technical evaluations confirmed the high precision of the robot's positioning, rotation, and extension, while highlighting limitations (e.g., challenges with simple visual interfaces).

• Limitations and Future Directions:

  • Limitations:

    • The system relies on pre-annotated screen interfaces in the database and does not support dynamically generated content.
    • The robot's size limits portability and requires improvement.
    • Limited capability to locate screens with minimal visual features.

  • Future Directions:

    • Hardware optimization (reducing device size, improving suction cup design) to enhance user experience.
    • Integration of intelligent vision algorithms to reduce dependence on pre-annotated databases.
    • Long-term deployment in real-world scenarios to collect broader user feedback and usage data.
    • Exploration of the technology's potential in other assistive contexts, such as aiding individuals with motor impairments or operating physical devices.