BrushLens: Hardware Interaction Proxies for Accessible Touchscreen Interface Actuation

Document Title

BrushLens: Hardware Interaction Proxies for Accessible Touchscreen Interface Actuation

Document Information

• Domain: Human-Computer Interaction and Assistive Technologies
• Keywords: touchscreen devices, accessibility design, interaction proxy, computer vision, touch actions

Research Background and Problem

• Identified Challenges:

  • The design of touchscreen devices often assumes users have normal vision and fine motor skills, making it difficult for individuals with visual and motor impairments to operate these devices independently.
  • Visually impaired users face challenges in recognizing screen information, navigating interfaces, and performing precise touch actions; motor-impaired users struggle with accurate touch due to issues such as hand tremors or muscle spasms.
  • Current solutions often require hardware or software modifications, which are not supported by many existing legacy touchscreen devices.

• Significance:

  • Touchscreen devices have become critical tools for daily interactions, including self-service kiosks in restaurants and airport check-in terminals. If these devices are inaccessible, it negatively impacts the independence and social participation of affected individuals.

• Research Motivation and Related Work:

  • While existing accessibility designs attempt to improve touchscreen usability through software modifications or guidance, they still demand high precision from users or require extensive hardware changes.
  • Inspired by research on interaction proxies and assistive tools, the authors propose an innovative approach using hardware proxies to address touchscreen inaccessibility, reducing user effort and enhancing interaction accuracy.

Solution

• Proposed Solution:

  • BrushLens: A hardware interaction proxy system integrated into a smartphone case that autonomously senses touchscreen interfaces, provides navigation guidance, and performs precise touch actions on behalf of users.

• Innovations:

  • BrushLens eliminates the need for modifications to existing touchscreen device hardware or software, enabling accessible interaction through hardware proxies.
  • It introduces two interchangeable touch actuators (electromagnetic pushrod and automatic clicker) to accommodate different types of touchscreen technologies (capacitive screens, resistive screens, and physical buttons).

• Implementation Steps and Key Technologies:

  • Interface Reverse Modeling: The system maintains a model of the interface layout, including button positions, UI element types, and state transitions.
  • Device Localization: Real-time device positioning is achieved using the smartphone's camera, IMU sensors, and dark-field sensors.
  • Touch Execution: Based on location data, the system selects the appropriate actuator to simulate touch actions and click target areas.
  • Accessible Interface Support: Interface information is presented in an accessible format on the smartphone, enabling visually and motor-impaired users to navigate and interact through guidance and feedback.

Research Outcomes

• Specific Outcomes:

  • BrushLens enables visually impaired users to navigate and operate previously inaccessible touchscreen devices, while significantly reducing erroneous touches and ineffective interactions for motor-impaired users.
  • Technical evaluations show that BrushLens's touch actuators achieve 100% accuracy on stationary devices, with accuracy rates of 74.2% for the electromagnetic pushrod and 88.3% for the automatic clicker during user movement.

• Advantages Comparison:

  • Compared to existing solutions requiring hardware or software modifications, BrushLens demands zero alterations to touchscreen devices, offering greater compatibility.
  • The automatic clicker exhibits significantly lower touch error rates than the electromagnetic pushrod, making it more suitable for fast user movements.

• User Experiment Results:

  • In trials involving 10 participants, visually impaired users successfully completed interface navigation and interaction tasks that were previously impossible.
  • For motor-impaired users, BrushLens significantly reduced erroneous touches and provided tactile feedback support for users with low finger sensitivity.

• Limitations and Future Directions:

  • The current device requires optimization in speed and portability to better meet diverse user needs in different scenarios.
  • Future improvements could include predictive algorithms for touch decision-making, extended support for more gestures (e.g., swiping and zooming), and automatic interface recognition functionality.

Conclusion

BrushLens introduces a novel solution for accessible touchscreen design through the concept of hardware interaction proxies, eliminating the need for traditional hardware modifications while enhancing user independence and confidence during interactions. With improvements in response speed and system optimization, it has the potential to become a practical accessibility tool across various scenarios.