TextFlow: Screenless Access to Non-Visual Smart Messaging

Document Title

Textflow: Screenless Access to Non-Visual Smart Messaging

Document Information

• Subject Area: Assistive technology, non-visual interaction, voice navigation, mobile smart devices
• Keywords: Text input, assistive technology, smart wearable devices, auditory navigation, ubiquitous smart environments, screenless interaction

Research Background and Problem

• Problems and Challenges:

  • Current screen-centric text input designs pose significant usability barriers for blind and visually impaired (BVI) users.
  • Screen-dependent interaction methods create operational difficulties for users who are mobile or need to hold other items.
  • While voice input can replace screen keyboards, it faces challenges such as privacy concerns and environmental noise affecting text recognition in public spaces.
  • Text input lacks flexibility, requiring manual input for repetitive text, which imposes a substantial burden on these users.

• Importance: Redesigning a more accessible and efficient text input solution is crucial for enhancing the daily life experiences and social integration of BVI users.

Solution

• Proposed Solution:

  • Introduce a system called TextFlow, a hybrid interaction approach based on context-awareness and proactive suggestions, providing BVI users with fully auditory text prompts.
  • Utilize wearable devices (TapStrap) to enable screenless navigation, allowing users to browse and send text options through finger tapping.
  • The system consists of three core modules:
    1. User Model: Builds a dynamic user profile, leveraging contextual information such as time, location, and activity to push relevant text options.
    2. Reasoning Model: Defines high-level rules to detect potentially important user scenarios and proactively recommend messages.
    3. Task Model: Supports users in browsing and selecting auditory prompts to complete text-sending tasks.

• Innovative Features:

  • Adheres to a fully screenless design principle, improving traditional keyboard and voice input modes through auditory streams and finger-touch interaction.
  • Combines contextual data, such as user location, time, and activity status, to generate suggested messages, reducing the burden of manual text construction.
  • Proactive push functionality is implemented through rule-based model design to avoid missing critical events.

• Implementation Steps and Technology:

  1. Generate contextual data using user location, activity detection, and schedule information.
  2. Produce relevant voice message options based on context priority, extracting candidate texts from datasets and optimizing them using pre-trained language models (RoBERTa).
  3. Enable rapid selection operations via the TapStrap wearable device.
  4. Complete screenless message sending.

Research Outcomes

• Specific Results:

  • Preliminary User Needs Assessment: Through interviews with 20 BVI users, 17 common text themes were identified, including notifications, assistance requests, schedule adjustments, location confirmations, etc.
  • Functionality Validation Tests: In field tests, 10 blind users attempted to use TextFlow to complete tasks in multiple simulated scenarios, achieving a task success rate of 88.66%.
  • User Experience Feedback:
    • Users generally felt that TextFlow reduced their reliance on smartphones.
    • The system demonstrated advantages in privacy protection and efficiency, especially in public spaces and mobile scenarios.

• Comparative Advantages Over Existing Solutions:

  • Reduces reliance on screens and visual sources, enhancing interaction experience through auditory methods and contextual data.
  • Improves privacy and accuracy in text selection compared to voice input.
  • Discrete operations supported by wearable devices are more "lightweight" than gestures and traditional keyboards.

• Experiment and Evaluation Results:

  • After training, users completed tasks in an average time of 17 to 43 seconds.
  • Both "step-by-step browsing" and "automatic playback" interaction modes were popular, but users preferred the former for maintaining active control.
  • TextFlow demonstrated high applicability in various scenarios (e.g., public transportation, crowded environments, work meetings).

• Limitations:

  • The current design does not support users adding detailed custom content (e.g., describing visual signals).
  • Text options lack personalization, with users expressing a desire for dynamic message sorting based on usage frequency.
  • Support for more complex text structures remains incomplete.

• Future Directions:

  • Enhance the message model to support user-defined needs, such as defining more complex composite message content.
  • Add machine learning components to optimize the use of user historical data, further improving interaction efficiency.
  • Design lighter and more functional wearable devices tailored for blind users.

Conclusion

The TextFlow system integrates context-awareness, auditory prompts, and screenless input technologies to significantly improve the experience of blind users in text message sending. In future development, the system's practicality and adaptability will depend on further refinement of personalized and diverse message generation functions, as well as the precision of contextual push notifications. This research provides important theoretical and technical support for service models designed for non-visual interfaces.