Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLM

Title of the Paper

Predicting Text Input Hint Text in Mobile Applications Using Large Language Models (LLM): Design and Evaluation of HintDroid

Paper Information

• Research Area: Human-Computer Interaction, Assistive Technology, Applications of Large Language Models
• Keywords: Mobile Application Design, Accessibility, User Interface, Large Language Models, Text Input, Feedback Mechanism, Visually Impaired Users

Research Background and Issues

• Problems and Challenges:

  1. Accessibility issues are prevalent in mobile applications, particularly the lack of hint text in text input components.
  2. An analysis of 4,501 Android applications revealed that over 76% of text input components lack hint text, creating barriers for visually impaired users relying on screen readers to understand input requirements.
  3. The absence of hint text is often due to developers' lack of awareness about accessibility or insufficient information descriptions.
  4. Existing research primarily focuses on generating accessibility labels for image components, with little attention to the automatic generation of text input hint text.

• Significance:

  1. According to the World Health Organization, at least 2.2 billion people worldwide face vision impairment, necessitating more inclusive mobile application designs.
  2. The lack of hint text prevents screen readers from conveying the semantic information of input fields, significantly hindering the interaction and experience of visually impaired users.
  3. Generating appropriate hint text can enhance the overall accessibility of mobile applications, providing better functional support for visually impaired users.

• Motivation and Related Work:

  1. Large Language Models (LLMs) have demonstrated outstanding performance in natural language processing and generation tasks, inspiring their application in accessibility-related text generation tasks.
  2. Very few studies have addressed semantic understanding and label generation for software interfaces, and no existing methods are tailored for generating text input hint text.
  3. HintDroid is proposed as a novel solution in this research domain to address these challenges.

Solution

• Method or Solution:

  1. A system named HintDroid is proposed, which combines LLMs with graphical user interface (GUI) information to generate hint text for text inputs through contextual learning.
  2. A feedback mechanism is introduced to optimize the generated hint text by determining whether the generated input content meets the requirements and whether it triggers the next page, providing feedback to refine the results.

• Innovations:

  1. A contextual learning approach based on LLMs transforms user interface information into comprehensible natural language descriptions.
  2. A feedback mechanism (including error information extraction) is designed to evaluate and improve the quality of the generated hint text.
  3. Hint text generation is integrated with input content generation to validate its relevance and improve outcomes.

• Implementation Steps:

  1. GUI Entity Extraction: Extract and organize information about the application, current page, and input components, including component text, IDs, relative positions, etc.
  2. Hint Generation: Use contextual learning to construct prompts with examples to help the LLM understand the task.
  3. Feedback Optimization: Evaluate the quality of hint text by detecting whether the input content triggers the next page and refine the generated results based on error information.
  4. Automated Experiments and Evaluation: Generate hint text using LLMs and validate its effectiveness through natural language evaluation metrics such as BLEU and ROUGE.

Research Outcomes

• Specific Results:

  1. HintDroid achieved leading performance in large-scale experiments, with BLEU@1 reaching 83%, ROUGE scoring 63%, and CIDEr scoring 62%.
  2. User studies showed that after using HintDroid, the rate of visually impaired users correctly filling in input content increased by 152%, while the coverage of exploring more page activities and states increased by 77% and 66%, respectively.
  3. The average time for generating hint text was 1.86 seconds per page, significantly improving development efficiency.

• Advantages Compared to Existing Solutions:

  1. Significantly outperforms existing methods, including rule-based, deep learning, and current LLM-based approaches, in terms of hint text generation accuracy and coverage.
  2. Provides a feedback-based optimization mechanism, making it adaptable to diverse application scenarios.
  3. Combines hint text generation with content generation, ensuring the practicality of the hint text and improving user experience.

• Experimental or Evaluation Results:

  1. Large-scale comparative experiments showed that HintDroid outperformed the best existing methods by over 82%.
  2. User experiment feedback indicated that using HintDroid reduced input completion time by 139%.

• Limitations and Future Directions:

  1. Limitations in generation accuracy: Hint text generation quality is affected when GUI contextual information is insufficient.
  2. Currently limited to the Android platform; future work could expand to iOS, web, and other platforms.
  3. Plans to enhance the model's contextual understanding capabilities and explore real-time interaction implementation.
  4. Users expressed a desire for more personalized and dynamically adjustable hints, such as providing input format or interactive feedback support.

Conclusion

HintDroid leverages the powerful natural language understanding capabilities of large language models, combined with contextual learning and feedback mechanisms, to effectively bridge the information gap for visually impaired users in mobile applications by providing accessible text input hints. Future work includes improving scenario adaptability, achieving cross-platform expansion, and incorporating personalized design features.