Finding the Needle in a Haystack: On the Automatic Identification of Accessibility User Reviews

Title of the Paper

Finding the Needle in a Haystack: On the Automatic Identification of Accessibility User Reviews

Paper Information

• Subject Area: Evaluation of mobile app accessibility and machine learning classification research
• Keywords: Accessibility evaluation, mobile apps, user reviews, machine learning, text classification, data-driven methods, automatic identification, usability testing, model optimization

Research Background and Issues

• Identified Problems or Challenges:

  • The accessibility of mobile applications is often poor, failing to meet the needs of individuals with disabilities.
  • User reviews serve as a critical data source for app improvement, but identifying accessibility-related reviews from a large volume of comments is time-consuming and challenging.
  • Manual identification of accessibility issues is inefficient, and automated keyword-matching methods have high false positive rates, making them impractical.

• Significance:

  • Accessibility optimization is crucial for enhancing user experience and fostering an inclusive society, yet developers often overlook accessibility issues in the software development process.
  • App store reviews reveal genuine user feedback, and identifying accessibility-related issues can help developers raise awareness and make improvements.

• Research Motivation and Related Work:

  • Current methods for identifying accessibility-related reviews rely primarily on manual processing or keyword-based matching algorithms, both of which fail to meet efficiency and effectiveness requirements.
  • There is a need for a robust solution capable of effectively distinguishing accessibility-related reviews from general reviews.
  • While some studies in the field have analyzed accessibility issues (e.g., keyword-matching recommendation methods), this paper integrates advanced machine learning to propose a more efficient solution.

Solution

• Proposed Method or Solution:

  • Transform the identification of accessibility-related reviews into a binary classification problem, utilizing supervised learning for automated detection.
  • Employ a Boosted Decision Trees (BDT) model and compare its performance with eight other machine learning classification algorithms to determine the optimal model.
  • Address semantic gaps and enhance model recognition by extracting actual features from user reviews rather than relying solely on accessibility guideline keywords.

• Innovations:

  • Extract text patterns from real user reviews instead of depending solely on technical keywords.
  • Propose an algorithmic model capable of automatically learning and distinguishing accessibility-related reviews from other comments, reducing the burden of manual filtering.
  • Demonstrate that high accuracy can be achieved with a small training dataset (1,500 reviews).

• Implementation Steps and Key Techniques:

  1. Data Collection: Use an existing manually verified dataset, including 2,663 accessibility-related reviews and 2,663 general reviews.
  2. Data Preprocessing: Apply natural language processing techniques (e.g., tokenization, lemmatization, stopword removal) to clean the review text.
  3. Feature Extraction: Utilize feature hashing for text pattern extraction, combined with N-Gram techniques.
  4. Model Selection and Parameter Optimization: Test nine classification algorithms and optimize performance through grid search, ultimately selecting the BDT model.
  5. Model Evaluation: Evaluate the classification model using 10-fold cross-validation and report performance metrics (Precision, Recall, F1-measure).

Research Findings

• Key Findings:

  • The BDT model successfully classified accessibility-related reviews from general reviews, achieving 90.6% accuracy and a 90.7% F1-measure.
  • Compared to baseline methods (keyword-matching algorithms and random classifiers), the BDT model improved the F1-measure by 1.574 times and 39.434 times, respectively.

• Experimental or Evaluation Results:

  • A small training dataset (1,500 reviews) was sufficient to achieve efficient classification, with an F1-measure exceeding 85%; adding more data further improved performance.
  • Analysis of feature words and bigrams in user reviews revealed diverse patterns of expressing accessibility feedback, providing valuable insights for developers.

• Limitations and Future Directions:

  • The dataset includes only reviews from Android open-source applications, which may limit the generalizability of the findings.
  • Reviews are restricted to English, making the approach inapplicable to scenarios involving other languages.
  • Future work is recommended to adopt active learning to optimize the training data collection process further and explore multi-class classification to categorize accessibility issues into specific domains.

Conclusion

This paper presents an efficient machine learning model for the automatic identification of accessibility issues in mobile app reviews, addressing the inefficiencies and high false positive rates of previous studies. The proposed solution offers developers a potential tool for improving and maintaining software accessibility.