Digital Comprehensibility Assessment of Simplified Texts among Persons with Intellectual Disabilities

Title of the Paper

Digital Reading Test: Assessing the Comprehensibility of Simplified Texts for Individuals with Intellectual Disabilities

Bibliographic Information

• Subject Area: Human-Computer Interaction and Natural Language Processing
• Keywords: Automatic Text Simplification, Comprehensibility Assessment, Intellectual Disabilities, Reading Behavior, Digital Testing, Dyslexia, Human-Computer Interface Design

Research Background and Problem Statement

• Identified Problems or Challenges: Automatic Text Simplification (ATS) is increasingly being utilized, but its evaluation is typically conducted by experts or crowd workers rather than the primary target users of simplified texts (e.g., individuals with intellectual disabilities). Existing research lacks detailed assessments and measurements of reading behavior specific to these target groups.
• Significance: The comprehensibility of simplified texts is crucial for target groups such as individuals with intellectual disabilities. With the growing prevalence of automatic simplification tools, applying digital assessments to target groups can not only improve evaluation efficiency but also capture detailed reading behaviors.
• Research Motivation and Related Work: Current studies mainly rely on expert ratings or subjective evaluations of text simplicity, overlooking grammatical errors, text fluency, and the actual reading behaviors of target users. This paper fills a methodological gap in the field by conducting the first human evaluation of German texts simplified by ATS for individuals with intellectual disabilities and a control group.

Proposed Solution

• Proposed Methods or Solutions: The authors designed a digital experiment using a mobile application to measure users' reading behavior and comprehension of original texts, manually simplified texts, and automatically simplified texts. The experiment included the following four types of measurements:
  1. Multiple-Choice Comprehension Test: Evaluates the ability to understand key details of the text through multiple-choice questions.
  2. Subjective Difficulty Rating: Users rate the difficulty of reading the text on a five-point scale.
  3. Response Time to Reading Questions: Measures the time taken to answer each question as a behavioral proxy for comprehension.
  4. Reading Speed: Calculates the number of words read per minute during the initial reading of the text.
• Innovative Aspects:
  1. Introduced a digital method for readability assessment specifically targeting individuals with intellectual disabilities for the first time.
  2. Utilized user interaction data (e.g., scrolling behavior, reading speed) to analyze reading behavior in depth.
  3. Compared the target group with a control group to reveal differences in reading behavior and comprehension abilities.
• Implementation Steps:
  1. Selected original and simplified texts in German simplified language and created comprehension test questions for each document.
  2. Developed a specially designed mobile application, "Okra," tailored to different user reading habits.
  3. Recruited participants from both the intellectual disability group and the control group to complete three experiments, including reading tasks and cognitive tasks.
  4. Analyzed comprehension performance across text versions using Bayesian statistical methods.

Research Outcomes

• Specific Findings:
  1. The accuracy rate for the multiple-choice comprehension test was 47.5% for the intellectual disability group, compared to 92.8% for the control group, highlighting significant heterogeneity in reading behavior and comprehension performance.
  2. Texts generated by ATS showed limited improvement in reading behavior for the target group, especially in subjective ratings.
  3. Automatic text simplification did not demonstrate significant comprehension improvements in the control group and was sometimes more difficult than the original texts.
  4. Manually simplified texts significantly improved comprehension performance for both groups.
• Advantages and Contributions:
  1. Proposed four measurement methods and validated the multiple-choice test as the most reliable, while difficulty ratings were less effective for the intellectual disability group.
  2. Compared the effects of manual and automatic text simplification, finding manual simplification to be more advantageous while identifying potential areas for ATS optimization.
  3. Used mobile devices to record reading behavior, providing data support for digital assessments and accessibility development.
• Limitations and Future Directions:
  1. The quality of automatically simplified texts remains limited, requiring further research to address grammatical and semantic errors.
  2. Did not deeply analyze differences in sensitivity to language errors between the target group and the control group.
  3. Future research could integrate advanced language models (e.g., large language models) to enhance ATS system performance.
  4. Recommended further exploration of reading interaction data patterns and their adaptability across different disability groups.

In conclusion, this paper combines digital methods with cognitive testing to reveal new pathways for evaluating simplified texts, laying the foundation for the development of future text simplification technologies and human-computer interaction research.