Older Adults' Think-Aloud Verbalizations and Speech Features for Identifying User Experience Problems

Title of the Paper

Older Adults’ Think-Aloud Verbalizations and Speech Features for Identifying User Experience Problems

Paper Information

• Research Area: User Experience (UX) Research and Usability Testing
• Keywords: Older Adults, Think-Aloud Method, Speech Features, User Experience Problems, Usability Testing, AI-Assisted UX Analysis, Human-Computer Collaboration

Research Background and Problem

• Problem and Challenges: Although previous studies have found that the think-aloud (TA) speech features and language patterns of younger users (aged 19-26) can reveal user experience problems, it remains unclear whether these patterns are applicable to older adults. Differences in task performance between older and younger users may be reflected in their language patterns, potentially affecting UX analysis.
• Research Significance: Older adults as a user group are increasingly gaining attention in technology interactions. Accurately identifying UX problems is crucial for improving product design and enhancing the usability of technology for older adults.
• Research Motivation and Related Work: Previous studies have explored the role of TA speech and language patterns in uncovering user experience problems, but they have primarily focused on younger populations. This study aims to fill this research gap and extend the analysis to older users.

Solution

• Methods and Steps:

  1. Design an experiment recruiting 10 older participants (average age 75) to complete usability testing using the think-aloud method. The test products include a coffee machine and two digital products (a pet adoption website and a food delivery app).
  2. Use Cooke’s language classification standards to categorize participants’ verbalizations into five types: Reading, Procedure, Observation, Explanation, and Others.
  3. Annotate the emotional valence of participants’ verbalizations (positive, negative, neutral).
  4. Extract speech features (loudness, pitch, and speech rate) and analyze their correlation with UX problems.
  5. Quantify the association between language and speech features and UX problems using Precision, Recall, and F-measure.
  6. Analyze the common vocabulary used by participants when encountering and not encountering problems.

• Innovations:

  1. This study is the first to quantitatively examine the correlation between older users’ TA language and speech features and user experience problems.
  2. It extends findings from previous studies on younger users, providing insights into age-related differences in UX analysis.
  3. Systematically compares the language and speech patterns of older and younger users during the think-aloud process.

Research Findings

• Specific Findings:

  1. Older adults are more likely to use observation-related language (Observation) when encountering UX problems, including negative emotional words, negations (e.g., "no"), and interrogatives (e.g., "why"). Their speech features are characterized by higher loudness, pitch, and speech rate.
  2. The proportion of Reading language used by older adults is lower than that of younger users, but the proportion of Observation language is higher.
  3. Analyzed the most frequently used vocabulary by older adults, revealing differences in language use across contexts, which provides inspiration for further UX analysis.

• Advantages Compared to Existing Solutions:

  1. Provides an analysis of older users’ language features across multiple products and scenarios.
  2. Quantitatively measures the precision and correlation of speech features (e.g., pitch and speech rate) with UX problems, highlighting diagnostically significant features.

• Experimental and Evaluation Results:

  • Language Classification: Observation-related language is the most indicative of UX problems, with an F-measure of approximately 0.64, higher than other categories.
  • Sentiment Analysis: Negative emotions are the most indicative of problems (F-measure of 0.7).
  • Speech Features: High loudness, pitch, and speech rate are strongly correlated with problems, while low speech rate showed no significant effect.
  • Word Frequency Analysis: Problem scenarios are more likely to involve negations, negative words, and interrogatives, while non-problem scenarios involve more task-related nouns and verbs.

• Limitations and Future Directions:

  1. The sample size is relatively small; future studies could expand the sample to validate the generalizability of the results.
  2. Older adults with physical or cognitive impairments were not included, which may influence language patterns.
  3. The range of test products was limited; future research should extend to a broader variety of product types.
  4. Proposes further exploration of the role of multimodal data, such as facial expressions and eye-tracking, in UX analysis.

Conclusion

• Research Significance: Confirms the impact of age on the indicative language and speech features of UX problems, providing a reliable basis for designing human-computer collaborative tools for UX problem diagnosis.
• Practical Implications: Recommends focusing on observation-related language and negative emotions in UX analysis, while designing AI-assisted tools to accelerate problem detection and reduce evaluators’ cognitive load.