Better Assumptions, Stronger Conclusions: The Case for Ordinal Regression in HCI

Paper Title

Better Assumptions, Stronger Conclusions: The Case for Ordinal Regression in HCI

Publication Info

• Topic area: Statistical methods for ordinal data analysis in HCI research
• Keywords: Ordinal data, Likert scales, HCI, statistical analysis, cumulative link models, ordinal regression, parametric methods, non-parametric methods, CLMM, CLM

Background and Problem

• Problem / challenge: There is a lack of consensus in HCI research on appropriate statistical methods for analyzing ordinal data, leading to inconsistent practices and potential errors in insights.
• Significance: Ordinal data, such as Likert-item responses, are widely used in HCI to measure subjective phenomena. Proper analysis is crucial for deriving accurate and reproducible insights.
• Motivation and related work: Prior work has highlighted the limitations of parametric methods (e.g., ANOVA) and non-parametric methods (e.g., Friedman test) for ordinal data, particularly due to metric assumptions. However, there has been limited adoption of more suitable methods like cumulative link (mixed) models (CLM/CLMM) in HCI.

Solution

• Proposed approach: The paper advocates for the use of cumulative link (mixed) models (CLM/CLMM) for analyzing ordinal data in HCI, leveraging their ability to treat ordinal data as categorical while accounting for inherent ordering.
• Novelty:
  1. A comprehensive review of statistical practices for ordinal data in HCI, identifying inconsistencies and limitations.
  2. Detailed explanation of the theory and mathematical foundations of CL(M)Ms.
  3. Practical demonstrations of CL(M)Ms using open-sourced HCI datasets and R software.
  4. Recommendations for reporting and interpreting CL(M)M results.
• Procedure and key techniques:
  • Review of 94 papers from CHI 2024 to assess statistical methods used for ordinal data.
  • Explanation of CL(M)Ms, including their latent variable model, cutpoints, and assumptions.
  • Reanalysis of two HCI datasets using CL(M)Ms to illustrate their application and advantages over traditional methods.

Results

• Concrete findings:
  • Non-parametric methods outnumber parametric methods in HCI (362 vs. 165 tests), but many impose metric assumptions (e.g., ART-ANOVA, Wilcoxon Signed Rank test).
  • CL(M)Ms are rarely used (10 instances out of 28 predictive models in the sample).
  • Reanalysis of datasets using CL(M)Ms revealed significant effects missed by traditional methods (e.g., learning condition effects on SIM-TLX sub-scales).
• Advantage over baselines:
  • CL(M)Ms provide greater statistical power than non-parametric methods and avoid the incompatible assumptions of parametric methods.
  • They appropriately model ordinal data as categorical with ordering, preserving information without imposing metric assumptions.
• Experiments / evaluation:
  • Review of 94 CHI papers to quantify statistical practices.
  • Reanalysis of datasets from Fitton et al. and Chen et al. using CL(M)Ms in R.
  • Comparison of CL(M)M results with traditional methods like ANOVA and Friedman tests.
• Limitations and future work:
  • CL(M)Ms require assumptions about latent variables, proportional odds, and equal variance, which must be tested and relaxed if necessary.
  • Limited applicability to ordinal data transformed into interval/ratio scales or with a large number of categories.
  • Future work could explore Bayesian implementations of CL(M)Ms and address challenges in applying these models to complex HCI datasets.

Summary

This paper highlights the inconsistent use of statistical methods for ordinal data in HCI and advocates for cumulative link (mixed) models (CL(M)Ms) as a more suitable alternative. Through a review of CHI papers and reanalysis of open-sourced datasets, the authors demonstrate that CL(M)Ms avoid the limitations of parametric and non-parametric methods by treating ordinal data as categorical with ordering. While CL(M)Ms require specific assumptions, their advantages in preserving data integrity and improving statistical power make them a valuable tool for HCI researchers. The paper provides practical guidance for applying and interpreting CL(M)Ms, aiming to standardize and improve ordinal data analysis in the field.