On the Computational Reproducibility of Human-Computer Interaction

Paper Title

On the Computational Reproducibility of Human-Computer Interaction

Publication Info

• Topic area: Computational reproducibility in Human-Computer Interaction (HCI) research.
• Keywords: Computational reproducibility, HCI, open science, CHI papers, data sharing, analysis code, guidelines, reproducibility rates, research transparency.

Background and Problem

• Problem / challenge: Despite increasing adoption of open science practices in HCI, reproducibility rates remain low, with only 49% of analyzed CHI papers being fully reproducible. There is a lack of clear guidelines, incentives, and community support for reproducible research.
• Significance: Reproducibility is essential for verifying research credibility, enabling reuse of materials, and fostering scientific progress. Low reproducibility rates hinder trust in findings and the ability to build upon prior work.
• Motivation and related work: Prior studies in other fields (e.g., psychology, political science) have reported similarly low reproducibility rates. HCI has unique challenges due to its interdisciplinary nature, rapid technological evolution, and limited adoption of transparency guidelines. Existing efforts, such as preregistrations and artifact badging systems, are insufficiently widespread.

Solution

• Proposed approach: Systematic evaluation of computational reproducibility in CHI papers, combined with surveys and interviews to understand author perspectives and obstacles. Development of practical recommendations for creating reproducible repositories.
• Novelty:
  1. Comprehensive analysis of 76 CHI papers with open repositories.
  2. Identification of barriers to reproducibility through surveys and interviews.
  3. Creation of detailed guidelines for improving reproducibility in HCI research.
  4. Advocacy for community-level changes to incentivize reproducible practices.
• Procedure and key techniques:
  • Extracted repository links from CHI papers (2007–2024) hosted on GitHub, OSF, and Zenodo.
  • Attempted reproduction of reported results using provided data and code.
  • Analyzed repositories based on dimensions such as documentation quality, data availability, and code executability.
  • Conducted surveys and interviews with authors to gather insights on motivations, challenges, and perceived reproducibility.
  • Synthesized findings into actionable recommendations for researchers and the HCI community.

Results

• Concrete findings:
  • 49% of CHI papers with open repositories were fully reproducible, 23% partially reproducible, and 28% non-reproducible.
  • Common issues included missing data (13 repositories), incomplete documentation (21 repositories), and persistent code errors (32 repositories).
  • Survey responses showed 60% of authors believed their work was fully reproducible, but actual reproducibility rates were lower.
• Advantage over baselines: CHI reproducibility rates (49%) are higher than those reported in Cognition (31%) and Science (26%), but lower than registered reports in psychology (58%).
• Experiments / evaluation:
  • Reproduction attempts followed a systematic workflow, analyzing repositories for documentation, data clarity, and code executability.
  • Survey of 63 authors (response rate: 59%) and interviews with 4 authors of exemplary repositories provided qualitative insights.
• Limitations and future work:
  • Reproducibility attempts may have been constrained by coding expertise and lack of author assistance.
  • Self-selection bias in survey participants limits generalizability.
  • Future work could explore motivations of researchers who do not share materials and develop automated tools for reproducibility checks.

Summary

This study systematically evaluated computational reproducibility in CHI papers, finding that only 49% of papers with open repositories were fully reproducible. Key barriers included missing data, incomplete documentation, and code errors. Surveys and interviews revealed that authors often lack training, guidelines, and incentives to prioritize reproducibility. The paper provides detailed recommendations for creating reproducible repositories and advocates for community-level changes, such as clearer guidelines, artifact evaluation committees, and recognition for reproducible practices. These findings aim to strengthen research credibility and foster a culture of transparency in HCI.