Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022

Title of the Paper

"Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022"

Paper Information

• Subject Area: Human-Computer Interaction (HCI), focusing on ethics, openness, and transparency in academic research.
• Keywords: Reproducibility, Replicability, Transparency, Ethics, Open Science, Data Availability, CHI.

Research Background and Issues

• Identified Problems or Challenges:

  1. While academia has increasingly emphasized promoting research ethics, openness, and transparency, the degree of implementation in practice remains unclear.
  2. Some studies face challenges such as resource limitations, imbalanced incentive mechanisms, and conflicts between ethics and transparency in certain research contexts.
  3. Existing research often relies on self-reported data from authors, lacking comprehensive evaluations of published papers.

• Significance:

  1. Academic Impact: Research ethics, openness, and transparency are crucial components of scientific reproducibility and social responsibility.
  2. Community Standards: Analyzing papers published at the top-tier CHI (Human-Computer Interaction Conference) can provide feedback to the HCI community, promoting higher standards in research practices.

• Research Motivation and Related Work:

  1. Previous studies have focused on specific topics such as statistical transparency, sample size reporting, and data sharing.
  2. A comprehensive evaluation of improvements in ethics, openness, and transparency among HCI researchers is needed.
  3. This study addresses gaps in existing research, providing data support and guidance for HCI research practices.

Solution

• Proposed Methods or Solutions:

  1. Define and operationalize 45 specific evaluation criteria for research ethics, openness, and transparency.
  2. Analyze 118 papers from CHI 2017 and 127 papers from CHI 2022.
  3. Develop a validation screening tool to explore the potential of automated technologies in supporting evaluation criteria.

• Innovations:

  1. Comprehensive evaluation criteria covering four main aspects: research design, data collection, data analysis, and results reporting.
  2. Propose an automated system combining natural language processing technology (BERT sentence similarity) to assess whether papers meet specific evaluation criteria.
  3. Provide preliminary empirical analysis of conflicts between ethics and transparency.

• Key Technologies:

  1. BERTScore for calculating sentence similarity to identify sentences that meet specific evaluation criteria.
  2. Zero-shot Classifier for further filtering content that satisfies the criteria.

Research Findings

• Specific Findings:

  1. Research Ethics:
     • CHI'22 showed significant improvements in ethical practices (e.g., ethical review, informed consent, participant compensation), reaching 50%-57%.
     • Approximately 29% of studies involved vulnerable populations, but their transparency (e.g., data sharing) lagged significantly.
  2. Openness:
     • CHI'22 saw progress in open access and data sharing, but only 62% of papers included any additional research materials.
     • Improvements in sharing practices following FAIR principles were not significant.
  3. Transparency:
     • Sharing of interview guides, qualitative data, and analysis code improved significantly (e.g., sharing of interview guides increased from 2% in CHI'17 to 25% in CHI'22).
     • Transparency in quantitative data and quantitative sample descriptions (e.g., reporting of statistical hypothesis tests) still requires improvement.
  4. Data indicates that ethical constraints may indeed limit data transparency in some papers.

• Comparison with Existing Research and Advantages:

  1. This study is more specific and comprehensive: it provides a fine-grained evaluation of actual papers.
  2. Compared to existing self-reported data from authors (e.g., higher transparency reporting rates), this study reflects more realistic practices.
  3. It is the first to explore potential trade-offs between ethics and transparency.

• Future Directions:

  1. Develop more detailed transparency standards tailored to HCI-specific research methods.
  2. Encourage academic conferences like SIGCHI to implement more specific review guidelines, making ethics and transparency evaluations part of the author and reviewer workflow.
  3. Further develop machine learning-based automated tools to enhance the ability to check research ethics, openness, and transparency.
  4. Investigate researchers' trade-offs between data openness and privacy protection in greater depth.

Output Format

• Clear and information-rich format, suitable as a reference for improving practices within the HCI research community, authors, and reviewers.
• Emphasizes actionable recommendations for improving research practices, supported by data-driven conclusions.