Are You Open? A Content Analysis of Transparency and Openness Guidelines in HCI Journals

Title of the Paper

Are You Open? A Content Analysis of Transparency and Openness Guidelines in HCI Journals

Paper Information

• Research Domain: Transparency and openness guidelines in the field of Human-Computer Interaction (HCI)
• Keywords: Editorial practices, Transparency and Openness Promotion Guidelines (TOP), Open science, HCI, Open data, Peer review policies

Research Background and Questions

• Problems or Challenges:
  1. In the context of the open science reform movement, the HCI field faces challenges regarding how to enhance transparency and openness.
  2. Publication venues, particularly journals, play a critical role in promoting open science practices, yet the extent to which HCI journals support open science practices remains largely unknown.
  3. The needs for open science vary across research domains, and some standards may not be applicable to the multidisciplinary nature of HCI.
• Significance:
  • Open science practices (e.g., data sharing, pre-registration of experiments) can significantly improve the credibility and transparency of research, especially in addressing issues of irreproducible results.
  • The lack of comprehensive transparency and openness standards in some HCI journals may negatively impact the credibility of research.
• Research Motivation:
  • The authors aim to understand the extent to which highly cited journals in the HCI field support Transparency and Openness Promotion (TOP) Guidelines, particularly focusing on editorial policies that emphasize open practices.

Solution

• Proposed Methods or Solutions:
  1. Analyze the TOP Guidelines to quantify the performance of highly cited HCI journals in supporting transparency and openness.
  2. Select publication data from first authors of CHI (Conference on Human Factors in Computing Systems) papers post-2016 to identify 51 frequently published journals as the sample.
  3. Use the 10 modules of the TOP Guidelines (e.g., data transparency, analytic methods transparency, research materials transparency) to code the editorial policies of the journals.
• Innovations:
  • Combining TOP Guidelines with the actual performance of HCI journals, this study provides the first systematic analysis of the consistency and implementation of open science standards in the field.
  • The authors propose using TOP scores as an alternative evaluation metric for assessing journal support for open science practices.
• Implementation Steps:
  1. Construct the sample set: Survey CHI first authors to identify journals frequently published in and filter journals meeting specific criteria (e.g., peer-reviewed, accessible author guidelines).
  2. Code journal transparency: Two researchers independently score each journal based on the TOP Guidelines and calculate inter-rater reliability (Cohen’s κ coefficient).
  3. Discuss and resolve discrepancies in coding results to reach consensus.
  4. Summarize the transparency and openness scores (TOP Factor) for each journal, with a maximum score of 29.

Research Findings

• Specific Results:
  1. The average TOP Factor of the sampled journals was 2.5 (SD = 3.6, maximum score = 15), with most journals scoring 0, indicating insufficient support for open science practices in the HCI field.
  2. Data citation transparency was the most common policy, with 51% of journals scoring on this standard.
  3. Standards such as pre-registration of analyses and experimental design transparency scored the lowest, with less than 5% of journals having relevant guidelines.
  4. Only six journals scored above 4 points, with Frontiers in Psychology and PLOS One achieving the highest scores.
• Advantages Compared to Existing Solutions:
  • Provides a systematic and efficient analysis of editorial policies regarding transparency and openness practices within the HCI community.
  • Offers the TOP Factor as an alternative metric for HCI practitioners to prioritize transparency when selecting journals for submission.
• Experimental or Evaluation Results:
  • Results indicate significant room for improvement in areas such as data standardization, open materials, and analytic methods within HCI journals.
  • Most HCI journals fail to meet core practices set by publishers (e.g., COPE) regarding data and reproducibility requirements.
• Limitations and Future Directions:
  • Limitations:
    1. Certain TOP standards (e.g., replicability and reporting guidelines) showed low coding consistency, suggesting the need for further refinement of scoring mechanisms.
    2. Some journals in the sample are not core HCI journals, which may impact the construct validity of the results.
  • Future Directions:
    1. Develop more tailored openness standards for specific HCI subfields, particularly for qualitative and engineering research.
    2. Encourage journals to improve the visibility and usability of guidelines to reduce the difficulty for researchers in adhering to transparency standards.
    3. Explore how transparency standards can be implemented in conference processes (e.g., CHI annual meetings).

Data and Code Availability

• Project data, code sets, and analysis scripts are available on Open Science Framework (https://osf.io/ck7em/).

Funding Information

This research is funded by:

1. UK Engineering and Physical Sciences Research Council (EPSRC) / Media and Arts Technology-related projects.
2. Digital Creativity Labs (funded by EPSRC/AHRC).