Apéritif: Scaffolding Preregistrations to Automatically Generate Analysis Code and Methods Descriptions

Title of the Paper

Apéritif: Scaffolding Preregistrations to Automatically Generate Analysis Code and Methods Descriptions

Paper Information

• Subject Area: Human-Computer Interaction (HCI), Open Science, and Experimental Design Tools
• Keywords: Preregistration, Experimental Design, Reproducibility, Data Analysis, Open Science, HCI, Methodology, Scientific Integrity

Research Background and Problem

• Problems and Challenges:
  • The adoption rate of preregistration in the field of Human-Computer Interaction (HCI) remains low, with only about 1.11% of papers involving preregistration. The level of detail in preregistrations and their consistency with the final papers are often insufficient.
  • Existing preregistration platforms (e.g., AsPredicted and OSF) lack clarity and integration in their template designs, making it difficult for users to determine the appropriate level of detail.
  • Changes during the experimental design and research process can lead to discrepancies between preregistrations and final papers, with a lack of version control and flexibility.
• Significance:
  • Preregistration is a key practice for enhancing transparency and reproducibility in scientific research. It reduces research vulnerabilities (e.g., HARKing: hypothesizing after results are known) and increases the credibility of studies.
  • However, the low adoption rate and insufficient formalization of preregistration hinder the realization of these potential benefits in the HCI domain.

Solution

• Methods and Tools: A research prototype tool called Apéritif was developed. It is a Chrome browser extension-based preregistration platform integrated with AsPredicted.
  • It structures and guides the preregistration process through scaffolding using question decomposition, interactive specification, and visualization tools.
  • It provides automation features, including the automatic generation of analysis code (supporting Python and R) and methodological descriptions.
  • It integrates a version control system (via GitHub API) to track changes in preregistration files.
• Innovations:
  • Introduces "scaffolded guidance" to frame and refine user input, improving the precision and informativeness of preregistration forms.
  • Incorporates version control to address changes during the research process while ensuring consistency with preregistration content.
  • Automatically generates analysis scripts and methodological sections as ready-to-use research tools, reducing cognitive load and time costs for users.
• Implementation Steps and Techniques:
  • Provides standardized templates and tools related to variable definitions, study design, hypothesis generation, and sample size calculations.
  • Utilizes the Tea framework to automatically select statistical analysis methods based on study design.
  • Tracks file changes and integrates the research process using GitHub.

Research Outcomes

• Specific Findings:
  • Empirical studies reveal that the adoption rate of preregistration in the CHI field is low, and existing preregistration templates lack sufficient detail and consistency with final papers.
  • Apéritif significantly reduces the time and cognitive burden during the experimental design process, enhancing research transparency and reproducibility.
  • Researchers who used Apéritif were more likely to adopt preregistration in future studies.
• Advantages:
  • Compared to traditional tools and templates, Apéritif's scaffolded structure and automation significantly lower the barrier to use.
  • The scaffolded guidance transforms preregistration from a "form-filling" task into a more interactive and educational process, increasing users' confidence in the scientific rigor of their research plans.
• Experimental and Evaluation Results:
  • Users of Apéritif spent less time completing preregistrations (approximately 11.25 minutes, about 37% faster than using AsPredicted).
  • The majority of evaluation participants (94%) believed the tool enhanced the rigor and transparency of their research plans, and about 82% indicated they would recommend the tool.
  • The automatically generated analysis code and methodological descriptions were highly praised by participants as powerful time-saving and learning support tools.
• Limitations and Future Directions:
  • Currently, the tool primarily supports quantitative research (particularly the NHST paradigm); support for qualitative and exploratory research is limited.
  • There is room for improvement in handling multivariable interactions and more complex experimental designs.
  • Future work could expand Apéritif to support more diverse analytical methods (e.g., Bayesian analysis) and interdisciplinary applicability.
  • Further exploration is needed to address the relationship between preregistration files and changes during the research process.

Summary and Discussion

This study diagnoses and addresses the low adoption rate and deficiencies in current preregistration practices within the HCI community, providing a viable technical solution to enhance research transparency and credibility. Apéritif's interactive tools and automation features introduce innovative approaches to planning and reporting scientific research while reducing researchers' time and cognitive costs. This work offers valuable directions for future preregistration practices in qualitative and diverse research contexts.