Standardizing Participant Compensation Reporting in HCI: A Meta-Review and Recommendations for the Field

Title of the Literature

Standardizing Reporting of Participant Compensation in HCI: A Systematic Literature Review and Recommendations for the Field

Literature Information

• Subject Area: Research and standardization of participant compensation strategies in Human-Computer Interaction (HCI)
• Keywords: meta-review, participant payment, HCI, CSCW, UBICOMP/IMWUT, UIST

Research Background and Issues

• Identified Problems or Challenges:

  • Participant compensation in HCI research lacks standardization and transparency.
  • Reporting on compensation strategies varies significantly and is often insufficient.
  • Inconsistent reporting of participant compensation may exacerbate existing ethical and reproducibility issues in science.

• Significance:

  • Participant compensation is crucial for defining ethical research practices and ensuring scientific reproducibility, yet the HCI field lacks systematic guidance.

• Research Motivation and Related Work:

  • Research on compensation strategies in HCI has primarily focused on micro-incentive schemes and motivational mechanisms, with limited systematic analysis of compensation levels.
  • Best practices for participant compensation have been established in fields like medicine and psychology, offering valuable insights for HCI.

Solution

• Methods or Solutions:

  • Systematic literature review: Analysis of user studies presented in four major HCI conferences (CHI, CSCW, Ubicomp/IMWUT, and UIST) from 2018-2019.
  • Structuring article evaluation using the PRISMA method.
  • Qualitative analysis of the rationale and frameworks for compensation.

• Innovative Contributions:

  • Addressing the lack of standardized frameworks for reporting participant compensation, the study provides recommendations and templates to standardize this practice.
  • Identifying common factors in participant compensation decisions, ethical considerations, and cultural context.
  • Advocating for a unified structure for reporting compensation to enhance transparency and standardization within the research community.

• Implementation Steps and Key Techniques:

  • Identification and classification of compensation data: Development of 17 categories (including participant count, activity type, compensation amount, and payment methods).
  • Data extraction and coding: Iterative analysis to extract compensation information and generate classification tables.
  • Data analysis: Quantitative analysis of compensation structures, amounts, and payment methods; qualitative analysis of compensation rationale.

Research Findings

• Specific Findings:

  • The literature review revealed that 57.9% of papers did not report whether compensation was provided, and 84.2% of studies did not sufficiently report compensation-related information.
  • Participant compensation amounts ranged from $0.05 to $700, hourly rates ranged from $2.86 to $240, and compensation methods included cash, gift cards, crowdsourcing platforms, etc.
  • Qualitative analysis identified six main categories of compensation rationale, including influence from previous studies, adherence to minimum wage standards, participant expertise, recommendations from external organizations, among others.

• Advantages of Existing Solutions:

  • This study is the first to systematically analyze participant compensation reporting in HCI and provide decision-making recommendations.
  • It offers comprehensive factors to consider when designing compensation strategies (e.g., cultural context, minimum wage standards), enhancing the rationality and reproducibility of compensation design.

• Experimental or Evaluation Results:

  • Various compensation models (e.g., fixed amount, task-based compensation, time-based calculation) were listed and illustrated with specific research cases.
  • Significant differences in compensation levels were observed across research locations and participant characteristics (e.g., vulnerable groups).

• Limitations and Future Directions:

  • Limitations: The study only covers data from 2018-2019, which may not fully reflect long-term historical trends; manual data extraction may introduce errors.
  • Future Work Suggestions:
    • Standardize compensation reporting methods to improve transparency.
    • Enhance participant involvement in compensation decision-making.
    • Evaluate the impact of compensation methods on participant behavior and research objectives.

Conclusion

This study contributes a comprehensive systematic literature review to the HCI field, analyzing the current state of participant compensation strategies and proposing recommendations for standardization. The findings not only highlight the diversity in compensation methods but also emphasize the importance of compensation in ethical design and research transparency. By adopting standardized practices, the HCI community can improve transparency in compensation decisions and enhance overall research quality.