Diagnosing Bias in the Gender Representation of HCI Research Participants: How it Happens and Where We Are

Document Title

Diagnosing Bias in the Gender Representation of HCI Research Participants: How it Happens and Where We Are

Document Information

• Subject Area: Gender representation and bias in HCI research
• Keywords: Gender, gender bias, user research, participants, HCI, datasets, meta-analysis, human-computer interaction, CHI, data structures

Research Background and Issues

• Problems or Challenges:

  • The imbalance in gender representation among HCI research participants may affect the generalizability of research findings, reinforce biases in data-driven practices, and pose potential risks to underrepresented groups.
  • Lack of attention to non-binary gender participants and the unrecognized phenomenon of data silence.
  • Insufficient understanding of the sources of gender bias in existing research.

• Importance of the Problem:

  • Failure to fairly represent different genders in technology design may result in new technologies being unsuitable or harmful to certain groups.
  • Diagnosing and addressing gender bias is critical for promoting inclusivity and equity in the HCI field.

• Research Motivation and Related Work:

  • Previous studies have shown that demographic characteristics of participants in HCI user research tend to skew male but have not identified specific variables contributing to bias.
  • By combining qualitative interviews and large-scale data analysis, this study aims to uncover potential factors behind participant gender bias and propose solutions.

Solutions

• Proposed Solutions:

  • Defined systematic data collection guidelines and data structures for extracting participant gender data from HCI research.
  • Provided extensive gender bias statistics through the analysis of 1,147 CHI (Conference on Human Factors in Computing Systems) papers.

• Innovations:

  • Proposed a data structure for gender and related variables, enabling the capture of complexity and flexibility in data-driven analysis.
  • Developed a tool (MAGDA) for extracting gender data, facilitating efficient and accurate data collection.
  • Used quantitative metrics (e.g., DER) to compare male and female representation in each paper.

• Implementation Steps and Key Techniques:

  • Qualitative Phase: Conducted interviews with 13 HCI researchers to identify potential variables (e.g., participant recruitment sources, research domains) associated with gender bias.
  • Data Collection: Designed a data structure and collected data from CHI conference papers published between 1981 and 2020.
  • Quantitative Analysis: Performed statistical tests on participant gender data, such as using DER to measure the degree of gender representation disparity in studies.

Research Findings

• Specific Findings:

  • Persistent underrepresentation of women in HCI research and the invisibility of non-binary gender participants.
  • Identified three key variables associated with gender bias: participant recruitment sources, methods of reporting gender data, and research domains.
  • Provided a gender dataset generated from 1,147 CHI papers, including gender reporting, participant numbers, and recruitment data.

• Advantages Over Existing Solutions:

  • Delivered a more comprehensive and detailed gender data analysis than previous studies (addressing both women and non-binary genders).
  • Offered a systematic data collection method and tools to improve efficiency and reduce errors.

• Experimental or Evaluation Results:

  • Trends showed no significant increase in female participant representation, while non-binary gender participants remained overlooked.
  • Over time, the proportion of male participants recruited via Amazon MTurk increased, exacerbating gender bias.
  • Certain research domains (e.g., programming tools, virtual environments) leaned toward male representation, while socially relevant domains (e.g., family and community) had higher female representation.

• Limitations and Future Directions:

  • Data collection relied on reported data in existing papers, constrained by reporting practices and terminology, and could not fully avoid a binary gender perspective.
  • Future research should explore variables related to non-binary genders and further optimize recruitment and gender reporting standards.
  • Recommended supplementary analysis of methodologies in HCI research to investigate interactions between methods and gender representation.

Recommendations and Actions

• Recommended Measures:

  • Encourage researchers to report gender and recruitment source information when documenting participant data.
  • Organize capacity-building workshops on gender inclusivity and equity to promote standardized practices across research domains.
  • The HCI community should implement comprehensive survey programs to identify participation barriers for all gender groups.

• Research Impact:

  • Provides a practical framework for understanding gender bias in HCI research.
  • Advances the improvement of population representation in the HCI field and enhances the fairness and inclusivity of new technologies.