How WEIRD is CHI?

Paper Title

How WEIRD is CHI?

Paper Information

• Authors: Sebastian Linxen, Vincent Cassau, Christian Sturm, Klaus Opwis, Florian Brühlmann, Katharina Reinecke
• Publication Year: 2021
• Conference: CHI Conference on Human Factors in Computing Systems (CHI `21), Yokohama, Japan
• Research Area: Sample representativeness and internationalization in HCI research
• Keywords: WEIRD, sample bias, generalizability, HCI research, geographic diversity

Research Background and Problem Statement

• Problem or Challenge:

  • Participant samples in CHI research predominantly come from Western societies (Western, Educated, Industrialized, Rich, Democratic, abbreviated as WEIRD), which account for only 12% of the global population. This bias may hinder the generalization of research findings to non-WEIRD societies.
  • The lack of globally representative research may limit the inclusivity and generalizability of technology design.

• Significance:

  • Technology users span the globe, yet research focuses on a narrow sample population, potentially leading to designs that fail to reflect the needs of the majority worldwide.
  • Issues of generalizability may impact the effectiveness of technology and social equity.

• Research Motivation and Related Work:

  • Behavioral sciences have long faced generalizability issues due to WEIRD samples, significantly influencing research conclusions in psychology, sociology, and related fields.
  • Similar challenges are prominent in HCI, with existing literature expressing concerns about uneven geographic sample distribution but lacking systematic analysis.
  • This study aims to quantify sample representativeness in CHI conferences, analyze the status of underrepresented countries and regions, and propose recommendations to enhance diversity.

Proposed Solution

• Method or Solution:

  • Employ the WEIRD framework proposed by Henrich et al. to conduct a content analysis of 2,768 papers published at CHI from 2016 to 2020, quantifying the geographic distribution and demographic characteristics of participant samples.
  • Analyze the Western attributes (Western), education level (Educated), industrialization (Industrialized), wealth (Rich), and democracy (Democratic) of sample countries, comparing sample proportions to global population data.

• Innovation:

  • Provides the first quantitative analysis of CHI sample geographic scope, explicitly highlighting significant sample bias.
  • Focuses not only on the overall WEIRD framework but also dissects individual variables to reveal finer sample characteristics.

• Implementation Steps and Key Techniques:

  • Manually encode participant sample information from CHI papers, extracting variables such as country, education, and income.
  • Standardize sample comparisons using data from organizations like the United Nations and World Bank to identify overrepresented or underrepresented countries.
  • Apply statistical methods such as Kendall rank correlation analysis to evaluate sample bias.

Research Findings

• Specific Findings:

  • Geographic Distribution Analysis: Over the past five years, 73% of research samples came from WEIRD countries, with the United States accounting for the highest proportion (45.82%), while 102 countries worldwide were not represented in the research.
  • Individual Variable Analysis:
    • Education: Approximately 70% of participants had a university degree or higher, significantly above the global average (8.4 years of schooling).
    • Industrialization and Wealth: Sample countries had GDP and GNI levels markedly higher than the world average.
    • Democracy: The vast majority of samples originated from highly democratic countries.
  • Trend Changes: The proportion of non-Western samples increased from 16.31% in 2016 to 30.24% in 2020.

• Advantages:

  • Provides a systematic framework and quantified data, enabling future research to track changes in sample diversity.
  • Highlights the issue of insufficient geographic diversity, fostering efforts to address generalizability challenges.

• Experimental or Evaluation Results:

  • Geographic diversity of samples is somewhat correlated with the CHI conference location but is primarily driven by academic preferences.
  • The increase in non-Western samples is partly attributed to online research methods, such as behavioral log analysis and cross-national experiments.

• Limitations and Future Directions:

  • Limitations: The study focuses solely on the WEIRD framework and does not consider other dimensions of participant identity; detailed personal information about many participants is difficult to obtain.
  • Suggestions for future work include:
    • Comparing data distributions from other academic conferences and journals.
    • Exploring sample types and individual participant differences.
    • Improving the standardization of sample statistics in research reporting.

Recommendations for Addressing WEIRD Sample Bias

• Increase the involvement of researchers from non-WEIRD countries to encourage cross-national collaboration and local sample recruitment.
• Support online research to reach a more diverse global participant pool.
• Develop methods tailored to cross-national sample studies and share experience reports to reduce barriers in cross-cultural research.
• Encourage replication and expansion studies of research findings.
• Clearly report sample attributes in papers to deepen awareness of generalizability issues.

Through this paper, the researchers have identified the issue of sample representativeness in the CHI field and called for more efforts in the HCI domain to achieve a global research perspective. The findings provide robust data support and strategic recommendations for expanding sample diversity in future studies.