A Qualitative Study on How Usable Security and HCI Researchers Judge the Size and Importance of Odds Ratio and Cohen's d Effect Sizes

Research Background and Problem

• What issues or challenges did the authors identify?

  1. In the fields of Human-Computer Interaction (HCI) and Usable Security and Privacy (USP) research, researchers often focus excessively on statistical significance (p-values) while neglecting the interpretation and reporting of effect sizes.
  2. Although effect sizes (e.g., Cohen's d and odds ratios) are crucial metrics for evaluating practical effects, they are rarely systematically reported or interpreted. This deficiency impacts the practical utility of research findings and their relevance to real-world applications.

• Why is this issue important?

  • Effect size analysis helps in understanding the practical significance of statistical results. However, if research literature fails to clearly report or interpret effect sizes, readers may misinterpret the findings or struggle to apply them in practice.
  • In security and privacy domains (e.g., password managers and browser warnings), misunderstandings and misjudgments can lead to negative consequences, including poor user decisions and actual harm.

• Research Motivation and Related Work

  1. The authors referenced prior studies on effect sizes, p-values, and statistical reporting, highlighting the widespread issues in reporting and interpreting effect sizes in HCI and USP fields.
  2. Existing literature predominantly focuses on p-value-driven significance while paying little attention to misunderstandings and practical implications of effect sizes.
  3. Addressing these issues can enhance research transparency and practical value, as well as strengthen the scientific foundation for designing and evaluating tools.

Solutions

• What methods or solutions did the authors propose?

  1. During the 2023 CHI and SOUPS conferences, the authors conducted surveys and in-depth interviews to explore HCI and USP researchers' understanding and misconceptions about effect sizes (primarily Cohen’s d and odds ratios).
  2. They designed various scenarios (e.g., password managers, slide consistency check tools, and browser warnings) with different effect size magnitudes to observe researchers' judgments.
  3. Based on the survey and interview results, they proposed specific recommendations for improving the presentation and interpretation of effect sizes in research reporting.

• What are the innovative aspects of the solution?

  1. The study systematically investigated researchers’ subjective judgments about the magnitude and importance of effect sizes in the USP and HCI fields.
  2. It identified specific misunderstandings associated with Cohen's d and odds ratios.
  3. The authors proposed a framework for interpreting effect sizes that integrates context, scope of impact, and user perspectives.

• What are the implementation steps? What key techniques were used?

  1. Data Collection: Surveys and interviews with 63 researchers were conducted to explore how they assess the magnitude and importance of effect sizes using both qualitative and quantitative methods.
  2. Scenario Design: Artificially constructed research scenarios were used to present descriptive statistics, p-values, and effect sizes. Effect size magnitudes were randomly assigned across different scenarios.
  3. Analysis: A constructivist grounded theory approach was applied to analyze interview transcripts and survey results, extracting patterns of misunderstanding and factors influencing judgments.

Research Findings

• What specific findings were obtained?

  1. Researchers exhibited widespread misunderstandings of effect sizes such as Cohen’s d and odds ratios, including unclear range definitions and incorrect semantic interpretations (e.g., misinterpreting odds ratios as probability ratios).
  2. Researchers’ judgments of effect size magnitude and practical importance were inconsistent with Cohen’s standards and were easily influenced by context and other statistical values (e.g., p-values).
  3. No significant differences were observed between researchers from HCI and USP backgrounds in their assessments of effect importance.

• What advantages does this solution have compared to existing ones?

  1. The authors proposed comprehensive recommendations for improving effect size reporting, including integrating contextual scenarios and explaining the reasons for practical impacts.
  2. They introduced a dual-track approach combining standardized effect size reporting with direct numerical effect size reporting to enhance intuitive understanding for readers.
  3. They suggested using data visualization and confidence intervals to better interpret the uncertainty of statistical results.

• What were the experimental or evaluation results?

  1. Over one-third of participants in the interviews and surveys demonstrated basic misunderstandings of effect sizes, particularly in interpreting odds ratios.
  2. Researchers’ "thresholds of importance" varied significantly across different scenarios, indicating a lack of consistency in effect size judgment and a strong dependence on context.
  3. The proposed recommendations were grounded in existing research methodologies and reporting practices and received broad support from participants and statistical experts.

• Limitations and Future Directions

  1. Limitations:
     • The study sample was limited, primarily consisting of researchers attending CHI and SOUPS conferences, which may not be fully representative.
     • Only two types of effect sizes (Cohen’s d and odds ratios) were studied, leaving other effect sizes and complex statistical practices unexplored.
  2. Future Directions:
     • Extend research to other effect sizes and explore their applicability and patterns of misunderstanding in various fields (e.g., medicine or education).
     • Develop interactive teaching tools and templates to help researchers better understand and report effect sizes.
     • Evaluate the impact of the proposed recommendations on the quality of HCI and USP research papers, such as their transparency and reproducibility within these fields.