Women Security Experts Are Not The Enemy: A Qualitative Study on Gender-Related Communication ChallengesEffective communication is crucial for meeting security needs, yet gender-related communication challenges faced by women security experts within software development remain underexplored. In an interview study with 25 women security experts, we investigated gender-related communication challenges hindering the adoption of security requirements, and strategies to overcome these. Key challenges included the undervaluation of women’s security expertise, communication barriers, resistance to women’s security-related suggestions, and instances of hostility. Communication challenges with stakeholders who were men disrupted team collaboration, resulting in delays, weakened security measures, and increased organizational risk. Consequently, women security experts often had to adopt strategies, such as leveraging allied men and overpreparing, to assert their security competence. We further offer insights into women’s participation in security studies. Based on our findings, we provide recommendations on how to address gender-related challenges.2025AYAsli Yardim et al.Ruhr University BochumGender & Race Issues in HCITechnology Ethics & Critical HCICHI
Small, Medium, Large? A Meta-Study of Effect Sizes at CHI to Aid Interpretation of Effect Sizes and Power CalculationStatistical reporting, especially of effect sizes, is at the root of many methodological issues in quantitative research at CHI. Effect sizes are necessary for assessing practical relevance of results, a-priori power analysis, and meta-analyses, but currently, they are often not reported. Interpretations in the context of the study and the research field are also rare. To aid to researchers in reporting and contextualizing their effect sizes within their research field as well as choosing effect sizes for power analysis, we conducted a meta-study of quantitative CHI papers. We extracted statistics from all quantitative CHI papers published between 2019-2023 (N=1692). Based on effect sizes and the papers' CCS categories, we present effect size distributions in 12 CHI research fields. Through an additional qualitative analysis of 67 quantitative CHI'23 publications, we identify five categories of approaches that researchers take when interpreting effect size: Comparing test-specific values, assigning size labels, using a statistical or methodological reference frame, comparing different observations and interpreting for the big picture.2025AOAnna-Marie Ortloff et al.University of BonnUser Research Methods (Interviews, Surveys, Observation)Computational Methods in HCIResearch Ethics & Open ScienceCHI
A Qualitative Study on How Usable Security and HCI Researchers Judge the Size and Importance of Odds Ratio and Cohen's d Effect SizesResearchers often place a strong focus on statistical significance when reporting the results of statistical tests. However, effect sizes are reported less frequently, and interpretation in the context of the study and the research field is even rarer. These interpretations of effect sizes are, however, necessary to understand the practical importance of a result for the community. To explore how Usable Security & Privacy (USP) and HCI researchers interpret effect sizes and make judgments on practical importance, we conducted survey and interview studies with a total of 63 researchers at CHI and SOUPS 2023. Our studies focused on Cohen's d and odds ratios in two USP and one HCI scenario. We analyzed which artifacts researchers consider when judging effect size, and found misconceptions and variation between the participants, highlighting how difficult judging statistics can be. Based on our findings, we make concrete recommendations for improved reporting practices around effect sizes.2025AOAnna-Marie Ortloff et al.University of BonnUser Research Methods (Interviews, Surveys, Observation)Research Ethics & Open ScienceCHI
Out of Sight, Out of Mind? Exploring Data Protection Practices for Personal Data in Usable Security & Privacy StudiesAdherence to data protection measures such as pseudonymization or anonymization is critical in human subjects research because it has a direct impact on the confidentiality of participants' sensitive information, trust in research practices, and compliance with ethical and legal standards. Regulations such as the General Data Protection Regulation (GDPR) and guarantees made by researchers in informed consent forms mandate strict protocols for data security. However, compliance with these is not always straightforward. To gain qualitative insights into data protection practices in the field of Usable Security and Privacy (USP), we conducted interviews with 22 practitioners (five professors, eight researchers, nine data protection officers) and one focus group with five researchers. Overall, our results show a high awareness of ethical and legal responsibilities but highlight many practical and procedural issues. Based on these, we make concrete recommendations on how to improve the protection of personal data in research.2025FMFlorin Martius et al.University of BonnAlgorithmic Transparency & AuditabilityPrivacy by Design & User ControlResearch Ethics & Open ScienceCHI
"They are responsible for ensuring that I can continue to use the service." Investigating Users' Expectations Towards 2FA Recovery in GermanyTwo-factor authentication is often recommended for increasing online security, and users often follow this by using their phones. If physical items become unavailable, there is a risk of losing access to the account due to missing authentication requirements. In such cases, users need a backup or help from the service. Previous work found no standardized approach to how services address this issue, assist users, or offer backup options. Until now, it is unclear how users handle backups and account recovery and what their expectations towards service providers are. To shed light on this, we conducted 16 interviews and a survey with 95 participants. We found that most had never considered how to access their accounts if the second factor was lost, and only a few had a backup plan. Instead, users often rely on website support, assuming that personal data will help them regain access. We give recommendations for services.2025ETEva Tiefenau et al.Fraunhofer FKIEPrivacy by Design & User ControlPasswords & AuthenticationCHI
Different Researchers, Different Results? Analyzing the Influence of Researcher Experience and Data Type During Qualitative Analysis of an Interview and Survey Study on Security AdviceWhen conducting qualitative research it is necessary to decide how many researchers should be involved in coding the data: Is one enough or are more coders beneficial? To offer empirical evidence for this question, we designed a series of studies investigating qualitative coding. We replicated and extended a usable security and privacy study by Ion et al. to gather both simple survey data and complex interview data. We had a total of 65 students and seven researchers analyze different parts of this data. We analyzed the codebook creation process, similarity of outcomes, inter-rater reliability, and compared the student to the researcher outcomes. We also surveyed five years of SOUPS-PC members about their views on coding. The reviewers view on coding practices for complex and simple data are almost identical. However, our results suggest that the coding process can be different for the two types of data, with complex data benefiting more from interaction between coders.2023AOAnna-Marie Ortloff et al.University of BonnUser Research Methods (Interviews, Surveys, Observation)Research Ethics & Open ScienceCHI
Less About Privacy: Revisiting a Survey about the German COVID-19 Contact Tracing AppThe release of COVID-19 contact tracing apps was accompanied by a heated public debate with much focus on privacy concerns, e.g., possible government surveillance. Many papers studied people's intended behavior to research potential features and uptake of the apps. Studies in Germany conducted before the app's release, such as that by Häring et al., showed that privacy was an important factor in the intention to install the app. We conducted a follow-up study two months post-release to investigate the intention-behavior-gap, see how attitudes changed after the release, and capture reported behavior. Analyzing a quota sample (n=837) for Germany, we found that fewer participants mentioned privacy concerns post-release, whereas utility now plays a greater role. We provide further evidence that the results of intention-based studies should be handled with care when used for prediction purposes.2023MHMaximilian Häring et al.University of BonnPrivacy by Design & User ControlPrivacy Perception & Decision-MakingContent Moderation & Platform GovernanceCHI
A Usability Evaluation of AFL and libFuzzer with CS StudentsIn top-tier companies and academia, fuzzing has established itself as a valuable tool for finding bugs. It is a tool created by experts for experts, and a lot of research is being invested into improving the power of fuzzing. However, the usability of fuzzing has not received much attention yet. To alleviate this, we evaluated the usability of two popular fuzzers: AFL and libFuzzer. In our fuzzing study, 47 computer science students each worked up to 20 hours in total. We found significant usability challenges for both fuzzers leading to only 17 participants who were able to finish all tasks. Even the successful participants struggled with some of the necessary steps and found them complex and confusing. While on the whole, AFL fared better than libFuzzer, both fuzzers have strengths and weaknesses and can be improved based on our results.2023SPStephan Plöger et al.Fraunhofer FKIEMental Health Apps & Online Support CommunitiesComputational Methods in HCICHI
On Conducting Security Developer Studies with CS Students: Examining a Password-Storage Study with CS Students, Freelancers, and Company DevelopersEcological validity is a major concern in usable security studies with developers. Many studies are conducted with computer science (CS) students out of convenience, since recruiting professional software developers in sufficient numbers is very challenging. In a password-storage study, Naiakshina et al. (CHI'19) showed that CS students behave similarly to freelance developers recruited online. While this is a promising result for conducting developer studies with students, an open question remains: Do professional developers employed in companies behave similarly as well? To provide more insight into the ecological validity of recruiting students for security developer studies, we replicated the study of Naiakshina et al. with developers from diverse companies in Germany. We found that developers employed in companies performed better than students and freelancers in a direct comparison. However, treatment effects were found to be significant in all groups; the treatment effects on CS students also held for company developers.2020ANAlena Naiakshina et al.University of BonnPasswords & AuthenticationUser Research Methods (Interviews, Surveys, Observation)CHI
"If you want, I can store the encrypted password": A Password-Storage Field Study with Freelance DevelopersIn 2017 and 2018, Naiakshina et al. (CCS'17, SOUPS'18) studied in a lab setting whether computer science students need to be told to write code that stores passwords securely. The authors' results showed that, without explicit prompting, none of the students implemented secure password storage. When asked about this oversight, a common answer was that they would have implemented secure storage - if they were creating code for a company. To shed light on this possible confusion, we conducted a mixed-methods field study with developers. We hired freelance developers online and gave them a similar password storage task followed by a questionnaire to gain additional insights into their work. From our research, we offer two contributions. First of all, we reveal that, similar to the students, freelancers do not store passwords securely unless prompted, they have misconceptions about secure password storage, and they use outdated methods. Secondly, we discuss the methodological implications of using freelancers and students in developer studies.2019ANAlena Naiakshina et al.University of BonnPasswords & AuthenticationUser Research Methods (Interviews, Surveys, Observation)CHI