Recruiting Participants With Programming Skills: A Comparison of Four Crowdsourcing Platforms and a CS Student Mailing List

Title of the Paper

Recruiting Participants With Programming Skills: A Comparison of Four Crowdsourcing Platforms and a CS Student Mailing List

Paper Information

• Research Area: Human-Computer Interaction, Crowdsourcing, Developer Studies
• Keywords: Recruitment, Developers, Crowdsourcing, Usable Privacy and Security, Programming, Dataset, Empirical Software Engineering

Research Background and Problem

• Finding reliable participants with programming skills for experimental studies remains a challenge.
• Crowdsourcing platforms (e.g., MTurk, Prolific) and Computer Science (CS) students are often used as sources for experimental participants; however, these channels have limitations in terms of external validity and the representativeness of experimental results.
• The goal of this study is to provide researchers with additional context on the best methods for recruiting participants with programming skills, focusing on the differences between crowdsourcing platforms and CS student groups.

Research Questions (RQs)

1. Which recruitment channels are suitable for recruiting participants with programming skills?
2. Which self-reported information is most strongly correlated with correctly answering all programming screening questions?
3. How do participants with programming skills recruited from different channels differ in terms of privacy and security attitudes and secure development self-efficacy?

Solution

• Method: A research tool consisting of a five-part survey questionnaire was developed, covering programming skills, privacy and security attitudes, and secure development self-efficacy. The questionnaire was distributed across five channels (Appen, Clickworker, MTurk, Prolific, and a university CS student mailing list).
• Innovation: This study is the first to comprehensively evaluate multiple established survey tools to compare differences across five recruitment channels.
• Implementation Steps:
  1. Screening Survey: Determine whether participants meet requirements for employment status, English proficiency, and programming experience.
  2. Main Survey: Use five existing survey tools (e.g., REALCODE, PROGEX, SSDSES, SEBIS, and IUIPC) to comprehensively assess participants' skills and attitudes.
  3. Perform descriptive statistical analysis on participants with lower programming skills.
  4. Use statistical models to analyze the relationship between independent variables (e.g., channel, occupational role, PROGEX score) and the ability to correctly answer all programming questions.

Research Findings

• Specific Findings:
  • Overall, 34.6% of participants correctly answered all programming questions (REALCODE). CS students had the highest accuracy rate (89%), followed by Clickworker (63.2%), Prolific (33.2%), while no Appen participants answered all questions correctly.
  • CS students provided the highest data quality at the lowest cost.
  • PROGEX results showed that MTurk participants, despite higher self-reported programming scores, performed poorly on actual programming tasks.
  • Prolific was more effective in recruiting samples with basic programming skills.
• Advantages Comparison:
  • Compared to existing solutions, recruiting through university CS students combined with the Prolific platform offers better cost-effectiveness and data quality.
  • Prolific has fewer duplicate entries and lower attention check error rates.
• Experimental Results:
  • Self-reported "programming experience" does not accurately reflect participants' programming skills, especially as the understanding of "having programming skills" varies across groups.
  • Age and occupational role are strongly associated with programming skills, particularly when participants are developers or have extensive software development experience.
• Limitations and Future Directions:
  • The study is limited to CS students from one university, and applicability to other schools or regions may vary.
  • Expanding the REALCODE tool's question bank is suggested to prevent data contamination from widespread use.
  • Future research could explore broader or more heterogeneous recruitment channels, such as social media.

Summary and Recommendations

This paper provides comprehensive data comparisons to support participant recruitment strategies in the developer research field. The study analyzes the performance and cost of recruitment strategies involving university students and multiple crowdsourcing platforms, recommending that future researchers fully utilize university resources while advising crowdsourcing platforms to optimize screening systems for more precise skill identification.