Towards Fair and Equitable Incentives to Motivate Paid and Unpaid Crowd Contributions

Research Background and Problem

• What issues or challenges did the authors identify?
  Researchers and practitioners rely on both paid crowdworkers and unpaid contributors, yet the motivations and performance of these two groups in editing high-quality data have rarely been studied in depth. Designing a system that can effectively motivate both groups while ensuring data quality is a significant challenge.

• Why is this problem important?
  Continuously maintaining dynamic information, such as tabular data about computer science professors, holds significant value for academia and research institutions. This includes supporting academic analysis, student-advisor selection, and diversity studies. However, due to insufficient or unevenly distributed contributor motivation, long-term system maintenance is difficult to sustain.

• Research Motivation and Related Work
  Previous studies have shown that people contribute data for various intrinsic (helping others, personal interest) and extrinsic (monetary rewards) motivations. However, little research has explored how to balance these driving forces on a single platform, particularly the effective integration of unpaid contributors (intrinsically motivated) and paid workers (extrinsically motivated).

Solution

• What methods or solutions did the authors propose?
  The authors developed and maintained Drafty, an open public editing platform that allows anyone to edit tabular data about academic personnel in the field of computer science. They conducted a comprehensive motivation analysis of users (including both paid and unpaid contributors) through a Discrete Choice Experiment (DCE) and combined this with their actual editing behavior to provide quantitative evidence.

• What is innovative about this solution?

  1. Mixed-method research: By combining discrete choice experiments with real operational behavior, the study provides a comprehensive analysis of motivations and performance.
  2. Fair incentive design recommendations: The authors offer detailed system design recommendations, quantitatively analyzing how fair and perceptible task structures can motivate different types of contributors.
  3. Validation through a real system: The study utilizes Drafty, a real system with nine years of operational history, to analyze existing editing records and user behavior, ensuring high ecological validity.

• What are the implementation steps and key technologies used?

  1. Publishing real tasks: Editing experiments were conducted within Drafty to collect contribution behavior from both paid and unpaid groups.
  2. Designing the Discrete Choice Experiment (DCE): A set of multi-attribute choice tasks (e.g., payment level, task completion time, task interest level) was constructed to quantify user preferences.
  3. Data analysis: The authors combined task completion accuracy with respondents' discrete choice experiment data to quantify the impact of intrinsic and extrinsic motivations on contributions.
  4. Proposing recommendations: Through statistical analysis, the study optimized motivational drivers (e.g., the importance of high pay for paid workers or emphasizing task interest for unpaid users).

Research Outcomes

• What specific results were achieved?

  1. Performance differences between groups:
     • Unpaid contributors demonstrated higher editing accuracy than paid crowdworkers, particularly for challenging tasks or those requiring domain knowledge.
     • Unpaid contributors were more strongly driven by intrinsic motivations (e.g., interest or community support), while paid crowdworkers prioritized extrinsic motivations (e.g., monetary rewards).
  2. Task driving factors:
     • Common motivational factors included task interest, the potential to help others, task duration, and payment.
     • Highly accurate contributors (whether paid or unpaid) were more likely to choose tasks aligned with intrinsic motivations.

• What advantages does it have compared to existing solutions?

  • The study provides specific guidance on system design, such as balancing intrinsic and extrinsic motivations.
  • It captures the behavior of both paid and unpaid users within the same system, offering new perspectives and methods for improving data quality and designing fair incentives.

• What were the experimental or evaluation results?

  • Paid workers' performance dropped sharply when payment was below $8/hour, but increasing payment to $16/hour did not significantly improve accuracy.
  • Unpaid contributors demonstrated a higher willingness to contribute to the academic community, with their editing accuracy surpassing that of paid workers across all task types studied.

• Limitations and future directions

  1. Limitations:
     • The data was sourced from a specific computer science academic community, which may limit the generalizability of the results.
     • The study focused on a single data platform, Drafty, and future research is needed to validate findings across other types of public data systems.
  2. Future directions:
     • Explore how to apply hybrid motivation-driven models across platforms.
     • Investigate how AI assistance and group collaboration can further enhance motivational factors.

Conclusion

This paper, through the Drafty system and discrete choice experiments, reveals significant differences in how intrinsic and extrinsic motivations drive paid and unpaid contributors to complete tasks. Its major contribution lies in proposing how to construct fairer incentive mechanisms to attract highly accurate contributors, whether they are paid crowdworkers or unpaid users. This research approach not only provides practical guidance for building specific public data systems but also offers new insights into how future participatory platforms can balance workforce composition and motivation alignment.