Exploring Trade-offs Between Learning and Productivity in Crowdsourced History

Crowdsourcing more complex and creative tasks is seen as a desirable goal for both employers and workers, but these tasks traditionally require domain expertise. Employers can recruit only expert workers, but this approach does not scale well. Alternatively, employers can decompose complex tasks into simpler microtasks, but some domains, such as historical analysis, cannot be easily modularized in this way. A third approach is to train workers to learn the domain expertise. This approach offers clear benefits to workers, but is perceived as costly or infeasible for employers. In this paper, we present CrowdSCIM, a workflow that teaches domain expertise (historical thinking skills) to novice crowd workers. We compare CrowdSCIM with two crowd learning techniques from prior work and a baseline to explore the trade-offs between learning and productivity. Our evaluation (n=360) shows that CrowdSCIM allows workers to learn domain expertise while producing work of equal or higher quality versus other conditions, but efficiency is slightly lower.