Two Tools are Better Than One: Tool Diversity as a Means of Improving Aggregate Crowd Performance

Crowdsourcing is a common means of collecting image segmentation training data for use in a variety of computer vision applications. However, designing accurate crowd-powered image segmentation systems is challenging because defining object boundaries in an image requires significant fine motor skills and hand-eye coordination, which makes tasks error-prone. Typically, special segmentation tools are created, and then answers from multiple workers are aggregated to generate more accurate results. However, individual tool designs can bias how people make mistakes, resulting in shared errors that remain even after aggregation. In this paper, we introduce a novel crowdsourcing workflow that leverages multiple tools for the same task to increase output accuracy by reducing systematic error biases introduced by the tools themselves. When a task can no longer be broken down to more tractable subtasks (the conventional approach taken by microtask workflows), our multi-tool approach can be used to improve accuracy further by assigning different tools to different workers. We present a series of studies that evaluate the feasibility of our multi-tool approach, and show that it is able to significantly improve aggregate accuracy in semantic image segmentation.