Origins of Algorithmic Instabilities in Crowdsourced Ranking

Crowdsourcing systems aggregate decisions of many people to help identify high-quality options, such as the best answers to questions or interesting news stories. A long-standing issue in crowdsourcing is how option quality and human judgement heuristics interact to affect collective outcomes, such as the perceived popularity of options. We address this limitation by conducting a controlled experiment where subjects choose between two ranked options whose quality can be independently varied. We use this data to construct a model that quantifies how judgement heuristics and option quality combine when deciding between two options. The model reveals popularity-ranking is unstable below a critical point: if the quality difference between the two options is sufficiently high, the higher quality option is eventually ranked on top, however, below a certain quality difference this is not guaranteed. To rectify the instability, we create an algorithm that accounts for judgement heuristics to infer the best option and ranks it first. This method is guaranteed to be optimal if data matches the model, and simulations show that this method performs better or at least as well as popularity-based and recency-based ranking for any two-choice question. Our work suggests that algorithms relying on inference of mathematical models of user behavior can substantially improve outcomes in crowdsourcing systems.