Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits

Video

DOI

While algorithm audits are growing rapidly in importance and commonality, relatively little scholarly work has gone toward synthesizing prior work and strategizing future research in the area. This systematic literature review aims to fill the gap, following PRISMA guidelines in a review of over 500 English articles that yielded 62 algorithm audit studies. The studies are synthesized and organized primarily by behavior (discrimination, distortion, exploitation, and misjudgement), with codes also provided for domain (e.g. search, vision, advertising, etc.), organization (e.g. Google, Facebook, Amazon, etc.), and audit method (e.g. sock puppet, direct scrape, crowdsourcing, etc.). Based on the review, previous audit studies have exposed powerful algorithms exhibiting problematic behavior, such as search algorithms culpable of distortion and advertising algorithms culpable of discrimination. The review also suggests some behaviors, domains, methods, and organizations that call for for future audit attention, such as problematic "echo chambers" and other distortion effects from advertising algorithms. The paper concludes by discussing algorithm auditing in the context of other research working toward algorithmic justice.