Exposing Error in Poverty Management Technology: A Method for Auditing Government Benefits Screening Tools

Public benefits programs help people afford necessities like food, housing, and healthcare. In the US, applicants must complete long forms to prove financial distress before receiving aid. Online benefits screening tools provide a gloss of such forms, advising households about their eligibility prior to completing full applications. If incorrectly implemented, they may discourage qualified households from applying for benefits. Unfortunately, errors are difficult to detect because they surface one at a time and difficult to contest because unofficial determinations do not generate a paper trail. We introduce a method for auditing such tools. We generate test households that automatically populate screening questions. To detect errors, we compare the returned determinations to predictions from our formal eligibility guidelines model. Illustrated on a real screening tool with households modeled from census data, our method exposes major errors with corresponding examples to reproduce them. Our work provides a necessary corrective to an already arduous benefits application process.