Methods for Evaluation of Imperfect Captioning Tools by Deaf or Hard-of-Hearing Users at Different Reading Literacy Levels

As Automatic Speech Recognition (ASR) improves in accuracy, it may become useful for transcribing spoken text in real-time for Deaf and Hard-of-Hearing (DHH) individuals. To quantify users' comprehension and opinion of automatic captions, which inevitably contain some errors, we must identify appropriate methodologies for evaluation studies with DHH users, including quantitative measurement instruments suitable to the various literacy levels among the DHH population. A literature review guided our selection of several probes (e.g. multiple-choice comprehension-question accuracy or response time, scalar-questions about user estimation of ASR errors or their impact, users' numerical estimation of accuracy), which we evaluated in a lab study with DHH users, wherein their literacy levels and the actual accuracy of each caption stimulus were factors. For some probes, participants with lower literacy had more positive subjective responses overall, and, for participants with particular literacy score ranges, some probes were insufficiently sensitive to distinguish between caption accuracy levels.