Beyond the Input Stream: Making Text Entry Evaluations More Flexible with Transcription Sequences

Text entry method-independent evaluation tools are often used to conduct text entry experiments and compute performance metrics, like words per minute and error rates. The input stream paradigm of Soukoreff & MacKenzie (2001, 2003) still remains prevalent, which presents a string for transcription and uses a serial character representation for encoding the text entry process. Although an advance over prior paradigms, the input stream paradigm is unable to support many modern text entry features. To address these limitations, we present “transcription sequences:” for each new input, a snapshot of the entire transcribed string unto that point is captured. By assembling transcription sequences and comparing adjacent strings, we can compute all prior metrics, reduce artificial constraints on text entry evaluations, and introduce new metrics. We conducted a study with 18 participants who typed 1620 phrases using a laptop keyboard, on-screen keyboard, and smartphone keyboard using features such as auto-correction, word prediction, and copy-and-paste. We also evaluated non-keyboard methods Dasher, gesture typing, and T9. Our results show that modern features and methods can be accommodated, prior metrics can be correctly computed, and new metrics can reveal insights. We validated our algorithms using ground truth based on cursor positioning, confirming 100% accuracy. We also provide a new tool, TextTest++, to facilitate web-based evaluations.