Rationalizer: Leveraging LLM to Support User Providing the Rationales Behind the Rating of Likert Scale Questionnaires

Surveys, especially Likert scale questionnaires, are widely used in HCI to capture users’ attitudes and experiences, but numeric ratings alone provide little insight into the rationales behind those ratings. While adding open-ended text fields or post-questionnaire interviews can elicit richer explanations, they often impose extra effort, leading to survey fatigue or recall bias. To address this gap, we proposed Rationalizer, an LLM-supported questionnaire system that generates contextualized rationales to support participants articulate their explanations alongside each Likert item and rating. In a user evaluation comparing it with the traditional questionnaire that included open-ended text fields, Rationalizer increased the percentage of Likert items with rationales, sustained participants’ willingness to provide self-input rationales, and supported them in articulating longer explanations within comparable writing durations as the study progressed. Quality analyses further showed that Rationalizer yielded higher-quality rationales (i.e., justification and relevance) than the traditional questionnaire. These findings highlight the potential of LLM-supported questionnaires to enrich Likert ratings with contextualized, richer explanations.