From Narrative to Numbers: Evaluating Survey Questionnaires with Large Language Models

As the field of intelligent interfaces is evolving, there is also a growing need for feedback mechanisms that are both expressive for participants and also contain reliable and useful information for the researcher conducting the study for survey data collection. Our study involves exploring two different kinds of survey methods: a standardized slider scale for web-based surveys and a free-form text input with a Large Language Model (LLM) acting as a backbone. The experiment includes 36 participants completing a 4×4 sliding-tile game at two different levels (easy and hard) with difficulty standardized via Manhattan-distance targets. The response mode order was counterbalanced across two sequences. This task aimed to evaluate the accuracy and quality of participant responses through different survey methodologies. Our key findings are that the LLM survey results are equivalent to the ones reported through the Web-Based slider scale questionnaire method. Our contribution is an intelligent framework that allows text-based reflections within an adaptive survey interface, helping both participants to express their experiences naturally and also researchers to gain valuable information about their system.