Chatbots for Data Collection in Surveys: A Comparison of Four Theory-Based Interview Probes

Research Background and Issues

• What problems or challenges did the authors identify?
  The authors pointed out that traditional open-ended questions in online surveys often fail to collect high-quality and rich data. These issues arise due to participants' lack of engagement or cognitive burden (e.g., survey fatigue), leading to low response quality or missing answers. Additionally, pre-set follow-up questions in traditional questionnaires cannot immediately address ambiguous responses or delve deeper into potential insights. While face-to-face interviews can overcome these issues, they are difficult to scale or standardize.

• Why is this issue important?
  Obtaining high-quality survey data is central to Human-Computer Interaction (HCI) research, as data quality directly impacts research outcomes and design decisions. If methods can be developed to combine the scalability of online surveys with the depth of interviews, the ability to generate research insights could be significantly enhanced.

• Research Motivation and Related Work
  Through a review of prior literature, the authors found that embedding chatbots in surveys can enhance participant engagement and data quality. Specifically, using theory-based interview probes (e.g., descriptive, idiographic, clarifying, and explanatory probes) may improve the depth and relevance of data across different research stages (e.g., exploration, requirements gathering, and evaluation). Furthermore, advancements in large language models (LLMs) have significantly improved chatbots' ability to interpret open-ended text, providing technical support for exploring this approach.

Solution

• What methods or solutions did the authors propose?
  The authors designed a chatbot based on large language models (LLMs) to simulate interview probes and embedded it into online surveys to collect qualitative data from open-ended questions. The study experimented with four theory-based interview probes—descriptive, idiographic, clarifying, and explanatory—integrated into three research stages: exploration, requirements gathering, and evaluation.

• What is innovative about this solution?
  The primary innovation lies in combining traditional social science interview probes with advanced natural language processing technology to enable real-time interactive data collection in surveys. Additionally, the study introduced Grice's maxims of communication (e.g., informativeness, relevance, clarity, and detail) as quantitative metrics to evaluate response quality. The solution also features a modular design, allowing surveys to be flexibly adapted to different research stages.

• What are the implementation steps and key technologies used?

  1. Developed a chatbot based on OpenAI GPT-4 and optimized its interview style through specific prompt engineering.
  2. Designed preparatory materials (e.g., images or videos) for different research stages to intuitively guide participants in reflecting on technology-related stress.
  3. Set up experimental conditions to compare the effects of different probe types and research stages on response quality.
  4. Used metrics such as Grice's maxims and Likert scales to analyze response quality and participant experience.
  5. Provided open-source code and modular design to ensure the solution's scalability and reusability.

Research Outcomes

• What specific outcomes were achieved?

  1. Among all interview probes, the idiographic probe performed best across various quality metrics (including relevance, detail, and clarity), making it particularly suitable for exploration, requirements gathering, and evaluation stages.
  2. In the exploration stage, both idiographic and descriptive probes effectively collected rich information related to user experiences.
  3. In the requirements gathering stage, descriptive probes captured broad task narratives, while idiographic probes collected detailed requirements.
  4. In the evaluation stage, idiographic probes excelled at eliciting specific feedback about products from users.

• What advantages does this solution have compared to existing approaches?

  1. Provides dynamic interaction capabilities, increasing participant reflection and data richness compared to traditional fixed-question surveys.
  2. Introduces classic probe methods from social sciences into intelligent survey tools, ensuring theoretical guidance for structured data collection.
  3. Avoids biases or high costs associated with manual interviews while improving participant comfort and acceptance.

• What were the experimental or evaluation results?
  Quantitative analysis (Grice's maxims) and user perception data revealed:

  • Theory-supported probes improve data quality, especially when tailored to different research stages.
  • Participants generally had positive experiences with the chatbot, though certain probes (e.g., explanatory probes) were perceived as repetitive.
  • The chatbot demonstrated flexibility in addressing diverse participant queries, further enhancing the depth of survey data.

• Limitations and Future Directions

  • Limitations include the relatively narrow scope of the experimental topic (technology-related stress) and insufficient exploration of the comparative effectiveness of traditional open-ended survey questions. Future work could extend applications to other HCI domains and further compare chatbots with traditional survey methods.
  • To address repetitive issues in chatbot conversations, future research could explore real-time topic quality monitoring mechanisms.
  • Investigating how to integrate multiple probe strategies to optimize long-term interactive surveys could also be a promising direction.

Through this study, the authors provided a novel tool for data collection in HCI research and inspired broader possibilities for improving qualitative data collection using natural language technologies.