Simulacrum of stories: Examining Large Language Models as Qualitative Research Participants

Research Background and Issues

• What problems or challenges did the authors identify?
  This study explores large language models (LLMs), particularly their application in qualitative research, where these models are used as synthetic research participants to replace real human participants. The main issue lies in whether this aligns with the core goals and ethical requirements of qualitative research, including privacy, participant autonomy, and the authenticity of knowledge production.

• Why is this issue important?
  Traditional qualitative research is deeply rooted in interactions with human participants, aiming to uncover profound and nuanced cultural, social, and emotional contexts. While LLMs offer the potential to reduce research time and costs, the ethical and epistemological challenges they pose could have significant implications for the core values of academia.

• Research Motivation and Related Work
  With the growing prevalence of generative AI, many studies have begun to discuss the potential applications of LLMs in qualitative research. Some works have demonstrated their utility in text generation, survey design, and even ethical analysis, but these approaches may face issues such as training data bias and oversimplification of complexity. The authors aim to delve into the perspectives of qualitative researchers to analyze the limitations and ethical challenges of using LLMs as substitutes for research participants.

Solutions

• What methods or solutions did the authors propose?
  The authors designed an exploratory approach, conducting in-depth interviews with qualitative researchers and introducing experimental tools (simulated LLM interview probes) to systematically analyze the application and ethical challenges of LLMs in qualitative research.

• What is innovative about this solution?
  Rather than merely testing the capabilities of LLMs, the study employs reflective discussions to uncover the deeper tensions between LLMs and the methodologies and ethics of qualitative research. It identifies six major limitations and uses theories from Science and Technology Studies (STS) and qualitative research to explain these issues.

• What are the implementation steps and key technologies used?

  1. Conducting semi-structured interviews with 19 qualitative researchers experienced in semi-structured interviewing.
  2. Using a programming tool (GPT-4-turbo API) to simulate virtual interviews, with prompts defining different character backgrounds.
  3. Analyzing participants' behaviors when generating data with LLMs, comparing output quality, and evaluating the ethical and practical utility of LLMs in research contexts.
  4. Conducting qualitative data analysis to identify themes, including in-depth observations and critiques of LLM limitations.

Research Findings

• What specific findings were obtained?
  The authors identified six core limitations of LLM applications:

  1. Lack of perceived authenticity (palpability): Generated content, while detailed to some extent, lacks the dynamism and nuance provided by real participants.
  2. Ambiguity in the model's cognitive framework: Generated data may integrate conflicting arguments, leading to unclear perspectives and contextual misalignment.
  3. Reinforcement of researcher bias (positionality): Researchers' prompt design can unintentionally amplify their assumptions, resulting in data biased toward "expected outcomes."
  4. Lack of informed consent and autonomy for participants: Generated content may include data obtained without permission, violating privacy rights.
  5. Erasure of community perspectives: Models may misinterpret or oversimplify the complex experiences of marginalized groups, reducing cultural identities to stereotypes.
  6. Threat to the legitimacy of qualitative knowledge: Qualitative research, already marginalized, may face further devaluation as LLMs shift focus toward rapid data generation and quantitative methods.

• What are the advantages compared to existing solutions?
  This study goes beyond simple technical performance comparisons by reflecting on the theoretical foundations of qualitative research. It delves deeper into the risks LLMs may pose to research from ethical and epistemological perspectives. The multi-faceted interviews reveal subtle insights and vulnerabilities in researcher-model interactions that are difficult to observe in laboratory tests.

• What were the experimental or evaluation results?
  The study shows that most participants were dissatisfied with the depth and reliability of simulated interview data. While the generated content appeared plausible, it lacked a solid grounding in complex social and cultural contexts. This superficial utility could pose long-term risks to research credibility and community collaboration.

• Limitations and Future Directions

  1. The study participants were primarily from academic backgrounds, which may not fully capture perspectives from non-academic fields.
  2. The tests focused on GPT-4 and did not validate other models or customized types.
  3. The study highlighted potential auxiliary uses of LLMs in scenarios such as teaching tools and pilot studies, but these also require deeper ethical and practical validation.
  4. Future research could expand to explore LLM behavior and community acceptance in multicultural contexts.

Through its analysis, this study provides a forward-looking evaluation of LLMs' involvement in qualitative research. The research clearly demonstrates that while LLMs have potential in text generation, their application is profoundly limited, especially when dealing with sensitive topics and social group contexts. The study offers valuable insights into the dialectical relationship between technology and humanism.