Large Language Models in Qualitative Research: Uses, Tensions, and Intentions

Research Background and Issues

• Issues and Challenges:

  • This paper focuses on the use of large language models (LLMs) in qualitative research, exploring potential tensions and ethical issues. While LLMs are considered to have the potential to enhance qualitative research, questions remain about their applicability, ethicality, and alignment with the goals and values of qualitative research.
  • LLMs, as general-purpose tools, allow for flexible handling of documents such as summarization and annotation. However, this flexibility may overly emphasize automation, reducing researchers' deep engagement with the data.
  • Ethical and technical guidelines for the use of LLMs in qualitative research are still in their infancy. Rapidly evolving technologies may conflict with research values such as deep understanding and participatory approaches.

• Significance:

  • The potential of LLMs to improve efficiency has garnered attention, but there is a lack of frameworks to ensure compatibility with participant privacy protection and ethical requirements.
  • As AI tools become increasingly widespread, it is imperative for researchers to carefully assess their impact on existing research methods to avoid an over-reliance on technology that may lead to a "positivist" trajectory.

• Research Motivation and Related Work:

  • This paper seeks to fill the gap in exploring the role of LLMs in qualitative research, responding to the longstanding academic discourse on the tension between qualitative and quantitative methods.
  • Building on existing work, such as research on human-AI collaboration, this paper focuses on how LLMs transform qualitative research processes and the philosophical and practical challenges they introduce.

Solution

• Methods or Solutions:

  • The paper employs interviews with 20 qualitative researchers to study how they adopt, perceive, and evaluate LLMs, aiming to uncover their current uses, challenges, and potential.
  • It identifies ethical dilemmas faced by researchers when leveraging LLMs and provides preliminary recommendations for addressing these dilemmas.

• Innovations:

  • Highlights the potential tensions between LLMs and qualitative research values, such as deep engagement and data interaction, and explores how LLMs can align with participant privacy protection, ethical considerations, and research goals.
  • Proposes specific design principles for creating AI tools that support researchers' deep engagement and interaction with data.

• Implementation Steps and Key Techniques:

  • The study collected data through semi-structured interviews and used an inductive thematic analysis framework to extract core themes.
  • The data analysis process was conducted using the Dedoose collaborative platform, emphasizing consistency in coding and research findings throughout the study.

Research Findings

• Specific Findings:

  • Current applications of LLMs in qualitative research are primarily focused on generating interview materials, assisting with data retrieval, and providing preliminary support in coding. However, they remain limited in deep analysis and complex data interpretation.
  • Interviews revealed researchers' significant concerns about privacy, bias, and ethical issues associated with LLMs, emphasizing the need for research tools to better support privacy protection and data transparency.
  • Specific guidelines for researchers were proposed, including intentional task selection, prioritization of privacy protection, and validation of LLM performance.

• Comparison with Existing Solutions:

  • Compared to earlier AI-assisted research tools, the natural language interaction and flexibility of LLMs enhance collaboration between researchers and tools. However, they also weaken the close experiential connection with data that many qualitative researchers aim to maintain.

• Experimental or Evaluation Results:

  • Data indicated that while many researchers expressed curiosity about the technology, they faced barriers related to time, skills, and tool selection.
  • Most participants preferred to maintain agency in the application of LLMs, such as having deeper manual involvement in model invocation or theme generation.

• Limitations and Future Directions:

  • Limitations:
    • Participants were primarily from North American higher education institutions, which may not represent non-Western or more localized academic perspectives.
    • The study focused on existing tools like ChatGPT and did not deeply explore fully controllable open-source model ecosystems.
  • Future Directions:
    • Explore the potential and challenges of LLMs in cross-cultural research, particularly for groups with diverse linguistic and cultural backgrounds.
    • Develop open LLM tools optimized for participant privacy protection and provide guidance on their integration methods.
    • Establish systematic validation frameworks to help researchers evaluate LLM performance while promoting transparency in research tools.

Conclusion

This paper provides qualitative researchers with a clear framework for understanding and evaluating the use of LLMs, offering a comprehensive discussion of their potential opportunities and limitations. It proposes principles for designing AI tools that more effectively support the values of qualitative research. These recommendations lay the groundwork for the academic community to further develop tools and standardize the use of LLMs in research.