Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review

Research Background and Issues

• Identified Problems or Challenges: The authors observed that research on Large Language Models (LLMs) in the field of Human-Computer Interaction (HCI) is rapidly growing. However, there is currently a lack of systematic understanding regarding the categorization, usage patterns, and potential limitations of these studies. Specifically, it is unclear how LLMs are applied in HCI papers, the types of contributions they make, and how scholars evaluate their limitations and risks.
• Significance: As LLMs are considered to have the potential to revolutionize the field of HCI, it is crucial to deeply understand their impact. LLMs not only reshape research on interfaces, design patterns, and socio-technical systems but also transform the methodological practices employed by researchers. Therefore, they have profound implications across the entire field of computer science.
• Research Motivation and Related Work: Given that the HCI research community is being shaped by the surge of interest in LLMs, the authors conducted a systematic literature review to explore the specific impacts of LLMs on HCI, identify research trends, and uncover potential research gaps.

Solution

• Proposed Method or Solution: The authors conducted a systematic literature review of 153 CHI (Conference on Human Factors in Computing Systems) papers related to LLMs published between 2020 and 2024, aiming to address the following research questions:
  1. In which domains and scenarios are LLMs applied?
  2. How do researchers utilize LLMs in their papers?
  3. What contributions do LLMs make to the HCI field?
  4. What limitations and risks of LLMs have scholars identified?
• Innovations: The authors employed a multidimensional coding approach to classify and analyze the application domains, roles, contribution types, and limitations/risks of LLMs in the reviewed papers. This comprehensive classification and the combination of quantitative and qualitative methods systematically reveal the multifaceted impact of LLMs on HCI.
• Implementation Steps:
  1. Collected full-text CHI conference papers from 2020 to 2024 and filtered those containing LLM-related research.
  2. Conducted preliminary filtering using keywords (e.g., "language model," "GPT," etc.).
  3. Performed iterative open coding and related techniques to analyze the content of the papers, ultimately forming a classification system for application scenarios, LLM roles, and descriptions of limitations.

Research Findings

• Specific Findings:
  1. Application Domains of LLMs: Identified 10 application domains, including communication and writing, education, programming, reliability and effectiveness, health and well-being, among others.
  2. Roles of LLMs: Determined five roles of LLMs in HCI research, such as serving as system engines, research tools, simulated participants/users, research objects, and studying user perceptions of LLMs.
  3. Types of Contributions: Found that LLM research primarily focuses on empirical and tool contributions, with fewer contributions in theory and methodology.
  4. Limitations and Risks: Enumerated 22 mainstream limitations and risks, including performance issues (e.g., hallucinations, non-determinism), resource constraints (e.g., computational and financial costs), research validity concerns, and potential societal consequences (e.g., misinformation propagation, bias).
• Advantages: Compared to traditional HCI research, this review highlights the extensive application scenarios and potential advantages of LLMs as tools while clarifying common issues and challenges in research, providing specific directions for future studies.
• Experimental or Evaluation Results: Through qualitative analysis and high-confidence coding consistency evaluation, the authors ensured the reliability of the classification system (e.g., Krippendorff's alpha for major classifications was close to or exceeded 0.8).
• Limitations and Future Directions:
  1. Limitations: The review only covered papers explicitly mentioning LLMs in their titles and abstracts; some studies involving LLMs but not explicitly labeled may have been overlooked. Additionally, integrative research themes, such as cross-domain applications of LLMs, were not deeply explored.
  2. Future Directions: The authors suggest further research on theoretical contributions and methodological issues to enhance the understanding and standardization of LLMs in the field. They also call on the HCI community to standardize considerations of paper impact and research ethics, such as reflecting on the societal consequences of LLM research through explicit ethical statement frameworks.

Conclusion

This paper provides a comprehensive framework and classification for understanding the impact of LLMs on HCI research, systematically analyzing aspects such as application domains, usage roles, limitations, and potential risks. Through a detailed literature review and analysis, the authors demonstrate how LLMs are transforming research practices, identify underexplored areas, and outline specific paths for advancing HCI toward greater theoretical and normative development. This work holds significant academic reference value for both the HCI field and the broader domain of computer science.