Literature Reviews in HCI: A Review of Reviews

Literature Title

Literature Reviews in HCI: A Review of Reviews

Literature Information

• Subject Area: Human-Computer Interaction (HCI)
• Keywords: Literature review, meta-review, meta-analysis, literature survey, methodology

Research Background and Issues

• Identified Problems/Challenges:

  • The interdisciplinary nature of HCI and the diversity of literature review types (ranging from systematic to critical reviews) have led to a lack of clear standards and consensus.
  • There is an absence of a shared theoretical framework and terminology system to guide, evaluate, and utilize HCI literature reviews.

• Significance:

  • Literature reviews are crucial for improving research quality and clarifying research directions.
  • By systematically organizing past research, literature reviews provide an essential knowledge foundation for the HCI field.

• Research Motivation:

  • To address the shortcomings of the current literature review system, this study aims to categorize the types and methods of HCI literature reviews and propose improvements.
  • Existing literature reviews exhibit significant quality variations, such as experimental methods, research scales, and ambiguous terminology usage.

Solutions

• Proposed Methods/Solutions:

  • Analyze 189 literature reviews published in SIGCHI conferences and ACM TOCHI journal up to August 2022.
  • Classify review contribution types and subject areas, clarifying the structure and methods of reviews.
  • Provide an online literature repository and design documentation to assist future literature review authors.

• Innovations:

  • Identified and defined five contribution types of HCI literature reviews: Empirical, Artefact, Methodological, Theoretical, and Opinion.
  • Proposed twelve review subject areas with detailed classifications.
  • Offered transparent literature review methods (e.g., PRISMA standards) and reflective design tools to improve the review, writing, and reading experience.

• Implementation Steps and Techniques:

  • Utilize the PRISMA-adapted process to detail the screening steps.
  • Employ qualitative coding and multi-step analysis to determine review topics and methods.
  • Provide an interactive online resource for dynamic updates and information filtering.

Research Outcomes

• Specific Outcomes:

  • Comprehensive categorization of literature review types and contributions in the HCI field.
  • Development of an online resource to help readers filter and query literature by topic, method, and type.
  • Promotion of a shared language and standards for literature reviews within the HCI community.

• Advantages Over Existing Solutions:

  • This study is the first to conduct a systematic analysis of literature reviews in the HCI field (Review of Reviews).
  • It proposes a more explicit classification system and corresponding methodological standards, addressing gaps in HCI literature review research.

• Experimental or Evaluation Results:

  • Distribution of contribution types in HCI literature reviews: Empirical (68), Methodological (55), Artefact (54), Theoretical (28), and Opinion (8).
  • Approximately 13% of the literature reported IRR (inter-rater reliability), and 43 reviews used PRISMA or other flowcharts.
  • Most literature reviews did not use standardized tools, highlighting the need for more unified methodological norms.

• Limitations and Future Directions:

  • Limitations: The study only covers SIGCHI conferences and TOCHI journal, excluding other significant HCI conferences or journals; IRR was not calculated as a consistency metric.
  • Future Directions:
    • Further research on the temporal trends of literature reviews and the academic impact of different review types.
    • More specific analyses of particular subject areas or high-impact literature.
    • Development of additional tools and frameworks to standardize literature reviews, enhancing transparency and efficiency.

Summary

This study provides valuable resources for establishing a shared language for literature reviews in the HCI field. It categorizes and summarizes review types and subject areas, proposing improvements to help researchers write and evaluate literature reviews more precisely. The study aims to foster academic discussions and enhance the quality and insights of HCI literature reviews.