What is User Engagement?: A Systematic Review of 241 Research Articles in Human-Computer Interaction and Beyond

Research Background and Issues

• Problem Identification and Challenges
  The authors point out that the concept of User Engagement (UE) has been frequently discussed in Human-Computer Interaction (HCI) research, but its definition, reliability, and application remain ambiguous and inconsistent. This fragmentation in definition not only affects its theoretical application in research but also hinders its practical utility in system design.

• Significance of the Problem
  A clear definition and consistent measurement of UE can help the HCI field more effectively compare research findings, develop standardized measurement methods, and improve research reproducibility. However, the current trajectory of UE development reveals issues of conceptual ambiguity and insufficient operationalization, directly limiting its practical application in user interface design and user behavior analysis.

• Research Motivation and Related Work
  The authors reviewed 241 articles on UE published between 1993 and 2023, aiming to summarize its definitions and measurement methods and analyze the alignment between conceptualization and evaluation approaches. Previous literature has also noted the ambiguity of UE, but lacks systematic analysis to provide clearer focus directions.

Proposed Solution

• Proposed Approach
  The authors suggest that UE should be treated as a "categorical label" rather than a unified construct until a more systematic framework can be developed. They advocate for the HCI community to adopt a "divergent perspective," contextualizing UE within specific research domains and focusing on measurable attributes aligned with specific practices and norms.

• Innovative Aspects
  The authors propose moving beyond traditional frameworks for defining and measuring UE by treating it as a flexible label and creating domain-specific definitions for different research areas. This approach is significant in addressing the inconsistencies in definition and measurement in current research.

• Implementation Steps and Techniques

  1. Systematic Literature Review: Examine 241 related articles indexed in ACM Digital Library, Web of Science, and Google Scholar.
  2. Extraction and Categorization: Manually extract definitions, measurement methods, and data collection approaches related to UE from the articles.
  3. Similarity and Thematic Analysis: Use semantic similarity, natural language processing tools, and open coding analysis to identify significant trends in definitions and evaluation methods.
  4. Recommendations: Propose community-based domain-specific definitions and measurement frameworks.

Research Findings

• Key Findings

  1. The study found that 95.8% of UE definitions exhibit low semantic similarity, with many definition categories being scattered and inconsistent. For example, recurring features in definitions include user behavior, perception, and technological affinity.
  2. UE has limited utility as a scientific construct, with unclear construct validity.
  3. UE measurement predominantly focuses on behavioral data (51.9%) and self-reported data (34.4%), but lacks standardized and consistent measurement methods.

• Strengths

  1. The systematic literature review is the first to comprehensively reveal the fragmentation and inconsistency in UE conceptualization and measurement.
  2. The proposed suggestion to treat UE as a categorical label offers flexibility, enabling researchers to quantify it within specific communities.

• Experimental or Evaluation Results

  • Analysis of 241 articles shows that the majority of studies lack a unified standard for UE definitions and measurements. For instance, in social media research, the similarity between definitions is also low, with only 20.4% of definition pairs having a semantic similarity above 0.5.
  • From a measurement perspective, most studies rely on individual measurement dimensions, with very few combining multiple measurement methods (only 1.2% used a comprehensive combination).

• Limitations and Future Directions
  Limitations include the inability to cover all relevant literature (e.g., UE research in non-HCI technical fields) and the focus of analysis being primarily on the linguistic and thematic characteristics of the existing sample. Future directions include:

  1. Establishing more precise theoretical frameworks for UE in specific domains.
  2. Validating the theoretical and practical value of UE through cross-domain comparisons.
  3. Exploring unified measurement methods that integrate psychological and behavioral dimensions of "user experience."

Conclusion

Overall, this study systematically analyzes the current state of UE application in the HCI field, revealing significant inconsistencies and fragmentation. The authors propose a key perspective of defining UE as a categorical label and creating domain-specific definitions within different HCI subfields. This new perspective provides a theoretical foundation for addressing current issues and offers practical directions for future research.