The Development and Validation of the Technology-Supported Reflection Inventory

Title of the Paper

The Development and Validation of the Technology-Supported Reflection Inventory

Publication Information

• Domain: Human-Computer Interaction (HCI), Technology-Supported Reflection
• Keywords: Personal Informatics, Reflection, Tracking, Construct, Reflective Technology, Scale Development, Validity Testing, Measurement Tool
• Conference: CHI Conference on Human Factors in Computing Systems (CHI '21)
• Publication Date: May 2021

Research Background and Problem Statement

• Identified Problems:

  • The HCI field increasingly focuses on designing technologies to support user reflection, but evaluating how and to what extent these technologies facilitate reflection remains a challenge.
  • The lack of evaluation methods for technology-supported reflection makes it difficult to compare different designs, hindering further technological innovation.
  • Existing evaluation methods (quantitative or qualitative) have limitations, such as focusing on user experience rather than reflection or the ambiguous definition of reflection.

• Research Significance:

  • Reflection is considered highly valuable for enhancing user well-being and personal growth, making it an important topic to explore how technology can enhance users' reflective abilities.
  • The absence of consistent and standardized evaluation criteria in existing reflective technology designs constrains the field's development.

• Research Motivation and Related Work:

  • Most existing reflection studies are based on traditional psychological definitions, but evaluation tools for "reflection support" in technology design are underdeveloped.
  • Although some technologies (e.g., personal informatics systems, health monitoring technologies) have been reported to successfully support reflection, quantifying the degree of support provided by these technologies remains unclear.
  • Baumer et al. (2014) highlighted the lack of consensus in evaluating reflection-supporting technologies, motivating the authors to develop a unified and effective measurement tool.

Proposed Solution

• Proposed Solution:

  • Developed a scale called the "Technology-Supported Reflection Inventory (TSRI)" to evaluate the extent to which systems support individual reflection.
  • TSRI was developed under expert guidance through a multi-step process, including literature review, item generation, factor analysis, and validation.

• Methodological Innovation:

  • The first systematically validated reflection evaluation scale centered on interactive systems, providing researchers with a standardized comparison tool.
  • Operationalized reflection into three dimensions (Insight, Exploration, Comparison), each containing items closely related to human-technology interaction design.

• Implementation Steps:

  1. Item Generation: Combined literature review and expert discussions to initially generate 115 potential scale items.
  2. Expert Review: Refined the items to 58 through two rounds of feedback from multidisciplinary experts.
  3. Factor Analysis: Used exploratory factor analysis to reduce the items, resulting in a final 9-item scale divided into 3 dimensions.
  4. Validity and Reliability Testing: Tested the robustness of the TSRI tool through construct validation (e.g., comparison with related scales) and test-retest reliability verification.

• Key Techniques:

  • Used a Likert scale to collect user data.
  • Applied statistical methods (e.g., factor analysis, reliability analysis) to optimize item quality and validate the scale model.
  • Designed experiments to test responses across different user groups.

Research Findings

• Specific Outcomes:

  • The TSRI scale includes 9 items divided into three dimensions: "Insight," "Exploration," and "Comparison," each named after a core function that helps users reflect.
  • Insight: Describes whether the system provides users with new perspectives and inspires shifts in thinking.
  • Exploration: Assesses the ease and appeal of analyzing and browsing personal data.
  • Comparison: Measures the ability of users to discuss and compare with others through the system.

• Advantages Over Existing Solutions:

  1. Focuses on reflection support in technology-user interaction rather than individual psychological traits, making it suitable for evaluating specific interactive systems.
  2. Underwent rigorous validation processes (e.g., construct validity, discriminant validity, test-retest reliability), ensuring scientific rigor and reliability.
  3. Can be used for rapid prototype evaluation, supporting direct comparisons across design solutions.

• Experimental or Evaluation Results:

  • Confirmatory factor analysis results showed good model fit (TLI=0.91, RMSEA=0.04).
  • TSRI distinguished between system designs with varying levels of reflection support, and construct validity indicated weak to moderate correlations between TSRI and individual reflective abilities.
  • Test-retest reliability results (κ=0.73, p<0.01) demonstrated the stability of TSRI across different time points.

• Limitations and Future Directions:

  1. Cultural Bias: The sample was concentrated in Western countries, lacking cross-cultural validity testing.
  2. Scope of Application: The scale currently emphasizes data-driven interactive technologies, with limited applicability to non-data-based reflection support technologies.
  3. Future Work:
     • Extend the scale to different types of interactive technologies.
     • Validate its cross-cultural consistency by incorporating more cultural contexts.
     • Apply and test the scale's effectiveness in more real-world usage scenarios.

Conclusion

This study developed and validated the TSRI, a tool for evaluating technology-supported reflection, addressing the gap in standardized reflection scales within the HCI field. It provides a critical reference tool for future design and research on reflection-supporting systems. Through rigorous development and validation methods, the scale ensures scientific rigor and practical utility.