Keeping Score: A Quantitative Analysis of How the CHI Community Appreciates Its Milestones

Research Background and Problem

• Identified Issues and Challenges: The authors point out that with the continuous evolution of the HCI field and the slowing pace of updates to its research topics, the CHI community's focus on historically significant contributions is gradually diminishing. Citing the "forgetting curve," they indicate that these knowledge milestones are fading from the community's collective memory.
• Why It Matters: These milestones are not only the foundation of the discipline's development but also provide critical guidance for future research directions. Ignoring these achievements could lead to a lack of historical depth in the community's future research.
• Research Motivation:
  • Recent changes in CHI (e.g., growth in publication volume and diversification of research areas) have blurred the community's knowledge focus.
  • The authors aim to quantitatively analyze how the CHI community engages with past contributions and to provoke reflection on knowledge memory and academic impact.

Solution

• Proposed Methods and Solutions:
  • The authors construct and analyze a "forgetting curve" to quantify the citation trends of historical milestone contributions within the CHI community.
  • They propose the "Milestone Coefficient" as a continuous metric to measure the impact of milestone papers.
• Innovations:
  • By introducing the "forgetting curve" and "Milestone Coefficient," the study captures the dynamic changes in the academic community's memory at a quantitative level.
  • The study proposes new metrics based on citation density and temporal dynamics, enabling more precise evaluations of research impact.
• Implementation Steps and Key Techniques:
  1. Collect citation data from CHI papers published between 1981 and 2024 (a total of 11,542 papers).
  2. Use growth-adjusted "forgetting curves" to analyze the annual citation proportion of past papers.
  3. Define and analyze the top 30 and top 300 papers as the primary milestone contributions of CHI.
  4. Employ techniques such as dynamic time warping clustering to distinguish milestone types and develop the "Milestone Coefficient" to generate quantitative impact metrics.

Research Outcomes

• Specific Findings:
  • Key Discoveries: The community's attention to milestone papers is primarily concentrated within 5-15 years after publication, followed by a gradual decline.
  • Milestone Types: Three types of milestones are identified:
    1. "Super Milestones" (e.g., Thematic Analysis proposed in 2006), which continue to grow in influence.
    2. "Declining Milestones," which were widely cited in the past but are now gradually forgotten.
    3. Stable milestones, which are consistently cited but do not reach "super" levels of influence.
  • A classification and detailed statistics of the top 300 milestone contributions are provided.
• Advantages Over Existing Approaches:
  • More effectively captures the temporal dynamics of paper citations compared to traditional metrics based on total citation counts.
  • Provides a detailed explanation of the phenomenon of milestone papers being "forgotten" in stages over time.
• Experimental or Evaluation Results: The proposed "forgetting curve" and "Milestone Coefficient" successfully characterize trends in community citation behavior. For example, the study reveals that since 2004, the community has shown a preference for citing more recent research, and it identifies significant long-tail distribution characteristics among different milestone types.
• Limitations and Future Directions:
  • Limitations:
    • The analysis is limited to the CHI conference and does not cover the entire HCI discipline.
    • Data processing issues include citation text errors and insufficient parsing of "author names."
    • The current method struggles to capture short-term citation peaks for works that are later forgotten.
  • Future Directions:
    • Expand the study to other subcommunities and conferences within HCI.
    • Develop more precise author and paper identification techniques.
    • Investigate the replacement of classic milestones by emerging research.