Past, Present, and Future of Citation Practices in HCI

Research Background and Problem

• What issues or challenges did the authors identify?
  The authors identified that in the field of Human-Computer Interaction (HCI), the number of references in papers has significantly increased since the introduction of new editorial policies at the ACM CHI conference in 2016. This change has led to a profound transformation in academic practices, with the average number of citations per paper showing a linear annual increase. This trend could result in an average of approximately 130 references per CHI paper by 2030.

• Why is this issue important?
  The rapid growth in the number of references has caused fatigue for both authors and reviewers, impacting the sustainability of academic work and the quality of the literature. Additionally, the citation culture has increasingly emphasized quantity over quality, meaning excessive citations may dilute their value and potentially undermine academic rigor.

• Research motivation and related work
  By analyzing CHI conference papers from 1981 to 2024, the authors aim to examine the impact of the 2016 editorial policy change on citation behaviors within the community. They also explore other potential factors, such as the number of co-authors, incentive mechanisms, and preprint citations, that may further drive this trend. This research falls within the domain of meta-research, shedding light on how academic policy decisions influence the broader research community.

Solutions

• What methods or solutions did the authors propose?
  The authors proposed an event study method combined with a linear regression model to quantify changes in citation behavior since the 2016 policy shift. They also explored a range of contextual factors that might influence citation counts, including incentive mechanisms, the growth of co-authorship, the use of literature reviews, and preprint citations.

• What is innovative about this solution?
  The authors utilized a combination of event studies and regression models to analyze the long-term effects of academic policy changes on citation behavior. This approach not only examines the direct outcomes of policy changes but also investigates the complex contextual factors influencing citation practices, such as academic incentives and publishing culture.

• What are the implementation steps and key techniques used?

  1. Data Collection: Extracted reference and author information from 11,542 CHI conference papers published between 1981 and 2024.
  2. Statistical Analysis: Predicted future citation trends using linear regression analysis and evaluated the impact of policy changes through event studies.
  3. Trend Exploration: Visualized and analyzed contextual factors influencing citation behaviors, including literature reviews, preprint citations, and references to code and data repositories.
  4. Potential Causal Analysis: Measured the impact of incentive mechanisms and other factors on citation counts.
  5. Proposed Solutions: Suggested potential measures to mitigate excessive citation growth, such as reintroducing limits and leveraging language models to assist authors in writing.

Research Findings

• What specific findings were obtained?

  1. The authors provided strong evidence that citation practices in the CHI community have significantly changed since the 2016 policy shift, with citation counts showing a linear increase.
  2. Data indicated that incentive mechanisms had a limited impact on citation growth, but other factors, such as preprint citations and potentially predatory publications, may have contributed to the increase.
  3. Literature reviews and machine learning-related citations have emerged as new trends in CHI papers.

• What advantages does this solution have compared to existing ones?
  The authors’ research comprehensively analyzed the contextual factors behind policy and behavioral changes, rather than merely observing overall trends. This multi-layered, in-depth analysis highlighted the far-reaching effects of policy changes, offering new perspectives on academic culture in the HCI field. Additionally, this approach provides a reference for other research domains to help plan and evaluate the impact of academic policies.

• What were the experimental or evaluation results?

  1. If the current trend continues, CHI papers are projected to include an average of approximately 130 references by 2030.
  2. Event studies demonstrated the significant impact of policy changes, while other contextual factors (e.g., the number of co-authors) contributed only marginally to citation growth.
  3. Notable contextual trends included a significant increase in references to open data and code repositories and a growing trend of citing "systematic literature reviews."

• Limitations and future directions

  1. Although the study comprehensively explored contextual factors, it may have overlooked other unexamined variables.
  2. Future research could delve deeper into the motivations behind author behaviors, potentially through interviews or the Delphi method.
  3. The authors suggested exploring how machine learning could be integrated into academic peer review to alleviate reviewer fatigue and developing more intelligent citation tools.

Through this paper, the authors call on the HCI community to reassess its citation practices, rebalancing the quantity and quality of references in academic papers. They propose several feasible solutions to promote the sustainable development of academic culture.