Toward Living Narrative Reviews: An Empirical Study of the Processes and Challenges in Updating Survey Articles in Computing Research

Research Background and Problem

• Problem or Challenge:
  In academic research, review papers are essential tools for building a knowledge foundation. However, due to the resource-intensive process of creating these papers and their tendency to become outdated over time, they often face challenges of obsolescence and inaccuracy. Literature indicates that some reviews may already be outdated at the time of publication, and over 25% of reviews require updates within two years of publication.

• Significance:
  Outdated information can lead scholars to miss research opportunities or mislead decision-makers. Maintaining the timeliness of review papers can better reflect the latest research trends, enhancing their scientific value and applicability.

• Research Motivation and Related Work:
  The proposed concept of "dynamic reviews" aims to address this issue. However, existing research primarily focuses on "dynamic systematic reviews," with limited support for updating semi-systematic and narrative reviews that involve expert interpretation and conceptual synthesis. Furthermore, with the development of artificial intelligence, particularly large language models, these technologies have the potential to improve literature discovery, evaluation, and updating processes.

Solution

• Proposed Solution by the Authors:
  Through in-depth interviews with 11 authors of review papers in the computing field, the study analyzes their workflows and challenges in writing and updating reviews, aiming to identify key technologies and methods to improve update efficiency and effectiveness.

• Innovative Contributions:
  The authors distinguish three types of updates—empirical updates, structural updates, and interpretative updates—and explore how artificial intelligence can assist with these updates to reduce costs and improve efficiency. This clear categorization and analysis of AI-assisted potential is an area that previous research has not sufficiently explored.

• Implementation Steps and Techniques:

  1. Interviews and Data Collection

     • Select authors of review papers published in the past 1-3 years and conduct semi-structured interviews to understand their practices in writing and updating reviews.
     • The interviews include open-ended questions, primarily tracking the detailed processes of literature search, screening, coding, writing, and revision.

  2. Data Analysis

     • Use thematic analysis to code and categorize the data, summarizing high-level themes.

  3. Exploration of AI-Assisted Potential

     • Analyze authors' perspectives on AI's roles in automation, collaboration, and as a "second opinion."
     • Propose specific functionalities for AI tools to support dynamic review updates, such as assisting in literature discovery, numerical updates, and trend prediction.

Research Findings

• Specific Findings:

  • Identified four core workflows in writing review articles (literature search, quality evaluation, organization and classification, interpretation and synthesis) and the challenges associated with each.
  • Analyzed three types of dynamic review updates—empirical updates, structural updates, and interpretative updates—each presenting unique challenges and opportunities for AI support.
  • Provided specific recommendations for AI-supported update processes, such as automated numerical updates, literature screening assistance, and revealing trends and changes in the field.

• Advantages Compared to Existing Solutions:

  • Focused on the practical needs of semi-systematic updates for computational review articles, rather than solely emphasizing more standardized systematic reviews.
  • Explored how technologies, including AI, can address the lack of incentives for updates in academia.

• Experimental or Evaluation Results:

  • Interviews revealed that researchers perceive significant value in updating reviews for the academic community. However, due to insufficient academic incentives and the time-consuming nature of the process, long-term updates are challenging to implement.
  • Artificial intelligence is seen as helpful for handling repetitive tasks but is considered less capable of performing creative tasks requiring deep understanding and critical thinking.

• Limitations and Future Directions:

  • Limitations:

    • The study relies on interviews based on researchers' recollections, which may be subject to recall bias.
    • It is limited to the computing field and does not comprehensively cover other disciplines.

  • Future Directions:

    • Conduct real-time observational studies to directly record researchers' practices during the review update process.
    • Explore the needs of review article readers and develop more interactive "dynamic review" output formats.
    • Design hybrid systems that leverage authors' explicit literature search and interpretation strategies to enhance AI's functionality in supporting dynamic review updates.

In summary, this paper provides significant insights into "dynamic narrative reviews," identifying key issues that hinder updates while proposing potential AI-assisted solutions, paving the way for the development of more efficient update systems.