Understanding and Supporting Peer Review Using AI-reframed Positive Summary

Research Background and Problem

• Problem or Challenge:

  • Peer review is crucial for improving writing quality and the effectiveness of research outcomes, but critical feedback can often leave authors feeling frustrated and demotivated.
  • In current practices, reviewers face significant cognitive burdens, needing to provide professional feedback while maintaining a positive tone. This often results in feedback being overly harsh or negative.
  • Existing research lacks empirical studies on improving positive feedback strategies, particularly the application of artificial intelligence in reframing critical feedback.

• Significance:

  • If critical feedback is not handled properly, it can reduce researchers' self-efficacy and motivation to revise, negatively impacting the healthy evolution of the research ecosystem, such as diminishing diversity within research communities.

• Research Motivation and Related Work:

  • Cognitive reframing has been shown to have potential in alleviating the negative emotions caused by criticism, but its application is complex and requires professional training. With the development of large language models (LLMs), using AI to reframe feedback has become feasible.
  • This study also incorporates the impact of "overall evaluation" (e.g., scoring), which is common in peer reviews, but its synergistic effect with feedback reframing has not been fully explored.

Solution

• Proposed Solution:

  • Use artificial intelligence to generate positive reframed summaries and append them to traditional critical feedback to improve authors' emotional, cognitive, and behavioral responses to criticism.

• Innovations:

  • Propose AI-assisted generation of "positive reframed summaries" as a novel supplement to traditional critical feedback.
  • Explore the combined effects of "overall evaluation scores" and AI reframed summaries on feedback reception and revision behavior.
  • Integrate attribution theory to analyze factors influencing revision motivation.

• Implementation Steps and Key Techniques:

  1. Use the GPT-4 model to generate critical feedback, adding AI-generated positive reframed summaries in certain groups.
  2. Design a 2×2 online experiment (presence/absence of AI reframed summaries × high/low overall evaluation scores) involving 137 participants.
  3. Use quantitative and qualitative methods to measure emotional changes (e.g., self-efficacy, emotions), perceptions of feedback (usefulness and fairness), and the quality and quantity of final revisions.

Research Findings

• Specific Findings:

  • Emotional and Cognitive Perception: AI reframed summaries significantly enhanced participants' positive emotions, sense of autonomy, and sense of competence, while improving their perception of the feedback and reviewers.
  • Revision Behavior: Low overall evaluation scores prompted authors to revise their texts more seriously, while AI reframed summaries mainly influenced participants' subjective perception of revision quality but did not significantly improve actual revision quality.
  • Factors Influencing Revision Motivation: Through qualitative analysis, seven key factors influencing revision motivation were identified:
    • Internal Attribution Factors (e.g., actionable feedback, alignment with personal goals, emotional reactions, motivation for learning and self-improvement).
    • External Attribution Factors (e.g., feedback quality, tone, and reviewers' professionalism and effort).

• Comparison with Existing Solutions and Advantages:

  • Building on existing studies that focus solely on positive praise or harsh criticism, this study empirically validates the potential of AI to "reframe" critical feedback.
  • Closely integrates LLM technology with cognitive reframing theory, applying it to the significant domain of academic peer review.

• Experimental and Evaluation Results:

  • After adding AI reframed summaries, participants generally perceived the feedback as fairer, though actual revision behavior showed no significant differences.
  • Low overall evaluation scores led to greater revision effort but might also demotivate participants with lower self-efficacy.

• Limitations and Future Directions:

  • Limitations:
    • The writing task focused on cover letters rather than academic articles, which, while easier to control in experiments, differs from actual academic review scenarios.
    • The study did not test the impact of transparently disclosing AI involvement, which could be critical for building trust with authors.
  • Future Directions:
    • Extend the study to real-world academic peer review contexts, exploring the potential of AI reframing using real feedback datasets.
    • Further optimize LLM reframing capabilities for personalized cognitive reframing strategies.
    • Investigate the impact of transparency regarding AI involvement in the feedback generation process on authors' perceptions and trust.

Conclusion

This study lays the foundation for the application of AI in improving critical feedback in academic peer review, demonstrating the potential advantages of AI reframing strategies in providing emotional support to authors. Combining low overall evaluation scores with AI reframed summaries may be an effective strategy to balance criticism and constructive feedback, contributing to a more supportive and constructive academic community culture.