Know Your Audience: The benefits and pitfalls of generating plain language summaries beyond the "general" audience

Title of the Paper

Know Your Audience: The Benefits and Pitfalls of Generating Plain Language Summaries Beyond the "General" Audience

Paper Information

• Research Domain: Human-Computer Interaction, Science Communication, Natural Language Processing
• Keywords: Language Complexity, Science Communication, Adaptive Language Generation, Plain Language Summaries, Generative AI, Academic Dissemination, Audience Background Awareness, Human-Computer Interface Design, Automated Summarization, AI-Assisted Science Communication

Research Background and Problem Statement

• Problems and Challenges:

  • In science communication, language barriers pose a significant challenge when conveying complex scientific content to general audiences. Existing studies often overlook the differences in background knowledge within the "general audience," relying on a single language simplification strategy.
  • Plain language summaries (PLS), while easy to understand, may lead to information loss or cause certain audiences to skip content.
  • Generating multi-version scientific summaries using language models (e.g., GPT series) shows promise, but the effectiveness, risks, and impact of background knowledge differences on the reading experience remain unclear.

• Importance of the Research:

  • Failures in scientific language communication may hinder public health decision-making and public engagement in science.
  • Multi-version, audience-adaptive scientific summaries can help disseminate accurate scientific information to a broader audience.

• Motivation and Related Work:

  • Existing research primarily focuses on single-version language simplification without adequately considering audience background knowledge differences.
  • This study aims to compare human expert-authored simplified texts with machine-generated simplified texts to examine audience perceptions and reactions to summaries of varying complexity. The findings will provide practical insights for developing interactive text reading interfaces tailored to diverse audience backgrounds.

Proposed Solution

• Research Methods:

  • Three user studies were designed to evaluate:
    1. Human expert-authored scientific summaries with varying levels of complexity (Study 1).
    2. Machine-generated scientific summaries without content constraints (Study 2).
    3. Machine-generated scientific summaries that aim to retain complete information content (Study 3).
  • In each study, participants read scientific summaries at three complexity levels (high: original article; medium: moderately simplified; low: plain language) and reported their reading experience (e.g., readability, comprehension, interest, perceived value) and behavioral data (e.g., skipping paragraphs, requesting original article links).

• Innovations:

  1. Proposed a strategy for scientific communication that adapts language complexity to diverse audience backgrounds.
  2. Compared the effectiveness of human-authored and machine-generated summaries in terms of language simplification and information retention.
  3. Introduced a model generation approach in Study 3 based on "extended explanation rather than information deletion," exploring the balance between simplified language and information completeness.

• Key Technologies:

  • Used GPT-3 and GPT-3.5 (a more advanced model) to generate scientific summaries.
  • Analyzed the quality of generated texts through automated and manual evaluation methods (e.g., content coverage ratio, sentence alignment evaluation).
  • Applied statistical mixed-effects models to analyze participant responses to summaries of varying complexity.

Research Findings

• Core Conclusions:

  1. Complexity Adaptation and Background Knowledge:
     • For audiences with limited knowledge of scientific topics, the lowest complexity summaries were most appealing, significantly enhancing the reading experience (e.g., ease of understanding, interest) and motivating more requests for the original article.
     • For audiences with deeper background knowledge (i.e., non-researchers but familiar with a specific field), low-complexity summaries increased paragraph skipping rates, particularly for critical information such as research limitations.
  2. Information Integrity and Extended Texts:
     • Study 3 demonstrated that low-complexity summaries generated with complete detail retention ("extended simplification") were more appealing only to readers with very limited background knowledge. For readers with richer background knowledge, such summaries were perceived as lengthy and inefficient.
  3. Comparison of Machine-Generated and Human-Written Texts:
     • GPT-generated texts are sufficiently capable of supporting low-complexity summaries but require expert validation to mitigate common issues such as factual inaccuracies or information loss.

• Advantages:

  • Identified a significant relationship between background knowledge and reading experience, providing a basis for designing personalized, dynamically adaptive reading interfaces.
  • Offered practical generation methods to optimize academic dissemination models (e.g., GPT-3.5) for higher reliability and better usability.

• Limitations:

  • Machine-generated texts still exhibit "hallucination" issues (e.g., incorrect facts), necessitating expert review.
  • The study relied on Amazon Mechanical Turk user samples, which may not represent the diversity of broader audience groups.
  • The actual effectiveness of adding details versus combating text length remains to be further validated.

• Future Directions:

  1. Develop dynamic interactive reading interfaces that allow users to choose their preferred complexity level.
  2. Investigate the dissemination effects of different presentation formats (e.g., incorporating infographics or multimodal interactions).
  3. Explore training AI generation models to enhance factual accuracy and reliability in related domains.

Structured Output Application

Through meticulous experimental design and data analysis, this study provides principles for academic language simplification tailored to audiences with varying background knowledge. These findings offer significant insights for science communication, human-computer interaction design, and generative AI applications.