The Effects of Perceived AI Use On Content Perceptions

Title of the Paper

The Effects of Perceived AI Use On Content Perceptions

Paper Information

• Domain: Perception of AI-generated content and user behavior
• Keywords: AI-generated content, credibility, human-computer interaction, information literacy, HCI
• Conference: CHI 2024
• DOI: https://doi.org/10.1145/3613904.3642076

Research Background and Issues

• Issues and Challenges:

  1. AI-generated content is becoming increasingly indistinguishable from human-created content, raising concerns about information accuracy and potential misinformation.
  2. Can the public judge whether content is AI-generated solely based on its characteristics? If not, does this affect their perception of the content?
  3. While society and regulatory bodies advocate for labeling AI-generated content, the actual impact of such labeling on user judgments remains unclear.

• Importance of the Research:

  • The rapid proliferation of AI-generated content could significantly alter content consumption patterns.
  • Misinformation may lead to severe consequences, such as poor decision-making and social disruption.
  • If labeling AI use affects content perception, it could influence the relationship between creators and audiences.

• Motivation and Related Work:

  • Extensive research has explored the impact of AI-generated content on credibility, creativity, and user opinions, but findings are inconsistent.
  • People may exhibit "algorithm aversion" or "algorithm appreciation" toward AI-generated content.
  • Most studies focus on distinguishing human and AI content, with less attention paid to user cognition and behavior when no discernible differences exist.

Solution

• Research Methods:

  • A 3×4 mixed experimental design was developed to test whether content was created entirely by humans, collaboratively by humans and AI, or entirely by AI, examining the effects of these three "labels" across four contexts: news, travel, health, and jokes.
  • A preliminary experiment was conducted to ensure minimal objective differences between content versions.

• Key Innovations:

  1. Investigated the impact of perceived AI use on content attributes (e.g., originality and credibility) and creator perceptions (e.g., effort invested).
  2. Analyzed how users interpret the specific meanings of "AI-assisted" or "AI-generated" content.
  3. Enhanced the realism of experimental settings by simulating actual online user environments and providing diverse content types (e.g., long-form text).

• Implementation Steps:

  1. Preliminary Experiment: Created different versions of original content and tested for objective differences between versions.
  2. Main Experiment:
     • Provided "content creator" labels categorized as "purely human-created," "human-assisted by AI," and "AI-generated."
     • Users rated content across multiple dimensions such as credibility, creativity, and trustworthiness.
     • Conducted preference selection and content evaluation.
  3. Open-ended Question Analysis: Asked users how they understood "AI-assisted" or "AI-generated" content.

Research Findings

• Specific Results:

  • Content Perception:

    • Content Evaluation: The perceived "content creator" label (purely human, AI-assisted, human-assisted) had no significant impact on the originality, credibility, or presentation of the content itself.
    • User satisfaction with content was significantly higher under the "purely human" label compared to AI-assisted or AI-generated labels.

  • Creator Perception:

    • Users perceived creators labeled as "purely human" to be more qualified and to have invested greater effort.
    • Labels indicating AI assistance or AI generation led users to hold more negative views of the creator's abilities and effort.

  • User Behavior:

    • Users primarily relied on personal experience and interests to select content rather than the "creator" label.
    • In the health domain, content labeled as human-created was slightly more likely to be shared, though the difference was not significant.

  • User Understanding of AI Use:

    • Identified five major user interpretations of "AI-assisted" and "AI-generated" content (e.g., "human provides ideas, AI writes" or "AI generates content without human intervention").
    • Users exhibited cognitive biases in understanding AI-generated content, such as believing AI-generated content to be more accurate and reliable.

• Comparison with Existing Solutions:

  • Results indicate that while disclosing AI-generated content labels may potentially lower user evaluations of creators, it has minimal impact on actual content judgments (e.g., credibility). This contrasts with some studies suggesting that AI labeling significantly influences content perception.

• Future Directions and Limitations:

  • Limitations:
    • The study only covered specific content formats (long text) and four domains, and results may not generalize to other formats (e.g., images, videos).
    • Certain experimental designs, such as "forced-choice questions," may deviate from real-world user behavior.
  • Future Directions:
    • Expand research to include different content types (e.g., videos, audio) and application scenarios (e.g., shopping, education).
    • Explore more suitable labeling designs to express "AI involvement levels" (e.g., specifying types of edits).
    • Investigate ways to improve user understanding of AI to address potential cognitive biases or misjudgments.

Conclusion

This study reveals the dual impact of perceived AI use on content and creator perceptions. While users generally hold critical attitudes toward AI-generated content, this does not significantly affect their judgments of the content itself. For policymakers and content creators, the findings suggest that discussions around disclosing AI use should focus on user trust, creator reputation, and educational strategies to help users better understand AI functionalities.