AI Personalization Paradox: Reading Highlights for Personalized AI-Assisted Writing Increases Engagement but Undermines Autonomy and Ownership

Paper Title

AI Personalization Paradox: Reading Highlights for Personalized AI-Assisted Writing Increases Engagement but Undermines Autonomy and Ownership

Publication Info

• Topic area: Personalization in AI-assisted writing and its effects on user behavior and perceptions.
• Keywords: AI-assisted writing, personalization, reading highlights, autonomy, ownership, engagement, argumentative writing, user behavior, cognitive processes, AI reliance.

Background and Problem

• Problem / challenge: Current personalized AI-assisted writing systems focus primarily on the writing phase, neglecting the intertwined nature of reading and writing. While personalization can enhance engagement, it risks over-reliance on AI and undermines user autonomy and ownership.
• Significance: Understanding how reading behaviors, such as highlighting, can inform personalization in AI-assisted writing is crucial for designing systems that balance support with user autonomy and ownership.
• Motivation and related work: Prior work has explored personalization in AI-assisted writing but has largely ignored the reading phase, despite its cognitive and behavioral links to writing. Existing systems like GhostWriter focus on writing interactions, leaving a gap in understanding how reading behaviors, such as highlighting, can influence personalized writing.

Solution

• Proposed approach: Integrating reading highlights into personalized AI-assisted writing systems to tailor AI suggestions based on user-selected content during reading.
• Novelty:
  1. Empirical investigation of how reading highlights influence behaviors and perceptions in AI-assisted writing.
  2. Identification of a paradox where personalization increases engagement but undermines autonomy, ownership, and self-credit.
  3. Exploration of the altered relationship between reading and writing due to personalization.
• Procedure and key techniques:
  • Conducted a between-subjects experiment with 46 participants tasked with writing argumentative essays.
  • Two conditions: Personalization (AI suggestions based on user highlights) and Baseline (AI suggestions based on full reading materials).
  • Measured reading and writing behaviors, AI reliance, and subjective perceptions using system logs and Likert-scale surveys.
  • Analyzed correlations between reading and writing metrics under both conditions.

Results

• Concrete findings:
  • Personalization participants made significantly more highlights (M = 22.96 vs. M = 7.23 in Baseline), with higher density (M = 0.452 vs. M = 0.051) and longer reading durations (M = 569.74 seconds vs. M = 392.72 seconds).
  • Personalization led to higher AI reliance (M = 0.924 vs. M = 0.710) and acceptance ratios (M = 0.489 vs. M = 0.307).
  • Baseline participants exhibited more edits (M = 1847.14 vs. M = 587.96) and longer essays (M = 514.64 words vs. M = 415.21).
  • Autonomy, ownership, and self-credit were significantly higher in Baseline (autonomy: M = 5.38 vs. M = 4.39; ownership: M = 4.80 vs. M = 3.72; self-credit: M = 5.42 vs. M = 4.64).
• Advantage over baselines:
  • Baseline condition supported deeper engagement in writing and stronger perceptions of authorship, autonomy, and ownership.
  • Personalization increased reading engagement but at the cost of turning highlighting into an instrumental act for AI input.
• Experiments / evaluation:
  • Participants wrote argumentative essays based on provided reading materials under controlled conditions.
  • Metrics included reading behaviors (e.g., highlight density), writing behaviors (e.g., edits, AI reliance), and subjective perceptions (e.g., autonomy, ownership).
  • Statistical analyses revealed significant differences between conditions and correlations between reading and writing metrics.
• Limitations and future work:
  • Focused on argumentative writing; other genres may yield different results.
  • Limited to highlights as personalization cues; other annotations like comments remain unexplored.
  • Explicit awareness of personalization effects may have influenced participant behavior; future studies should explore implicit or partially transparent personalization.

Summary

This study investigated the use of reading highlights for personalized AI-assisted writing, revealing that while personalization increased reading engagement, it shifted highlighting from a comprehension tool to an instrumental act for AI input. This led to higher AI reliance and diminished autonomy, ownership, and self-credit during writing. In contrast, the Baseline condition fostered deeper engagement and stronger perceptions of authorship. The findings highlight the risks of personalization in undermining intrinsic motivation and suggest design strategies to preserve user autonomy and ownership, such as separating cognitive highlights from personalization cues and embedding reflective reading practices.