From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants

Paper Title

From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants

Publication Info

• Topic area: The impact of mental models on user interaction with AI-based writing assistants.
• Keywords: AI writing assistants, mental models, user behavior, oversight, trust, interaction design, writing quality, human-computer interaction, system safety, user experience.

Background and Problem

• Problem / challenge: While mental models are known to influence user interaction with complex systems, their role in shaping engagement with AI-based writing assistants remains underexplored, particularly in contexts involving system errors.
• Significance: Understanding how mental models affect user oversight and writing outcomes is critical for designing AI systems that support effective and safe user interaction, especially as such systems become ubiquitous in everyday tasks.
• Motivation and related work: Prior research in human-computer interaction and system safety has shown that accurate mental models improve trust, satisfaction, and decision-making. However, gaps remain in understanding how functional versus structural mental models influence user behavior and writing quality in AI-assisted writing.

Solution

• Proposed approach: A controlled experiment comparing the effects of functional and structural mental models on user interaction with an AI-based writing assistant.
• Novelty:
  1. Empirical analysis of how mental models influence user behavior and writing quality in the presence of AI-generated errors.
  2. Development of a controlled experimental framework to manipulate and test mental models using instructional priming.
  3. Insights into the role of interaction affordances in shaping user perceptions of control and ownership.
• Procedure and key techniques:
  • Participants (N = 48) were primed with functional or structural mental models via instructional videos.
  • Participants completed a cover letter writing task using a modified CoAuthor platform with deliberately injected grammatical errors.
  • Behavioral data, final writing quality, and self-reported experiences were analyzed to assess control behavior, writing outcomes, and user perceptions.

Results

• Concrete findings:
  • Participants in the structural condition produced letters with more grammatical errors (calibrated corrections: t(46) = −2.316, p < 0.05, d = 0.67).
  • Structural participants accepted more erroneous suggestions (errors per word: U = 170.0, p < 0.05, r = 0.352).
  • No significant differences were observed in broader writing quality dimensions (relevance, flow, tone/style).
  • Structural participants rated the system as easier to use (t(46) = −2.07, p < 0.05, d = 0.60).
• Advantage over baselines:
  • Structural mental models increased perceived ease of use but also led to higher error acceptance, suggesting a tradeoff between trust and oversight.
  • Interaction affordances (e.g., user control over suggestions) played a more significant role in fostering ownership and control than mental model depth.
• Experiments / evaluation:
  • Writing task: Participants wrote a cover letter for a fictional job posting.
  • Evaluation metrics: Behavioral logs (e.g., suggestion requests, acceptance ratio), Grammarly correctness scores, qualitative rubric-based assessments, and self-reported surveys.
  • Statistical analyses: Independent-samples t-tests, Mann–Whitney U tests, and thematic analysis of interviews.
• Limitations and future work:
  • Mental models were assessed at a single point in time and may evolve dynamically.
  • Participant variability (e.g., writing proficiency, native language) may have influenced results.
  • Future research should explore adaptive scaffolding, progressive disclosure, and interaction techniques to support ongoing mental model calibration.

Summary

This study investigated how functional and structural mental models influence user interaction with AI-based writing assistants. Structural mental models improved perceived ease of use but led to higher acceptance of erroneous suggestions, resulting in more grammatical errors in final outputs. Interaction affordances, rather than mental model depth, were key to fostering user control and ownership. These findings highlight the need for designing AI systems with robust interaction mechanisms and adaptive feedback to support effective oversight and safe use. Future work should explore dynamic mental model calibration and its impact on user behavior in diverse writing contexts.