An Empirical Study to Understand How Students Use ChatGPT for Writing Essays

Paper Title

An Empirical Study to Understand How Students Use ChatGPT for Writing Essays

Publication Info

• Topic area: Investigating the use of ChatGPT by students in academic essay writing.
• Keywords: ChatGPT, generative AI, essay writing, writing education, cognitive process theory, student behavior, writing self-efficacy, technology acceptance, AI-assisted writing, educational technology.

Background and Problem

• Problem / challenge: The detailed functional use of ChatGPT by students in writing tasks, and its impact on writing processes and perceptions, remains underexplored. Existing studies often rely on self-reported data or lack academic context.
• Significance: Understanding how students use ChatGPT is critical for educators to assess its impact on learning, guide its integration into writing pedagogy, and address concerns about academic integrity and critical engagement.
• Motivation and related work: Prior research highlights both the benefits and challenges of generative AI in education, including improved productivity and risks of reduced critical thinking. While datasets exist on AI usage, they often lack detailed interaction data or focus on non-native English speakers. This study builds on Flower and Hayes’ Cognitive Process Theory of Writing to analyze ChatGPT’s role in writing.

Solution

• Proposed approach: A study observing 77 college students writing essays using an in-house ChatGPT platform that captured detailed interaction data, including queries, responses, and keystrokes.
• Novelty:
  1. Development of a taxonomy categorizing ChatGPT queries into Planning, Translating, Reviewing, and All categories.
  2. Identification of distinct student usage patterns, including a “Vibe Writing” mode where students iteratively guide ChatGPT to produce essays.
  3. Analysis of how individual characteristics (e.g., writing self-efficacy, technology acceptance) predict ChatGPT usage patterns.
  4. Creation of a publicly available dataset capturing detailed writing interactions.
• Procedure and key techniques:
  • Students wrote essays on a custom platform integrating ChatGPT (OpenAI API, model 3.5-turbo).
  • Queries were categorized using Flower and Hayes’ framework.
  • Surveys measured writing self-efficacy (SEWS) and technology acceptance (TAM).
  • Essays were analyzed for word count, authorship distribution, and readability (Flesch-Kincaid, Dale-Chall).
  • Clustering analysis identified six distinct usage patterns.

Results

• Concrete findings:
  • 361 queries were categorized into Planning (35 participants), Translating (10 participants), Reviewing (34 participants), and All (50% of participants used at least one All query).
  • 19.5% of students submitted essays entirely written by ChatGPT, with 87.1% of words in some essays being AI-generated.
  • Essays written with ChatGPT had higher readability scores (Flesch-Kincaid and Dale-Chall).
  • Lower writing self-efficacy predicted higher ChatGPT usage, especially for Translating and Reviewing tasks.
  • Perceived usefulness of ChatGPT (TAM PU) correlated with All queries, while ease of use (TAM PEOU) correlated negatively with All queries.
• Advantage over baselines: The study provides empirical, process-level data on ChatGPT usage, surpassing prior self-reported or limited-context studies. It also introduces a taxonomy and clustering approach to analyze usage patterns.
• Experiments / evaluation:
  • Participants: 77 college students (ages 18–64, diverse demographics).
  • Metrics: Query counts, word authorship distribution, readability scores, perceived ownership, and creativity support.
  • Tools: Generalized linear models, Kruskal-Wallis tests, and clustering analysis.
• Limitations and future work:
  • Ecological validity: Conducted in a low-stakes environment, which may not fully replicate classroom settings.
  • Lack of human evaluation of essay quality and originality.
  • Future work includes deploying ChatGPT-integrated platforms in real classrooms and studying instructor perceptions of AI-assisted writing.

Summary

This study investigates how college students use ChatGPT for essay writing, revealing distinct usage patterns categorized into Planning, Translating, Reviewing, and All tasks. Students with lower writing self-efficacy relied more on ChatGPT, and essays generated with AI exhibited higher readability scores. The study highlights concerns about critical engagement and academic integrity, particularly with the emergence of “Vibe Writing,” where students iteratively guide ChatGPT to produce essays. The findings provide actionable insights for educators and contribute a publicly available dataset for further research into AI-assisted writing.