Intelligent Support Engages Writers Through Relevant Cognitive Processes

Document Title

Intelligent Support Engages Writers Through Relevant Cognitive Processes

Document Information

• Subject Area: Human-Computer Interaction, Intelligent Writing Support, Learning Technologies
• Keywords: Intelligent Writing Support, Cognitive Processes, Peer Review by Students, Large Language Models, Educational Technology, Human-Computer Interaction, User Experience, Cognitive Writing Theory

Research Background and Issues

• Identified Problems or Challenges:

  • Peer review by students is increasingly common in online education (especially MOOCs), but students face time-consuming tasks and "writer's block."
  • A lack of effective tool support may lead to decreased willingness to participate and lower review efficiency.
  • While large language models (LLMs) such as OpenAI's GPT framework have the potential to provide support, designs that are not user-centered may result in excessive cognitive load.

• Research Significance:

  • With the massive scale of MOOC users (reaching 220 million in 2021), efficient writing support is crucial for improving learning outcomes.
  • Designing intelligent tools that target critical cognitive processes in writing can significantly enhance students' participation in peer review and the quality of their output.

• Research Motivation and Related Work:

  • The Cognitive Process Theory of Writing emphasizes the internal cognitive processes of writers, providing theoretical guidance for designing support tools.
  • Current writing support tools often focus on surface-level functions like grammar checking, lacking support for complex writing tasks such as idea generation and evaluation.
  • Combining the generative capabilities of LLMs with cognitive writing theory is a promising research direction.

Solution

• Proposed Solution:

  • Developed an intelligent writing support tool based on GPT-3.5-Turbo, focusing on assisting two key cognitive processes: ideation and evaluation.
  • The tool was designed with a user-centered approach to ensure that its features closely align with students' actual needs.

• Innovations:

  • Introduced cognitive writing theory as the core of the design, limiting and optimizing intelligent features to meet the specific cognitive demands of tasks.
  • Conducted experiments to verify that intelligent support significantly impacts task completion time and students' tool usage habits.
  • Integrated new features of LLMs to provide personalized, interactive feedback and support.

• Implementation Steps:

  1. Analyzed three core cognitive processes of writing—planning, translating, and reviewing—based on cognitive writing theory.
  2. Designed corresponding functions for ideation and evaluation (e.g., a generation button and an automatic suggestion box).
  3. Experimental setup: Employed a fully randomized 2x2 design, with four participant groups receiving different combinations of static or intelligent support.
  4. Data collection: Recorded writing time, button click frequency, and changes in cognitive process stages (exploration vs. exploitation).

Research Findings

• Key Findings:

  • Intelligent writing support significantly increased task duration (e.g., the ideation phase increased by an average of 100 seconds), indicating improved student engagement and focus.
  • When static content (e.g., pre-generated suggestions) was provided, students significantly reduced their use of intelligent features (a 39% decrease in ideation support and a 59% decrease in evaluation support).
  • In the submitted task texts, approximately 77% of the generated ideas and 34% of the evaluation suggestions were adopted after using the tool.

• Comparison with Existing Solutions and Advantages:

  • Compared to traditional tools limited to grammar checking, the intelligent tool in this study provided more support for users' cognitive processes.
  • The intelligent writing support demonstrated clear advantages in offering specific and diverse suggestions.
  • The tool's design, grounded in cognitive theory, avoided cognitive overload caused by excessive functionality.

• Experimental Results and Evaluation:

  • Clarified the role of intelligent support in different writing task stages, such as ideation and evaluation.
  • Students generally found the tool intuitive and easy to use, and the time differences during task operations were interpreted as higher task engagement.
  • User experience data indicated that the intelligent tool enhanced user satisfaction and the quality of task completion.

• Limitations and Future Directions:

  • Limitations:
    • Participants were primarily Prolific platform users or students, and the results may not generalize to a broader group of professional writers.
    • The 250-word task requirement used in the study was reported as unnatural, potentially affecting participants' genuine responses to the task.
    • The study did not include longitudinal research, leaving the long-term effects of tool usage unexamined.
  • Future Research Directions:
    • Expand research to participants from different fields and demographics, such as professional writers and non-native speakers.
    • Explore the long-term effects of the tool on task pacing and process transitions, and design support tools that better align with natural writing workflows.
    • Investigate ethical issues related to intelligent tools, such as potential plagiarism and the generation of inaccurate information, in greater depth.