Instructional Mechanisms for Professional Writing: A Comparison of Scaffolded Annotation and ChatGPT

Paper Title

Instructional Mechanisms for Professional Writing: A Comparison of Scaffolded Annotation and ChatGPT

Publication Info

• Topic area: Professional writing instruction and digital tool evaluation
• Keywords: Scaffolded annotation, professional writing, Lettersmith, ChatGPT, cognitive processes, writing quality, instructional mechanisms, generative AI, digital writing tools, cover letters

Background and Problem

• Problem / challenge: Many students lack sufficient guidance to develop professional writing skills, and existing tools like ChatGPT often fail to effectively support the cognitive processes involved in writing. Prior research on Lettersmith has shown its efficacy but has not examined its individual instructional mechanisms or compared it to generative AI tools.
• Significance: Professional writing is critical for securing employment opportunities, yet many students struggle to meet employers' expectations. Effective instructional tools are needed to bridge this gap, particularly for novice writers with limited experience.
• Motivation and related work: Previous studies have demonstrated the benefits of scaffolded annotation in Lettersmith for improving writing quality and understanding. However, the individual contributions of its instructional mechanisms and its comparative efficacy against tools like ChatGPT remain unexplored.

Solution

• Proposed approach: The study evaluates the instructional mechanisms of scaffolded annotation in Lettersmith and compares their efficacy to the unstructured use of ChatGPT for improving cognitive processes and writing quality in professional writing.
• Novelty:
  1. Empirical evaluation of individual instructional mechanisms (checklist, annotated examples, tagging) within scaffolded annotation.
  2. Comparison of scaffolded annotation in Lettersmith to unstructured use of ChatGPT.
  3. Insights into how novice writers engage with generative AI tools like ChatGPT.
  4. Identification of the instructional mechanisms most effective for improving writing quality.
• Procedure and key techniques:
  1. Conducted a lab experiment with 146 first-year college students tasked with writing and revising a cover letter.
  2. Participants were assigned to six conditions, each featuring different instructional mechanisms or unstructured use of ChatGPT.
  3. Measured changes in cognitive processes and writing quality using surveys and coding of cover letters.
  4. Analyzed qualitative themes from participants’ interactions with ChatGPT.

Results

• Concrete findings:
  • Lettersmith’s full scaffolded annotation (checklist, annotated examples, tagging) led to the greatest improvement in understanding what constitutes a quality cover letter.
  • Combining a checklist with another mechanism (e.g., examples or tagging) improved writing quality more than using a checklist or example alone.
  • ChatGPT’s unstructured use resulted in the least improvement in writing quality and cognitive processes.
  • Novice writers with no prior cover letter experience benefited most from scaffolded annotation.
• Advantage over baselines:
  • Lettersmith outperformed ChatGPT in improving both cognitive processes and writing quality.
  • The checklist was identified as the most impactful individual mechanism, especially when combined with other features.
• Experiments / evaluation:
  • Participants wrote and revised cover letters in six conditions: checklist, non-annotated example, checklist + example, checklist + tagging, full Lettersmith, and unstructured ChatGPT.
  • Writing quality was assessed based on nine key components of a cover letter, and cognitive processes were measured using pre- and post-surveys.
• Limitations and future work:
  • Focused on first-year college students; results may not generalize to experienced writers.
  • Lab-based setting may not fully reflect real-world writing contexts.
  • Future work could explore structured ChatGPT use, integration of GenAI with scaffolded annotation, and field experiments in classrooms or employment centers.

Summary

This study evaluated the instructional mechanisms of scaffolded annotation in Lettersmith and compared them to unstructured use of ChatGPT for professional writing instruction. Results showed that combining mechanisms like checklists, annotated examples, and tagging within Lettersmith significantly improved students’ understanding and execution of quality cover letters, particularly for novice writers. In contrast, unstructured use of ChatGPT did not outperform scaffolded annotation. These findings highlight the importance of structured guidance in digital writing tools and suggest opportunities for integrating generative AI with pedagogical scaffolding to further enhance professional writing instruction.