Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models

Title of the Paper

Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models

Paper Information

• Subject Area: Human-Computer Interaction (HCI), Artificial Intelligence, and Collaborative Writing
• Keywords: AI-generated text, collaborative writing, human-AI collaboration, generative language models, writing assistants

Research Background and Problem

• Identified Issues or Challenges: With the advancements in generative language models (e.g., GPT-3), their potential in collaborative writing has garnered significant attention. However, there is still a lack of comprehensive understanding of how varying levels of AI support influence the human writing process and outcomes.
• Significance: AI writing tools hold immense potential to enhance creativity and facilitate expression. However, their design must ensure human agency to maintain user satisfaction and a sense of ownership over the text. Research on balancing AI support with human autonomy is crucial for the future design of writing tools.
• Research Motivation and Related Work:
  1. Traditional writing support tools typically offer simple completion functions but fail to effectively assist with complex writing needs.
  2. Previous studies have shown the potential of generative AI to improve efficiency in areas such as drafting help requests, storytelling, and academic writing. However, the impact of different levels of AI support on user experience remains underexplored.
  3. Related works referenced include studies on WordCraft and CoAuthor, which analyze the capabilities and applicability of generative AI in collaborative writing.

Solution

• Proposed Methods or Solutions:
  1. Develop an experimental platform using a generative language model (GPT-3) to simulate human-AI collaborative writing scenarios.
  2. Design three levels of AI support: no AI support (baseline condition), single-sentence suggestions (low-level support), and full-paragraph suggestions (high-level support). Analyze their effects on writing quality and user experience.
  3. Conduct classroom experiments with 131 participants to collect real-world writing data and subjective feedback.
• Innovations:
  1. Developed a multi-level scaffolding framework for AI-assisted writing.
  2. Used a U-shaped relationship model to analyze the impact of low and high levels of support on writing quality, user satisfaction, and productivity.
  3. Quantified how technical proficiency and writing skills influence outcomes in human-AI collaboration.
• Implementation Steps and Key Technologies:
  1. Employed a Latin Square Design in the experiments to balance conditions and minimize order effects.
  2. Utilized multiple measurement methods, including human ratings, automated tools (e.g., TAACO and TAALES), and NASA Task Load Index.
  3. Collected behavioral data on user editing habits and evaluated user satisfaction and sense of ownership in AI collaboration.

Research Findings

• Specific Findings:
  1. Provided evidence on how varying levels of scaffolding influence writing quality: low-level scaffolding (single-sentence suggestions) may disrupt text coherence, while high-level scaffolding (paragraph suggestions) significantly improves writing quality.
  2. High-level scaffolding notably enhanced the writing performance of non-expert and less tech-savvy users.
  3. While AI suggestions increased productivity (words per unit time), they also led to a decrease in user satisfaction and sense of ownership over the text.
  4. No significant impact on cognitive load was observed.
• Advantages: Compared to the no-support condition, high-level scaffolding offered a more coherent article structure, significantly reduced the amount of content participants needed to edit, and improved quality scores.
• Experimental or Evaluation Results:
  1. The U-shaped relationship showed that low-level scaffolding reduced text output quality, while high-level scaffolding significantly improved it.
  2. Data indicated increased user reliance on paragraph-level AI suggestions, which may reduce writing autonomy.
• Limitations and Future Directions:
  1. Did not explore complex scaffolding systems that dynamically adjust support levels.
  2. The experiment was limited to English-speaking users and did not include multilingual or low-writing-skill groups.
  3. The experimental scenario was confined to short-term testing, without observing the long-term impact of AI tools on writing habits.
  4. Future research could expand to more diverse writing tasks (narrative, technical writing, etc.) and real-world writing contexts (academic, journalistic writing, etc.).

Through this study, the authors emphasize the necessity of designing user-centered AI writing tools. These tools should dynamically adjust support levels based on diverse user needs to better foster creativity and productivity while maintaining user satisfaction and engagement.