Wordcraft: Story Writing With Large Language Models

Document Title

Wordcraft: Story Writing With Large Language Models

Document Information

• Topic Area: Applications of machine learning and natural language processing in human-computer collaborative creation
• Keywords: Large language models, GPT-3, LaMDA, human-computer collaboration, creative writing, natural language generation, text editing, prompt engineering, human-computer interaction

Research Background and Problem

• Problems and Challenges Identified by the Authors:

  • Although large language models (such as GPT-3) have demonstrated exceptional language understanding and generation capabilities, how to effectively utilize these models for creative writing remains an open question.
  • Current AI collaboration tools are overly narrow, focusing primarily on "accuracy" metrics such as grammar correction and text auto-completion, failing to fully leverage the potential flexibility of language models.
  • There is a lack of empirical research on the practical use and effectiveness of these models in the domain of writing.

• Importance of the Research:

  • Literary creation is a significant cultural activity, and human-computer collaboration has the potential to inspire creativity and transform traditional writing methods.
  • Designing more advanced collaborative tools could drive widespread adoption of AI in content creation and other domains.

• Motivation and Related Work:

  • Previous research and applications have primarily focused on limited domains (e.g., painting, music, game design), with insufficient exploration of the writing domain.
  • This study is inspired by recent interactive writing tools like Write With Transformer and AI Dungeon, aiming to further investigate how AI impacts the writing process.
  • The goal is to address challenges in story generation through direct collaboration with large language models and to develop tools that support customized writing tasks.

Solution

• Proposed Solution:

  • Developed a web application called Wordcraft, enabling users to collaborate with large language models (LaMDA) to complete story writing.
  • Provides multiple built-in text editing control features (e.g., filling, continuation, rhetorical style transformation) and allows users to create custom operations on demand to meet specific needs.

• Innovative Aspects of the Solution:

  • Utilizes LaMDA's powerful conversational capabilities to achieve flexible natural language generation through few-shot learning and relevant example prompting techniques.
  • Offers a comprehensive user experience design to enhance the intuitiveness and efficiency of human-AI collaboration.
  • Supports users in creating custom requests in real-time during the writing process, completing complex tasks without the need to construct prompts independently.

• Implementation Steps and Key Technologies:

  • Integrated LaMDA's conversational capabilities into a text editor.
  • Employed few-shot prompting methods to provide predefined writing tasks.
  • Developed "relevant example prompting" techniques to support users in conveying unstructured requests instantly.
  • Designed meta-prompting features to suggest possible actions users could take.

Research Outcomes

• Specific Outcomes:

  • Wordcraft increased participants' engagement in writing and received higher helpfulness ratings without diminishing the pride in autonomous creation.
  • Compared to baseline tools (simple continuation functionality or conversational interfaces), participants using Wordcraft produced longer stories and accepted more AI-provided suggestions.
  • Users benefited from Wordcraft not only at the beginning of the creative process but also in inspiration generation, detail completion, and language style adjustment.

• Advantages Over Existing Solutions:

  • Wordcraft offers multiple interactive control options beyond simple "text continuation."
  • Achieved more flexible user control through customizable prompt functionality.
  • User studies significantly demonstrated its enhancement of writing efficiency and enjoyment.

• Experimental or Evaluation Results:

  • Participants using Wordcraft accepted an average of 7 AI suggestions per story, compared to only 4 with baseline tools.
  • AI-suggested text accounted for 13.2% of the final story, significantly higher than the 1.3% with baseline tools.
  • Users reported enjoying collaboration with AI while retaining sufficient pride and ownership of the final work.

• Limitations and Future Directions:

  • Limitations:
    1. AI still struggles with context handling, and generated content may be inconsistent with the existing narrative.
    2. For certain tasks (e.g., text filling), LLM performance may be inferior to specialized smaller models.
    3. LLM training data may contain biases, potentially affecting content quality.
  • Future Directions:
    1. Explore collaborative modes for non-fiction writing and professional reports.
    2. Expand support for custom prompts, allowing users to express more complex creative goals.
    3. Investigate how user interactions with LLMs evolve as unsupervised language models continue to develop.
    4. Develop context-aware writing assistants capable of better understanding users' current needs.

This document establishes a new benchmark for human-computer collaboration technologies and provides guidance for future research, particularly on how to fully harness the potential of large language models to support complex writing tasks.