Investigating How Computer Science Researchers Design Their Co-Writing Experiences With AI

Research Background and Issues

• What problems or challenges did the authors identify?
  With the development of AI technology, an increasing number of intelligent writing assistants are being used to support scientific writing. However, there is a lack of empirical research on how researchers collaborate with AI in scientific writing. Particularly, it remains unclear how this collaboration unfolds when dealing with the complexity of research processes and the use of multiple tools. Challenges faced by researchers include interruptions in the writing workflow, over-reliance on AI, and the generation of inaccurate or false information (e.g., fabricated citations).

• Why is this issue important?
  Scientific writing involves multiple iterative and non-linear stages, from idea formation to evidence collection, and the organization and expression of text. It serves as a core tool for the dissemination of scientific knowledge and is subject to strict requirements for accuracy, originality, and ethical standards. Understanding how humans collaborate with AI in writing not only aids in the development of more effective writing assistants but also supports the responsible use of these technologies in scientific domains.

• Research Motivation and Related Work
  The authors were inspired by prior studies on the potential of AI in scientific workflows (e.g., content generation and assistance with literature reviews), while also addressing concerns about AI-induced disruptions, the generation of false information, and the degradation of writing skills. Specifically, the exploration of how researchers "design" their collaborative experiences in scientific writing and how they utilize existing intelligent writing assistants remains an area in need of deeper investigation.

Solutions

• What methods or solutions did the authors propose?
  The authors adopted a "design-in-use" perspective, studying how 19 computer science researchers collaborate with intelligent writing assistants through remote observations and semi-structured interviews. They analyzed how researchers adjust writing assistants to solve problems and meet personal needs in their workflows.

• What is innovative about this solution?
  The "design-in-use" approach fundamentally views users as active participants in designing their own experiences, rather than passive consumers of technology. The authors uncovered researchers' customization practices, including teaching, resisting, repurposing, orchestrating, and complying with intelligent writing tools. These practices demonstrate how researchers flexibly adapt AI assistance to the iterative and complex processes of writing.

• What are the implementation steps and key technologies used?

  1. Participant Recruitment: Select computer science researchers with at least one year of research experience who are actively engaged in scientific writing and regularly use intelligent writing assistants.
  2. Observation Phase: Record participants' interactions with intelligent assistants during their normal writing processes via screen sharing.
  3. Interview Phase: Conduct semi-structured interviews to understand researchers' overall perceptions of the tools and specific application scenarios.
  4. Data Analysis: Employ a mixed coding method, combining inductive and deductive approaches to extract observations from the data and validate categorization schemes.

Research Findings

• What specific findings were obtained?
  The study identified five styles of design-in-use: Teaching, Resisting, Repurposing, Orchestrating, and Complying, encompassing 14 specific practices. These styles highlight how researchers optimize their writing workflows through tool combinations, address common issues with AI-generated content, and leverage AI tools to support their needs.

• How does it compare to existing solutions?
  This study is the first to investigate researchers' AI-assisted writing behaviors in actual tasks through remote field observations, complementing previous studies that relied solely on user feedback from surveys. It reveals the complexity of user behavior and captures practical strategies for tool combination and AI customization.

• What are the experimental or evaluation results?
  The study identified the following trends:

  1. Researchers tend to use AI critically, avoiding direct adoption of AI-generated text.
  2. Tool combination is common, such as using ChatGPT to generate preliminary text and Grammarly for grammar checks.
  3. Advantages of AI assistance include improved writing efficiency, overcoming writing bottlenecks, and addressing language barriers; disadvantages include workflow interruptions and quality issues arising from AI suggestions.

• Limitations and Future Directions
  Limitations:

  • The participant sample was skewed toward early-career researchers and lacked gender balance, potentially limiting generalizability to other academic fields.
  • The study focused on the computer science domain, where researchers may have a higher level of technical familiarity and a more critical perspective on AI usage compared to other fields.
  • The relatively short observation period may have constrained the depth of interaction analysis.

  Future Directions:

  • Expand the participant pool to include researchers with diverse academic backgrounds and technical expertise.
  • Investigate the use of writing assistants in other fields (e.g., literature or social sciences).
  • Develop new integrated ecosystems for cross-tool and collaborative design.
  • Explore the ethical relationship between users and AI, addressing issues such as text originality and privacy concerns.