Exploring the Needs of Practising Musicians in Co-Creative AI Through Co-Design

Research Background and Issues

• What problems or challenges did the authors identify?

  • Despite significant advancements in generative AI music technologies and their widespread use as collaborative tools in music creation, there is currently limited research on the deep involvement of actual musicians in the design process. Most studies involve musicians only after the tools have been developed, rather than fully integrating their needs during the design phase.
  • AI researchers may develop tools that fail to meet the actual needs of musicians due to their own biases or limited understanding of musical practices. This issue is common in technology development.

• Why is this issue important?

  • Music creation is a highly complex and human-centric practice that often takes years to master. Incorporating the experiences of actual musicians into the design process of generative AI tools ensures that the tools genuinely meet their needs, avoiding mismatches between design and requirements, and preventing negative impacts on the culture of music creation.

• Research Motivation and Related Work

  • This study employs a co-design approach to involve practicing musicians in the development of AI music tools, aiming to better understand their needs for generative tools and explore how AI can be embedded into their creative workflows. Previous studies have emphasized the importance of user experience, transparency, and controllability, but most have focused on tool evaluation rather than early-stage user involvement in development.

Solution

• What methods or solutions did the authors propose?

  • The authors adopted a co-design approach to design a music variation tool centered on the needs of musicians. This method includes early insights, feature definition, and ecosystem evaluation, integrating musician feedback throughout the process.
  • The design philosophy is rooted in the actual practices of musicians, emphasizing the retention of control over the creative process and positioning the tool as an assistant rather than a collaborative partner.

• What is innovative about this solution?

  • Through the co-design approach involving deep participation in the design process, this study not only developed a music tool but also uncovered musicians' deeper needs in creative practices. These needs include ownership of the entire creative process, intuitive control options, and personalized features tailored to different musical contexts.
  • The study proposed bridging traditional musical terminology with machine language in generative AI tools to address musicians' potential difficulties in understanding technical jargon.

• What are the implementation steps and key technologies used?

  • Phase 1: Conducted the first round of workshops to understand musicians' creative workflows and attitudes toward AI collaboration, clarifying the design direction.
  • Phase 2: Developed a prototype of the music variation tool, hosted workshops, and collected design suggestions.
  • Phase 3: Deployed the tool in musicians' ecosystems for a two-week experiment and refined the tool design based on feedback.
  • Technologically, the study utilized MusicTransformer and MusicBERT models, employing specialized encoding and masking functions to generate variations of music fragments. The optimized tool allows users to select specific variation attributes and adjust the complexity of the variations according to their needs.

Research Outcomes

• What specific outcomes were achieved?

  • Experiments and ecosystem evaluations demonstrated that the tool could provide inspiration during the ideation phase of music creation without disrupting musicians' overall control of the creative process.
  • The tool received positive feedback from practicing musicians, with some expressing interest in incorporating it into their daily creative workflows.

• What advantages does it have compared to existing solutions?

  • Unlike fully automated music generation tools, this tool preserves users' creative agency through "variations" rather than replacing or undermining musicians' creative autonomy.
  • The integration of user-controllability design ensures that the tool's generative process can more directly meet the diverse needs of different musicians.

• What were the experimental or evaluation results?

  • Ecosystem experiments revealed that musicians primarily used the tool during the early ideation phase, where it provided unexpected inspiration points. Musicians maintained control over the creative process by selecting and refining the tool-generated content.
  • Feedback collected provided clear guidance for enhancing model functionality and developing the next phase of a plugin-based version.

• Limitations and Future Directions

  • Limitations: The study lacked sufficient representation of older musicians, potentially missing their unique needs. The tool currently supports only limited variation types and has not yet been integrated with mainstream Digital Audio Workstations (DAWs).
  • Future Directions: Plans include developing a plugin version of the tool to integrate with mainstream DAWs for practical music creation. Additionally, the tool's variation functionality will be expanded to support more complex music creation needs.

Through a comprehensive co-design approach, targeted technical optimization, and in-depth ecosystem evaluation, this study explored the alignment between musicians' needs and the design of generative AI tools, providing valuable insights for the future development of AI music creation tools.