Sound Designer-Generative AI Interactions: Towards Designing Creative Support Tools for Professional Sound Designers

Title of the Paper

Sound Designer-Generative AI Interactions: Towards Designing Creative Support Tools for Professional Sound Designers

Paper Information

• Research Domain: Human-Computer Interaction, Creative Tool Design, Sound Design
• Keywords: Audio, Generative AI, Sound Design, Creative Support Tools, Mixed-Initiative Creative Interfaces

Research Background and Issues

• Identified Problems or Challenges

  • Existing generative AI applications are predominantly focused on music creation, with limited research dedicated to environmental sound design.
  • The needs of expert users in sound design practices have been insufficiently addressed, as most studies target beginners.
  • Sound design requires complex sound manipulation and creative processes to meet specific perceptual demands, but AI tools often lack usability and adaptability.

• Significance

  • Sound design holds significant artistic and technical importance in entertainment fields such as film, music, and gaming. Developing robust AI tools can greatly enhance efficiency and outcomes in these domains.
  • Exploring AI's collaborative potential in audio design can contribute to new workflow models and enrich design methodologies.

• Research Motivation and Related Work

  • This study aims to investigate how generative AI can provide practical support for professional sound designers.
  • Two interactive generative AI models are proposed and evaluated as creative support tools to explore AI's potential in assisting sound design.
  • The research leverages emerging sound generation models (e.g., GAN, StyleGAN) to offer novel interaction methods and practical insights.

Solution

• Methodology or Solution

  • Two interactive generative AI tools (Creative Support Tools, CST) are proposed, utilizing:
    • Domain-Specific Control (Interface-1): Guided generation using sound parameters (e.g., frequency, pulse width, filter order).
    • Technology-Specific Control (Interface-2): Semantic editing control achieved by decomposing latent vectors within StyleGAN's latent space.
  • Ambiguity factors are introduced in the design, allowing users to flexibly define and interpret system outputs during exploration.

• Innovative Contributions

  • Exploration of generative AI feasibility in practical audio design through "domain-specific control" and "technology-specific control."
  • Deployment of AI tools directly in sound designers' real-world work scenarios to reflect on their limitations and opportunities.
  • Integration of the Mixed-Initiative Creative Interfaces concept, enabling iterative interaction with sound designers.

• Implementation Steps and Key Technologies

  • Design and Implementation: Development of StyleGAN-based generators and corresponding interactive interfaces for each tool.
  • Data Collection: Recruitment of nine professional sound designers, divided into two groups to test the tools, followed by subjective interviews post-task completion.
  • Data Analysis: Inductive thematic analysis of interview results to identify participants' exploration strategies, user experiences, and personalized needs.

Research Outcomes

• Specific Results

  • Identified advantages of AI tools in supporting rapid iteration, reducing manual recording requirements, and generating virtual yet perceptible sound effects during sound design processes.
  • Summarized designers' exploration patterns and system usage feedback, emphasizing the importance of uncertainty in creative discovery.
  • Proposed five design recommendations for sound design tools, such as "intuitive control design" and "balancing generality with specificity."

• Advantages Compared to Existing Solutions

  • Highlights the creative blending of sound layers rather than merely replicating real audio.
  • Provides real-time exploration and editing of latent spaces, enabling users to unlock more personalized and diverse audio effects.

• Experimental and Evaluation Results

  • Participants were able to generate sounds for sci-fi scenarios or unreal yet believable soundscapes by exploring AI-generated sound elements.
  • When the system introduced uncertainty, designers employed trial-and-error, min-max approaches, and other methods to efficiently understand the AI model's capabilities.
  • While ambiguity stimulated creativity, it occasionally led to discomfort and reduced efficiency when precise control was required for specific tasks.

• Limitations and Future Directions

  • The study's scope was limited to nine participants, and the tools have not been tested in long-term or real project cycles.
  • Future research should explore the potential of other generative models (e.g., diffusion models) and focus on optimizing fine-grained control mechanisms in audio design processes.
  • Integration of tools for visualizing sound associations to further assist users in understanding AI-generated outputs.