AMUSE: Human-AI Collaborative Songwriting with Multimodal Inspirations

Research Background and Issues

What problems or challenges did the authors identify?

1. Lack of multimodal inspiration and creative support:

   • The authors point out that songwriting is often driven by multimodal inspiration (e.g., images, narratives, or music), but existing music AI systems fail to adequately support creators in integrating these multimodal inputs into the creative process.
   • Currently, generated music is typically non-editable audio files rather than symbolic music elements (e.g., chords or melodies) that can be reused in iterative creation.
   • Most symbolic music generation systems only provide suggestions based on existing musical context, lacking support for multimodal inspiration.

2. Lack of training data:

   • There is a shortage of training data that aligns multimodal inputs with symbolic music elements, hindering the development of generation models based on multimodal inputs.

Why is this issue important?

• Gap in inspiration transformation: Creators are inspired by multimodal inputs but are unable to efficiently translate them into concrete musical elements, potentially limiting creativity and efficiency in the creative process.
• Lack of creative control: Music creators need editable, modular tools to maintain ownership and flexibility over large-scale compositions, rather than relying on uncontrollable, monolithic generated outputs.

Research Motivation and Related Work

• Multimodal generation systems have been applied in artistic creation (e.g., illustration, writing), and the music domain is also exploring text-based music generation.
• However, most music generation tools either output non-editable audio files or lack integration of multimodal inspiration.
• This study addresses the gap in symbolic music generation tools by supporting multimodal inspiration while enabling iterative control by creators.

Solution

What methods or solutions did the authors propose?

1. Design of the Amuse Tool:

   • Developed a collaborative songwriting assistant called Amuse, which integrates users' multimodal inputs (e.g., images, text, audio) to generate reusable chord progressions.
   • Includes two main functional modules:
     • Chord Generator: Extracts musical keywords from users' multimodal inputs and generates chord progressions that align with the keywords' style.
     • Chord Transcriber: Transcribes chord progressions from audio.

2. Innovative generation methods:

   • Multimodal language model generation: Utilizes large multimodal language models (e.g., GPT-4o) to generate chord progressions related to the extracted keywords.
   • Rejection sampling filtering: Employs a single-modal model trained on real music data to filter the chord progressions generated by the LLM, enhancing musical coherence and diversity.

What are the innovative aspects of this solution?

• Generation method without aligned training data: Proposes a method that does not require aligned training data, combining the generalization capabilities of LLMs with the accuracy of single-modal models.
• Integration of multimodal and symbolic music generation: For the first time, explores chord generation based on multimodal inputs such as images, text, and audio, building upon existing symbolic music generation technologies.
• User guidance and transparency: Enhances the explainability and user control of the generation process through the use of keywords as an intermediary layer.

What are the implementation steps and key technologies used?

1. User input: Users provide multimodal content (e.g., images, text, or audio).
2. Keyword extraction: A large language model extracts musical keywords from the input.
3. Chord generation:
   • Based on the keywords, GPT-4o generates initial chord progressions.
   • A note-based language model (trained on real music data) applies rejection sampling to retain chord progressions that align with musical distributions.
4. Chord transcription: Audio files are transcribed into symbolic chords.

Research Outcomes

What specific outcomes were achieved?

1. Technical evaluation:

   • Chord progressions generated by Amuse outperformed baseline methods in terms of diversity, relevance, and musical coherence.
   • User studies demonstrated that Amuse effectively supports the transformation of multimodal inspiration into concrete musical elements.

2. User experience:

   • Users felt a greater sense of control and creative agency when using Amuse.
   • The tool's keyword feature was perceived as intuitive and transparent, significantly contributing to the explainability of the generated chords.

What advantages does it have compared to existing solutions?

• Multimodal support: Directly integrates multimodal inspiration rather than being limited to musical context.
• Editability and modularity: Provides symbolic music elements that users can control and flexibly apply.
• No need for aligned data: Solves the data scarcity issue through innovative methods without requiring large-scale multimodal-music alignment data.

What were the experimental or evaluation results?

1. Technical evaluation:

   • Chord progressions generated by Amuse demonstrated superior musical coherence compared to traditional methods (validated using Jensen-Shannon Divergence).
   • In user preference tests, Amuse-generated chords outperformed baselines in terms of keyword relevance.

2. User studies:

   • Users experienced greater freedom and exploration in the creative process when using Amuse.
   • The Chord Generator module was more popular, while the Chord Transcriber was used less frequently, indicating a preference for more flexible functionalities in actual creation.

Limitations and Future Directions

1. Tool integration gaps: Users reported discontinuity when switching from Amuse to other tools (e.g., Aria), indicating a need for better integration of multimodal and contextual conditions.
2. Limited task coverage: The current tool primarily supports chord generation and does not cover melody or lyric generation. Future work could expand to other musical elements.
3. Scenario testing limitations:
   • This study primarily tested in static environments and did not fully explore real-time creative scenarios.
   • Long-term adoption has not yet been evaluated, and future research should explore more realistic and diverse use cases.

In summary, Amuse provides a novel approach and potential for human-computer collaborative music creation. Its support for multimodal inspiration significantly enhances the freedom and personalization of the creative process.