MVPrompt: Building Music-Visual Prompts for AI Artists to Craft Music Video Mise-en-scène

Research Background and Problem

• What problems or challenges did the authors identify?
  Music video production requires combining music with visual elements, emphasizing visual content to convey information. However, despite advancements in generative AI technologies (e.g., text-to-video models) that lower the barrier to video creation, accurately reflecting the "mise-en-scène" in music videos remains challenging. This mise-en-scène demands specialized knowledge, and AI artists often lack guiding tools, making it difficult to express their creative intentions.

• Why is this problem important?
  Mise-en-scène is critical for maintaining visual consistency and delivering a rich aesthetic experience in music videos. As the demand for AI artists to create music videos using generative AI tools grows, guidance tools for mise-en-scène can not only enhance artistic quality but also support creators in achieving more complex artistic expressions.

• Research Motivation and Related Work
  While some studies have developed tools to support generative AI in image creation (e.g., optimizing text-to-image prompts), these tools primarily focus on static content rather than dynamic video. Video content requires dynamic information (e.g., character movements and camera transitions), making prompt design for mise-en-scène more complex and demanding higher levels of tools and knowledge.

Solution

• What methods or solutions did the authors propose?
  The authors developed MVPrompt—a tool designed to help AI artists generate prompts that align with the visual mise-en-scène of music videos. This tool supports creative generation through a two-stage design: (1) defining visual themes based on music analysis; (2) assisting users in concretizing visual scenes and expressing details (e.g., background, colors, object arrangement, and camera language) through interactive dialogue.

• What are the innovative aspects of this solution?

  1. Directly associates musical characteristics (rhythm, emotion, lyrics) with visual themes, providing clear contextual support for music video creation.
  2. Introduces a four-stage dialogue mechanism (inquiry, selection, suggestion, confirmation) to help AI artists progressively clarify mise-en-scène elements and concretize prompt content based on music and user preferences.
  3. Establishes a creative generation process starting from music, automatically recommending visual themes (e.g., via the Aesthetic Wiki database) and guiding creators in exploration.

• What are the implementation steps and key technologies used?

  1. Music Analysis Stage: The tool receives music input, extracts musical features (e.g., rhythm, timbre) using the Librosa library, and analyzes the association between music and aesthetic themes via the OpenAI API.
  2. Scene Concretization Stage: Based on user-selected thematic images, the tool facilitates defining objects, backgrounds, colors, lighting, arrangements, and camera designs in the scene through multiple rounds of interactive dialogue.
  3. Final Prompt Generation: Combines user-confirmed mise-en-scène details to generate text prompts for generative AI tools (e.g., Runway Gen-3 Alpha model) to create videos.

Research Outcomes

• What specific outcomes were achieved?
  In user studies and experiments, MVPrompt enhanced AI artists' ability to create music video scenes, particularly excelling in creative collaboration, idea discovery, and clarity of prompt expression. Through staged design, MVPrompt breaks down the complex task of music video scene creation into manageable steps.

• What advantages does it have compared to existing solutions?

  1. MVPrompt outperformed baseline systems and two simplified versions (w/o Visual Scene & Grammar, w/o Visual Theme) in dimensions such as collaboration, exploration, and clarity.
  2. It enables users to explore visual themes based on musical characteristics and progressively clarify mise-en-scène details through guided dialogue, functionalities that baseline systems cannot provide.

• What were the experimental or evaluation results?

  1. In a user study involving 24 AI artists, MVPrompt received the highest user preference ranking for music video scene creation (10 participants ranked it first).
  2. In user experience surveys, MVPrompt significantly outperformed other systems in creativity discovery, clarity of prompt expression, and practical usability.
  3. Statistical analyses using Friedman and Conover tests demonstrated significant advantages for MVPrompt in "collaboration" and "discovery" metrics.

• Limitations and Future Directions

  1. Limitations: The current study is limited to a specific text-to-video generation model (Runway Gen-3 Alpha), and future research needs to test its effectiveness across other models. Additionally, short-term user studies lack insights into long-term user experiences.
  2. Future Directions: Exploring MVPrompt's application in non-music video scenarios (e.g., films or advertisements) while optimizing the balance between free dialogue and guided suggestions to meet diverse creators' needs for tool flexibility and guidance.

Through this research and development, MVPrompt effectively addresses the gap in guiding AI artists in music video mise-en-scène creation, providing new support tools and design methods for future generative AI artistic practices.