“Is Text-Based Music Search Enough to Satisfy Your Needs?” A New Way to Discover Music with Images

Title of the Paper

“Is Text-Based Music Search Enough to Satisfy Your Needs?” A New Way to Discover Music with Images

Paper Information

• Research Area: Human-Computer Interaction and Music Information Retrieval
• Keywords: Image-Music Retrieval, Music Search, Multimodal, User Experience, System Usability

Research Background and Problem

• Background: Music plays a significant role in human experiences, but users often feel confused when searching for music that matches a specific context or mood due to vague search terms. Traditional music search systems rely on text-based retrieval methods, which provide limited support for complex emotional and contextual needs.
• Challenges and Importance:
  • Difficulty in handling vague or unclear search queries.
  • A need for a more intuitive approach that allows users to find suitable music without requiring explicit song information.
  • Improving user experience and satisfaction in music search.
• Research Motivation:
  • To explore whether image-based music retrieval methods can better meet user needs.
  • To design a novel music search system tailored for scenarios involving ambiguous queries and compare its effectiveness with traditional text-based search systems.

Solution

• Methods or Solutions:
  • Proposed three music search systems:
    1. TTTune: A traditional text-based music search system.
    2. VisTune: An image-based music search system where users search by selecting images related to their context or mood.
    3. VTTune: A hybrid music search system combining images and text, allowing users to express initial needs through images while enriching search information with text.
• Innovations:
  • Introduced images as a medium for music search, exploring whether the ambiguity of images can better express users' emotions and contexts.
  • Developed a hybrid method combining images and text to further enhance search effectiveness and user satisfaction.
• Implementation Steps and Technology:
  • The MTG-Jamendo music dataset was used.
  • TTTune employed natural language processing tools (KeyBERT) to extract keywords and map them to music tags.
  • VisTune and VTTune utilized an image-to-music tag mapping algorithm, searching for content similar to the input image in a pre-constructed image database and linking it to music.
  • The system prototype was developed using the popular open-source framework Streamlit.

Research Findings

• Specific Findings:
  • User Experience: Image-based music search systems (VisTune and VTTune) outperformed the traditional text-based search system (TTTune) in terms of user experience scores and system usability.
  • Music Matching Accuracy: Text-based search systems performed better in terms of music matching accuracy, but image-based systems were more favored in terms of user experience.
  • Age Influence: VisTune showed higher satisfaction among users aged 40 and above.
  • Experiments and Evaluation:
    • Two standardized measurement tools (UEQ-S and PSSUQ) were used to evaluate user experience and usability.
    • 236 participants were recruited, with some participating in follow-up interviews to explore usage scenarios in depth.
• Key Advantages:
  • Image-based music search systems are more intuitive, especially for users who face difficulties with text input.
  • The hybrid search system (VTTune) balances user experience and music matching accuracy, improving overall satisfaction.
  • The system has potential for expanding user music preferences and handling ambiguous queries.
• Limitations and Future Directions:
  • Limitations:
    • Image tags need optimization to better capture user intent.
    • The system interface requires further improvement, particularly in image display and interaction design.
    • The provided scenarios are limited and need to be expanded to meet more real-world user needs.
  • Future Directions:
    • Develop personalized interfaces for specific user groups (e.g., older adults).
    • Use generative models to directly create music that matches user expectations.
    • Test system performance in real-world applications and explore integration with social media scenarios.

Discussion and Contributions

• Design Guidelines: Provided interface design recommendations for image-based music retrieval systems:
  • Simplify the image selection process, such as adopting a swipe selection mechanism.
  • Consider allowing users to upload custom images or automatically capture surrounding environments from their devices as query options.
  • Optimize interface design for older users by increasing image view size and interaction convenience.
• Theoretical and Practical Contributions:
  • Proposed an entirely new music search paradigm, offering innovative ideas for the field of visual-based music retrieval.
  • Provided detailed empirical data and optimization suggestions for music search interface design and user experience research.