User Experience of LLM-based Recommendation Systems: A Case of Music Recommendation

Research Background and Problem

• What problems or challenges did the authors identify?
  The authors pointed out that traditional recommender systems (RS) typically operate by passively delivering recommendations to users. This limits the user's role, making it difficult for them to actively participate or express complex needs. Additionally, while existing conversational recommender systems (CRS) enhance user engagement through multi-turn interactions, they are still constrained by predefined interaction patterns, making it difficult to flexibly meet diverse user needs.

• Why is this problem important?
  Recommender systems have become an integral part of digital life, but their design often focuses on optimizing recommendation performance while neglecting users' ability to actively express needs and explore. This results in recommendation experiences being confined to system logic, making them less personalized and potentially incapable of supporting users in self-reflection or discovering new preferences.

• Research Motivation and Related Work
  With advancements in large language models (LLM), recommender systems now have new possibilities: users can design their own recommendation services through more open interaction modes. This study aims to explore the differences in user experiences between LLM-driven CRS and traditional RS, as well as how users utilize LLM to create personalized recommendation services.

Solution

• What methods or solutions did the authors propose?
  The authors designed a customized music conversational recommender system based on LLM (GPT) and conducted a three-week diary study to observe how users personalize their recommendation services. Users were allowed to freely choose interaction methods during the three stages of the recommendation process: preference elicitation, recommendation presentation, and user feedback.

• What are the innovative aspects of this solution?

  • It provides flexible user interaction methods, enabling users to express implicit needs, explore new preferences, and engage in self-reflection.
  • It allows users to design their own recommendation logic, making the system more tailored to individual needs rather than adhering to preset interaction patterns.
  • It analyzes the potential of LLM-driven CRS from the perspective of user experience rather than system performance.

• What are the implementation steps and key technologies used?
  The study was divided into the following three stages:

  1. Exploration Stage: Users experienced different interaction scenarios to form initial ideas about recommender system interactions.
  2. Selection Stage: Users summarized and selected the optimal recommendation logic and received recommendations through a customized GPT.
  3. Optimization Stage: Based on their experiences in the second stage, users adjusted the recommendation logic to further refine interaction methods.
     The diary study was implemented using a GPT customization tool, where users recorded system interaction screenshots and feedback at each stage.

Research Findings

• What specific findings were achieved?
  The study revealed that LLM-driven CRS can provide users with three new experiences:

  1. Helping users express implicit needs by transforming vague demands into specific preferences through diverse expression methods (e.g., images, emotional descriptions).
  2. Supporting unique exploration, allowing users to break free from traditional systems' constraints on existing preferences, customize new recommendation logic, and access novel content.
  3. Enhancing a deeper understanding of music preferences, enabling users to more precisely define and expand their tastes through system analysis of preference details.

• What advantages does it have compared to existing solutions?

  • It breaks the limitations of fixed logic in traditional RS, allowing users to directly control the recommendation process.
  • It assists users in self-discovery and personal reflection, offering higher user engagement and design autonomy.
  • It significantly enriches recommendations and user experience through personalized interactions.

• What were the experimental or evaluation results?
  The study collected multi-round feedback data and improvement suggestions from participants. The analysis showed that:

  • LLM-driven CRS significantly outperformed traditional RS in meeting vague needs, providing exploratory content, and explaining recommendation rationales.
  • Most users found that such systems significantly improved the quality of personalized recommendations but also noted potential issues, such as system dependency and cumbersome interaction processes.

• Limitations and Future Directions

  • The customized GPT was not integrated with real-world music applications, leading to occasional inaccuracies or false recommendations.
  • The participant sample was concentrated on young users, lacking representation of broader demographic characteristics.
  • The three-week short-term study period could not observe how users develop long-term preferences or how the system evolves with prolonged use.
    Future research could focus on the following directions:
  1. Long-term observation of user interactions with LLM recommender systems.
  2. Designing systems integrated with real-world music platforms to enhance recommendation accuracy.
  3. Expanding participation to users of different ages and backgrounds to improve the generalizability of the research.

This study establishes LLM-driven CRS as a powerful alternative to traditional recommender systems. Its emphasis on user-centered design and flexible interaction modes provides significant guidance for the future development of recommendation technologies.