Social-RAG: Retrieving from Group Interactions to Socially Ground AI Generation

Research Background and Problem

• Problem and Challenges Identified: The authors observed that AI agents often provide ineffective or unwelcome suggestions in online collaborative spaces due to a lack of understanding of group preferences and social behaviors, which can even disrupt group dynamics. These systems lack the flexibility to adapt to evolving group preferences and social contexts.
• Significance: Social dynamics in online collaborative environments are highly complex, and inappropriate AI interventions may reduce collaboration efficiency, lead users to abandon the system, or ignore its outputs. This issue directly impacts the integration and utility of AI in human teams.
• Related Work and Motivation: Large Language Models (LLMs) offer the ability to generate flexible, human-like text but lack social knowledge to align generated content with group preferences. The authors analyzed how Retrieval-Augmented Generation (RAG) techniques embed external knowledge but found their applications are typically limited to factual knowledge, with little exploration of how to apply social knowledge to generate socially appropriate content.

Solution

• Proposed Approach: The authors proposed a framework called Social-RAG, which retrieves social interaction information from groups and embeds "social facts" from past social dynamics into the context of generative language models to produce socially appropriate content.
• Innovations:
  • Extending Retrieval-Augmented Generation techniques to the domain of social knowledge.
  • Building a dynamic "social knowledge base" that integrates historical and real-time interaction signals from groups, reducing the burden on users to actively provide preferences.
  • Developing a contextual generation method based on social signals to avoid information overload and conflicts with group interactions.
• Implementation Steps:
  1. Collection and Indexing: Extract content from group interactions to build a social knowledge base, including historical chat logs, user reactions, and comments.
  2. Retrieval of Social Signals: Extract social signals related to recommended content from the knowledge base, including relevant historical discussions, metadata, and potentially interested group members.
  3. Information Generation: Use large language models to generate concise and socially adapted messages, ensuring the format, tone, and information density align with group norms.
  4. Publishing and Feedback: Publish the generated messages in group channels and collect group comments and reactions to further optimize the knowledge base.

Research Outcomes

• Specific Results:
  • Implemented the PaperPing system, which recommends papers and provides socially contextualized explanations in academic Slack groups.
  • Through a three-month real-world deployment, PaperPing demonstrated the ability to learn group preferences and provide accurate recommendations, significantly outperforming generic paper summaries.
• Advantages:
  • Reduces the burden on users to provide feedback and integrates with existing social habits.
  • Compared to traditional systems, PaperPing's generated content was perceived as more relevant and helpful to researchers' projects and interests.
• Experimental and Evaluation Results:
  • Among 18 participating groups, over 75% of users found PaperPing's recommendations highly aligned with group preferences, and 82% discovered papers they had previously struggled to find.
  • Approximately 54% of users found the summaries generated by PaperPing relevant to their personal interests.
  • Some users noted that PaperPing enhanced their understanding of group members' interests, helping to establish common ground.
• Limitations and Future Directions:
  1. Recommendation quality declines in inactive groups or those with limited interaction history.
  2. The system faces challenges in balancing individual and group interests, such as overfitting to the preferences of specific group members.
  3. Some users still trust human-recommended content more, believing machine-generated content requires verification.
  4. Future directions include exploring more flexible feedback mechanisms, consulting finer-grained social norms, and expanding to other group application scenarios, such as meeting summarization tools or misinformation correction systems in social communities.

Conclusion

Social-RAG provides a novel approach to addressing the issue of social adaptation for AI in group collaboration spaces by leveraging group social dynamic data to generate content that better aligns with group interests and habits. This research demonstrates the potential of Retrieval-Augmented Generation techniques in the social domain, advancing a new direction for socially aware and technologically adaptive systems while emphasizing the importance of design transparency and privacy protection.