Social Simulacra: Creating Populated Prototypes for Social Computing Systems

Title of the Paper

Social Simulacra: Creating Populated Prototypes for Social Computing Systems

Paper Information

• Domain: Human-Computer Interaction (HCI), Social Computing Design
• Keywords: Social computing, prototype design, large language models, system population, design iteration, GPT-3, Reddit, diverse behavior generation, harmful behavior, design tools

Research Background and Problem

• Problem or Challenge: Traditional social computing prototype design often relies on small-scale user testing, which struggles to simulate the complex behaviors of a system once it is populated at scale, particularly antisocial behaviors. Additionally, designers currently lack tools to predict the dynamic behaviors of communities after large-scale user participation.
• Importance: With the proliferation of social computing systems, poor design decisions can lead to community breakdown or user harm. Designers need to predict potential issues before deployment and iterate on their designs.
• Motivation and Related Work: The authors observed that existing tools are inadequate for addressing the dynamics of complex social behaviors in current social computing design practices. They were inspired by the potential of large language models like GPT-3, which they believe can serve as a foundation for generating diverse social behaviors.

Solution

• Proposed Method: Social Simulacra, a novel prototyping technique that uses large language models to create populated prototypes of social computing systems. This solution generates realistic social interactions—including posts, replies, and antisocial behaviors—based on community descriptions provided by designers (e.g., goals, rules, user roles).
• Innovations:
  1. Utilizing GPT-3 to generate a large number of virtual users and their interactive behaviors.
  2. Introducing "Prompt Chains" to control the generated output.
  3. Designing a "Multiverse" model to explore the uncertainty of social dynamics through multiple possible outcomes.
  4. Providing a tool that enables designers to iterate on their designs before deployment, reducing the cost of reactive fixes later.
• Implementation Steps:
  1. Role Expansion: Generate hundreds or thousands of diverse and highly relevant user roles from initial seed roles.
  2. Content Generation: Create top-level posts and replies within the community, ensuring alignment with goals and rules.
  3. Scenario Inference and Dynamic Exploration: Use "WhatIf" and "Multiverse" features to explore different design possibilities.
  4. Tool Development: Build SimReddit, a simulated Reddit interface that allows designers to test their designs.

Research Outcomes

• Specific Results:
  1. Developed the SimReddit tool, which generates highly realistic social content closely resembling real communities.
  2. Experiments demonstrated that users struggled to distinguish SimReddit-generated content from real community content.
  3. Designers effectively improved their community designs using SimReddit.
• Advantages: Compared to traditional small-group testing, this method simulates the complex behaviors of large-scale participation. It can also preemptively identify potential issues, such as antisocial behaviors or the isolation of marginalized groups.
• Experiment/Evaluation Results:
  1. In technical evaluations, participants were unable to significantly distinguish between real community content and SimReddit-generated content, with error rates close to random guessing levels (41%).
  2. In designer evaluations, 16 participants generally found the generated content realistic and inspiring for new design ideas.
  3. Designers significantly improved their community designs through iteration, such as adding clearer rules and fostering community culture.
• Limitations and Future Directions:
  1. The model cannot predict the actual future but can only provide potential design insights.
  2. Currently supports only English text, lacking multi-language and cross-cultural design capabilities.
  3. Limited by GPT-3's technical capabilities and data cutoff, which may affect the scope and realism of generated content.
  4. Future research could expand to other platforms (e.g., Facebook groups) and explore multi-modal generation (text + video).

This paper introduces a novel perspective on social computing system design. By leveraging behavior simulations generated by large language models, it helps designers anticipate potential issues and identify areas for improvement, significantly enhancing design efficiency and enabling designers to better address social complexity in their work.