Botender: Supporting Communities in Collaboratively Designing AI Agents through Case-Based Provocations

Paper Title

Botender: Supporting Communities in Collaboratively Designing AI Agents through Case-Based Provocations

Publication Info

• Topic area: Collaborative design of AI agents for online communities.
• Keywords: AI agents, collaborative design, large language models, online communities, case-based provocations, bot customization, participatory design, prompt engineering, Discord, human-computer interaction.

Background and Problem

• Problem / challenge: Bots in online communities are often designed by external developers or a few tech-savvy members, leading to limited customization and misalignment with community-specific needs. Non-AI experts face challenges in collaboratively designing bots, including narrow focus on single scenarios, lack of consensus-building processes, and technical barriers.
• Significance: Misaligned bots can lead to undesirable community-wide consequences, such as discouraging participation or enforcing norms ineffectively. A participatory approach could help communities design bots that align with their collective values and needs.
• Motivation and related work: Prior research has explored individual prompt design, collaborative ML model design, and the use of concrete cases in iterative design. However, tools enabling non-experts to collaboratively design bots tailored to their communities remain underexplored.

Solution

• Proposed approach: Botender, a system enabling communities to collaboratively design LLM-powered bots through case-based provocations, which are interaction scenarios designed to spark reflection and discussion about bot behavior.
• Novelty:
  1. Introduction of case-based provocations for iterative and collaborative bot design.
  2. Integration of a coordinated workflow for proposing, iterating, and deploying bot behaviors.
  3. Seamless integration into community platforms like Discord to encourage participation.
  4. Validation of the approach through an algorithm study and a real-world field study.
• Procedure and key techniques:
  1. Users propose bot behavior changes via a web interface or playground feature.
  2. Botender generates case-based provocations to highlight ambiguities, overly narrow prompts, or unintended consequences.
  3. Community members review, discuss, and iteratively refine bot prompts using test cases.
  4. Proposals are deployed upon reaching a consensus, with notifications integrated into Discord.

Results

• Concrete findings:
  • Botender’s case-based provocations were rated higher in provocativeness (4.1 vs. 3.5) and controversialness (3.9 vs. 3.2) compared to standard test cases.
  • In a field study, participants created 137 proposals and 800 test cases across six Discord communities, deploying 69 customized tasks.
  • 96% of participants expressed interest in continuing to use Botender.
• Advantage over baselines: Botender’s provocations revealed more opportunities for bot improvement, surfaced disagreements, and supported iterative design better than standard test cases.
• Experiments / evaluation:
  • Validation study with 90 participants compared Botender’s provocations to standard test cases.
  • Field study with six Discord communities (5–429 members each) over five days to test real-world applicability.
  • Metrics included participant ratings on provocativeness, controversialness, coverage, and diversity of test cases.
• Limitations and future work:
  • Current provocations do not address task conflicts or deeply surface disagreements in homogeneous groups.
  • Limited to single-turn bot interactions; future work could expand to multi-turn conversations and richer context.
  • Scaling to larger communities and addressing hierarchical power dynamics remain challenges.

Summary

Botender is a system that enables online communities to collaboratively design LLM-powered bots by proposing, iterating, and deploying behaviors tailored to their needs. Its case-based provocations effectively reveal opportunities for improvement and facilitate consensus-building. Validation and field studies demonstrate its utility in real-world settings, with participants successfully creating diverse tasks that reflect their community norms. Future work could enhance case generation, expand bot capabilities, and address scalability and power dynamics in larger or hierarchical communities.