"Bespoke Bots'': Diverse Instructor Needs for Customizing Generative AI Classroom Chatbots

Paper Title

“Bespoke Bots”: Diverse Instructor Needs for Customizing Generative AI Classroom Chatbots

Publication Info

• Topic area: Customization of generative AI chatbots for educational contexts.
• Keywords: AI chatbots, education technology, customization, modular design, pedagogy, STEM education, course management, personalization, agentic workflows, instructional design.

Background and Problem

• Problem / challenge: Current AI chatbots for classrooms lack the flexibility to meet the diverse and context-specific needs of instructors. Existing systems are often monolithic, making it difficult to adapt them to varying course sizes, disciplines, and teaching styles.
• Significance: Customizable AI chatbots could reduce teaching bottlenecks, enhance pedagogy, and align with instructors’ unique course requirements, but their potential is hindered by the lack of modular and adaptable design frameworks.
• Motivation and related work: Prior research has shown the value of customizing static educational tools like LMS, but chatbots introduce new challenges due to their nondeterministic behavior. While some tools for chatbot customization exist, they are not designed to accommodate the dynamic and heterogeneous nature of instruction. This paper builds on the idea of modular and agentic workflows to address these gaps.

Solution

• Proposed approach: The concept of "teachable teammates" — modular AI chatbots that can be customized to specific course rules, roles, and materials, enabling flexibility and control for instructors.
• Novelty:
  1. Empirical identification of instructors’ customization priorities for classroom AI chatbots.
  2. Synthesis of ten feature categories for chatbot customization, derived from content analysis of 182 prompts and interviews with STEM instructors.
  3. Design implications for modular and agentic workflows in educational chatbot systems.
• Procedure and key techniques:
  • Conducted content analysis of 182 public educational prompts to identify customization categories.
  • Performed semi-structured interviews with 10 STEM instructors, including a card-sorting activity to prioritize customization features.
  • Analyzed convergence and divergence in instructor needs across teaching contexts.

Results

• Concrete findings:
  • Instructors consistently prioritized customization of course materials, pedagogical strategies, and constraints/guardrails.
  • Features like persona/tone and content format were de-prioritized.
  • Divergent priorities emerged for personalization, course management, and user environment, reflecting differences in course size, format, and teaching style.
• Advantage over baselines: The study highlights the inadequacy of one-size-fits-all chatbot designs and proposes modularity as a solution to balance common needs and flexibility for diverse contexts.
• Experiments / evaluation:
  • Participants: 10 STEM instructors from North America and Europe, teaching courses of varying sizes and formats.
  • Methods: Card-sorting activity to rank ten customization categories, followed by thematic analysis of interview transcripts.
  • Metrics: Prioritization rankings and qualitative insights into instructors’ reasoning and trade-offs.
• Limitations and future work:
  • Findings are limited to STEM university instructors and may not generalize to K-12 or non-STEM contexts.
  • The card-sorting method may have introduced minor interpretive variations.
  • Future work should explore global perspectives, institutional barriers to sharing, and the development of modular chatbot systems.

Summary

This paper investigates how STEM instructors want to customize AI chatbots for their classrooms, finding convergence on embedding course materials, pedagogical strategies, and guardrails, while identifying divergence in needs like personalization and course management. The study proposes "teachable teammates," modular chatbot agents that can be configured for specific roles and tasks, as a solution to the limitations of monolithic chatbot systems. By emphasizing modularity, agentic workflows, and sharing mechanisms, the findings offer a roadmap for designing adaptable and sustainable AI tools that align with the dynamic and diverse nature of instruction.