Can Crowds Customize Instructional Materials with Minimal Expert Guidance? Exploring Teacher-guided Crowdsourcing for Improving Hints in an AI-based Tutor

DOI

AI-based educational technologies may be most welcome in classrooms when they align with teachers’ goals,preferences, and instructional practices. Teachers, however, have scarce time to make such customizationsthemselves. How might the crowd be leveraged to help time-strapped teachers? Crowdsourcing pipelineshave traditionally focused on content generation. It is, however, an open question how a pipeline might bedesigned so the crowd can succeed in a revision/customization task. In this paper, we explore an initial versionof a teacher-guided crowdsourcing pipeline designed to improve the adaptive math hints of an AI-basedtutoring system so they fit teachers’ preferences, while requiring minimal expert guidance. In two experimentsinvolving 144 math teachers and 481 crowdworkers, we found that such an expert-guided revision pipelinecould save experts’ time and produce better crowd-revised hints (in terms of teacher satisfaction) than twogeneration conditions. The revised hints however, did not improve on the existing hints in the AI tutor, whichwere already highly rated, though with room for improvement and customization. Further analysis revealedthat the main challenge for crowdworkers may lie in understanding teachers’ brief written comments andimplementing them in the form of effective edits, without introducing new problems. We also found thatteachers preferred their own revisions over other sources of hints, and exhibited varying preferences over hintsin AI-tutor. Overall, the results confirm that there is a clear need for customizing hints to individual teachers’preferences, but also highlight the need for more elaborate scaffolds so the crowd has specific knowledge ofthe requirements that teachers have for hints. The study represents a first exploration in the literature of howto support crowds with minimal expert guidance in revising and customizing instructional materials.