TutorCraftEase: Enhancing Pedagogical Question Creation with Large Language Models

Research Background and Issues

• Problems and Challenges:
  The authors point out that generating a large number of high-quality instructional questions is challenging for teachers, especially novice educators. Even experienced teachers often struggle to create a substantial number of effective questions within limited time constraints.

  • Traditional Intelligent Tutoring System (ITS) question-generation tools require significant time and technical expertise; for example, developing one hour of instructional content can take approximately 300 hours.
  • Existing generation tools often have steep learning curves, requiring programming or specialized editing skills, which can create psychological burdens for teachers.
  • Large Language Models (LLMs), while used in educational contexts, typically generate a limited range of questions, often restricted to simple Q&A formats, lacking step-by-step guidance and auxiliary materials such as "hints" and "scaffolds."

• Importance of the Issue:
  Instructional questions play a critical role in classrooms, serving to assess students' comprehension, foster critical thinking, and stimulate learning interest. However, existing resources and technologies are insufficient to effectively support teachers in accomplishing this task.

• Research Motivation and Related Work:
  Based on LLMs (e.g., GPT-4), "TutorCraftEase" aims to overcome the limitations of traditional tools, enhance the efficiency of instructional question generation, and meet the practical needs of modern teaching scenarios. This research builds on prior ITS tools (e.g., CTAT, OATutor) and explores the potential applications of LLMs in education.

Solution

• Method and Solution:
  The authors propose an interactive generation tool called "TutorCraftEase," which leverages the generative capabilities of LLMs to assist teachers in quickly creating multi-level, high-quality instructional questions while providing editing and improvement functionalities.

• Innovations:

  1. Hierarchical Generation Architecture: Questions are structured based on "title-body-solution steps" and support step-by-step guidance (e.g., hints and scaffolds), making the generated questions more aligned with students' actual learning needs.
  2. Interactive Editing: Teachers can not only generate questions but also customize them by adjusting attributes (e.g., difficulty, type, and knowledge points) or manually modifying the questions.
  3. Generation Optimization Techniques:
     • Utilizes prompt chaining to generate hierarchical content.
     • Introduces the RICTEF (Role, Input, Constraints, Task, Examples, Format) template to ensure model outputs meet academic requirements.
     • Employs a tree-structured decomposition method to break down complex questions into multiple sub-questions, generating hints and scaffolds for each.

• Implementation Steps and Key Technologies:

  1. Text Analysis: By referencing selected sections of textbooks, the model generates instructional questions based on user input.
  2. Attribute Editing: Users can select content attributes such as grade level, difficulty, and knowledge points before question generation, instantly producing tailored instructional questions.
  3. Real-Time Preview: A preview panel allows users to instantly view and test the generated questions.
  4. Data Output: Supports exporting completed instructional questions to specific ITS systems (e.g., OATutor).

Research Outcomes

• Specific Results:

  1. The quality of instructional questions generated by TutorCraftEase is comparable to those created by experienced teachers.
  2. In a user study involving 39 participants, TutorCraftEase significantly improved question creation efficiency compared to traditional spreadsheet tools and basic LLM tools, while reducing teachers' cognitive load.
  3. Participants generally found TutorCraftEase easy to use, streamlined, and effective in enhancing human-computer collaboration and expanding educational perspectives.

• Advantages:

  1. Significant Time Savings: The time required to generate questions was reduced by over 40% compared to traditional methods (e.g., spreadsheets).
  2. Diversity and Completeness: Generated questions featured diverse formats with more comprehensive step-by-step guidance, hints, and scaffolds.
  3. Positive User Experience: Scored significantly higher than competing tools in dimensions such as usability, efficiency, and satisfaction.

• Experimental or Evaluation Results:

  • Blind Testing Results: Three experienced teachers evaluated 90 randomly generated instructional questions, finding that TutorCraftEase's quality was on par with manually created questions and superior to basic LLM tools.
  • User Survey: TutorCraftEase significantly outperformed other tools in terms of helpfulness, usability, and efficiency.

• Limitations and Future Directions:

  1. Limitations:
     • Some generated questions exhibited instability in quality, with issues in logic or complexity.
     • Lack of support for complex input formats (e.g., tables or advanced templates), limiting adaptability in specific scenarios.
     • Heavy reliance on LLMs led some users to feel constrained in their creative freedom.
  2. Future Directions:
     • Optimize the stability and quality of generated questions by introducing smarter output correction and secondary review mechanisms.
     • Add design features to support complex question structures, complemented by dynamic templates to enhance generation diversity.
     • Explore methods to better evaluate students' acceptance and learning outcomes of the generated questions in real classroom or ITS environments.

The insights summarized above highlight the significant potential of TutorCraftEase in improving the efficiency of instructional question generation and reducing teachers' workload. This research also provides valuable experience and solutions for the deeper application of LLMs in the field of education.