ReadingQuizMaker: A Human-NLP Collaborative System to Support Instructors Design High Quality Reading Quiz Questions

Title of the Paper

ReadingQuizMaker: A Human-NLP Collaborative System that Supports Instructors to Design High-Quality Reading Quiz Questions

Paper Information

• Domain: Human-Computer Collaboration, Natural Language Processing, and Educational Technology
• Keywords: Reading Quiz, Active Learning, Human-Computer Collaboration, Automatic Question Generation, Educational Technology

Research Background and Problem

• Problem Identification:

  • Students often lack active reading skills; only 20%-30% of college students complete assigned reading materials.
  • Existing social annotation tools may extend reading time but fail to provide substantial feedback on content comprehension.
  • Designing high-quality, thought-provoking reading quiz questions is labor-intensive, time-consuming, and requires domain-specific expertise and technical support.
  • Current automatic question generation tools are widely regarded as low-quality and often limited to specific domains.

• Significance:

  • Well-designed reading quiz questions can promote active learning and content comprehension while providing students with immediate feedback.
  • Developing robust support tools for educators can address low engagement in academic reading and improve teaching effectiveness.

• Research Motivation and Related Work:

  • Automatic question generation technologies are primarily applied in language learning and mathematics education, failing to address higher-order cognitive tasks.
  • Human-computer collaboration has the potential to compensate for the shortcomings of purely automated methods, especially in creative human tasks.

Solution

• Proposed Method:

  • Developed a system named ReadingQuizMaker to assist instructors in designing high-quality reading quiz questions.
  • The system is based on a human-computer collaboration model, integrating natural language processing (NLP) technologies to assist instructors while maintaining their full control.

• Innovations:

  • Provides workflow-based NLP support, allowing instructors to select model inputs and edit generated outputs.
  • Offers real-time AI suggestions (e.g., sentence rewriting, summary generation, negation generation) to optimize the question creation process.
  • Features a visualization interface to help instructors assess content coverage and question quality.

• Implementation Steps and Techniques:

  • Reading materials are uploaded to the system interface.
  • Instructors can highlight text to generate question options and preview AI suggestions.
  • The system includes a question template library, an NLP toolbox (supporting rewriting, negation, etc.), and exportable quiz packages.
  • Utilizes pre-trained models like BART and PEGASUS for text summarization, sentence rewriting, and negation generation.

Research Outcomes

• Specific Results:

  • The ReadingQuizMaker system significantly reduced the time required for instructors to design questions while improving question quality.
  • In evaluation studies, all participating instructors successfully created satisfactory questions and found AI suggestions helpful for inspiration.
  • The system was endorsed by instructors as a replacement for traditional question generation methods and as a tool to support higher-level teaching objectives.

• Advantages:

  • The human-computer collaboration model outperforms automated generation methods, offering instructors greater flexibility and control.
  • Compared to existing tools, the system aligns more closely with natural workflows and significantly enhances the quality of generated quiz questions.

• Experimental and Evaluation Results:

  • A total of 89 questions were created during the study, with 60% of AI suggestions adopted by instructors.
  • Under baseline conditions, instructors generally found automatically generated questions to be of low quality, lacking context, and limiting creativity.

• Limitations and Future Directions:

  • Generating distractors remains the most challenging task for instructors, highlighting the need to refine model performance.
  • The system is currently more suitable for academic articles but could be expanded to include more informal texts (e.g., news, tutorials).
  • Future work could explore optimizing the visualization of AI outputs (e.g., summaries, rewritten results) to enhance interpretability and usability.
  • Longitudinal studies are needed to understand the development of instructors' trust in the system and its long-term benefits.

Conclusion and Insights

• Designing reading quizzes is a highly cognitive and creative task, and the human-computer collaboration model significantly improves instructors' efficiency and enhances question quality.
• This study highlights the importance of allowing users to provide input and maintain control in high-stakes tasks (e.g., educational content creation), offering critical insights for the design of educational technologies.