Codesigning Ripplet: an LLM-Assisted Assessment Authoring System Grounded in a Conceptual Model of Teachers’ Workflows

Paper Title

Codesigning Ripplet: an LLM-Assisted Assessment Authoring System Grounded in a Conceptual Model of Teachers’ Workflows

Publication Info

• Topic area: Educational assessment authoring with AI support
• Keywords: LLMs, assessment authoring, education technology, human-AI collaboration, reusable edits, formative assessments, iterative workflows, codesign, teacher tools, multilevel interactions

Background and Problem

• Problem / challenge: Teachers face significant challenges in creating high-quality assessments due to time constraints, lack of training, and insufficient tools. Existing AI-based solutions focus on individual question generation and fail to address the multilevel, iterative nature of assessment authoring.
• Significance: High-quality assessments are critical for student learning and teacher accountability. Addressing the inefficiencies in assessment creation can improve educational outcomes and reduce disparities between schools.
• Motivation and related work: Prior work on automatic question generation (AQG) and human-AI collaboration has shown promise but remains limited to individual questions. There is a lack of tools that integrate AI into teachers’ holistic workflows for creating, adapting, and refining assessments.

Solution

• Proposed approach: Ripplet, a web-based system leveraging large language models (LLMs) to support multilevel, iterative assessment authoring workflows.
• Novelty:
  1. Development of a conceptual model capturing teachers’ iterative dual process of authoring assessments and refining requirements.
  2. Introduction of multilevel reusable interactions, enabling teachers to adapt and reuse edits across questions and assessments.
  3. Integration of AI tools to support question generation, adaptation, and quality assurance while embedding teachers’ requirements into the workflow.
  4. Demonstration of Ripplet’s impact through codesign with 13 teachers and a controlled study with 15 additional teachers.
• Procedure and key techniques:
  • Conducted a three-phase, seven-month codesign process with teachers to develop the conceptual model, design objectives, and refine Ripplet.
  • Implemented features such as multilevel reusable edits, question generation from diverse inputs, and tools for restructuring and exporting assessments.
  • Evaluated Ripplet’s effectiveness through independent classroom adoption and a controlled user study.

Results

• Concrete findings:
  • Ripplet improved assessment quality (+1.11 on a 10-point scale, p = 0.032) and teacher satisfaction with the effort-to-result ratio (+1.93, p = 0.012).
  • Teachers reported time savings, e.g., creating a 28-question exam in 30 minutes instead of five hours.
  • Multilevel reusable edits supported efficient adaptation and reuse of content.
• Advantage over baselines:
  • Ripplet outperformed teachers’ current practices in enjoyment (+2.70, p = 0.003), exploration (+2.43, p = 0.012), and assessment quality.
  • Enabled teachers to create assessments they would not have otherwise made, shifting their role from generation to curation.
• Experiments / evaluation:
  • Codesign with 13 teachers across eight schools, followed by independent use in classrooms.
  • Controlled user study with 15 teachers comparing Ripplet to their current practices using surveys and qualitative feedback.
• Limitations and future work:
  • Limited representation of new and non-STEM teachers in the codesign process.
  • Need for safeguards to mitigate risks of AI reliance, especially for less experienced teachers.
  • Future work includes deeper evaluations of multilevel reusable interactions and expanding the conceptual model to diverse teaching contexts.

Summary

This paper presents Ripplet, an LLM-assisted system for assessment authoring, developed through a seven-month codesign process with teachers. Ripplet supports iterative, multilevel workflows by enabling question generation, adaptation, and reuse of edits. It improved assessment quality and teacher satisfaction in both codesign and controlled studies. While demonstrating significant educational benefits, future work will address broader teacher demographics and refine safeguards for AI reliance.