Will They Try Again? A Large-Scale RCT on Scaffolds that Support Persistence in an Intelligent Tutoring System

Paper Title

Will They Try Again? A Large-Scale RCT on Scaffolds that Support Persistence in an Intelligent Tutoring System

Publication Info

• Topic area: Persuasive design interventions in educational technology to promote persistence after failure.
• Keywords: Intelligent tutoring systems, persistence, failure, nudges, prompts, persuasive design, randomized controlled trial, education technology, mastery learning, interface design.

Background and Problem

• Problem / challenge: Students in intelligent tutoring systems often disengage after making mistakes, undermining the benefits of persistence and corrective feedback.
• Significance: Persistence after failure is critical for learning, as it enables mastery through repeated practice and error correction.
• Motivation and related work: Prior research highlights the benefits of productive struggle and mastery learning but lacks scalable, adaptive interventions to promote persistence. While persuasive design techniques like nudges and prompts have shown promise in other domains, their combined effects in education remain unexplored.

Solution

• Proposed approach: Two interventions were tested: (1) a visual default nudge that highlights the retry option, and (2) a persuasive prompt encouraging students to try again after failure.
• Novelty:
  1. Large-scale randomized controlled trial involving 164,532 students and 17 million practice problems.
  2. Comparison of implicit (nudge) and explicit (prompt) strategies, examining their independent and combined effects.
  3. Analysis of spillover effects, fade-out patterns, and downstream learning outcomes.
• Procedure and key techniques:
  • Students were randomly assigned to one of four conditions: control, nudge-only, prompt-only, or combined interventions.
  • Nudges were consistently delivered as visual cues, while prompts were probabilistically shown to reduce fatigue.
  • Behavioral data (e.g., repeated problem attempts, mastery rates) were analyzed using mixed-effects models.

Results

• Concrete findings:
  • Nudges increased persistence by 9 percentage points, prompts by 2 points, and their combination by 11 points.
  • Spillover effects were limited, with prompts showing slightly greater generalization than nudges.
  • Both interventions maintained effectiveness over time, with nudges slightly increasing and prompts stabilizing after consistent delivery.
  • Mastery rates improved modestly: nudges increased problem-solving success by 1.6 percentage points, prompts by 0.6 points.
• Advantage over baselines: Both interventions significantly outperformed the control condition, with additive effects when combined.
• Experiments / evaluation:
  • Conducted in Siyavula’s intelligent tutoring system for Grades 8–12 math and science.
  • Analyzed nearly 2.4 million errors and 17 million problem attempts using regression models.
• Limitations and future work:
  • Probabilistic delivery of prompts may have limited their impact; future research could explore optimal frequencies.
  • Results were domain-specific (math/science); testing in other contexts is needed.
  • Heterogeneous effects across problem types, subjects, and student characteristics warrant further investigation to enable personalized interventions.

Summary

This study demonstrates that implicit nudges and explicit prompts independently and additively promote persistence after failure in intelligent tutoring systems. Nudges produced larger immediate effects, while prompts showed slightly greater generalization to non-targeted contexts. Both interventions retained their effectiveness over time, and modest improvements in mastery rates were observed. These findings advance theories of persuasive design by showing that implicit and explicit strategies operate through distinct mechanisms and can be effectively combined. The results provide actionable guidance for designing scalable, ethically grounded interventions to support persistence in educational and other interactive systems.