When AI Gets It Wrong: Scaffolding AI Hallucination Detection for Children Through Chatbot Creation

Paper Title

When AI Gets It Wrong: Scaffolding AI Hallucination Detection for Children Through Chatbot Creation

Publication Info

• Topic area: AI literacy and hallucination detection in children through chatbot development.
• Keywords: AI literacy, hallucination detection, chatbot development, generative AI, children-AI interaction, educational scaffolds, trustworthy AI, iterative design, critical thinking, middle school learners.

Background and Problem

• Problem / challenge: Generative AI systems often produce hallucinations—factually incorrect or misleading outputs—which are particularly problematic for children due to their developmental characteristics, such as heightened trust in authoritative-seeming technologies and overestimation of AI capabilities. Existing AI literacy tools rarely address hallucinations or involve children in the process.
• Significance: Addressing this issue is crucial to prevent the reinforcement of misconceptions, erosion of trust in technology, and to promote critical thinking and responsible AI engagement among children.
• Motivation and related work: Prior work has focused on AI literacy frameworks and tools for supervised learning or classification but has not adequately addressed hallucinations in generative AI. Scaffolds like confidence indicators and fact-checking have been effective for adults but are underexplored for children. This paper builds on these gaps by designing and evaluating a child-friendly chatbot development environment.

Solution

• Proposed approach: A scaffolded chatbot-building tool called LUMI, which integrates hallucination-awareness features to help children detect and respond to AI hallucinations during chatbot creation.
• Novelty:
  1. Development of a scaffolded chatbot-builder prototype tailored for children, embedding five hallucination-awareness features.
  2. Empirical insights into children’s strategies for detecting and responding to hallucinations.
  3. Design implications for creating child-centered AI literacy tools that balance creativity with reliability.
• Procedure and key techniques:
  • Design of LUMI with features like response confidence indicators, fact-checking, document verification, model comparison, and repeated questioning.
  • A two-day study with 48 middle school learners (ages 10–14) across two camps, comparing a baseline chatbot builder with one that includes hallucination-awareness scaffolds.
  • Data collection through pre-/post-surveys, focus groups, and analysis of learner-created chatbot prompts.

Results

• Concrete findings:
  • Significant pre-to-post gains in AI knowledge, hallucination awareness, and confidence in building trustworthy chatbots across both groups.
  • Learners effectively used hallucination-awareness features like confidence indicators and fact-checking but struggled with deeper integration of these tools into systematic workflows.
  • Chatbot prompts created by students were generally of high quality, with 60% rated as "Excellent."
• Advantage over baselines:
  • Both versions of the chatbot builder supported learning gains, but no statistically significant differences were observed between the baseline and awareness-supported versions.
• Experiments / evaluation:
  • Study involved 48 participants across two camps, with one group using the baseline builder and the other using the awareness-supported builder.
  • Evaluation metrics included pre-/post-surveys, focus group thematic analysis, and rubric-based scoring of chatbot prompts.
• Limitations and future work:
  • Self-reported measures may lack sensitivity to detect nuanced learning shifts.
  • Group-level interface assignment and unequal sample sizes may have introduced bias.
  • Limited long-term follow-up and inconsistent hallucination elicitation due to advanced LLM capabilities.
  • Future work should explore robust assessments, individual randomization, and long-term impacts.

Summary

This study introduces LUMI, a scaffolded chatbot-building tool designed to help children detect and respond to AI hallucinations. Through a two-day study with 48 middle school learners, significant learning gains in AI knowledge, hallucination awareness, and chatbot-building confidence were observed. While learners engaged effectively with features like confidence indicators and fact-checking, challenges included over-reliance on surface cues, fragmented workflows, and a tension between creativity and reliability. The findings highlight opportunities to design AI literacy tools that foster critical thinking, iterative workflows, and developmental appropriateness, empowering children as active creators and evaluators of AI systems.