ScamPilot: Simulating Conversations with LLMs to Protect Against Online Scams

Paper Title

ScamPilot: Simulating Conversations with LLMs to Protect Against Online Scams

Publication Info

• Topic area: Cybersecurity education and scam prevention using conversational AI.
• Keywords: Online scams, cybersecurity, large language models, inoculation theory, experiential learning, scam recognition, conversational user interfaces, scam resilience training, scam education, scam simulation.

Background and Problem

• Problem / challenge: Automated fraud detection systems and awareness campaigns are limited in their ability to adapt to evolving scam tactics and engage users effectively. Existing training systems often lack realism, adaptability, and detailed feedback, failing to prepare users for real-world scam scenarios.
• Significance: Online scams cause significant financial and emotional harm, with billions lost annually. Training users to recognize and resist scams is critical to complement automated detection systems.
• Motivation and related work: Prior work has explored static scam simulations, gamified systems, and inoculation theory, but these approaches often lack engagement, adaptability, and actionable feedback. This paper builds on these foundations by leveraging large language models (LLMs) to create dynamic, realistic scam simulations.

Solution

• Proposed approach: ScamPilot, a conversational user interface powered by LLMs, simulates realistic scam scenarios through dynamic interactions between a scammer agent, a target agent, and a human user acting as an advisor.
• Novelty:
  1. Introduction of a three-way conversational interaction between two LLM agents and a human user.
  2. Integration of inoculation theory with experiential learning principles (learning-by-doing and learning-by-teaching).
  3. Dynamic, phase-specific feedback and embedded multiple-choice quizzes to reinforce scam recognition and response strategies.
• Procedure and key techniques:
  • Users observe a simulated conversation between a scammer and a target agent.
  • They provide real-time advice to the target agent and answer multiple-choice questions about scam tactics.
  • Feedback is provided after each phase, summarizing the conversation and offering actionable insights.
  • The system uses prompt engineering to ensure realistic and consistent behavior from the LLM agents.

Results

• Concrete findings:
  • The quiz+advice interface increased scam recognition by 8%, response efficacy by 9%, and self-efficacy by 19% compared to the control group.
  • No significant decrease in legitimate message recognition was observed.
• Advantage over baselines:
  • Dynamic advice-based interfaces outperformed static quiz-only and control interfaces in scam recognition and situational judgment tasks.
  • The quiz+advice condition balanced improved scam recognition with minimal impact on legitimate message discernment.
• Experiments / evaluation:
  • A between-subjects study with 150 participants across four conditions (control, quiz, advice, quiz+advice).
  • Metrics included scam/legitimate discernment, situational judgment, self-efficacy, and response efficacy.
  • Mixed-methods analysis combining quantitative performance measures and qualitative advice coding.
• Limitations and future work:
  • Limited to imposter scams; applicability to other scam types needs validation.
  • Real-world effectiveness and long-term knowledge retention were not tested.
  • Potential ethical concerns with LLM-generated scam simulations.
  • Future work includes longitudinal studies, testing with diverse demographics, and expanding to other scam types.

Summary

ScamPilot leverages large language models to simulate realistic scam conversations, enabling users to practice scam recognition and resistance through interactive advice-giving and quizzes. The quiz+advice interface was the most effective, improving scam recognition (+8%), response efficacy (+9%), and self-efficacy (+19%) without significantly increasing false positives for legitimate messages. This study demonstrates the potential of combining inoculation theory with experiential learning in cybersecurity education. Future work will explore broader applications, long-term retention, and ethical considerations.