Helping Johnny Make Sense of Privacy Policies with LLMs

Paper Title

Helping Johnny Make Sense of Privacy Policies with LLMs

Publication Info

• Topic area: Enhancing user comprehension and engagement with privacy policies using Large Language Models (LLMs).
• Keywords: Privacy policies, LLMs, user-centered design, transparency tools, retrieval-augmented generation, privacy awareness, browser extension, user study, data protection, interactive systems.

Background and Problem

• Problem / challenge: Privacy policies are complex, lengthy, and written for legal compliance rather than user comprehension, leaving users unable to make informed decisions about their data.
• Significance: With increasing data collection and privacy risks, tools that make privacy policies accessible are critical for empowering users and addressing informational asymmetries.
• Motivation and related work: Prior tools for privacy policy analysis rely on outdated standards, static criteria, or legal compliance checks, limiting adaptability and user engagement. LLMs offer potential for dynamic, user-friendly policy analysis, but existing solutions lack integration into natural browsing experiences and fail to address issues like hallucinations and adversarial robustness.

Solution

• Proposed approach: PRISMe (Privacy Risk Information Scanner for Me), a browser extension that combines LLM-based privacy policy assessment with interactive features like dashboards and chat interfaces.
• Novelty:
  1. Integration of LLM-based privacy policy analysis directly into the browsing experience.
  2. Layered interface design enabling quick overviews, detailed dashboards, and conversational exploration.
  3. Retrieval-Augmented Generation (RAG) refinement to mitigate hallucinations and improve adversarial robustness.
  4. Mixed-methods user study identifying user interaction patterns and challenges.
• Procedure and key techniques:
  • PRISMe dynamically assesses privacy policies using GPT-4o, rating criteria on a 5-point Likert scale.
  • Users interact via a layered interface: visual cues (smiley icons), a dashboard for detailed assessments, and a chat for tailored queries.
  • Post-study, RAG was integrated to ground responses in policy text, with fallback mechanisms for insufficient evidence.

Results

• Concrete findings:
  • PRISMe improved users’ perceived understanding of privacy policies, with an average SUS score of 88.9/100.
  • RAG eliminated hallucinations and euphemistic language, improving response fidelity.
  • Chat response times averaged 1.87–8.32 seconds, while initial policy assessments took 19.94 seconds without caching.
• Advantage over baselines:
  • PRISMe’s interactive and adaptive features outperformed static, rule-based tools by enabling dynamic exploration and personalized engagement.
  • RAG refinement addressed key LLM limitations, such as hallucinations and vague responses.
• Experiments / evaluation:
  • Conducted a lab-based user study (N=22) with three scenarios: privacy exploration, policy comparison, and free exploration.
  • Participants represented diverse backgrounds and privacy knowledge levels.
  • Data collection included interviews, questionnaires, telemetry, and thematic analysis.
• Limitations and future work:
  • Small sample size limits generalizability; future studies should include larger, longitudinal, and real-world evaluations.
  • Missing comparative baselines against other tools or full policy reading.
  • Technical limitations include incomplete policy scraping and runtime delays.

Summary

PRISMe is an interactive browser extension designed to enhance user comprehension and engagement with privacy policies by leveraging LLMs. It provides layered interaction through visual cues, dashboards, and conversational interfaces, enabling users to explore policies at varying levels of detail. A user study demonstrated PRISMe’s effectiveness in raising privacy awareness and understanding, though challenges like hallucinations and adversarial robustness were identified. Post-study integration of RAG successfully mitigated these issues, improving response fidelity. PRISMe highlights the potential of LLM-based tools to empower users in navigating complex privacy policies, with future work needed to refine scalability, usability, and real-world applicability.