PrivacyAkinator: Articulating Key Privacy Design Decisions by Answering LLM-Generated Multiple-choice Questions

Paper Title

PrivacyAkinator: Articulating Key Privacy Design Decisions by Answering LLM-Generated Multiple-choice Questions

Publication Info

• Topic area: Privacy risk assessment and developer tools
• Keywords: Privacy design, NIST PRAM, large language models, privacy risk assessment, novice developers, multiple-choice questions, privacy representation, stakeholder interactions, privacy design space, usability evaluation

Background and Problem

• Problem / challenge: Existing privacy risk assessment frameworks, such as NIST’s PRAM, are complex, require significant expertise, and are challenging for novice developers to use effectively. Key issues include difficulty in articulating privacy design decisions, vague terminology, and open-ended structures that lead to incomplete assessments.
• Significance: Lowering the barriers for privacy risk assessments is critical for startups and small organizations that lack dedicated privacy professionals. Effective tools can help developers identify and mitigate privacy risks, ensuring compliance and protecting user data.
• Motivation and related work: Prior frameworks like PRAM, LINDDUN, and PIA are cumbersome and require expertise. Tools such as Coconut and PARROT support experienced developers but are not tailored for novices. Existing privacy question-answering systems focus on user-facing policies rather than developer-centric design decisions. This paper addresses the gap by creating a structured, guided tool for novices.

Solution

• Proposed approach: PrivacyAkinator, an interactive tool that helps developers articulate key privacy design decisions by answering multiple-choice questions generated by large language models (LLMs).
• Novelty:
  1. A universal privacy representation that abstracts design decisions into data flows and stakeholder interactions.
  2. A domain-aware privacy design space constructed by mining 10K privacy-related news articles.
  3. A dynamic question-generation workflow that prioritizes relevant questions based on prior responses and system context.
• Procedure and key techniques:
  • Developers provide a high-level system description.
  • PrivacyAkinator expands the description into detailed functional requirements with highlighted design choices.
  • Developers answer LLM-generated multiple-choice questions to articulate design decisions.
  • The tool organizes decisions into a structured privacy representation and maps them to PRAM worksheets.
  • Questions are dynamically generated using co-occurrence statistics and domain-specific context.

Results

• Concrete findings:
  • PrivacyAkinator enabled developers to identify 47% more key privacy decisions in 73% less time compared to PRAM worksheets.
  • The tool achieved 93.67% coverage of key design decisions and 77.33% coverage of actual choices in case studies, outperforming baseline LLMs (coverage ranged from 43.91% to 55.22% for key decisions).
• Advantage over baselines:
  • Significantly higher coverage of privacy design decisions compared to GPT-4o, Claude 3.7, and Gemini 2.5 Flash.
  • Reduced cognitive load and improved usability for novice developers.
• Experiments / evaluation:
  • Observational study with 12 participants identified challenges in using PRAM.
  • User study with 24 participants showed PrivacyAkinator’s effectiveness in improving efficiency and decision coverage.
  • Case studies on 30 real-world data practices validated the tool’s ability to identify key privacy design decisions.
• Limitations and future work:
  • Subjectivity in risk evaluation remains unaddressed.
  • Over-reliance on LLM-generated questions may limit creativity.
  • Potential biases in the news-based design space and geographic underrepresentation.
  • Future work could extend support for risk prioritization and explore applicability for privacy experts.

Summary

PrivacyAkinator is a tool designed to help novice developers articulate privacy design decisions by answering LLM-generated multiple-choice questions. It addresses challenges in using traditional frameworks like PRAM by introducing a structured privacy representation, a domain-aware design space, and a dynamic question-generation workflow. Evaluation results show that PrivacyAkinator significantly improves efficiency and coverage of key privacy decisions compared to PRAM. While the tool is tailored for novices, its components could also benefit experts. Future work will focus on addressing subjectivity in risk evaluation and expanding the design space to include diverse domains and regulatory environments.