Tinker, Tailor, Trust: How Developers Create Privacy Policies With and Without AI

Paper Title

Tinker, Tailor, Trust: How Developers Create Privacy Policies With and Without AI

Publication Info

• Topic area: Privacy policy creation processes in mobile app development, with and without AI tools.
• Keywords: Privacy policies, mobile app development, large language models, AI-assisted writing, compliance, privacy regulations, SDKs, app stores, validation, developer workflows.

Background and Problem

• Problem / challenge: Developers often create privacy policies without legal assistance, leading to inaccuracies and non-compliance. The emergence of LLMs introduces new challenges, such as reliance on potentially outdated or incorrect outputs.
• Significance: Privacy policies are legally required and critical for user trust, yet inaccuracies can lead to regulatory violations and undermine transparency.
• Motivation and related work: Previous research has focused on analyzing privacy policies or automating their generation but has not extensively studied how developers actually create these documents, especially with the integration of LLMs. This study addresses this gap by examining real-world practices.

Solution

• Proposed approach: A mixed-methods study involving interviews and LLM demonstrations with 20 mobile developers to understand their privacy policy creation processes.
• Novelty:
  1. First empirical study of how developers create privacy policies with and without LLMs.
  2. Insights into the challenges of using LLMs for legally sensitive tasks.
  3. Identification of gaps in validation processes and reliance on app store acceptance.
  4. Recommendations for hybrid tools combining structured workflows with LLM flexibility.
• Procedure and key techniques:
  • Conducted semi-structured interviews with developers from diverse regions and industries.
  • Observed participants using an LLM (Claude Sonnet 3.5) to create privacy policies for their apps.
  • Analyzed workflows, prompting strategies, validation methods, and trust levels.

Results

• Concrete findings:
  • Developers primarily use templates, generators, or LLMs to create privacy policies, rarely writing from scratch.
  • LLMs save time but often produce generic, incomplete, or outdated outputs.
  • Most participants trusted LLM-generated policies, with 10/20 rating their comfort level at 8–10 on a 10-point scale.
  • Developers rely on app store acceptance as a primary validation method, despite minimal substantive checks by platforms.
• Advantage over baselines: LLMs offer flexibility and customization compared to traditional generators but require sophisticated prompting to avoid generic outputs. Hybrid approaches may combine the strengths of both.
• Experiments / evaluation:
  • 20 interviews with developers from five regions (Asia, North America, Europe, Middle East, Africa).
  • Observed LLM usage patterns, including initial prompts, follow-up strategies, and validation approaches.
  • Analyzed challenges like SDK data uncertainty, outdated regulatory information, and incorrect assumptions.
• Limitations and future work:
  • Study design was exploratory and not fully generalizable.
  • Did not assess the legal compliance or quality of generated policies.
  • Future work should focus on automated accuracy and compliance tests, as well as more ecologically valid studies.

Summary

This study investigates how mobile developers create privacy policies, highlighting the growing use of LLMs alongside traditional methods like templates and generators. Developers value LLMs for their time-saving capabilities but face challenges with outdated information, incorrect assumptions, and inadequate SDK coverage. Validation often relies on app store acceptance rather than legal review, raising concerns about policy accuracy. The findings suggest opportunities for hybrid tools that combine structured workflows with LLM flexibility. Platforms and SDK publishers could further support developers by providing transparent validation processes and machine-readable privacy documentation.