"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents

Title of the Paper

“It’s a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents

Paper Information

• Research Domain: User privacy and sensitive information disclosure behavior in large language models (LLM)
• Keywords: Large language models, AI conversational agents, privacy risks, data usage, user behavior, contextual integrity, privacy design, human-computer interaction, empirical study

Research Background and Problem Statement

• What problems or challenges did the authors identify?

  • LLMs (e.g., ChatGPT) are widely used in high-risk domains (e.g., healthcare, finance, mental health counseling), requiring users to disclose sensitive information (e.g., medical records, financial data, personal experiences), which raises significant privacy and security risks, including data breaches, misuse of personal information, and model memory risks.
  • Current research on LLM privacy issues is predominantly technical, lacking focus on user-centric issues such as subjective perceptions of privacy risks and motivations for disclosure behavior.

• Why is this problem important?

  • LLM conversational agents have become essential tools in daily life, and their openness and human-like design encourage users to share more, often escalating to sensitive information.
  • Protecting user privacy is not only a technical design requirement but also closely tied to policy-making and user rights.

• Research Motivation and Related Work

  • Existing studies mainly focus on technical privacy protection measures (e.g., differential privacy algorithms, data cleaning, and knowledge learning), neglecting users' actual behaviors.
  • This study adopts a user-centered perspective for the first time, exploring how users balance risks and benefits when disclosing private information.

Solutions

• What methods or solutions did the authors propose?

  • The study consists of two parts:
    1. Qualitative analysis of publicly available ChatGPT conversation history data (ShareGPT52K dataset) to explore users' sensitive information disclosure behaviors.
    2. Semi-structured interviews with 19 LLM users to further investigate users' perceptions of privacy risks, motivations, and coping strategies.
  • Proposed designing more privacy protection tools and providing progressive privacy controls to help users better manage privacy risks.

• What is innovative about this solution?

  • Systematically explores LLM user privacy challenges based on actual user data and behaviors for the first time.
  • Provides direction for designing more targeted and effective privacy protection measures.

• What are the implementation steps? What key technologies were used?

  1. Data Analysis: Used Microsoft Presidio to detect PII (Personally Identifiable Information) in the ShareGPT52K dataset, then categorized users' actual conversational behaviors into multiple dimensions and generated a typology of scenarios.
  2. User Interviews: Conducted open-ended questions and drawing tasks to understand users' privacy protection behaviors and their cognitive models of system operations (mental models).
  3. Inductive Design Recommendations: Based on identified privacy issues and user needs, developed a multi-layered privacy protection design framework (e.g., automated de-sensitization processes and flexible selective data usage controls).

Research Findings

• What specific findings were achieved?

  • Identified key factors influencing users' information disclosure behaviors, including trust in the agent's capabilities, operational convenience, perception of data sensitivity, and a behavioral trend of "actively accepting privacy risks."
  • Proposed a detailed classification of user privacy risks, including institutional privacy, memory risks, and interdependent privacy issues (involving third-party data).
  • Discovered multiple barriers preventing users from adopting existing privacy controls (e.g., dark patterns in design) and suggested directions for improving privacy support.
  • Found confusion in users' mental models, with significant misunderstandings about LLM training and data memory mechanisms, affecting privacy decision-making.

• How does it compare to existing solutions?

  • Provides a more comprehensive coverage of dynamic user behavior, offering direct guidance for technology and interface design.
  • Links technical issues to real-world scenarios, providing concrete evidence for social and policy interventions.

• What are the experimental or evaluation results?

  • Interview data revealed widespread user misunderstanding of privacy controls, with most users unaware of effective privacy protection tools (e.g., Opt-out features).
  • Data analysis showed users gradually disclose more information during interactions, directly influenced by the human-like conversational style of LLMs.

• Limitations and Future Directions

  • Limitations:
    • Focuses on ChatGPT, potentially lacking comprehensive insights into other LLM platforms.
    • Interview sample may have representational bias, as participants with highly sensitive information might opt out.
    • Quantitative modeling requires further research.
  • Future Directions:
    • Investigate privacy issues among different user groups, particularly vulnerable populations.
    • Test improved privacy protection tool designs and conduct longitudinal studies to observe changes in public privacy attitudes over time.