Who Controls the Conversation? User Perspectives On Generative AI (LLM) System Prompts

Paper Title

Who Controls the Conversation? User Perspectives on Generative AI (LLM) System Prompts

Publication Info

• Topic area: User-centered design and governance of generative AI system prompts.
• Keywords: Generative AI, system prompts, transparency, user control, large language models, AI governance, value-sensitive design, AI alignment, AI ethics, user preferences.

Background and Problem

• Problem / challenge: System prompts, which significantly influence AI behavior, remain opaque to users, limiting transparency and accountability. Current practices lack empirical grounding in user needs and preferences.
• Significance: System prompts shape millions of daily interactions with generative AI systems, embedding values and constraints that affect user trust, safety, and agency.
• Motivation and related work: Prior research has focused on user-issued prompts but neglected the impact of system prompts imposed by developers. Calls for transparency and accountability have emerged, but systematic understanding of user perspectives on system prompts is lacking.

Solution

• Proposed approach: A systematic empirical investigation combining taxonomy development of system prompts and a user survey to understand perceptions, preferences, and transparency/control desires.
• Novelty:
  1. Developed a seven-topic taxonomy of system prompts based on manual and computational analysis.
  2. Conducted a survey (N = 109) to capture user perspectives on system prompt design, transparency, and control mechanisms.
  3. Identified user-centered design values and actionable insights for aligning system prompts with user expectations.
• Procedure and key techniques:
  • Compiled a dataset of 1,309 system prompts from official sources and community repositories.
  • Conducted manual thematic coding and computational concept induction using LLooM to develop the taxonomy.
  • Designed a seven-stage survey to assess user awareness, design preferences, transparency desires, and control mechanisms.
  • Analyzed survey responses to identify key findings and implications for system prompt design and governance.

Results

• Concrete findings:
  • Seven taxonomy topics identified: AI Role & Identity, Capabilities & Domain Specifics, Communication Style & Structure, Compliance, Safety & Security, Deployment & Operation, Intrinsic Values & Principles, Response Quality.
  • 89% of participants desired transparency, with preferences for summaries (23%) or full system prompts (27%).
  • 79% wanted control mechanisms, favoring structured options like setting preferences (20%) over unrestricted editing (8%).
  • Privacy and freedom from bias were rated as the most important design values.
• Advantage over baselines:
  • Provides empirical evidence of user preferences for transparency and control, addressing gaps in prior research.
  • Introduces a comprehensive taxonomy applicable across diverse system prompt types.
• Experiments / evaluation:
  • Mixed-methods approach: manual coding and LLooM computational analysis for taxonomy development; survey with 109 participants to assess user perspectives.
  • Metrics: comfort ratings, importance of design values, transparency and control preferences.
• Limitations and future work:
  • Limited access to proprietary system prompts; findings based on publicly available and community-sourced data.
  • English-speaking participant sample may limit generalizability to non-Western contexts.
  • Future work should explore behavioral studies, cross-cultural perspectives, and finer-grained differences in domain-specific prompts.

Summary

This study investigates the design and governance of system prompts in generative AI systems, combining taxonomy development and user survey analysis. It identifies seven core topics of system prompts and reveals strong user preferences for transparency and control mechanisms, emphasizing values like privacy and freedom from bias. The findings highlight the need for participatory approaches to system prompt design and governance, with implications for transparency, accountability, and user agency in AI systems. Future research should address technical challenges, cross-cultural perspectives, and evolving practices in system prompt implementation.