When Generative AI Is Intimate, Sexy, and Violent: Examining Not-Safe-For-Work (NSFW) Chatbots on FlowGPT

Paper Title

When Generative AI Is Intimate, Sexy, and Violent: Examining Not-Safe-For-Work (NSFW) Chatbots on FlowGPT

Publication Info

• Topic area: Analysis of NSFW generative AI chatbots and their implications for design, safety, and moderation.
• Keywords: NSFW, generative AI, chatbots, FlowGPT, content moderation, user safety, technosexuality, virtual intimacy, harmful content, AI ethics.

Background and Problem

• Problem / challenge: Limited understanding of how generative AI (GenAI) chatbots are used to create and share NSFW content, including their functions, user interactions, and associated risks.
• Significance: NSFW chatbots represent a new medium for delivering explicit content, raising concerns about user safety, content moderation, and the ethical implications of AI-mediated sexuality.
• Motivation and related work: Previous studies have focused on human-generated NSFW content on platforms like Tumblr, Reddit, and Instagram, but the dynamics of AI-generated NSFW content, especially on platforms like FlowGPT, remain underexplored. This paper builds on Paasonen’s framework of NSFW as boundary work, engagement device, and framing device to analyze these chatbots.

Solution

• Proposed approach: A data-driven analysis of NSFW chatbots on FlowGPT, categorizing their functions, examining their engagement strategies, and identifying harmful content dynamics in user-chatbot interactions.
• Novelty:
  1. Categorization of NSFW chatbots into four types: AI Characters, Story Generators, Image Generators, and Do-Anything-Now (DAN) bots.
  2. Analysis of chatbot identity and behavioral traits as mechanisms for user engagement.
  3. Identification of harmful content patterns (e.g., sexual, violent, and insulting content) in user-chatbot interactions.
  4. Exploration of the implications of NSFW chatbots for user safety, content moderation, and AI ethics.
• Procedure and key techniques:
  • Thematic analysis of 376 NSFW chatbots and 307 public conversation sessions on FlowGPT.
  • Categorization of chatbot types, identities, and behavioral traits.
  • Use of tools like ChatGPT, Google SafeSearch, and Azure Content Safety to detect harmful content in user prompts and chatbot outputs.
  • Manual and automated annotation of harmful content across conversations.

Results

• Concrete findings:
  • 74.2% of NSFW chatbots are AI Characters, followed by Story Generators (16.8%), Image Generators (5.6%), and DAN bots (4.0%).
  • 40.9% of AI Characters adopt fantasy or subculture identities, while others represent professional figures, close relationships, or strangers.
  • Behavioral traits include hangout (38.4%), flirting (33.0%), sexual interaction (24.7%), and rejection (3.9%).
  • Over 40% of conversations contain sexual content, with violent and insulting content also present.
  • Chatbots sometimes generate sexual content even without explicit user prompts.
• Advantage over baselines: Provides a comprehensive categorization and analysis of NSFW chatbots, highlighting their unique functionalities and risks compared to traditional NSFW content on social media.
• Experiments / evaluation:
  • Data collection from FlowGPT using multiple researcher accounts to reduce retrieval bias.
  • Manual and automated annotation of harmful content using established taxonomies and tools.
  • Cross-validation of harmful content detection using multiple methods.
• Limitations and future work:
  • Lack of analysis of multi-turn interactions between users and chatbots.
  • Dependence on LLM accuracy for harmful content detection.
  • Dataset limited to FlowGPT, which may not generalize to other platforms.
  • Future work should explore user motivations, real-world behavioral impacts, and improved moderation strategies.

Summary

This paper investigates NSFW chatbots on FlowGPT, categorizing them into four types and analyzing their engagement strategies and harmful content dynamics. The findings reveal that AI Characters dominate the NSFW chatbot landscape, often adopting fantasy or subcultural identities and engaging users through hangout, flirtation, or explicit sexual interactions. Harmful content, including sexual, violent, and insulting material, is prevalent, with some chatbots generating explicit content even without user prompts. The study highlights the challenges of moderating NSFW chatbots and the ethical implications of AI-mediated intimacy, calling for further research on user safety, content moderation, and the societal impact of these technologies.