PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions

Document Title

PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions

Document Information

• Subject Area: Human-Computer Interaction, Text-to-Image Generation Models, AI-Driven Interaction Design
• Keywords: Text-to-Image Generation, Generative Models, Human-Computer Interaction, Painting Interaction, Generative Art, Image Editing, Diffusion Models, Model Steering, Traversing Semantic Space

Research Background and Problem

• Problems and Challenges:

  • Diffusion-based Text-to-Image (T2I) models are powerful in generating high-quality images but are difficult to control and iterate effectively through language alone.
  • Users often face challenges in describing complex or hard-to-express artistic concepts.
  • Most existing T2I systems rely on end-to-end generation, lacking flexible interaction tools for incremental creation.
  • Users desire features like mixed styles and iterative adjustments rather than relying solely on textual descriptions.
  • The randomness and complexity of generative models can lead to discrepancies between user intent and final output.

• Significance of the Research:

  • Integrating better interaction methods into the creative process can help users more effectively express their intentions and gradually craft artworks that meet their expectations.
  • Expanding the application of T2I technology from "black-box" generation (inputting prompts only) to a more transparent and controllable tool.

• Motivation and Related Work:

  • Related work has provided technical support for various aspects of artistic creation, such as image editing and rich interaction controls, but most studies still focus on enhancing technical features rather than interaction model design.
  • Simulating the interactive characteristics of artistic creation through painting-like methods to enhance control over generated images offers a new approach to addressing the interaction limitations of existing systems.

Solution

• Proposed Solution:

  • A tool named PromptPaint is proposed, offering painting-like interaction for diffusion-based T2I models.
  • Users can utilize not only textual prompts but also a "palette-like" interaction to mix different prompts or flexibly adjust prompt content during the generation process.

• Innovations:

  • Introducing "semantic palette" (prompt mixing) to blend semantics in vector space, simulating color mixing and providing users with greater freedom to explore transitions between semantics.
  • Incorporating "directional prompts," allowing for specific visual changes such as color intensity and style adjustments.
  • Breaking the limitations of end-to-end generation by enabling flexible prompt insertion or adjustment during the generation process (prompt intervention).
  • Supporting "prompt stencils" to generate visual content for specific areas of the canvas, enabling users to incrementally create and modify artworks section by section.

• Implementation Steps and Techniques:

  1. Prompt Mixing: Interpolating discrete prompts (e.g., two styles) on a palette, allowing users to explore transitions between semantics.
  2. Directional Prompt: Providing directional vectors to add extra attributes (e.g., transitioning style from "modern" to "impressionist").
  3. Prompt Stencil: Allowing users to select generation areas on the canvas and adjust "coverage" for overlay generation or modification of existing images.
  4. Prompt Intervention: Dynamically adjusting prompts during image generation to steer the conceptual direction of the image.

• Key Technologies:

  • Based on diffusion models (e.g., Stable Diffusion), combining spatial vector interpolation and feature control techniques to achieve semantic blending and fine-tuning.
  • Introducing multi-layer control, real-time generation feedback, and noisy image display functionalities.

Research Outcomes

• Specific Outcomes:

  • A novel interaction method (PromptPaint) is provided, enabling users to create original visual works effectively through natural methods akin to mixing and layering paint.
  • The system helps diverse users (from visual art enthusiasts to novices) intuitively understand and control AI-generated content.
  • Comparative experiments demonstrate that PromptPaint's blending methods (e.g., prompt mixing, stencils) are more flexible and expressive than traditional direct prompt concatenation.

• Experiments and User Studies:

  • Experiments analyzed the expressive capabilities of prompt mixing techniques, evaluating performance in maintaining original attributes while adding new visual features.
  • User studies showed that interactive tools enhance users' psychological ownership and sense of control over generated results, though balancing automation and manual operation remains a user preference challenge.

• Limitations and Future Directions:

  • Limitations:
    • The randomness and complexity of T2I model outputs can lead to inconsistencies between user intent and actual results.
    • The current system lacks more diverse data generation control methods, such as negative prompt control.
  • Future Directions:
    • Extending PromptPaint to more generative scenarios (e.g., 3D model or video generation).
    • Improving the user interface to make intermediate generation results more intuitive and comprehensible.
    • Exploring how AI collaborative creation tools can enhance users' sense of ownership and manage the legitimacy of rights over generated content.