Prompting for Discovery: Flexible Sense-Making for AI Art-Making with Dreamsheets

Title of the Paper

Prompting for Discovery: Flexible Sense-Making for AI Art-Making with DreamSheets

Bibliographic Information

• Field of Study: Human-Computer Interaction and Generative AI Art Creation
• Keywords: Generative AI, Text-to-Image, Design Space Exploration, Prompt Engineering, Visualization

Research Background and Problem Statement

• Identified Problems or Challenges:

  • The input and output spaces of Text-to-Image (TTI) models are vast and opaque, making it difficult for users to understand the relationship between inputs and outputs.
  • Commercial TTI interfaces offer limited support for exploratory tasks and prompt engineering, with most existing tools providing only basic prompt text boxes and minimal output display.
  • There is a lack of comprehensive interface tools to systematically assist users in exploring the prompt space to achieve desired results or understand the mechanisms behind model generation.

• Importance of the Problem:

  • As generative AI models become increasingly prevalent in art creation, understanding and mastering the input-output mapping of TTI models has become a core challenge. Effective exploration is essential for users to achieve creative goals and drive innovative applications of generative AI.

• Research Motivation and Related Work:

  • Existing studies have focused on improving the interaction interfaces of generative models (e.g., Promptify, GanZilla), but these tools often prioritize optimizing single-generation tasks rather than supporting large-scale exploration.
  • Related literature highlights the potential of prompt engineering techniques and large-scale result visualization, but lacks long-term analysis of comprehensive exploration interfaces for TTI models.
  • This study aims to address the gaps in existing research and tools by introducing the DreamSheets tool, which provides flexible support for TTI exploration and sense-making.

Solution

• Proposed Method or Solution:

  • Develop DreamSheets, an innovative spreadsheet-based tool that allows users to define prompt exploration systems, combining customizable prompt generation and image creation functionalities.
  • Provide a series of spreadsheet functions powered by large language models (LLMs) (e.g., generating synonyms, alternative descriptions, or semantic expansions) to support prompt diversity and exploration.
  • Integrate a dynamic two-dimensional display interface, enabling users to easily compare input-output relationships or perform batch generation and image classification through drag-and-drop and auto-fill features.

• Innovative Aspects:

  1. Offers a highly flexible exploration interface, supporting users in comprehending and controlling the entire process from prompt input to image output.
  2. Combines spreadsheet formula construction with LLM-assisted functionalities, establishing scalable "prompt exploration axes" and layouts for large-scale dynamic image generation.
  3. Focuses on the "sense-making" phase of the exploration process, supporting users in building exploration structures and developing intuition for input-output relationships through iterative generation.

• Implementation Steps and Key Technologies:

  1. Feature Design: Embed TTI() function in Google Sheets for image generation, along with GPT() functions for prompt manipulation or expansion.
  2. System Implementation: Utilize Google Apps Script and a backend proxy server to connect Stability.ai and OpenAI APIs for efficient multi-threaded processing of TTI and LLM calls.
  3. User Evaluation: Conduct preliminary lab studies and a two-week expert study to explore the exploration habits and workflow construction methods of novice and expert users.

Research Outcomes

• Specific Achievements:

  • DreamSheets enables users to build dynamic prompt templates, facilitating multidimensional large-scale exploration and comprehensive evaluation of generated results.
  • The proposed tool successfully supports users in targeted image generation by combining multidimensional exploration axes (e.g., seeds, semantic transformations, different prompt components).
  • Identified a series of exploration patterns, including iterative prompt optimization, parameter transformations (seeds and CFG sliders), and LLM-supported semantic exploration.

• Advantages Over Existing Solutions:

  • Compared to traditional "single input-output" TTI tools, DreamSheets supports multidimensional, layered, and repeatable exploration, allowing users to gain deeper insights into TTI model semantics.
  • The system combines flexibility and composability, enabling users to customize workflows and conduct extensive generative experiments.

• Experimental or Evaluation Results:

  • In the lab study, 12 participants successfully completed prompt engineering tasks, even though most lacked prior TTI experience.
  • In the expert study, five analysts independently extended and improved the DreamSheets system, with the most active user achieving over 11,000 generation calls, demonstrating the tool's high adaptability for free exploration.

• Limitations and Future Directions:

  • Limitations:
    • The study primarily focuses on specific TTI models and limited parameter dimensions; future work should consider supporting more complex parameter spaces.
    • The tool currently lacks direct support for advanced generation controls (e.g., ControlNet, image editing).
  • Future Directions:
    • Expand support for spatial conditional inputs, diversified modifications based on generated images, and image-to-text conversion functionalities.
    • Enhance interoperability between different AI models and strengthen workflow modularity.
    • Investigate deeper support for creators in flexibly switching between "exploration depth" and "exploration breadth."