Noise Pilot: Enabling Artistic Workflow Composition with Diffusion-Based Image Generation

Paper Title

Noise Pilot: Enabling Artistic Workflow Composition with Diffusion-Based Image Generation

Publication Info

• Topic area: Creativity support tools for diffusion-based image generation.
• Keywords: Diffusion models, image generation, creativity support tools, node-based programming, artistic workflows, materiality, glass-box abstractions, generative AI, iterative denoise, computational art.

Background and Problem

• Problem / challenge: Existing AI creativity tools treat diffusion models as black-box systems, limiting artists to indirect manipulation (e.g., prompting) or requiring advanced ML-engineering skills for customization. This restricts creative control and exploration.
• Significance: Enabling deeper interaction with diffusion processes can empower artists to create novel and bespoke outputs, fostering innovation in computational art.
• Motivation and related work: Prior tools like DALL-E and ComfyUI offer limited control over diffusion internals. Techniques for customizing diffusion exist in ML literature but are inaccessible to non-programmers. This work aims to bridge this gap by providing a multi-layered interface for artistic exploration.

Solution

• Proposed approach: Noise Pilot, a node-based programming environment for authoring diffusion-based image generation workflows, allowing artists to manipulate diffusion processes at varying levels of depth.
• Novelty:
  1. Introduces a multi-layered interface supporting prompting, diffusion pipelining, and diffusion customization.
  2. Implements the diffusion process (DDPM) as editable nodes, enabling granular control.
  3. Demonstrates the tool’s capability to replicate state-of-the-art techniques and develop novel ones.
  4. Conducts a 2-week user study with artists to evaluate the tool’s impact on creative practice.
• Procedure and key techniques:
  1. Represent the DDPM algorithm as nodes in a node-based interface.
  2. Allow users to compose workflows combining diffusion and image processing operations.
  3. Provide templates and tutorials for common diffusion techniques (e.g., visual anagrams, generative fill).
  4. Enable users to customize the iterative denoise process for bespoke outputs.

Results

• Concrete findings:
  • Participants created 167 workspaces and generated 7,295 images during the study.
  • 3,183 prompting changes, 3,794 pipeline edits, and 2,364 diffusion customizations were observed.
  • Artists produced outputs that were impossible to achieve with prompting alone, such as visual anagrams and hybrid images.
• Advantage over baselines:
  • Noise Pilot supports deeper interaction with diffusion processes compared to tools like ComfyUI, enabling novel artistic workflows.
  • Artists reported a sense of material engagement, akin to sculpting, rather than merely issuing prompts.
• Experiments / evaluation:
  • A 2-week deployment study with 9 computational artists.
  • Participants used the tool for ~10 hours, submitted reflections, and participated in interviews.
  • Analysis revealed diverse strategies across three levels of interaction depth (prompting, pipelines, customization).
• Limitations and future work:
  • Limited to 256x256 resolution, which constrained some styles (e.g., photorealism).
  • Does not support model training or dataset-level manipulation.
  • Future work could explore higher resolutions, alternative models, and disentangling the effects of model vs. interface design.

Summary

Noise Pilot is a node-based creativity support tool that enables artists to manipulate diffusion-based image generation processes at varying levels of depth. By implementing the DDPM algorithm as editable nodes, the tool allows for prompting, workflow composition, and direct customization of the diffusion process. A 2-week study with 9 artists demonstrated its ability to support novel artistic workflows and foster material engagement with diffusion as a medium. While limited by resolution and lack of training-level control, Noise Pilot highlights the potential for designing generative AI tools that empower artists through visibility, flexibility, and material-like manipulation.