Protosampling: Enabling Free-Form Convergence of Sampling and Prototyping through Canvas-Driven Visual AI Generation

Paper Title

Protosampling: Enabling Free-Form Convergence of Sampling and Prototyping through Canvas-Driven Visual AI Generation

Publication Info

• Topic area: Generative AI tools for creative processes
• Keywords: Protosampling, generative AI, creativity, visual design, prototyping, sampling, canvas interfaces, provenance, creative workflows

Background and Problem

• Problem / challenge: Current generative AI tools either focus on simplified prompt-based interfaces or complex node-based systems, leaving a gap for creative practitioners who need a balance of control and expressiveness while working in a medium that supports their intent.
• Significance: Bridging sampling and prototyping into a unified process is critical for enhancing creative workflows, especially with the accelerated capabilities of generative AI to generate and remix media.
• Motivation and related work: Prior systems like MidJourney and ComfyUI cater to specific ends of the spectrum but fail to integrate sampling and prototyping seamlessly. Mood boards and generative AI interfaces provide inspiration but lack tools for iterative refinement and integration. This paper builds on these systems to propose a new approach that operationalizes the convergence of sampling and prototyping.

Solution

• Proposed approach: Atelier, a canvas-like system designed to operationalize Protosampling by blending sampling and prototyping into a unified creative process.
• Novelty:
  1. Characterization of Protosampling as a lens for treating sampling and prototyping as a joint activity.
  2. Design and implementation of Atelier, a media-first canvas system supporting visual creation with generative AI.
  3. Development of abstractions and workflows for generating AI images and videos with enhanced creative controllability.
• Procedure and key techniques:
  • Media-first infinite canvas for arranging and manipulating collected and generated assets.
  • Modular "easels" encapsulating generative workflows for tasks like drawing, painting, tracing, modifying, and animating.
  • Provenance tools for tracking asset lineage, history, and connections.
  • Organization features like collections, search, and tagging for sensemaking and reflection.

Results

• Concrete findings:
  • Participants used Atelier to create diverse projects, including film scenes, product shots, and animations, demonstrating its flexibility.
  • Post-study survey showed increased sentiment for malleability, agency, expressiveness, and ability to fix outputs compared to current AI tools.
• Advantage over baselines:
  • Compared to prompt-based systems, Atelier provided more creative control and agency by enabling direct manipulation of assets.
  • Compared to node-based systems, Atelier offered a more accessible and intuitive interface while retaining advanced capabilities.
• Experiments / evaluation:
  • Conducted a 4-hour first-use study with 5 creative professionals, exploring how they appropriated Atelier for their workflows.
  • Participants created canvases for individual projects, organized their workspace spatially, and used provenance tools to reflect on their process.
• Limitations and future work:
  • Technical limitations include reliance on local models and hardware, leading to potential memory constraints and slower model swapping.
  • Conceptual limitations include risks of design fixation and saturation from excessive generations.
  • Future work includes exploring video generation, expanding model flexibility, and studying long-term use of Atelier for creative processes.

Summary

This paper introduces Protosampling, a concept that integrates sampling and prototyping into a unified creative activity, and operationalizes it through Atelier, a canvas-driven system for visual AI generation. Atelier supports freeform exploration, modular workflows, and provenance tracking, enabling creative professionals to blend thinking and making seamlessly. A first-use study demonstrated its ability to enhance creative control, agency, and expressiveness compared to existing AI tools. While limitations in technical robustness and conceptual challenges remain, Atelier represents a significant step toward empowering creators in the generative AI era.