Spellburst: a node-based interface for exploratory creative coding with natural language prompts

Document Title

Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts

Document Information

• Subject Area: Human-Computer Interaction, Computational Generative Art, Creative Coding Tools
• Keywords: Large Language Models, Exploratory Programming, Creative Coding, Generative Art, Prompt Engineering, Semantic and Syntactic Space

Research Background and Problem

• Problems and Challenges:

  1. Creative coding tasks are often exploratory, and artists face difficulties in translating semantic intentions into programming syntax.
  2. Current programming tools lack support for rapid iteration and generative art exploration.
  3. Existing version control systems are complex to operate and fail to meet the needs of artists for quickly generating multiple versions during iterative experimentation.
  4. Comparing different code versions or output results is challenging, and tools are often disconnected from the programming environment.
  5. While generative AI using natural language prompts is powerful, fine-grained adjustments can be cumbersome.

• Importance of Research: Creative coding is crucial for artistic creation, but tools need to efficiently support rapid semantic and syntactic exploration, fostering creative iteration and inspiration generation for artists. Researching how to design modern creative tools that provide greater flexibility and support for artists holds significant importance.

• Motivation and Related Work: Based on the characteristics of creative coding (such as bridging semantics and syntax) and the limitations of related tools (e.g., version control), the authors identified deficiencies in current tools regarding support for generative art exploration, history version recording, and rapid iteration through literature review and artist interviews.

Solution

• Methods and Solutions:

  1. Proposed a tool named Spellburst, an exploratory creative coding tool that combines a node-based interface with natural language prompt-based code generation.
  2. The tool offers the following core features:
     • Node-based Interface: Presents creative paths in a tree structure, with each node representing generated sketches or code.
     • Natural Language Interaction: Enables rapid semantic-level programming jumps using large language models.
     • Dynamic Semantic-driven Interface: Provides parameter adjustment sliders for fine-grained modifications.
     • Version Management Functionality: Seamlessly records and visualizes exploration history with a clear and intuitive framework.

• Innovations:

  • Integrates generative large language models into creative art programming, combining semantic operations with programming syntax operations.
  • Implements AI-based automatic "semantic merging" of code in creative tools for the first time, allowing the synthesis of multiple creative paths.
  • Supports flexible conversion between semantics and syntax through prompt engineering-based natural language generation.
  • Introduces a dynamic, non-linear visual layout, breaking away from traditional "linear-parallel" branching layout design methods.

• Implementation Steps and Key Technologies:

  • System architecture combines the React online editing library, d3.js hierarchical tree visualization, and OpenAI API calls.
  • Designed prompt templates with additional constraints to guide ChatGPT in code generation and auto-completion.
  • Developed semantic parameter mapping and "semantic-syntax" interaction optimization techniques for real-time adjustments.
  • Testing plan: comparison with existing tools, expert testing, and user surveys.

Research Outcomes

• Specific Outcomes:

  1. Developed a complete prototype of Spellburst and conducted user testing to preliminarily evaluate its effectiveness and user experience.
  2. Users were able to easily perform significant creative leaps (e.g., directly generating new forms of artistic sketches through natural language) and minor syntactic detail optimizations using Spellburst.
  3. Participants generally found Spellburst highly suitable for rapid generation of artistic prototypes, aiding in understanding the possibilities of different exploratory directions.

• Comparison with Existing Solutions:

  1. Compared to terminal-based ChatGPT or version control tools like Git, Spellburst allows artists to organize and track exploration results in a highly visual, non-linear manner.
  2. Spellburst simplifies the synthesis of AI-generated outputs with artistic experimentation while providing greater control at the syntax level.

• Experimental or Evaluation Results:

  • Engaged over 10 experienced artists for testing, and results showed participants gained significant exploratory inspiration and improved work efficiency using Spellburst.
  • Users appreciated Spellburst's visualized "history tree" after natural language prompts, dynamic slider settings, and diverse generation results.
  • In cognitive load evaluation (NASA-TLX), users reported low physical burden for the interface but higher mental demand (requiring more familiarity).

• Limitations and Future Directions:

  1. Occasionally, the code generated by Spellburst encounters runtime issues, requiring manual debugging by users.
  2. Some users expressed a desire for filtering non-essential nodes in the visual interface to reduce "loading overload."
  3. Extensive global samples and complex scenario testing are needed in future efforts to enhance reliability and diversity.
  4. For cross-team collaboration, long-term debugging, and multi-artist co-creation scenarios, future research should delve deeper into the design of visual extensions.