InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMs

Research Background and Problem

• Identified Problems or Challenges:
  This paper focuses on enabling users to quickly construct machine learning (ML)-based workflows in visual programming. Existing visual programming platforms often require users to build entire pipelines from scratch, which is particularly challenging for novice users as it involves manually selecting nodes, designing workflow structures, and connecting nodes. While template-based methods can reduce the initial learning curve, they lack flexibility and fail to meet users' specific needs.

• Importance:
  With the development of machine learning technologies, constructing and customizing ML pipelines has become a common demand. However, the complexity of technical workflows and interfaces hinders non-technical users. Simplifying these processes not only enhances usability for new users but also accelerates the realization of creative ideas.

• Research Motivation and Related Work:
  Tools like GitHub Copilot and other LLM (Large Language Model)-based systems have demonstrated that natural language prompts can significantly simplify code generation. While these tools have been successful in linear code editing, they have yet to be widely applied in the domain of visual programming. Additionally, recent works like LangChain showcase the potential of AI chains, providing direction for developing more user-friendly construction tools.

Solution

• Proposed Method or Solution:
  This paper introduces InstructPipe, a visual programming AI assistant that semi-automatically generates ML pipelines through natural language instructions. Starting from user-provided textual instructions and labels, InstructPipe employs a two-stage LLM module to generate pseudocode, which is then rendered into a pipeline diagram.

• Innovations:

  1. Utilizes a two-stage LLM module (Node Selector and Code Writer) to parse user instructions effectively, addressing the data limitations of a finite node library.
  2. Introduces pseudocode to significantly reduce the computational effort required for generating complex JSON files, optimizing the process of rendering into graphical interfaces.
  3. Incorporates interactive design, allowing users to adjust the generated pipeline, fostering a unique human-AI collaborative workflow.

• Implementation Steps and Key Technologies:

  1. Node Selector: Identifies nodes matching user requirements by accurately locating functional nodes through brief descriptions.
  2. Code Writer: Generates pseudocode based on the selected nodes, including connection logic and topology structure.
  3. Code Interpreter: Parses the pseudocode into a DAG (Directed Acyclic Graph) for the visual programming platform and optimizes the initial layout through automatic rendering.
  4. Users can fine-tune parameters or correct connection errors in the rendered interface.

Research Outcomes

• Specific Achievements:

  1. InstructPipe achieves highly accurate pipeline generation, significantly reducing user interaction burden. In technical evaluations, it reduced user interaction requirements to 18.9% of what would be needed to build from scratch (an 81.1% reduction).
  2. Experimental results indicate that even when the generated pipeline is not entirely correct, the initial structure provided by InstructPipe significantly lowers the learning curve for novice users and supports further creative exploration.

• Advantages Over Existing Solutions:

  1. Compared to traditional methods of building from scratch, InstructPipe offers significantly lower entry barriers and interaction costs.
  2. In addition to automatic pipeline generation, it provides users with an intuitive interface for rapid editing and adjustments, enhancing user control and flexibility.
  3. Unlike static templates, InstructPipe's dynamic generation adapts to more open-ended creative use cases.

• Experimental or Evaluation Results:

  1. Technical Evaluation (N=48 pipelines):
     • On average, only 18.9% of user interaction was required to complete pipeline construction. The accuracy was particularly high for multimodal pipelines (e.g., image and text processing), requiring only 20.8% user interaction.
  2. User Study (N=16):
     • Through questionnaires (NASA-TLX) and observational records, the study showed that InstructPipe significantly reduced users' physical and time burdens, with high overall satisfaction.
  3. Exploratory Creativity Testing:
     Participants used InstructPipe to quickly prototype potential business or daily applications, demonstrating the tool's usability and creative potential for novices and non-technical users.

• Limitations and Future Directions:

  1. Cognitive Load: Users may feel burdened when writing high-quality instructions, as it requires an understanding of visual programming and AI instruction writing.
  2. Dynamic Node Library Expansion: Currently supports only 27 nodes; future work could explore more flexible dynamic expansion.
  3. Node Parameter Optimization: The system does not fully automate parameter tuning, requiring manual adjustments by users. Future research could focus on LLM-based parameter generation.
  4. Security and Compatibility: The system lacks content safety filtering mechanisms, and its openness to extensions may pose misuse risks.

In summary, InstructPipe introduces a natural language-driven generative approach to visual programming, offering a beginner-friendly experience and open-ended creative potential.