ChainBuddy: An AI-assisted Agent System for Generating LLM Pipelines

Research Background and Problem

• Identified Problem: With the rapid development of large language models (LLMs), users often face challenges when setting up complex evaluation tasks or constructing LLM workflows, such as the "blank page problem" (not knowing where to start).
• Significance: Designing and developing LLM pipelines is crucial for studying their performance and behavior, as well as for creating efficient applications. However, most current tools require users to have a high level of technical expertise, limiting the widespread adoption of LLM technologies.
• Research Motivation and Related Work: Existing solutions include tools with specific functionalities (e.g., LangGraph, EvalLM, Flowise) and interactive models for automated analysis (e.g., AutoML) and user requirement collection. However, these tools either focus on single LLM operations (e.g., prompt engineering) or struggle to support complex automated tasks. This study aims to design a novel AI assistant tool, ChainBuddy, to help users efficiently initiate and maintain tasks for LLM pipeline creation and evaluation.

Solution

• Proposed Approach: ChainBuddy is an AI assistant integrated into the ChainForge platform that interacts with users through natural language to automatically generate LLM pipelines tailored to specific needs. It includes an interactive requirement collection system and a multi-agent architecture to support workflow generation.
• Innovations:
  • Generates complex workflows through a layered architecture, breaking tasks into input layers, intermediate processing layers, and evaluation layers for better management and scalability.
  • Incorporates a multi-agent system where each agent is responsible for specific tasks (e.g., data generation, prompt creation, and evaluation).
  • Enables users to describe their needs through a simple conversational interface, eliminating the need to write complex code directly.
• Implementation Steps:
  1. Users input their requirements via a chat interface.
  2. The system interacts with users through proactive questioning to clarify needs.
  3. The multi-agent system includes:
     • Planner Agent: Plans and decomposes workflows into layers based on user requirements.
     • Task Agents: Refines specific nodes (e.g., input generation, prompt processing, model evaluation).
     • Connector and Layout Agents: Handle node connections and layout arrangement.
  4. The system automatically generates and displays the workflow, allowing users to further edit and refine it.
• Key Technologies: Built on LangGraph's multi-agent architecture, combining lightweight reinforcement learning concepts and task decomposition strategies.

Research Outcomes

• Specific Achievements:
  • ChainBuddy significantly reduces the difficulty of designing LLM pipelines from scratch, providing users with editable initial workflows.
  • In experiments, workflows generated by ChainBuddy achieved an average quality score of 4.09 (out of 5), with most issues fixable through minor adjustments.
• Comparison with Existing Solutions:
  • Compared to similar LLM tools (e.g., Flowise), ChainBuddy places greater emphasis on user experience, actively guiding users to clarify their requirements.
  • Compared to baseline systems requiring fully manual workflow design, ChainBuddy significantly reduces user workload, especially in creating evaluation nodes and chain structures.
• Experimental and Evaluation Results:
  • User experiments showed that participants experienced lower mental and physical workload (NASA TLX scores) when using ChainBuddy and reported higher satisfaction and confidence in task completion.
  • Compared to the baseline system, workflows designed with ChainBuddy's assistance were significantly better. Independent expert evaluations indicated higher scores for workflows generated using ChainBuddy.
  • However, in subjective user ratings, users' perceived success did not fully align with objective scores (Dunning-Kruger effect).
• Limitations and Future Directions:
  • Limitations:
    • The current system does not support editing existing workflows; users must start from scratch.
    • Users may exhibit over-reliance on AI-generated results, and the system may have limitations for different task types.
    • Experiments involved a limited range of task types and small sample sizes, not fully covering all potential user needs and scenarios.
  • Future Directions:
    1. Enable users to upload external data and collaborate with the system.
    2. Improve the system's interface performance in scenarios like "data import" and "editing existing workflows."
    3. Provide diverse workflow templates while reducing over-reliance on default solutions.
    4. Explore better requirement collection interaction designs to minimize excessive Q&A sessions that consume user time.
    5. Develop algorithms to help users compare multiple generated solutions, enhance evaluation transparency, and encourage independent decision-making.

This research demonstrates the potential of AI-supported LLM pipeline creation through ChainBuddy, offering a new paradigm for achieving automated, user-friendly, and efficient LLM operations.