PlanTogether: Facilitating AI Application Planning Using Information Graphs and Large Language Models

Research Background and Problem

• What problems or challenges did the authors identify?
  Clients, during the planning phase of collaborating with AI experts to develop AI applications, often need to draft an initial plan (referred to as a "pre-collaboration plan"). However, due to a lack of knowledge about AI terminology and capabilities, as well as insufficient understanding of AI experts' information needs, clients struggle to translate their requirements and expectations into concrete, actionable plans. This leads to multiple iterative steps, additional time consumption, and frustration on both sides.

• Why is this problem important?
  High-quality pre-collaboration plans are the foundation of successful collaboration. They not only provide depth for subsequent discussions but also reduce the need for repeated adjustments to requirements and technical feasibility during the development process, which has the potential to significantly improve efficiency and reduce misunderstandings.

• Research Motivation and Related Work
  Current tools (e.g., AINeedsPlanner and People+AI Workbook) provide static guidelines but fail to flexibly cover a wide range of AI application scenarios or dynamically offer the guidance clients need. Moreover, these tools do not effectively leverage information dependency relationships to organize and enhance the quality of clients' plans.

Solution

• What methods or solutions did the authors propose?
  The authors developed a system called PlanTogether, which leverages information graphs and large language models (LLMs) to support clients in drafting pre-collaboration plans. PlanTogether not only dynamically generates guidance but also helps clients iteratively refine their plans through a series of graphical interfaces and personalized suggestions.

• What are the innovative aspects of this solution?

  1. Planning Information Graph: Nodes in the graph represent planning information, and edges represent dependencies between pieces of information. Using this graph, the system prioritizes questions relevant to the current issue.
  2. Dynamic Suggestion Generation: By combining LLMs with the graph structure, the system generates personalized prompts and suggestions in real-time, providing contextual guidance for each question.
  3. Comprehensive Plan Overview: The system generates a progress overview and a structured plan view, helping clients grasp the overall plan and its completion status in real-time.
  4. Navigation Functionality: Users can flexibly navigate between related questions and expand cases based on the graph structure.

• What are the implementation steps and key technologies used?

  1. Graph Structure Construction:
     • Decompose planning information into nodes and define their interdependencies.
     • Predefine question types (core questions or supplementary questions).
  2. Handling Question Dependencies:
     • Use graph traversal algorithms (based on depth-first search) to dynamically adjust the order of questions, prioritizing dependent questions.
  3. Dynamic Prompt Generation:
     • Provide context-based prompts and diverse suggestions based on previous answers, using LLMs to generate this content.
  4. Experimental Validation and User Research:
     • Conduct user and expert evaluations to compare PlanTogether with static baseline tools.

Research Outcomes

• What specific outcomes were achieved?
  User studies revealed that PlanTogether significantly improved the quality of pre-collaboration plans, including:

  1. Plan Feasibility: Plans became more specific and actionable, reducing the number of iterations needed during expert discussions.
  2. Reflection of Client Domain Knowledge: Documents better captured clients' domain expertise, enhancing communication effectiveness.
  3. Improved Client Experience: Study participants found the prompts, suggestions, navigation features, and overview highly helpful for the overall process.

• What advantages does it have compared to existing solutions?

  1. The dynamically generated personalized support is contextually adaptive, responding flexibly to the information provided by clients.
  2. The graph-based navigation and smooth information flow design address the limitations of existing tools in effectively linking complex tasks.
  3. The comprehensive plan overview feature helps clients track the overall progress of the creation process.

• What were the experimental or evaluation results?
  Both user studies and expert evaluations showed that the quality of plans generated by PlanTogether was significantly higher than those produced by baseline tools (median scores of 4 versus 3, out of 5). Experts highlighted the system's superior performance in terms of plan executability and reflection of domain knowledge. On the client side, users frequently utilized personalized suggestions and navigation features to enhance their independent thinking efficiency.

• Limitations and Future Directions

  1. Addressing the issue of users overly relying on model-generated content during initial use, with suggestions to improve early-stage prompt generation strategies.
  2. Exploring further support for highly customized client domain needs, such as enabling users to fine-tune models with domain-specific data.
  3. Conducting additional research on real-world deployment, including collaborations between clients and various types of AI experts, as well as adaptation to legal environments.
  4. Introducing quality feedback and dynamic optimization mechanisms to help automatically detect and resolve potential issues in documents.

Through this research, PlanTogether demonstrates the potential of leveraging technology to enhance the efficiency of the planning phase in AI application development and opens new directions for future research.