Beyond Code Generation: LLM-supported Exploration of the Program Design Space

Research Background and Problem

• What problems or challenges did the authors identify?
  The authors observed that current programming assistants based on large language models (LLMs) primarily focus on code generation and are typically limited to solving specific problems. This approach restricts the exploration of the programming design space, particularly in scenarios requiring user needs analysis, problem reframing, and starting from high-level design goals. Additionally, the code generation process may involve numerous implicit decisions without adequately revealing how these decisions impact the final design goals. Furthermore, information overload is a common issue for users of existing tools, especially when systems generate a large number of potential design solutions, making it difficult for users to focus.

• Why is this problem important?
  Programming is not just about code implementation; it also involves complex problem definition, high-level goal setting, and user needs evaluation. Tools that focus solely on specific code generation may limit innovation and fail to support comprehensive exploration during the design process. Addressing this issue is critical for enhancing the application value of generative AI in design activities and mitigating user distraction caused by information overload.

• Research Motivation and Related Work
  Drawing on design theories (e.g., the "Double Diamond Design Model") and the limitations of existing AI programming assistance tools, the authors aim to explore a new approach that supports the expansion of the programming design space and optimizes the design process. Additionally, related work in the field indicates that organizing information and exploring problems in the design domain can better support AI-based interactive design solutions.

Solution

• What methods or solutions did the authors propose?
  The authors introduced a prototype integrated development environment (IDE) called "Pail," designed to support users in exploring a broader design space through LLM-based programming assistance. This tool not only facilitates code generation but also generates alternative problem definitions and design solutions, helping users think about problems and design goals at a higher level of abstraction.

• What are the innovative aspects of this solution?

  1. High-level abstraction support: Pail encourages users to go beyond specific problems, explore the problem space, and think about solution design through structured design problems and goals.
  2. Design space tracking and comparison: Pail provides tracking and comparative views of design problems, identified requirements, implicit decisions, and abstract concepts.
  3. Interactive design and operability: Pail proposes alternative design solutions interactively, allowing users to quickly experiment and roll back design options.
  4. Mechanisms to prevent information overload: By providing easy-to-scan summaries and lists of alternative options, Pail helps users explore the design space more efficiently.

• What are the implementation steps and key technologies used?

  1. Design goals: Four design goals were summarized based on user needs, including support for generating alternative design solutions, task tracking, visualizing abstract decisions, and more.
  2. System functionality design: Pail incorporates three main agents (ConversationAgent, DesignAgent, and ReflectionsAgent) responsible for user interaction, design optimization, and reflective generation, respectively.
  3. Interface layout: The tool features four main areas: a code panel, an output panel, a conversation panel, and a design panel containing design problems, requirement confirmations, implicit decisions, and abstractions.
  4. Technical implementation: Pail is implemented using the React framework and leverages GPT models for dialogue and code generation, with a differential update mechanism introduced to reduce irrelevant code generation.

Research Outcomes

• What specific outcomes were achieved?
  User studies revealed that Pail enables users to quickly explore the programming design space, considering potential user needs and high-level design goals. The tool helps users seamlessly switch between the divergent and convergent phases of the Double Diamond Design Model. Users expressed satisfaction with the design panel's summarization and tracking features, noting that it alleviated the information burden in complex design tasks. In practical use, all study participants identified at least one alternative design solution they had not previously considered.

• What advantages does it have compared to existing solutions?
  Compared to traditional code generation tools, Pail offers the following key advantages:

  • Supports higher-level abstract thinking, not limited to code-level solutions.
  • Provides design support, including generating problem definitions and alternative design solutions, encouraging users to consider different design possibilities.
  • Mitigates information overload issues present in current LLM tools through more efficient information presentation mechanisms.

• What were the experimental or evaluation results?
  User experiments demonstrated that Pail performs well in encouraging users to iterate quickly and explore design solutions more broadly. Participants praised the tool's summarization features and visualization of implicit decisions but also noted issues with attention distraction due to the volume of information. Additionally, the experiments revealed that the design process becomes compressed due to the information generation capabilities of LLMs, raising future research questions on better managing user attention and data presentation.

• Limitations and Future Directions
  Limitations:

  • The current tool is not suitable for complex programming design tasks due to its reliance on LLM capabilities, which may be constrained by the model's background knowledge.
  • The tool requires a relatively high level of design literacy from participants, limiting its adaptability for users outside the design domain.

  Future Directions:

  • Explore how to design tools to better manage users' attention burden, balancing information generation with effective information presentation.
  • Expand the tool's applicability to cover more programming design tasks and further develop mechanisms for information convergence and selection.
  • Investigate design methods that support cross-abstraction-level work, helping users transition and operate more fluidly within multi-level problem spaces.