Cells, Generators, and Lenses: Design Framework for Object-Oriented Interaction with Large Language Models

Title of the Paper

Cells, Generators, and Lenses: Design Framework for Object-Oriented Interaction with Large Language Models

Paper Information

• Domain: Human-Computer Interaction and the Design and Application of Large Language Models (LLMs)
• Keywords: Large Language Models, Generative Models, Writing Assistance Tools, Reification, Human-Computer Interaction Design

Research Background and Problem

• Problems or Challenges:

  • LLMs have significantly reduced users' writing burden, but existing writing interfaces limit users' ability to experiment with and iterate on generation configurations (e.g., inputs and model parameters).
  • Users' generation configurations are often overwritten, making it difficult to track, compare, and combine multiple configurations, thereby hindering flexible exploration and experimentation.
  • The black-box nature and non-deterministic characteristics of the models leave users without effective means to "correct" model outputs, requiring additional effort for experimentation.

• Research Importance:

  • Iteration and experimentation are critical for exploring the design space in creative tasks. Improving writing interfaces to support parallel prototyping with multiple configurations can significantly enhance users' productivity and creativity.

• Research Motivation and Related Work:

  • Existing studies (e.g., tools like GANSliders and PromptMaker) have improved the configurability of certain components of generative models but lack support for cross-component customization.
  • Research on LLMs has focused on optimizing input configurations (e.g., prompt engineering) and generated outputs but has not addressed debugging and comprehensive configuration of the overall generation process.
  • Human-AI writing tools have demonstrated AI's complementary role in writing but have not fully considered supporting users' needs for iteration and experimentation.

Solution

• Proposed Method or Solution:

  • The authors propose the "Cells, Generators, and Lenses" framework, which materializes the main components of generation configurations (input, model, output) as interactive objects.
  • The interactive objects in the design framework are persistent, reusable, and composable. Through these objects, users can create multiple configuration variants and conduct parallel prototyping.

• Innovations:

  • The introduction of an object-oriented interaction design framework enables users to flexibly combine input, model, and output components, supporting diverse and experimental generation configurations.
  • The design incorporates the concept of "object reification," transforming task elements (e.g., text snippets, model parameters) into intuitive and user-friendly interactive objects.

• Implementation Steps and Key Technologies:

  1. Cells:
     • Materialized representations of input units, such as sentences or phrases.
     • Users can create, modify, duplicate, and combine multiple cells to construct generation inputs.
  2. Generators:
     • Materialized representations of model settings, including model types and parameters.
     • Users can independently modify each generator's configuration, link it to cells to generate outputs, and track its configuration history.
  3. Lenses:
     • Materialized representations of generated outputs, used for organizing and visualizing outputs (e.g., lists, scatterplots).
     • Users can flexibly assemble lenses to view multiple representations of the same output simultaneously.

Research Outcomes

• Specific Outcomes:

  • The framework was applied to redesign three interfaces: for ad writing, email composition, and story creation. Each interface demonstrated the framework's applicability to different tasks.
  • Experimental results showed that interfaces designed based on the framework significantly encouraged users to generate more outputs, experiment with more inputs, and adopt model suggestions more frequently in final outputs.
  • A design workshop revealed that the framework not only provided designers with concrete interaction design components but also inspired them to further modularize the generation process.

• Advantages over Existing Solutions:

  • Provides comprehensive control and personalized support for users' generation processes, significantly improving their iterative and experimental experiences.
  • Addresses the limitations of existing generative tools that only support single-component customization by integrating comprehensive interaction support for inputs, model settings, and outputs.

• Experimental and Evaluation Results:

  • A user study compared the framework-based interface with traditional configuration interfaces, finding that users generated more outputs in the former (M=9.78 vs. M=6.33, p=0.006) and used a higher number of unique inputs (M=5.89 vs. M=3.06, p<0.000).
  • Participants using the framework had a significantly higher adoption rate of generated outputs (BLEU score analysis: M=0.884 vs. M=0.768, p=0.045).

• Limitations and Future Directions:

  • Limitations include the difficulty of predicting the impact of model parameters and the lack of intelligent suggestions for modifying generation configurations.
  • The authors suggest future research focus on improving the interpretability of generator parameters, developing new lenses that balance output breadth and review efficiency, and optimizing the automatic generation and recommendation of cells.

Open Resources

• The authors have released an open-source ReactJS library for quickly integrating "Cells, Generators, Lenses" components, facilitating interface design for developers (URL: https://github.com/kixlab/llm-ui-objects).