Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Model

Research Background and Problem

• Problem and Challenges: Current user interface (UI) design is mostly based on fixed models, making it difficult to flexibly respond to users' evolving information needs. Although generative UI can produce code through user prompts, the code generation approach makes it challenging for end-users to iteratively customize interfaces to adapt to dynamic requirements.
• Significance: Interfaces that support dynamic tasks and personalized needs represent the future direction of human-computer interaction. Traditional applications fail to meet these demands, and the limitations of existing generative technologies further restrict users' customization experiences.
• Research Motivation and Related Work: Drawing on studies in Model-Based UI Development, Specification-Based UI Generation, and End-User Programming, the authors propose a method for dynamic generation and evolution of interface design.

Solution

• Method Overview: A technical pipeline based on a Task-Driven Data Model is proposed, using task-driven data models generated by large language models (LLMs) as the foundation for UI generation and evolution. The pipeline allows dynamic adjustments to models and interfaces through natural language prompts and direct manipulation.
• Innovations:
  1. Employing task-driven data models to represent the core entities, relationships, and attributes of user tasks.
  2. Mapping data models to UI specifications to establish standardized component generation rules, ensuring consistency.
  3. Enabling users to flexibly adjust models and interfaces via natural language and direct manipulation.
• Implementation Steps:
  1. Data Model Generation: Using LLMs to analyze user prompts and generate object-associated data models (e.g., entities, attribute relationships, and structured data).
  2. UI Specification Generation: Generating UI specifications based on the data model, including component mapping and state management rules.
  3. UI Rendering: Rendering the interface according to UI specifications using predefined rules.
  4. User-Customized Interaction: Allowing users to update data models and UI through continuous prompts or direct manipulation.

Research Outcomes

• Specific Results:
  1. Developed a prototype system, "Jelly," capable of dynamically generating user interfaces and supporting continuous customization.
  2. Technical evaluations demonstrate that LLM-generated data models perform well in task relevance and accuracy.
  3. User studies show that generative and plastic interfaces help users efficiently handle open-ended information tasks, providing a seamless and personalized interactive experience.
• Advantages and Comparisons: Compared to traditional GUIs and LLM-driven chat interfaces, Jelly offers a structured information space with continuous personalized customization, allowing users to dynamically expand interfaces based on specific tasks.
• Experimental and Evaluation Results:
  1. Technical evaluation results indicate that LLMs can generate high-quality entities and attributes (over 94% of attributes were deemed necessary and meaningful).
  2. User studies reveal that Jelly helps users complete tasks and adapt dynamically to changing needs. Users expressed satisfaction with the system's flexibility and transparency in digesting information and customizing interfaces.
  3. Failure cases mainly involved handling highly customized UI requests or complex attribute operations, pointing to limitations in the current specification scope.
• Limitations and Future Directions:
  1. Current interface design specifications limit the diversity of information representation (e.g., lack of advanced visualization options).
  2. Interaction efficiency could be improved through more efficient model extension mechanisms and direct manipulation.
  3. Future work includes introducing external data source support, modeling more complex task dependencies, and enabling context-based interface personalization.

Through this study, the authors provide a clear framework for the development of generative and plastic interfaces and demonstrate effective practices for implementing dynamic task-driven data models and UI generation.