Canvil: Designerly Adaptation for LLM-Powered User Experiences

Research Background and Problem

• What problems or challenges did the authors identify?
  In the era of rapid advancements in artificial intelligence, large language models (LLMs) have become a crucial technology driving user experience. However, the authors identified several challenges faced by designers in their work, such as difficulties in effectively participating in the adaptation of models to adjust their behavior to meet design requirements. Additionally, there is a lack of tools and methods to translate designers' needs into specific model adaptation outcomes. Designers urgently need a workflow that enables bidirectional translation between user needs and model behavior to address specific user scenarios.

• Why is this problem important?
  Improper integration of LLMs can lead to biases or safety issues in user experiences, potentially harming user well-being. Moreover, designers play a critical role in product development, and their human-centered perspective can help mitigate these risks. Empowering designers to participate in model adaptation can assist technical teams in better meeting user needs, creating more inclusive and safer user experiences.

• Research Motivation and Related Work
  The research motivation stems from the challenges designers face when interacting with LLMs. Previous work has focused on providing tools to help designers understand AI behavior, such as visualization tools or interaction methods for model exploration. However, these tools often lack integration with the design process. Furthermore, existing methods emphasize technical parameter adjustments, which are not easily accessible to designers without technical backgrounds. The authors aim to propose a new method that more comprehensively supports designers in participating in model adaptation.

Solution

• What methods or solutions did the authors propose?
  The authors proposed a new method called "Designerly Adaptation." This method is a design code aimed at achieving bidirectional translation between design requirements and LLM behavior. To validate this method, the authors developed a technical tool named Canvil, a Figma-based plugin that helps designers adapt model behavior more conveniently while creating UX designs.

• What is innovative about this solution?
  Designerly Adaptation breaks down design requirements into multiple dimensions (e.g., target user personas, core instructions, and safety measures) through structured forms and helps designers define and adjust model behavior in a user-friendly way. The Canvil tool is integrated into the design environment, supporting real-time iteration and incorporating collaboration and version control features, overcoming the limitations of traditional AI tools that cater primarily to technical developers.

• What are the implementation steps and key technologies used?

  1. Understand user needs: Conduct user research to extract scenario requirements, such as user goals, cultural preferences, and usage constraints.
  2. Translate needs into model behavior: Use Canvil's structured forms to input model instructions, such as setting tone, task content, and safety measures.
  3. Co-evolve design and model: Calibrate model behavior through rapid iteration while optimizing UX design.
  4. Share adaptation outcomes within the team: Leverage Canvil's collaboration features to facilitate more effective communication and adjustments among designers, engineers, and product managers.

Research Outcomes

• What specific outcomes were achieved?
  Experiments with Canvil demonstrated that designers could efficiently translate user needs into LLM behavior while optimizing UX design based on model outputs. Additionally, at the team collaboration level, designers found that Canvil could serve as a knowledge-sharing and communication tool, improving the overall efficiency of model adaptation.

• What advantages does it have compared to existing solutions?
  Canvil significantly lowers the barrier for designers to use LLMs while integrating model behavior analysis with UX design capabilities. Designers can not only iterate on model behavior but also perform UI design on the same canvas, enhancing team communication through collaboration features.

• What were the experimental or evaluation results?
  Design experiments showed that designers using Canvil could more quickly understand and adapt LLM behavior, achieving an average System Usability Scale (SUS) score of 69.94, indicating "above average" usability. Furthermore, designers in the experiments were able to create customized LLM solutions and design interfaces that better aligned with user needs. Team members also actively explored using Canvil for knowledge sharing and version management.

• Limitations and Future Directions

  1. The experimental scenarios simulated real-world conditions but could not yet cover more granular needs in complex industries, such as in-depth user research or cross-national deployments.
  2. The current Canvil templates rely on natural language for LLM configuration and do not incorporate higher-precision techniques, such as sparse autoencoders or feature clamping methods.
  3. Future research is recommended to adopt longitudinal studies to observe designers' adaptation behaviors during real product development cycles. Additionally, exploring Canvil's application in early design stages (e.g., brainstorming) could be valuable.

The new method and tool proposed in this study—Designerly Adaptation and Canvil—offer significant new perspectives in the fields of technology and design, laying the foundation for building human-centered LLM solutions.