A Framework for Efficient Development and Debugging of Role-Playing Agents with Large Language Models

DOI

We propose a framework that leverages large language models (LLMs) to semi-automate the development and debugging of role-playing agents, reducing the need for extensive manual effort. Role-playing agents powered by LLMs offer scalable solutions that enhance communication and interaction in various applications, such as employee training, healthcare, and software development. However, creating prompts manually is a time-consuming process, and sequential debugging increases the difficulty of anticipating conversation flow, resulting in increased cognitive load. Our framework addresses these challenges by generating and summarizing dialogue examples, providing a clearer overview of conversation flow and reduce mental workload. It also enhances role-playing quality by mitigating LLMs’ tendency to produce generic or vague responses. In a user study, the proposed method significantly improved perceived workload and five of the six NASA-TLX dimensions. Moreover, it can generate agents comparable to those created with expertly crafted prompts. This framework is model-agnostic, enabling integration of advancements in LLM capabilities and prompting techniques, and is applicable to diverse domains.