Evaluating Non-AI Experts' Interaction with AI: A Case Study In Library Context

Research Background and Problem

• Problem or Challenge: Public libraries face challenges such as labor shortages, tight budgets, and overburdened staff, necessitating 24/7 response solutions like Conversational Agents (CAs) to address these issues. However, the development of CAs currently requires professional AI developers and programming skills, which most library staff lack. Additionally, the unique characteristics of library operations (e.g., low-risk, community-oriented services) demand customized AI solutions. Existing research has not sufficiently explored the design requirements and evaluation criteria for CA development by non-AI experts.

• Significance: Libraries are crucial hubs for community building and knowledge sharing, offering services such as digital access, information support, and educational activities. Enabling library professionals to create their own CAs can help alleviate staff workload, improve service efficiency, and enhance the social responsibility and value of libraries.

• Research Motivation and Related Work: This study aims to fill the research gap regarding the design requirements and evaluation standards for non-AI experts developing customized CAs. Prior research has explored CA design in high-risk environments (e.g., healthcare, emergency services), but solutions for CA development in low-risk environments (e.g., libraries) remain scarce. The democratization of generative AI presents new opportunities for non-AI experts to create CAs using accessible tools, while also accommodating community-specific and personalized needs.

Solution

• Proposed Solution: The authors designed a prototype tool, AgentBuilder, a no-code development platform that supports non-AI experts in creating conversational agents. Its goal is to meet the needs of library professionals in defining, customizing, and evaluating CAs.

• Innovations: AgentBuilder includes the following functional modules:

  1. Boundary Setting Module: Helps users define the scope and role of the CA by uploading knowledge base files and personalizing settings to construct a foundational service framework.
  2. Knowledge Synchronization Module: Provides knowledge transparency and editing capabilities, enabling library professionals to review and modify the quality of CA responses, fostering "human-AI collaborative learning."
  3. Role Simulation and Evaluation Module: Uses virtual roles for interaction simulations to test the CA's adaptability to diverse user needs.

• Implementation Steps and Technologies:

  1. Users can upload PDF documents to build a knowledge base.
  2. Natural language prompts are used to define the CA's role and behavior, allowing non-AI users to customize the agent through simple conversational settings.
  3. The RAG (Retrieval-Augmented Generation) framework combines user-uploaded content with internal AI knowledge to enhance response accuracy.
  4. Through simulation features, users can observe interactions between the CA and virtual users, iteratively adjusting and optimizing the CA's behavior.

Research Findings

• Specific Findings:

  1. Through two user studies (involving 23 participants), three key design goals for non-AI experts creating CAs in library environments were identified:
     • Goal 1: Retain control over the CA's service scope.
     • Goal 2: Ensure AI responses align with library professionals' expectations through collaborative processes.
     • Goal 3: Address diverse user groups and cover various scenario needs.
  2. Key evaluation criteria for library professionals assessing CAs were identified, including accurate understanding of user intent, precise citation of knowledge sources, alignment of emotional expression, faithful paraphrasing of original information, and effective handling of "unknown answers."

• Advantages and Comparisons: Compared to existing commercial CA tools, AgentBuilder is more suitable for non-technical users, allowing them full control over the customization process. Additionally, the tool enhances the flexibility and transparency of AI to meet community needs.

• Experimental Results and Limitations: Users found AgentBuilder's features relatively intuitive, but several limitations were noted:

  • Dependence on the completeness of user-uploaded documents; incomplete elements (e.g., images or partial descriptions) may lead to inaccurate responses.
  • Designing recursive communication capabilities for AI (e.g., follow-up clarification questions) remains a technical challenge.
  • The reliance on precise user input for prompts makes it difficult for less proficient users to utilize effectively.

• Future Directions:

  1. Expand the research scope to other public service domains (e.g., healthcare, education).
  2. Enhance system automation to reduce the time investment required during user iterations.
  3. Improve the agent's adaptability to informal language in role simulations, approximating real-world conversations.
  4. Develop comprehensive guidance tools specifically for non-AI experts, such as pre-designed templates and prompt frameworks.

Through this study, the authors not only proposed an effective CA creation tool for library services but also explored best practices for using generative AI in low-risk environments. This work provides significant guidance for the democratization of AI and its application across diverse domains.