Towards Designing a Question-Answering Chatbot for Online News: Understanding Questions and Perspectives

Title of the Paper

Towards Designing a Question-Answering Chatbot for Online News: Understanding Questions and Perspectives

Bibliographic Information

• Research Domain: Human-Computer Interaction (HCI), Question-Answering Systems, Large Language Models (LLMs)
• Keywords: Online news, chatbot, question-answering systems, large language models, user behavior analysis, reader interaction, news credibility, news recommendation, human-computer interaction, news ethics

Research Background and Issues

• What issues or challenges did the authors identify?

  1. Readers often have questions after reading news and use comment sections, emails, and social media to ask journalists. However, due to the high volume of questions, limited time, and lack of direct incentives, journalists struggle to respond comprehensively.
  2. Although news organizations have expressed interest in using chatbots and AI technologies, they lack systematic design approaches for building a question-answering chatbot that meets both journalist and reader needs.
  3. Existing chatbot and news interaction technologies primarily focus on news dissemination and recommendation but are less adept at handling open-ended questions, especially subjective ones.

• Why is this issue important?

  • Engaging in conversational interactions with readers helps build trust and enhances the appeal of news content.
  • The rise of LLMs opens possibilities for developing intelligent chatbots, but their limitations (e.g., hallucination generation, inaccurate answers) raise ethical concerns that require further exploration.
  • The functional needs of readers and journalists for chatbots may differ; ignoring these differences could result in mismatched designs.

• Research Motivation and Related Work

  • Motivation: To build a user-friendly question-answering (QA) chatbot that fulfills interaction needs in the news industry while reducing journalists' burden of answering repetitive questions.
  • Related Work: Explored user expectations for chatbots, the evolution of reader interaction in the news industry, and advancements in NLP question-answering technologies.

Solutions

• What methods or solutions did the authors propose?

  1. Conducted semi-structured interviews with six journalists to understand their current reader interaction practices, challenges, and perspectives on chatbots.
  2. Conducted an online experiment on Amazon Mechanical Turk (N=124) to analyze readers' questioning patterns regarding news and compare their interactions with journalists and chatbots.
  3. Proposed a framework for designing QA chatbots tailored to the news industry based on findings from interviews and experiments.

• What are the innovative aspects of this solution?

  1. Conducted a two-way examination of QA chatbot functionalities from both journalist and reader perspectives.
  2. Proposed a design framework that integrates the requirements of news organizations, user needs, current technological capabilities, and legal regulations—an unprecedented approach in the field of news AI.
  3. Systematically analyzed question types (e.g., factual, subjective) and the complexity of questioner behavior.

• Implementation Steps

  1. Interview Study: Recorded and thematically analyzed the current state of journalist-reader interactions, revealing that journalists primarily need chatbots to answer "factual" and "repetitive" questions to save time.
  2. Experimental Study: Participants read news articles from health, politics, and environmental domains, posed questions to either the news author or a chatbot, and categorized the types, complexity, and content of the questions.
  3. Framework Development: Combined journalist and reader perspectives to explore the feasibility and design principles of various modules for QA chatbots.

Research Outcomes

• What specific outcomes were achieved?

  1. Journalists' Needs: Journalists want chatbots to answer simple factual or repetitive reader questions but do not want chatbots to replace direct communication, especially for subjective explanations.
  2. Readers' Behavior Patterns: Readers tend to ask short factual questions to chatbots but pose more complex, subjective, or multidimensional questions to authors.
  3. Question patterns were classified into seven categories: factual, detailed, evidence-based, closed-ended, subjective opinions, explanations, and other types.

• What advantages does it have compared to existing solutions?

  • Conducted extensive two-way analysis of functional needs for journalists and readers, addressing interaction demand matching issues better than traditional technologies.
  • Proposed a classification and routing strategy-based design framework that ensures the chatbot's compliance and practicality in functionality.

• What were the experimental or evaluation results?

  1. Readers' questions primarily fell into two categories: information retrieval (including factual and detailed questions) and subjective interpretation (opinions and explanations).
  2. When news quality was low, readers were more likely to pose critical questions to authors, but criticism significantly decreased when interacting with chatbots.
  3. Bloom's Taxonomy complexity analysis showed that the complexity of questions posed to authors (average score: 2.58) was significantly higher than those posed to chatbots (1.86).

• Limitations and Future Directions

  1. Limitations:
     • The chatbot experiment did not provide real-time responses, which may not fully reflect readers' actual interaction behaviors.
     • Only a limited range of news topics was studied; future research should expand to more types (e.g., breaking news, investigative journalism).
  2. Future Directions:
     • Develop prototype chatbots with QA functionality and conduct interaction studies.
     • Investigate the QA performance of various LLMs (e.g., GPT and LLaMa) to assess technological applicability.
     • Explore the differing needs of various reader groups (e.g., age, cultural background).