TalkTive: A Conversational Agent Using Backchannels to Engage Older Adults in Neurocognitive Disorders Screening

Title of the Paper

TalkTive: A Conversational Agent Using Backchannels to Engage Older Adults in Neurocognitive Disorders Screening

Paper Information

• Research Area: Human-Computer Interaction, Conversational Agents, Neurocognitive Disorders Screening
• Keywords: Backchanneling, Conversational Agents, Neurocognitive Disorders, Older Adults, Speech-based Interaction

Research Background and Problem

• Problems and Challenges:

  • With the intensification of global aging, neurocognitive disorders (NCDs) such as dementia pose significant socioeconomic and personal health burdens.
  • Existing screening techniques (e.g., MoCA tests) require face-to-face assessments by clinical professionals, which are difficult to scale due to limited medical resources and mobility issues among participants.
  • Machine learning and speech interaction technologies may provide solutions for NCD screening, but designing systems that effectively interact with older adults and stimulate meaningful language input remains an unresolved challenge.

• Research Motivation:

  • Investigate whether participatory interaction through conversational agents (CAs) can effectively support NCD screening.
  • Draw inspiration from communication strategies used by human professional assessors, such as backchanneling, to engage in more meaningful dialogue with older adults.

• Related Work:

  • CAs have been applied in various health management domains, but there is limited research on their specific pragmatic behaviors in cognitive screening.
  • While some engineering efforts have explored backchanneling techniques, their active functions (e.g., "encouraging continued expression") have not been adequately modeled and evaluated.

Solution

• Main Approach:

  • Proposed two types of backchanneling: reactive backchannels (RBC, e.g., "Hmm") and proactive backchannels (PBC, e.g., "Please continue").
  • Analyzed 246 real MoCA test audio recordings to identify the timing and types of these backchannels and developed data-driven, algorithm-based models.
  • Designed and implemented the conversational system "TalkTive," capable of intelligently generating appropriate RBCs and PBCs.

• Innovations:

  • Developed a backchannel generation algorithm combining real-time speech signal analysis and multi-task prediction, particularly for proactive backchannels (PBC), with response strategies based on task progress, participant characteristics, and pause duration.
  • Conducted one of the few studies on backchanneling in Cantonese, integrating machine learning and acoustic feature engineering to provide insights for low-resource language settings.

• Implementation Steps and Key Technologies:

  • Data Analysis: Annotated MoCA test audio data, encoding pauses in participant speech and types of assessor responses.
  • Feature Extraction: Utilized the large-scale acoustic feature set ComParE and applied stability selection to identify features suitable for Cantonese backchannel detection.
  • Dual-Module Model:
    • RBC Detection Module: Used a Support Vector Machine (SVM) classifier to determine whether a backchannel response should occur.
    • PBC Detection Module: Predicted response timing through weighted calculations of progress scores, pause scores, and participant characteristic scores.
  • System Integration: Implemented "TalkTive," combining real-time speech processing with a pre-defined Cantonese backchannel library.

Research Outcomes

• Specific Results:

  • In experiments, over 89% of backchannels generated by TalkTive were deemed appropriate by professional assessors; PBCs received particularly positive feedback from older users.
  • User studies confirmed that TalkTive effectively designed task-driven dialogues and that its backchanneling strategies were not perceived as intrusive.

• Advantages Compared to Existing Solutions:

  • Provides a backchannel implementation combining data-driven models and task-adaptive strategies, making conversations more engaging.
  • Offers a speech interaction design solution for Cantonese, a low-resource language.

• Experimental or Evaluation Results:

  • In an experiment involving 36 older participants from Hong Kong, three conditions were tested during MoCA tasks: no backchannels (baseline group), RBC only (condition 1), and both RBC and PBC (condition 2). Quantitative and qualitative studies showed that PBC significantly enhanced participant experience.
  • Older participants particularly favored PBCs, as proactive verbal encouragement improved interaction when tasks were challenging or pauses occurred.

• Limitations and Future Directions:

  • Limited sample size and experimental scope confined to laboratory settings.
  • System responses cannot fully replicate the flexibility of human conversation and have limited semantic processing capabilities for complex tasks.
  • Future research directions include:
    • Expanding the system to mobile scenarios for easier home use.
    • Integrating semantic understanding modules to enhance response accuracy through optimized ASR or semantic decoding technologies.
    • Exploring adaptive models to emulate personalized response logic of assessors.

This study demonstrates the significant potential of conversational AI in cognitive screening for older adults and provides empirical support and theoretical guidance for designing responsive conversational systems.