Supporting Effective Goal Setting with LLM-Based Chatbots

Paper Title

Supporting Effective Goal Setting with LLM-Based Chatbots

Publication Info

• Topic area: Human-computer interaction and behavioral interventions using AI.
• Keywords: Goal setting, implementation intentions, large language models, chatbots, behavioral interventions, HCI, social presence, feedback mechanisms, adaptive suggestions, psychological theory.

Background and Problem

• Problem / challenge: Despite the effectiveness of psychological theories like goal setting and implementation intentions, individuals often struggle to apply these frameworks independently. Existing technological solutions, such as rule-based systems, lack adaptability and nuanced support.
• Significance: Enhancing goal-setting processes can improve outcomes in domains like health, education, and workplace productivity. Scalable, accessible interventions are needed to bridge the gap between theoretical frameworks and practical application.
• Motivation and related work: Previous studies have shown that human coaches and rule-based systems can support goal setting, but these approaches are limited in scalability and personalization. Large language models (LLMs) offer potential for more dynamic and human-like interaction, yet their specific mechanisms for effective goal setting remain underexplored.

Solution

• Proposed approach: Development and evaluation of LLM-based chatbots incorporating three design features—guidance, adaptive suggestions, and feedback—to support effective goal setting and implementation intention formation.
• Novelty:
  1. Systematic comparison of individual and combined effects of guidance, suggestions, and feedback in LLM-based interventions.
  2. Empirical evidence on how LLMs improve goal specificity and implementation intention quality.
  3. Exploration of the trade-offs between goal quality and user motivation.
  4. Mediation analysis of social presence as a mechanism influencing goal commitment.
• Procedure and key techniques:
  • Five chatbot conditions were developed: ControlBot (static input form), GuidanceBot (step-by-step guidance), SuggestionBot (guidance with adaptive suggestions), FeedbackBot (guidance with iterative feedback), and GenBot (combining all features).
  • Participants (N = 543) interacted with one chatbot to set a personally meaningful goal and implementation intention.
  • Outcomes were measured via self-reports (e.g., goal commitment, social presence) and expert-coded ratings (e.g., goal specificity, implementation intention quality).

Results

• Concrete findings:
  • Guidance improved goal specificity (z = 3.42, p = .001) and implementation intention quality (z = 5.91, p < .001), but not perceived goal difficulty.
  • Feedback further enhanced goal specificity (z = 4.34, p < .001) and implementation intention quality (z = 7.37, p < .001).
  • Suggestions did not significantly improve outcomes beyond guidance.
  • Combining all features (GenBot) yielded the highest goal specificity (z = 6.35, p < .001) but did not consistently improve goal difficulty or implementation intention quality compared to FeedbackBot.
• Advantage over baselines:
  • LLM-based feedback and combined features significantly outperformed static and rule-based systems in generating specific and high-quality goals.
  • Social presence mediated higher goal commitment for LLM-based chatbots, though commitment was lower overall compared to the control condition.
• Experiments / evaluation:
  • Randomized controlled experiment with five chatbot conditions.
  • Measures included goal specificity (coded 0–4), implementation intention quality (coded 0–4), perceived social presence, goal commitment, and intention to act.
  • Statistical analyses included Kruskal–Wallis tests, Dunn’s post-hoc comparisons, and mediation analysis.
• Limitations and future work:
  • Study focused on immediate effects; long-term goal attainment was not assessed.
  • Ceiling effects in goal difficulty and commitment measures may have obscured differences.
  • Future research should explore goal selection processes and autonomy-preserving designs to mitigate motivational trade-offs.

Summary

This study demonstrates that LLM-based chatbots can effectively support goal setting by improving goal specificity and implementation intention quality through guidance and feedback mechanisms. However, adaptive suggestions showed limited impact, and more sophisticated chatbot designs reduced user commitment and intention to act, highlighting a quality–motivation trade-off. Social presence mediated positive effects on goal commitment but was insufficient to counteract negative direct effects. The findings provide actionable insights for designing scalable, domain-agnostic AI interventions that balance clarity and user autonomy while leveraging psychological frameworks.