Mind the Gap! Choice Independence in Using Multilingual LLMs for Persuasive Co-Writing Tasks in Different Languages

Research Background and Issues

• What problems or challenges did the authors identify?

  • Multilingual large language models (LLMs) exhibit significant performance disparities across different languages, particularly in low-resource languages, where issues such as fluency and contextual adaptability are more pronounced.
  • Human users in multilingual environments may experience cross-task performance differences when using LLMs, which can reduce their utilization of the models.
  • Users tend to predict the performance of LLMs in high-resource language tasks based on their prior experiences with the models in low-resource languages. This behavior violates the classical principle of choice independence.

• Why is this issue important?

  • The multilingual capabilities of LLMs play a crucial role in globalized work environments. Performance inconsistencies may have profound impacts on human behavior and productivity, especially in cross-cultural and cross-linguistic collaborations.
  • Violations of choice independence may lead to suboptimal use of tools, reducing the potential benefits of AI-assisted systems in multilingual contexts. This is particularly detrimental to regions with low language resources, exacerbating global inequalities.

• Research Motivation and Related Work

  • Existing literature indicates that humans tend to generalize observed errors in computer-generated content to unrelated, task-independent domains.
  • Few studies have examined the real-world impact of such choice independence violations, particularly in scenarios involving human-AI collaborative persuasive writing tasks.

Solutions

• What methods or solutions did the authors propose?

  • The study designed two experiments to evaluate the performance of multilingual LLMs in collaborative writing tasks in English and Spanish and to explore changes in user behavior after exposure to LLM performance in different languages.
  • It investigated how the usage patterns of LLMs in different languages influence the persuasiveness of advertisements and users' donation behaviors toward AI-generated content.

• What are the innovative aspects of this solution?

  • The study extended the concept of choice independence from abstract decision-making tasks in laboratory settings to real-world, complex application scenarios, such as persuasive advertisement writing and donation contexts.
  • It considered user adaptation to AI-generated content in multilingual settings, addressing gaps in research on multilingual collaboration.
  • It explored how cultural and gender factors influence attitudes toward AI-generated content, which holds significance for donation behavior and social interactions.

• What are the implementation steps and key technologies used?

  1. Experiment 1: Persuasive Writing Task
     • Task: Participants were asked to write charity advertisements in English and Spanish using LLMs, with sequence variables (e.g., English first, then Spanish, or vice versa) introduced to measure exposure effects between LLM performances.
     • Tool: An improved version of the ABScribe tool equipped with specific AI modification modes, such as positive framing and negative framing.
     • Key parameters: Recorded participants' frequency of using AI-generated features and the similarity metrics between final texts and AI-generated content.
  2. Experiment 2: Donation Behavior Experiment
     • Used advertisement texts from different sources (pure AI-generated, human-generated, AI-human collaborative) to measure their impact on donation behavior.
     • Recorded participants' beliefs about the source of the advertisements (human or AI-generated) and how these beliefs influenced donation decisions.

Research Outcomes

• What specific outcomes were achieved?

  1. Violations of choice independence were confirmed—after initial exposure to low-resource language (Spanish) LLMs, participants showed a significant decrease in usage of high-resource language (English) LLMs.
  2. The donation experiment found no significant differences in the persuasiveness of advertisements based on changes in LLM utilization, suggesting that choice independence violations have minimal direct effects on persuasive outcomes.
  3. Participants generally struggled to accurately distinguish between human- and AI-generated texts, but their donation behavior was influenced by erroneous beliefs about the source of the advertisements:
     • Female Spanish-speaking participants who believed the advertisements were AI-generated donated significantly less on average, with a notable increase in selfish behavior (complete refusal to donate).
  4. Participants from different backgrounds exhibited cultural and gender differences in their attitudes toward LLM-generated content.

• What advantages does this solution offer compared to existing approaches?

  • Expanded the framework of user behavior to more representative real-world application scenarios, providing new theoretical and practical insights into the use of multilingual AI assistants.
  • Carefully designed experiments captured the asymmetry in LLM performance in complex multilingual environments and its subtle impacts on user behavior.

• What were the experimental or evaluation results?

  • Deep involvement of AI-generated content showed no significant advantage in advertisement persuasiveness, but erroneous perceptions of AI sources profoundly influenced individual donation behavior.
  • Exposure to low-resource languages significantly reduced trust and reliance on LLMs for cross-language tasks.

• Limitations and Future Directions

  • Limitations:
    • The study used a single language pair (English and Spanish), which may provide only a limited behavioral baseline. Future research should explore languages with even lower resources.
    • The writing tasks were relatively simple; future studies could extend to more complex or higher-level writing scenarios.
    • Cultural variables were not explicitly manipulated or thoroughly explained, though significant language and gender effects were observed.
  • Future Directions:
    • Expand to global multilingual contexts to study how to better design and optimize multilingual LLMs to balance resource asymmetries.
    • Investigate the impact of educational interventions (e.g., AI capability awareness) on user rationality and decision-making.
    • Explore cross-cultural decision-making biases and their underlying cognitive mechanisms.