Verbal vs. Visual: How Humans Perceive and Collaborate with AI Teammates Using Different Communication Modalities in Various Human-AI Team Compositions

DOI

As AI becomes more advanced in natural language processing, the research on AI's communication in teaming environments is getting more attention in CSCW/HCI. Even though AI's communication serves as an essential part of facilitating team coordination and shaping team outcomes, the impact of AI's communication modality on human-AI teamwork is still understudied. Using a mixed-design experiment and follow-up group interviews with 100 participants, we investigate the impact of AI's communication modality, one of AI's most essential communication characteristics, on team coordination and team outcomes in two different human-AI team compositions, human-human-AI teams, and human-AI-AI teams. Our findings highlight the trade-offs between AI's verbal communication and visual communication, which inspire two design recommendations on how to apply AI's verbal and visual communication to support human-AI coordination effectively. Our study generates an initial understanding of the role that AI's communication will have in ensuring team effectiveness in human-AI teams in the CSCW/HCI field.