Through the Lens of Human-Human Collaboration: An Configurable Research Platform for Exploring Human-Agent Collaboration

Paper Title

Through the Lens of Human-Human Collaboration: A Configurable Research Platform for Exploring Human-Agent Collaboration

Publication Info

• Topic area: Human-Agent Collaboration in HCI and CSCW
• Keywords: Human-agent collaboration, LLM agents, CSCW, HCI, research platform, interaction controls, experimental design, agent integration, workspace awareness, communication modalities

Background and Problem

• Problem / challenge: Existing intelligent systems are primarily tools rather than collaborators, lacking mutual awareness, adaptivity, accountability, and interdependence. Current research platforms are not designed for systematic, reproducible studies of human-agent collaboration, particularly with LLM agents.
• Significance: Addressing these gaps is critical for designing effective human-agent systems, especially in high-stakes domains like healthcare and crisis management, where collaboration dynamics significantly impact outcomes.
• Motivation and related work: Prior research in HCI and CSCW has identified key variables for effective human-human collaboration, such as communication modalities and workspace awareness. However, these insights have not been systematically tested in human-agent contexts due to the lack of suitable platforms. Existing platforms either focus on human-human experiments or multi-agent simulations, failing to integrate agents into controlled human collaboration studies.

Solution

• Proposed approach: An open, configurable research platform designed for systematic, reproducible experiments on human-agent collaboration, inspired by classic CSCW paradigms like Shape Factory and Hidden Profile.
• Novelty:
  1. Introduction of the Agent Context Protocol (ACP) to ensure parity between human and agent participants.
  2. Modular architecture enabling adaptation of classic experimental paradigms for human-agent studies.
  3. Declarative Experiment Configuration Language (ECL) for flexible experiment design.
  4. Theory-driven interaction controls for manipulating variables like communication modalities and workspace awareness.
• Procedure and key techniques:
  • Four core components: researcher interface, participant interface, experiment controller, and ACP.
  • Researchers configure experiments using ECL and the researcher interface, which includes live monitoring and result analysis.
  • Interaction controls allow manipulation of communication, action structure, social framing, and agent responsiveness.
  • Evaluation through case studies (Shape Factory and Hidden Profile) and participatory cognitive walkthroughs with HCI researchers.

Results

• Concrete findings:
  • In the Shape Factory experiment, disabling private chat increased participants' wealth (+$40.6 on average) and reduced communication overhead.
  • In the Hidden Profile experiment, proactive agents improved team-level decision accuracy (16.7% vs. 8.3% in the passive condition).
  • Participants reported higher perceived trust and workspace awareness in conditions with richer communication and awareness support.
• Advantage over baselines:
  • The platform supports systematic manipulation of interaction variables, enabling controlled comparisons between human-human and human-agent collaboration.
  • Demonstrated ability to re-implement classic paradigms with LLM agents and capture nuanced behavioral and perceptual differences.
• Experiments / evaluation:
  • Shape Factory: 16 participants, two factors (communication level, awareness level), between-subjects design.
  • Hidden Profile: 16 participants, agent responsiveness (proactive vs. passive), between-subjects design.
  • Cognitive walkthrough: Five HCI researchers evaluated the researcher interface, identifying usability challenges and informing iterative improvements.
• Limitations and future work:
  • Current LLM agents lack deeper social and cognitive grounding, limiting generalizability.
  • Evaluation focused on platform design, not empirical validation of findings across paradigms.
  • Future work includes systematic replication of CSCW theories and advancing agent design with cognitive models.

Summary

This paper introduces a configurable research platform for studying human-agent collaboration, addressing gaps in existing tools by enabling systematic, reproducible experiments with LLM agents. The platform features modular architecture, the Agent Context Protocol for agent-human parity, and theory-driven interaction controls. Evaluation through case studies (Shape Factory, Hidden Profile) and cognitive walkthroughs demonstrates its efficacy in isolating interaction variables and supporting diverse experimental paradigms. While current LLM agents have limitations, the platform provides a foundation for advancing human-agent collaboration research and re-examining classic CSCW theories in this new context.