Sorry, Your HIT Is Overbooked – Investigating the Use of Crowdsourcing HIT CatchersIn microtask crowdsourcing, Human Intelligence Tasks (HITs) are commonly allocated on a first-come, first-served basis: they are published on the platform and the fastest workers select the most attractive ones first. This step has not received much attention from the scientific community yet, though it can become particularly taxing for workers when they compete to secure the most sought-after tasks. There are many strategies to ensure one's access to tasks and their effects on the labour process as a whole are not well understood. For instance, platforms with a sizeable task reservation queue allow workers to gain preferential access to a large number of tasks, which in turn may cause a shortage of work for the rest of the crowd. For the requesters, this means lower rates of completion and a lack of worker diversity. We explore workers' strategies for accessing and reserving tasks using monitoring techniques from both client and server sides. We investigate how these strategies affect task execution, in terms of availability, completion time, and answer quality, by deploying 1000 image annotation HITs in Amazon Mechanical Turk including objective and subjective tasks. We observe that workers who do not use automated catching techniques tend to have higher annotation quality, are more focused, spend more effort on text editing, and provide a higher diversity of output than workers using such tools. This study also reveals the tragedy of the commons effect among platform members due to the use of catching techniques: workers using automated catching techniques reserve and complete a substantially higher portion of the available tasks, but the over-reservation of HITs restricts all workers of reservation opportunities, and compromise their own future labour capacity as well. We observe a high inefficiency in job completions, as the majority of the times a task is being reserved by a worker, it will not get actually performed and will need to be republished for further allocation. Finally, we propose solutions to mitigate the negative effects of these phenomena on the labour process.2025EMEddy Maddalena et al.The Gig EconomyCSCW
Metaphors for Good Digital IdentitiesDigital identities are often discussed or explained as digital versions of physical documents such as passports. This metaphor tends to ignore, intentionally or not, the social challenges associated with real-world implementation of these technologies. This paper presents eight alternative metaphors for “good" digital identities which are derived from a 12-month Research-through-Design process. This process is presented as an annotated portfolio showcasing insights from a variety of design activities and stakeholder engagements, including design sprints, workshops, an artist residency and an exhibition, with the metaphors operating as “meta-annotations" on the portfolio. The eight metaphors intend to provoke and enable wider conversation with various stakeholders including academics, non-profits, industry professionals and policy makers about what “good" digital identities might mean, by focusing on societal rather than common technical concerns.2025KSKim Snooks et al.Online Identity & Self-PresentationInclusive DesignParticipatory DesignDIS
Advancing Affective Intelligence in Virtual Agents Using Affect Control TheoryAffective Intelligent Virtual Agents (AIVAs) has emerged as a research domain that integrates artificial intelligence, affective computing, and virtual agent technology. This fusion aims to develop interactive systems capable of perceiving, interpreting, and responding to human emotions. Affect Control Theory (ACT), a theoretical framework developed by Heise (1977) and adapted for virtual agent applications by Robillard and Hoey (2018) proposes that individuals unconsciously compare their own affective behavior with that of their interlocutor, forming predictions about the latter. Satisfaction and psychological stress levels are then influenced by the extent to which the interlocutor’s behavior aligns with these expectations. In this paper we introduce an AIVA that employs ACT concepts to interpret user text and generate emotionally-aligned responses, facial expressions, and gestures for an animated virtual character, AvataRena, that we are developing to act as a virtual life coach. Using the DeepMoji network, user textual input is mapped to emojis and then to a three-dimensional affect space. We then use prompt engineering to create ChatGPT responses that are moderated by ACT analyses to deliver emotionally-aligned textual and non-verbal responses. This alignment adheres to the principle of deflection within ACT, positing that lower deflection values correspond to heightened positivity in elicited emotions. To validate the model we performed a controlled simulation using 1480 questions derived from counselor-patient interactions to explore differences between prompt-engineered ChatGPT-generated responses with, and without, ACT moderation. Specifically, we found significantly lower deflection measures for the ACT-moderated AIVA responses, indicating that the moderated system adheres more closely to expected affective behavior than unmoderated ChatGPT. This was a large effect (t(1479)=-33.03, p<.001, Cohen's d = 0.862). Future work will investigate whether this promising result transfers to enhanced user satisfaction and response alignment during extended interactions in the life coach setting.2025ELEvdoxia Eirini Lithoxoidou et al.Intelligent Voice Assistants (Alexa, Siri, etc.)Agent Personality & AnthropomorphismGenerative AI (Text, Image, Music, Video)IUI
MedAI-SciTS: Enhancing Interdisciplinary Collaboration between AI Researchers and Medical ExpertsIntegrating AI in healthcare requires effective interdisciplinary collaboration, yet challenges like methodological differences, terminology barriers, and divergent objectives persist. To address the issues, we introduce MedAI-SciTS, a structured approach combining a theoretical framework and a toolkit to improve collaboration across disciplines. The framework builds on a formative study (N=12) and literature review, identifying the key challenges and potential solutions in medical-AI projects. We further develop an innovative toolkit with twelve tools, featuring an AI-enhanced research glossary with personalized analogies, an agile co-design platform, and an integrated resource management system. A three-month case study involving AI and medical professionals (N=16 total) applying a segmentation algorithm for adrenal CT images confirmed the toolkit’s effectiveness in enhancing team engagement, communication, trust, and collaboration outcomes. We envision MedAI-SciTS could potentially be applied to a wide range of medical applications and facilitate broader medical-AI collaboration.2025CCChen Cao et al.university of sheffield, Information schoolEV Charging & Eco-Driving InterfacesHand Gesture RecognitionKnowledge Worker Tools & WorkflowsCHI
DeliData: A dataset for deliberation in multi-party problem solvingGroup deliberation enables people to collaborate and solve problems, however, it is understudied due to a lack of resources. To this end, we introduce the first publicly available dataset containing collaborative conversations on solving a well-established cognitive task, consisting of 500 group dialogues and 14k utterances. In 64% of these conversations, the group members are able to find a better solution than they had identified individually, and in 43.8% of the groups who had a correct answer as their final solution, none of the participants had solved the task correctly by themselves. Furthermore, we propose a novel annotation schema that captures deliberation cues and release all 14k utterances annotated with it. Finally, we use the proposed dataset to develop and evaluate two methods for generating deliberation utterances. The data collection platform, dataset and annotated corpus will be made publicly available.2023GKGeorgi Milev Karadzhov et al.Collaboration ICSCW
The Barriers to Online Clothing Websites for Visually Impaired People: An Interview and Observation Approach to Understanding NeedsVisually impaired (VI) people often face challenges when performing everyday tasks and identify shopping for clothes as one of the most challenging. Many engage in online shopping, which eliminates some challenges of physical shopping. However, clothes shopping online suffers from many other limitations and barriers. More research is needed to address these challenges, and extant works often base their findings on interviews alone, providing only subjective, recall-biased information. We conducted two complementary studies using both observational and interview approaches to fill a gap in understanding about VI people's behaviour when selecting and purchasing clothes online. Our findings show that shopping websites suffer from inaccurate, misleading, and contradictory clothing descriptions; that VI people mainly rely on (unreliable) search tools and check product descriptions by reviewing customer comments. Our findings also indicate that VI people are hesitant to accept assistance from automated, but that trust in such systems could be improved if researchers can develop systems that better accommodate users' needs and preferences.2023AAAmnah Alluqmani et al.Visual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)DIS
Integrative Objects in Sociotechnical Contexts: Constructing Digital Well-Being with Generic EpistemologyThis paper presents a generative approach to interdisciplinary collaboration based on generic epistemology. Informed by the work of philosopher Anne-Françoise Schmid, we introduce the concept of the integrative object as a means to reorient interdisciplinary collaboration toward the requirements of the object of research itself, rather than via the requirements of particular disciplinary languages, methods, or operative logics. We show how such an approach is useful for research into sociotechnical phenomena that exceed the boundaries of discrete disciplines and their convergence. We introduce digital well-being as a case study, drawing on the authors' own interdisciplinary collaborative experiences in this area as its empirical matter. From this, and in order to aid future research into similarly complex sociotechnical objects, we then provide practical tools to help those in the HCI community prepare and conduct interdisciplinary research in a similarly generative, non-dogmatic, and non-hierarchical manner.2023MKMagdalena Krysztoforska et al.University of NottinghamInclusive DesignParticipatory DesignCHI
CrowdCO-OP: Sharing Risks and Rewards in CrowdsourcingPaid micro-task crowdsourcing has gained in popularity partly due to the increasing need for large-scale manually labelled datasets which are often used to train and evaluate Artificial Intelligence systems. Modern paid crowdsourcing platforms use a piecework approach to rewards, meaning that workers are paid for each task they complete, given that their work quality is considered sufficient by the requester or the platform. Such an approach creates risks for workers; their work may be rejected without being rewarded, and they may be working on poorly rewarded tasks, in light of the disproportionate time required to complete them. As a result, recent work has shown that crowd workers may tend to choose specific, simple, and familiar tasks and avoid new requesters. In this paper, we propose a novel crowdsourcing reward mechanism that allows workers to share these risks and achieve a standardized hourly wage equal for all participating workers. Reward-focused workers can thereby take up challenging and complex HITs without bearing the financial risk of not being rewarded for completed work. We experimentally compare different crowd reward schemes and observe their impact on worker performance and satisfaction. Our results show that 1) workers clearly perceive the benefits of the proposed reward scheme, 2) work effectiveness and efficiency are not affected as compared to those of the piecework scheme, and 3) the presence of slow workers is limited and does not disrupt the proposed cooperation-based approaches.2020SFShaoyang Fan et al.Crowds and CollaborationCSCW
Nationality and Gender Biases in Multicultural Online Learning Environments: The Effects of AnonymityOnline learning environments eliminate geographical barriers and enable new forms of collaboration between students at large scale. Self-presentation within such environments affects how students interact with learning content and with each other. We explore how anonymity/identifiability in user profile design impacts student interactions in a large multicultural classroom across two geographical locations. After triangulating 150,000 online interactions with questionnaires and focus groups, we provide three major findings. First, being identifiable had a significant impact on how students accessed and rated content created by their peers. Second, when identifiable, cultural differences became more prominent, leading some students to avoid content created by classmates of certain nationalities. Finally, when students interacted with their real identities, there were significant and negative gender effects which were absent when students were anonymous. These findings contribute to our understanding of social dynamics within multicultural learning environments, and raise practical implications for tool design.2020GMGabriela Morales-Martinez et al.The University of SheffieldOnline Learning & MOOC PlatformsGender & Race Issues in HCIParticipatory DesignCHI
Understanding The Messages Conveyed by Automated VehiclesEfficient and safe interactions between automated vehicles and other road users can be supported through external Human-Machine Interfaces (eHMI). The success of these interactions relies on the eHMI signals being adequately understood by other road users. A paired-comparison forced choice task (Task 1), and a 6-point rating task (Task 2) were used to assess the extent to which ten different eHMI signals conveyed three separate messages, ‘I am giving way’, ‘I am in automated mode’ and ‘I will start moving’. The different eHMI options consisted of variations of a 360° lightband, a single lamp, and an auditory signal. Results demonstrated that the same eHMI format could convey different messages equally well, suggesting a need to be cautious when designing eHMI, to avoid presenting misleading, potentially unsafe, information. Future research should investigate whether the use of an eHMI signal indicating a change in the AV’s behaviour is sufficient for conveying intention.2019YLYee Mun Lee et al.External HMI (eHMI) — Communication with Pedestrians & CyclistsAutoUI
Deadline-Aware Fair Scheduling for Multi-Tenant Crowd-Powered SystemsCrowdsourcing has become an integral part of many systems and services that deliver high-quality results for complex tasks such as data linkage, schema matching, and content annotation. A standard function of such crowd-powered systems is to publish a batch of tasks on a crowdsourcing platform automatically and to collect the results once the workers complete them. Currently, these systems provide limited guarantees over the execution time, which is problematic for many applications. Timely completion may even be impossible to guarantee due to factors specific to the crowdsourcing platform, such as the availability of workers and concurrent tasks. In our previous work, we presented the architecture of a crowd-powered system that reshapes the interaction mechanism with the crowd. Specifically, we studied a push-crowdsourcing model whereby the workers receive tasks instead of selecting them from a portal. Based on this interaction model, we employed scheduling techniques similar to those found in distributed computing infrastructures to automate the task assignment process. In this work, we first devise a generic scheduling strategy that supports both fairness and deadline-awareness. Second, to complement the proof-of-concept experiments previously performed with the crowd, we present an extensive set of simulations meant to analyze the properties of the proposed scheduling algorithms in an environment with thousands of workers and tasks. Our experimental results show that, by accounting for human factors, micro-task scheduling can achieve fairness for best-effort batches and boosts production batches.2019DDDjellel Difalla et al.Crowds and participatory sensingCSCW
Phone vs. Tangible in Museums: A Comparative StudyDespite years of HCI research on digital technology in museums, it is still unclear how different interactions impact on visitors’. A comparative evaluation of smart replicas, phone app and smart cards looked at the personal preferences, behavioural change, and the appeal of mobiles in museums. 76 participants used all three interaction modes and gave their opinions in a questionnaire; participants interaction was also observed. The results show the phone is the most disliked interaction mode while tangible interaction (smart card and replica combined) is the most liked. Preference for the phone favour mobility to the detriment of engagement with the exhibition. Different behaviours when interacting with the phone or the tangibles where observed. The personal visiting style appeared to be only marginally affected by the device. Visitors also expect museums to provide the phones against the current trend of developing apps in a “bring your own device” approach.2018DPDaniela Petrelli et al.Sheffield Hallam UniversityHuman Pose & Activity RecognitionMuseum & Cultural Heritage DigitizationCHI