"I Didn't Know I Looked Angry": Characterizing Observed Emotion and Reported Affect at WorkWith the growing prevalence of affective computing applications, Automatic Emotion Recognition (AER) technologies have garnered attention in both research and industry settings. Initially limited to speech-based applications, AER technologies now include analysis of facial landmarks to provide predicted probabilities of a common subset of emotions (e.g., anger, happiness) for faces observed in an image or video frame. In this paper, we study the relationship between AER outputs and self-reports of affect employed by prior work, in the context of information work at a technology company. We compare the continuous observed emotion output from an AER tool to discrete reported affect obtained via a one-day combined tool-use and diary study (N=15). We provide empirical evidence showing that these signals do not completely align, and find that using additional workplace context only improves alignment up to 58.6%. These results suggest affect must be studied in the context it is being expressed, and observed emotion signal should not replace internal reported affect for affective computing applications.2022HKHarmanpreet Kaur et al.University of MichiganHuman Pose & Activity RecognitionExplainable AI (XAI)CHI
Large Scale Analysis of Multitasking Behavior During Remote MeetingsVirtual meetings are critical for remote work because of the need for synchronous collaboration in the absence of in-person interactions. In-meeting multitasking is closely linked to people's productivity and wellbeing. However, we currently have limited understanding of multitasking in remote meetings and its potential impact. In this paper, we present what we believe is the most comprehensive study of remote meeting multitasking behavior through an analysis of a large-scale telemetry dataset collected from February to May 2020 of U.S. Microsoft employees and a 715-person diary study. Our results demonstrate that intrinsic meeting characteristics such as size, length, time, and type, significantly correlate with the extent to which people multitask, and multitasking can lead to both positive and negative outcomes. Our findings suggest important best-practice guidelines for remote meetings (e.g., avoid important meetings in the morning) and design implications for productivity tools (e.g., support positive remote multitasking).2021HCHancheng Cao et al.Stanford UniversityRemote Work Tools & ExperienceNotification & Interruption ManagementWorkplace Wellbeing & Work StressCHI
Scraps: Enabling Mobile Capture, Contextualization, and Use of Document ResourcesPeople often capture photos or notes from their phones to integrate later into a document. But current mobile capture tools can make this hard, with the captured information ending up fragmented and decontextualized. This paper explores how to help document authors capture, contextualize, and use document-related information. A survey of 66 information workers reveals that document-focused information capture differs from other types of mobile information capture, and that while people capture a broad range of information types while mobile, most document-related capture comes in the form of photos, notes, and bookmarks. Based on this survey we built Scraps, which consists of two parts: 1) a mobile app that makes it easy for people to capture and add context to information from their phone, and 2) a Word sidebar that helps them later link that information to a document on their desktop. In a field study with 11 information workers, we find that Scraps streamlined the process of capturing and using document-related information, and enabled people to focus on writing over integrating captured information.2021ASAmanda M Swearngin et al.University of WashingtonInteractive Data VisualizationKnowledge Worker Tools & WorkflowsCHI
UIST+CSCW: A Celebration of Systems Research in Collaborative and Social ComputingThis joint panel between UIST and CSCW brings together leading researchers at the intersection of the conferences—systems researchers in collaborative and social computing—to engage in a discussion and retrospective. Pairs of panelists will represent each decade since the founding of the conferences, sharing a brief retrospective that surveys the most influential papers of that decade, the zeitgeist of the problems that were popular that decade and why, and what each decade's work has to say to the decades that came before and after. The panel is intended as a space to celebrate advances in the field, and reflect on the burdens and opportunities that it faces ahead.2020MBMichael S. Bernstein et al.UIST+CSCW: A Celebration of Systems Research in Collaborative and Social ComputingCSCW
Optimizing for Happiness and Productivity: Modeling Opportune Moments for Transitions and Breaks at WorkInformation workers perform jobs that demand constant multitasking, leading to context switches, productivity loss, stress, and unhappiness. Systems that can mediate task transitions and breaks have the potential to keep people both productive and happy. We explore a crucial initial step for this goal: finding opportune moments to recommend transitions and breaks without disrupting people during focused states. Using affect, workstation activity, and task data from a three-week field study (N=25), we build models to predict whether a person should continue their task, transition to a new task, or take a break. The R-squared values of our models are as high as 0.7, with only 15% error cases. We ask users to evaluate the timing of recommendations provided by a recommender that relies on these models. Our study shows that users find our transition and break recommendations to be well-timed, rating them as 86% and 77% accurate, respectively. We conclude with a discussion of the implications for intelligent systems that seek to guide task transitions and manage interruptions at work.2020HKHarmanpreet Kaur et al.University of MichiganNotification & Interruption ManagementWorkplace Wellbeing & Work StressCHI
Sketching NLP: A Case Study of Exploring the Right Things To Design with Language IntelligenceThis paper investigates how to sketch NLP-powered user experiences. Sketching is a cornerstone of design innovation. When sketching, designers rapidly experiment with a number of abstract ideas using simple, tangible instruments such as drawings and paper prototypes. Sketching NLP-powered experiences, however, presents challenges, i.e. How to visualize abstract language interaction? How to ideate a broad range of technically feasible intelligent functionalities? As a first step towards understanding these challenges, we present a first-person account of our sketching process when designing intelligent writing assistance. We detail the challenges we encountered and emergent solutions, such as a new format of wireframe for sketching language interactions and a new wizard-of-oz-based NLP rapid prototyping method. Drawing on these findings, we discuss the importance of abstraction in sketching and other implications.2019QYQian Yang et al.Carnegie Mellon UniversityHuman-LLM CollaborationAI-Assisted Creative WritingCHI
An Exploration of Speech-Based Productivity Support in the CarIn-car intelligent assistants offer the opportunity to help drivers productively use previously unclaimed time during their commute. However, engaging in secondary tasks can reduce attention on driving and thus may affect road safety. Any interface used while driving, even if speech-based, cannot consider non-driving tasks in isolation of driving---alerts for safer driving and timing of the non-driving tasks are crucial to maintaining safety. In this work, we explore experiences with a speech-based assistant that attempts to help drivers safely complete complex productivity tasks. Via a controlled simulator study, we look at how level of support and road context alerts from the assistant influence a driver's ability to drive safely while writing a document or creating slides via speech. Our results suggest ways to support speech-based productivity interactions and how speech-based road context alerts may influence driver behavior.2019NMNikolas Martelaro et al.Accenture Technology LabsVoice User Interface (VUI) DesignAI-Assisted Decision-Making & AutomationCHI
Casual Microtasking: Embedding Microtasks in FacebookMicrotasks enable people with limited time and context to contribute to a larger task. In this paper we explore casual microtasking, where microtasks are embedded into other primary activities so that they are available to be completed when convenient. We present a casual microtasking experience that inserts writing microtasks from an existing microwriting tool into the user's Facebook feed. From a two-week deployment of the system with nine people, we observe that casual microtasking enabled participants to get things done during their breaks, and that they tended to do so only after first engaging with Facebook's social content. Participants were most likely to complete the writing microtasks during periods of the day associated with low focus, and would occasionally use them as a springboard to open the original document in Word. These findings suggest casual microtasking can help people leverage spare micromoments to achieve meaningful micro-goals, and even encourage them to return to work.2019NHNathan Hahn et al.Carnegie Mellon UniversityCrowdsourcing Task Design & Quality ControlOpen-Source Collaboration & Code ReviewNotification & Interruption ManagementCHI
Guidelines for Human-AI InteractionAdvances in artificial intelligence (AI) frame opportunities and challenges for user interface design. Principles for human-AI interaction have been discussed in the human-computer interaction community for over two decades, but more study and innovation are needed in light of advances in AI and the growing uses of AI technologies in human-facing applications. We propose 18 generally applicable design guidelines for human-AI interaction. These guidelines are validated through multiple rounds of evaluation including a user study with 49 design practitioners who tested the guidelines against 20 popular AI-infused products. The results verify the relevance of the guidelines over a spectrum of interaction scenarios and reveal gaps in our knowledge, highlighting opportunities for further research. Based on the evaluations, we believe the set of design guidelines can serve as a resource to practitioners working on the design of applications and features that harness AI technologies, and to researchers interested in the further development of human-AI interaction design principles.2019SASaleema Amershi et al.MicrosoftVoice User Interface (VUI) DesignAI-Assisted Decision-Making & AutomationAlgorithmic Fairness & BiasCHI
Creating Better Action Plans for Writing Tasks via Vocabulary-Based PlanningWhile having a step-by-step breakdown for a task—an action plan—helps people complete tasks, prior work has shown that people prefer not to make action plans for their own tasks. Getting planning support from others could be beneficial, but it is limited by how much domain knowledge people have about the task and how available they are. Our goal is to incorporate the benefits of having action plans in the complex domain of writing, while mitigating the time and effort costs of creating plans. To mitigate these costs, we introduce a vocabulary—a finite set of functions pertaining to writing tasks—as a cognitive scaffold that enables people with necessary context (e.g. collaborators) to generate action plans for others. We develop this vocabulary by analyzing 264 comments, and compare plans created using it with those created without any aid, in an online study with 768 comments (N=145) and a lab study with 96 comments (N=8). We show that using a vocabulary reduces planning time and effort and improves plan quality compared to unstructured planning, and opens the door for automation and task sharing for complex tasks.2018HKHarmanpreet Kaur et al.Language and LinguisticsCSCW
Multitasking with Play Write, a Mobile Microproductivity Writing ToolMobile devices offer people the opportunity to get useful tasks done during time previously thought to be unusable. Because mobile devices have small screens and are often used in divided attention scenarios, people are limited to using them for short, simple tasks; complex tasks like editing a document present significant challenges in this environment. In this paper we demonstrate how a complex task requiring focused attention can be adapted to the fragmented way people work while mobile by decomposing the task into smaller, simpler microtasks. We introduce Play Write, a microproductivity tool that allows people to edit Word documents from their phones via such microtasks. When participants used Play Write while simultaneously watching a video, we found that they strongly preferred its microtask-based editing approach to the traditional editing experience offered by Mobile Word. Play Write made participants feel more productive and less stressed, and they completed more edits with it. Our findings suggest microproductivity tools like Play Write can help people be productive in divided attention scenarios.2018SIShamsi T. Iqbal et al.Intelligent Voice Assistants (Alexa, Siri, etc.)Human-LLM CollaborationKnowledge Worker Tools & WorkflowsUIST
Supporting Workplace Detachment and Reattachment with Conversational IntelligenceResearch has shown that productivity is mediated by an individual’s ability to detach from their work at the end of the day and reattach with it when they return the next day. In this paper we explore the extent to which structured dialogues, focused on individuals’ work-related tasks or emotions, can help them with the detachment and reattachment processes. Our inquiry is driven with SwitchBot, a conversational bot which engages with workers at the start and end of their work day. After preliminarily validating the design of a detachment and reattachment dialogue frame-work with 108 crowdworkers, we study SwitchBot’s use in-situ for 14 days with 34 information workers. We find that workers send fewer e-mails after work hours and spend a larger percentage of their first hour at work using productivity applications than they normally would when using SwitchBot. Further, we find that productivity gains were better sustained when conversations focused on work-related emotions. Our results suggest that conversational bots can be effective tools for aiding workplace detachment and reattachment and help people make successful use of their time on and off the job.2018AWAlex C Williams et al.University of WaterlooConversational ChatbotsNotification & Interruption ManagementCHI