Crowdsourced Think-Aloud StudiesThe think-aloud (TA) protocol is a useful method for evaluating user interfaces, including data visualizations. However, TA studies are time-consuming to conduct and hence often have a small number of participants. Crowdsourcing TA studies would help alleviate these problems, but the technical overhead and the unknown quality of results have restricted TA to synchronous studies. To address this gap we introduce CrowdAloud, a system for creating and analyzing asynchronous, crowdsourced TA studies. CrowdAloud captures audio and provenance (log) data as participants interact with a stimulus. Participant audio is automatically transcribed and visualized together with events data and a full recreation of the state of the stimulus as seen by participants. To gauge the value of crowdsourced TA studies, we conducted two experiments: one to compare lab-based and crowdsourced TA studies, and one to compare crowdsourced TA studies with crowdsourced text prompts. Our results suggest that crowdsourcing is a viable approach for conducting TA studies at scale.2025ZCZach Cutler et al.University of Utah, Visualization Design LabTime-Series & Network Graph VisualizationUser Research Methods (Interviews, Surveys, Observation)Prototyping & User TestingCHI
Designed & Discovered Euphoria: Insights from Trans-Femme Players' Experiences of Gender Euphoria in Video GamesMany transgender (and cisgender) people experience gender euphoria -- satisfaction and relief caused by self-actualization and gender congruence -- a term that has been overlooked by the design community. Video games create intense experiences involving identities, bodies, and social interaction, providing opportunities to empower people through gender euphoria. We develop themes for creating and supporting gender euphoria in games within the Design, Dynamics, Experience Game Design Framework from a reflexive thematic analysis of 25 games, with an in-depth analysis of four of them. The analysis combines the authors' positionalities as trans gamers with close reading and content analysis of the games, employing perspectives from critical discourse analysis. We contribute an operational understanding of gender euphoria to support design, in-depth case studies of particularly euphoric game experiences, and identify themes that designers and researchers can use to develop new games and analyze existing ones.2025SLShano Liang et al.Worcester Polytechnic InstituteMultiplayer & Social GamesRole-Playing & Narrative GamesGender & Race Issues in HCICHI
Perceptions and Preferences: Deaf ASL-Signing Users' Insights on Video Elements, Styles and LayoutsVideo components are a central element of user interfaces that deliver content in a signed language (SL), but the potential of video components extends beyond content accessibility. SL videos may be designed as user interface elements: layered with interactive features to create navigation cues, page headings, and menu options. To be effective for signing users, novel SL video-rich interfaces require informed design choices across many parameters. To align with the specific needs and shared conventions of the Deaf community and other ASL-signers in this context, we present a user study involving deaf ASL-signers who interacted with an array of designs for SL video elements. Their responses offer some insights into how the Deaf community may perceive and prefer video elements to be designed, positioned, and implemented to guide user experiences. Through a qualitative analysis, we take initial steps toward understanding deaf ASL-signers’ perceptions of a set of emerging design principles, paving the way for future SL-centric user interfaces containing customized video elements and layouts with primary consideration for signed language-related usage and requirements.2025KAKhulood Alkhudaidi et al.Worcester Polytechnic InstituteForce Feedback & Pseudo-Haptic WeightVoice AccessibilityDeaf & Hard-of-Hearing Support (Captions, Sign Language, Vibration)CHI
This Game SUX: Why & How to Design Sh@*!y User ExperiencesWhile normative – "good" – game design and user experiences have been established, we look to games that challenge those notions. Intentional frustration and failure can be worthwhile. Through a reflexive thematic analysis of 31 games we identify how intentionally non-normative design choices lead to meaningful experiences. Working within the established Mechanics Dynamics Aesthetics (MDA) Game Design Framework, we lay out themes to design Shitty User Experiences (SUX). We contribute SUX MDA themes for designers and researchers to counter the status quo and identify new forms of play and interaction.2025MCMichelle V Cormier et al.Monash University, Exertion Games LaboratoryGame UX & Player BehaviorSerious & Functional GamesGamification DesignCHI
Is Resistance Futile?: Early Career Game Developers, Generative AI, and Ethical SkepticismThis paper presents a study that examines developer perceptions and usage of generative AI (GAI) in a summer professional development program for game development interns focused on mobile game design. GAI applications are in common usage worldwide, yet the impacts of this technology in game development remain relatively underexplored. Through a qualitative study using ethnographic interviews and participatory observation, this paper explores how GAI impacted the workflows, creative processes, and professional identities of early career game developers. We present a case of GAI integration that was not a straightforward adoption. Focusing on the interns' resistance, negotiation, and reimagining, we show that the interns were actively developing a new professional culture both with and against generative AI. For the interns, their ethical commitments to fellow game developers and the future of their profession were as important as their practical concerns about usability, utility, and efficacy of GAI tools.2024JBJosiah D Boucher et al.WPIBrain-Computer Interface (BCI) & NeurofeedbackGenerative AI (Text, Image, Music, Video)AI Ethics, Fairness & AccountabilityCHI
How Can Deep Neural Networks Aid Visualization Perception Research?: Three Studies on Correlation Judgments in ScatterplotsHow deep neural networks can aid visualization perception research is a wide-open question. This paper provides insights from three perspectives—prediction, generalization, and interpretation—via training and analyzing deep convolutional neural networks on human correlation judgments in scatterplots across three studies. The first study assesses the accuracy of twenty-nine neural network architectures in predicting human judgments, finding that a subset of the architectures (e.g., VGG-19) has comparable accuracy to the best-performing regression analyses in prior research. The second study shows that the resulting models from the first study display better generalizability than prior models on two other judgment datasets for different scatterplot designs. The third study interprets visual features learned by a convolutional neural network model, providing insights about how the model makes predictions, and identifies potential features that could be investigated in human correlation perception studies. Together, this paper suggests that deep neural networks can serve as a tool for visualization perception researchers in devising potential empirical study designs and hypothesizing about perpetual judgments. The preprint, data, code, and training logs are available at https://doi.org/10.17605/osf.io/exa8m.2023FYFumeng Yang et al.Northwestern UniversityVisualization Perception & CognitionCHI
Smooth as - The Effects of Frame Rate Variation on Game Player Quality of ExperienceFor gamers, high frame rates are important for a smooth visual display and good quality of experience (QoE). However, high frame rates alone are not enough as variations in the frame display times can degrade QoE even as the average frame rate remains high. While the impact of steady frame rates on player QoE is fairly well-studied, the effects of frame rate variation is not. This paper presents a 33-person user study that evaluates the impact of frame rate variation on users playing three different computer games. Analysis of the results shows average frame rate alone is a poor predictor of QoE, and frame rate variation has a significant impact on player QoE. While the standard deviation of frame times is promising as a general predictor for QoE, frame time standard deviation may not be accurate for all individual games. However, 95% frame rate floor -– the bottom 5% of frame rates the player experiences –- appears to be an effective predictor of both QoE overall and for the individual games tested.2023SLShengmei Liu et al.Worcester Polytechnic InstituteGame UX & Player BehaviorGamification DesignCHI
Estimating 3D Finger Pose via 2D-3D Fingerprint MatchingTouchscreens have become the primary input devices for smartphones, tablet computers, and other intelligent devices over the past decades. While for the most pervasive commercial devices, only 2D touch positions on the screen are utilized as interaction inputs. To extend the richness of the input vocabulary, some researchers have proposed several innovative interaction techniques, e.g. finger pose. However, due to the low resolution and lacking in information of capacitive images, only two angles, pitch and yaw, are considered in most finger pose estimation algorithms, and the accuracy is not sufficiently high for large scale applications in smartphones. With the rapid development of under-screen fingerprint sensing technology, a new input modality, fingerprint image, for 3D finger pose estimation is available from these fingerprint sensors. In this paper, we propose a finger specific algorithm for estimating 3D finger pose including roll, pitch, and yaw from fingerprint images. 3D finger surface is first reconstructed based on sequential fingerprint images captured in enrollment, and given this 3D surface model, 3D finger pose of a test fingerprint is estimated by matching keypoints between the 2D image and 3D point cloud and minimizing the projection error. The proposed approach is a non-learning algorithm with good generalization ability and robustness in real applications. To evaluate the performance of our method, a dataset of fingerprint images with their corresponding ground truth 3D angles is collected. Experimental results on this dataset demonstrate the effectiveness of introducing reconstructed 3D finger surface shape in 3D finger pose estimation. The average absolute errors of three angles are 10.74 for roll, 8.25 for pitch, and 7.38 for yaw, respectively. Extensive experiments are also conducted to explore the impact of touching area size and gallery size on performance.2022YDYongjie Duan et al.Hand Gesture RecognitionHuman Pose & Activity RecognitionIUI
The Impact of Latency on Navigation in a First-Person Perspective GameCompetitive first-person shooter games are played over a network, where latency can degrade player performance. To better understand latency's impact, a promising approach is to study how latency affects individual game actions, such as moving and shooting. While target selection (aiming and shooting at an opponent) is fairly well studied, navigation (moving an avatar into position) is not. This paper presents results from a 30-person user study that evaluates the impact of latency on first-person navigation using a custom ``hide and seek'' game that isolates avatar movement in a manner intended to be similar to movement in a first-person shooter game. Analysis of the results shows latency has pronounced effects on player performance (score and seek positioning), with subjective opinions on Quality of Experience following suit.2022SLShengmei Liu et al.Worcester Polytechnic InstituteGame UX & Player BehaviorMultiplayer & Social GamesCHI
Interaction with Touch-Sensitive Knitted Fabrics: User Perceptions and Everyday Use ExperimentsRecent work has investigated the construction of touch-sensitive knitted fabrics, capable of being manufactured at scale, and having only two connections to external hardware. Additionally, several sensor design patterns and application prototypes have been introduced. Our aim is to start shaping the future of this technology according to user expectations. Through a formative focus group study, we explore users' views of using these fabrics in different contexts and discuss potential concerns and application areas. Subsequently, we take steps toward addressing relevant questions, by first providing design guidelines for application designers. Furthermore, in one user study, we demonstrate that it is possible to distinguish different swipe gestures and identify accidental contact with the sensor, a common occurrence in everyday life. We then present experiments investigating the effect of stretching and laundering of the sensors on their resistance, providing insights about considerations necessary to include in computational models.2022DMDenisa Qori McDonald et al.Drexel UniversityHaptic WearablesShape-Changing Interfaces & Soft Robotic MaterialsCHI
Towards Sign Language-Centric Design of ASL Survey ToolsQuestionnaires are fundamental learning and research tools for gathering insights and information from individuals, and now can be created easily using online tools. However, existing resources for creating questionnaires are designed for written languages (e.g. English) and do not support sign languages (e.g. American Sign Language). Sign languages (SLs) have unique visual characteristics that do not fit into user interface paradigms designed for written, text-based languages. Through a series of formative studies with the ASL signing community, this paper takes steps towards understanding the viability, potential benefit, challenges, and user interest in SL-centric surveys, a novel approach for creating questionnaires that meet the needs of deaf individuals using sign languages, without obligatory reliance on a written language to complete a questionnaire.2022SMShruti Mahajan et al.Worcester Polytechnic InstituteMultilingual & Cross-Cultural Voice InteractionVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Augmentative & Alternative Communication (AAC)CHI
Exploring A Reporting Tool to Empower Individuals with Intellectual and Developmental Disabilities to Self-Report AbuseIn the US, abuse of individuals with intellectual and developmental disabilities (I/DD) is at epidemic proportions. Further, abuse inci- dents of individuals with I/DD are woefully under-reported. We surveyed practitioners who help individuals with I/DD post-abuse to get a broader context on the problem. We found that abuse of individuals with I/DD was often reported by someone other than the survivor as survivors faced impediments in reporting. Conse- quently, we argue for developing a mobile-computing-based reporting tool for empowering individuals with I/DD to self-report abuse. Next, we conducted focus groups of individuals with I/DD to evaluate the tool’s viability, with respect to their ability to recognize/report abuse and use mobile-computing devices. We found individuals with I/DD could recognize/report abuse well when they received appropriate training. We also found individuals with I/DD could independently use their devices though they shared access to them with family. Based on these findings, we call for several lines of accessibility research in designing an abuse self-reporting tool for individuals with I/DD.2021KVKrishna Venkatasubramanian et al.University of Rhode IslandCognitive Impairment & Neurodiversity (Autism, ADHD, Dyslexia)Special Education TechnologyInclusive DesignCHI
Lower is Better? The Effects of Local Latencies on Competitive First-Person Shooter Game PlayersVideo game play is among the most popular forms of entertainment in the world and eSports is a multi-billion dollar industry. Esports gamers, and competitive gamers more broadly, want fast game systems to maximize their chances of winning. In general, the faster the game system, the lower the latency between a player's action and the intended outcome. But how much small reductions in local latencies benefit competitive players is not known. This paper presents results from a 43-person user study that evaluates the impact of system latencies for high-end gaming systems (below 125 ms) on experienced Counter-strike: Global Offensive (CS:GO) players. Analysis of the results show pronounced benefits to CS:GO player performance (accuracy and score) for even small reductions in latency, with subjective opinions on Quality of Experience following suit.2021SLShengmei Liu et al.Worcester Polytechnic InstituteGame UX & Player BehaviorMultiplayer & Social GamesCHI
reVISit: Looking Under the Hood of Interactive Visualization StudiesQuantifying user performance with metrics such as time and accuracy does not show the whole picture when researchers evaluate complex, interactive visualization tools. In such systems, performance is often influenced by different analysis strategies that statistical analysis methods cannot account for. To remedy this lack of nuance, we propose a novel analysis methodology for evaluating complex interactive visualizations at scale. We implement our analysis methods in reVISit, which enables analysts to explore participant interactions performance metrics, and responses in the context of users' analysis strategies. Replays of participant sessions can aid in identifying usability problems during pilot studies and make individual analysis processes salient. To demonstrate the applicability of reVISit to visualization studies, we analyze participant data from two published crowdsourced studies. Our findings show that reVISit can be used to reveal and describe novel interaction patterns, to analyze performance differences between different analysis strategies, and to validate or challenge design decisions.2021CNCarolina Nobre et al.Harvard UniversityInteractive Data VisualizationVisualization Perception & CognitionCHI
Exploring How Game Genre in Student-Designed Games Influences Computational Thinking DevelopmentGame design is increasingly used in modern education to foster Computational Thinking (CT). Yet, it is unclear how and if the game genre of student-designed games impact CT and programming. We explore how game genre impacts CT development and programming routines in Scratch games designed by 8th-grade students using a metrics-based approach (i.e., Dr. Scratch). Our findings show that designing particular games (e.g., action, storytelling) impact CT and programming development. We observe, for instance, that CT skills develop and consolidate fast, after which students can focus on aspects more specific to game design. Based on the results, we suggest that researchers and educators in constructionist learning consider the impact of game genre when designing game-based curricula for the learning of programming and CT.2020GTGiovanni Maria Troiano et al.Northeastern UniversityGamification DesignProgramming Education & Computational ThinkingSTEM Education & Science CommunicationCHI
Evaluating Multivariate Network Visualization Techniques Using a Validated Design and Crowdsourcing ApproachVisualizing multivariate networks is challenging because of the trade-offs necessary for effectively encoding network topology and encoding the attributes associated with nodes and edges. A large number of multivariate network visualization techniques exist, yet there is little empirical guidance on their respective strengths and weaknesses. In this paper, we describe a crowdsourced experiment, comparing node-link diagrams with on-node encoding and adjacency matrices with juxtaposed tables. We find that node-link diagrams are best suited for tasks that require close integration between the network topology and a few attributes. Adjacency matrices perform well for tasks related to clusters and when many attributes need to be considered. We also reflect on our method of using validated designs for empirically evaluating complex, interactive visualizations in a crowdsourced setting. We highlight the importance of training, compensation, and provenance tracking.2020CNCarolina Nobre et al.University of UtahInteractive Data VisualizationTime-Series & Network Graph VisualizationCHI
Friend, Collaborator, Student, Manager: How Design of an AI-Driven Game Level Editor Affects CreatorsMachine learning advances have afforded an increase in algorithms capable of creating art, music, stories, games, and more. However, it is not yet well-understood how machine learning algorithms might best collaborate with people to support creative expression. To investigate how practicing designers perceive the role of AI in the creative process, we developed a game level design tool for Super Mario Bros.-style games with a built-in AI level designer. In this paper we discuss our design of the Morai Maker intelligent tool through two mixed-methods studies with a total of over one-hundred participants. Our findings are as follows: (1) level designers vary in their desired interactions with, and role of, the AI, (2) the AI prompted the level designers to alter their design practices, and (3) the level designers perceived the AI as having potential value in their design practice, varying based on their desired role for the AI.2019MGMatthew Guzdial et al.Georgia Institute of TechnologyGame UX & Player BehaviorAI-Assisted Creative WritingCreative Collaboration & Feedback SystemsCHI
Evaluating Preference Collection Methods for Interactive Ranking AnalyticsRankings distill a large number of factors into simple comparative models to facilitate complex decision making. Yet key questions remain in the design of mixed-initiative systems for ranking, in particular how best to collect users' preferences to produce high-quality rankings that users trust and employ in the real world. To address this challenge we evaluate the relative merits of three preference collection methods for ranking in a crowdsourced study. We find that with a categorical binning technique, users interact with a large amount of data quickly, organizing information using broad strokes. Alternative interaction modes using pairwise comparisons or sub-lists result in smaller, targeted input from users. We consider how well each interaction mode addresses design goals for interactive ranking systems. Our study indicates that the categorical approach provides the best value-added benefit to users, requiring minimal effort to create sufficient training data for the underlying ranking algorithm.2019CKCaitlin Kuhlman et al.Worcester Polytechnic InstituteInteractive Data VisualizationComputational Methods in HCICHI
Bridging a Bridge: Bringing Two HCI Communities TogetherACM SIGCHI is the largest association for professionals in HCI that bridges computer science, information science, as well as the social and psychological sciences. Meanwhile, a parallel HCI community was formed in 2001 within the Association of Information Systems (AIS SIGHCI) community. While some researchers have already bridged these two HCI sub-disciplines, the history and core values of these respective fields are quite different, offering new insights for how we can move forward together to sustain the future of HCI research. The main goal of this workshop is to begin building a bridge between these two communities to maximize the relevance, rigor, and generalizability of HCI research.2018SDSoussan Djamasbi et al.Worcester Polytechnic InstituteUser Research Methods (Interviews, Surveys, Observation)Computational Methods in HCICHI
Using Animation to Alleviate Overdraw in Multiclass Scatterplot MatricesThe scatterplot matrix (SPLOM) is a commonly used technique for visualizing multiclass multivariate data. However, multiclass SPLOMs have issues with overdraw (overlapping points), and most existing techniques for alleviating overdraw focus on individual scatterplots with a single class. This paper explores whether animation using flickering points is an effective way to alleviate overdraw in these multiclass SPLOMs. In a user study with 69 participants, we found that users not only performed better at identifying dense regions using animated SPLOMs, but also found them easier to interpret and preferred them to static SPLOMs. These results open up new directions for future work on alleviating overdraw for multiclass SPLOMs, and provide insights for applying animation to alleviate overdraw in other settings.2018HCHelen Chen et al.University of San FranciscoInteractive Data VisualizationVisualization Perception & CognitionCHI