Feminist Interaction Techniques: Social Consent Signals to Deter NCIM ScreenshotsNon-consensual Intimate Media (NCIM) refers to the distribution of sexual or intimate content without consent. NCIM is common and causes significant emotional, financial, and reputational harm. We developed Hands-Off, an interaction technique for messaging applications that deters non-consensual screenshots. Hands-Off requires recipients to perform a hand gesture in the air, above the device, to unlock media—which makes simultaneous screenshotting difficult. A lab study shows that Hands-Off gestures are easy to perform and reduce non-consensual screenshots by 67%. We conclude by generalizing this approach and introduce the idea of Feminist Interaction Techniques (FIT), interaction techniques that encode feminist values and speak to societal problems, and reflect on FIT’s opportunities and limitations.2024LQLi Qiwei et al.Cyberbullying & Online HarassmentEmpowerment of Marginalized GroupsTechnology Ethics & Critical HCIUIST
Authors' Values and Attitudes Towards AI-bridged Scalable Personalization of Creative Language ArtsGenerative AI has the potential to create a new form of interactive media: AI-bridged creative language arts (CLA), which bridge the author and audience by personalizing the author's vision to the audience's context and taste at scale. However, it is unclear what the authors' values and attitudes would be regarding AI-bridged CLA. To identify these values and attitudes, we conducted an interview study with 18 authors across eight genres (e.g., poetry, comics) by presenting speculative but realistic AI-bridged CLA scenarios. We identified three benefits derived from the dynamics between author, artifact, and audience: those that 1) authors get from the process, 2) audiences get from the artifact, and 3) authors get from the audience. We found how AI-bridged CLA would either promote or reduce these benefits, along with authors' concerns. We hope our investigation hints at how AI can provide intriguing experiences to CLA audiences while promoting authors' values.2024TKTaewook Kim et al.Northwestern UniversityGenerative AI (Text, Image, Music, Video)AI-Assisted Creative WritingCHI
PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like InteractionsWhile diffusion-based text-to-image (T2I) models provide a simple and powerful way to generate images, guiding this generation remains a challenge. For concepts that are difficult to describe through language, users may struggle to create prompts. Moreover, many of these models are built as end-to-end systems, lacking support for iterative shaping of the image. In response, we introduce PromptPaint, which combines T2I generation with interactions that model how we use colored paints. PromptPaint allows users to go beyond language and mix prompts to express challenging concepts. Just as we iteratively tune colors by the layered placement of paint on a physical canvas, PromptPaint similarly allows users to apply different prompts to different parts of the generative process and canvas areas. Through a set of studies, we characterize different approaches for mixing prompts, design trade-offs, and technical and socio-technical challenges to the use of generative models. With PromptPaint we provide insight into future steerable generative tools.2023JCJohn Joon Young Chung et al.Generative AI (Text, Image, Music, Video)AI-Assisted Creative WritingUIST
Artinter: AI-powered Boundary Objects for Commissioning Visual ArtsWhen commissioning visual art, clients and artists communicate to agree on what is to be created. This often requires bridging a language gap in how they conceive art. To arrive at a mutual understanding, they leverage boundary objects---organized language and artifact instances. However, building and working with such objects is hard due to their innate subjectivity and ambiguity. Moreover, acquiring artifact instances, such as references and sketches, requires effort. We introduce Artinter, an AI-powered commission-support system for sharing, concretizing, and expanding boundary objects. Artinter helps artists and clients develop a mutually understood `language' by allowing them to define concepts with artifacts (e.g., what they mean by `happy’). The system provides two AI-powered approaches for expanding commission boundary objects: 1) guided search with user-defined concepts and 2) instance generation by mixing concepts and artifacts. Our studies identify how AI features can support commissions and reveal future directions for AI-powered collaborative art-making.2023JCMichelle Chung et al.Generative AI (Text, Image, Music, Video)Creative Collaboration & Feedback SystemsDIS
VideoSticker: A Tool for Active Viewing and Visual Note-taking from VideosVideo is an effective medium for knowledge communication and learning. Yet active viewing and note-taking from videos remain a challenge. Specifically, during note-taking, viewers find it difficult to extract essential information such as representation, composition, motion, and interactions of graphical objects and narration. Current approaches rely on creating static screenshots, manual clipping, and manual annotation/transcription. Additionally, note-takers may need to repeatedly pause and rewind the video, disrupting their active viewing process. We propose VideoSticker, a tool designed to support visual note-taking by extracting expressive content from videos as 'motion stickers'. VideoSticker implements automated object detection and tracking, linking objects to the transcript, and rapid extraction of stickers across space, time, and events of interest. VideoSticker's two-pass approach allows viewers to capture high-level information uninterrupted and later extract specific details. We demonstrate the usability of VideoSticker for a variety of videos and note-taking needs.2022YCYining Cao et al.Recommender System UXData StorytellingIUI
Solving Separation-of-Concerns Problems in Collaborative Design of Human-AI Systems through Leaky AbstractionsIn conventional software development, user experience (UX) designers and engineers collaborate through separation of concerns (SoC): designers create human interface specifications, and engineers build to those specifications. However, we argue that Human-AI systems thwart SoC because human needs must shape the design of the AI interface, the underlying AI sub-components, and training data. How do designers and engineers currently collaborate on AI and UX design? To find out, we interviewed 21 industry professionals (UX researchers, AI engineers, data scientists, and managers) across 14 organizations about their collaborative work practices and associated challenges. We find that hidden information encapsulated by SoC challenges collaboration across design and engineering concerns. Practitioners describe inventing ad-hoc representations exposing low-level design and implementation details (which we characterize as leaky abstractions) to "puncture" SoC and share information across expertise boundaries. We identify how leaky abstractions are employed to collaborate at the AI-UX boundary and formalize a process of creating and using leaky abstractions.2022HSHariharan Subramonyam et al.Stanford UniversityHuman-LLM CollaborationExplainable AI (XAI)AI-Assisted Decision-Making & AutomationCHI
FlatMagic: Improving Flat Colorization through AI-driven Design for Digital Comic ProfessionalsCreating digital comics involves multiple stages, some creative and some menial. For example, coloring a comic requires a labor-intensive stage known as 'flatting,' or masking segments of continuous color, as well as creative shading, lighting, and stylization stages. The use of AI can automate the colorization process, but early efforts have revealed limitations---technical and UX---to full automation. Via a formative study of professionals, we identify flatting as a bottleneck and key target of opportunity for human-guided AI-driven automation. Based on this insight, we built FlatMagic, an interactive, AI-driven flat colorization support tool for Photoshop. Our user studies found that using FlatMagic significantly reduced professionals' real and perceived effort versus their current practice. While participants effectively used FlatMagic, we also identified potential constraints in interactions with AI and partially automated workflows. We reflect on implications for comic-focused tools and the benefits and pitfalls of intermediate representations and partial automation in designing human-AI collaboration tools for professionals.2022CYChuan Yan et al.George Mason UniversityGenerative AI (Text, Image, Music, Video)AI-Assisted Creative WritingCHI
TaleBrush: Sketching Stories with Generative Pretrained Language ModelsWhile advanced text generation algorithms (e.g., GPT-3) have enabled writers to co-create stories with an AI, guiding the narrative remains a challenge. Existing systems often leverage simple turn-taking between the writer and the AI in story development. However, writers remain unsupported in intuitively understanding the AI’s actions or steering the iterative generation. We introduce TaleBrush, a generative story ideation tool that uses line sketching interactions with a GPT-based language model for control and sensemaking of a protagonist’s fortune in co-created stories. Our empirical evaluation found our pipeline reliably controls story generation while maintaining the novelty of generated sentences. In a user study with 14 participants with diverse writing experiences, we found participants successfully leveraged sketching to iteratively explore and write stories according to their intentions about the character’s fortune while taking inspiration from generated stories. We conclude with a reflection on how sketching interactions can facilitate the iterative human-AI co-creation process.2022JCJohn Joon Young Chung et al.University of MichiganGenerative AI (Text, Image, Music, Video)AI-Assisted Creative WritingPrototyping & User TestingCHI
Towards a Process Model for Co-Creating AI ExperiencesThinking of technology as a design material is appealing. It encourages designers to explore the material's properties to understand its capabilities and limitations--a prerequisite to generative design thinking. However, as a material, AI resists this approach because its properties only emerge as part of the user experience design. Therefore, designers and AI engineers must collaborate in new ways to create both the material and its application experience. We investigate the co-creation process through a design study with 10 pairs of designers and engineers. We find that design 'probes' with user data are a useful tool in defining AI materials. Through data probes, designers construct designerly representations of the envisioned AI experience (AIX) to identify desirable AI characteristics. Data probes facilitate divergent design thinking, material testing, and design validation. Based on our findings, we propose a process model for co-creating AIX and offer design considerations for incorporating data probes in AIX design tools.2021HSHariharan Subramonyam et al.Human-LLM CollaborationPrototyping & User TestingDIS
The Intersection of Users, Roles, Interactions, and Technologies in Creativity Support ToolsCreativity Support Tools (CSTs) have become an integral part of artistic creation. The range of CST technologies is broad---from fabricators to generative algorithms to robots. The interaction approaches for CSTs are accordingly broad. CSTs combine specific technologies and interaction types to serve a spectrum of roles and users. In this work, we tackle a comprehensive understanding of how the intersections of users, roles, interactions, and technologies form a design space for CSTs. We accomplish this by reviewing 111 art-creation CSTs from HCI and computing research and analyzing how diverse aspects of CSTs relate to each other. Our findings identify patterns for designing CSTs, which can give guidance to future CST designers. We also highlight under-explored types of CSTs within the HCI community, providing future directions that CST researchers can pursue given the current trajectory of technological advancement. This work contributes an integrating perspective to understand the landscape of art-creation CSTs.2021JCJohn Joon Young Chung et al.EV Charging & Eco-Driving InterfacesElectrical Muscle Stimulation (EMS)Brain-Computer Interface (BCI) & NeurofeedbackDIS
ProtoAI: Model-Informed Prototyping for AI-Powered InterfacesWhen prototyping AI-powered interfaces, designers seek to build useful and usable ways to support end-user tasks with AI capabilities. In the process, they need to consider the potential capabilities and constraints of the AI services and build interface adaptations such as explainability, error recovery, and feedback. Unfortunately, in assuming a black-box view of AI, current prototyping tools fail to support the design of AI capabilities and adaptations. Designers need to work with separate tools to explore machine learning models, understand model performance across diverse inputs, and align model behavior and interface choices. This introduces friction to rapid and iterative prototyping. We propose Model-Informed Prototyping (MIP), a workflow that combines model exploration with UI prototyping tasks. Our system, ProtoAI, allows designers to directly incorporate model outputs into interface design, evaluate design choices across different inputs, and iteratively revise their design by analyzing model breakdowns. We demonstrate how ProtoAI can readily operationalize several human-AI design guidelines. Our user study finds that designers can effectively engage in MIP to create AI-infused prototypes.2021HSHariharan Subramonyam et al.Generative AI (Text, Image, Music, Video)AI-Assisted Decision-Making & AutomationPrototyping & User TestingIUI
Plotting with Thread: Fabricating Delicate Punch Needle Embroidery with X-Y PlottersPunch needle embroidery is a unique type of embroidery that uses loops of threads to create designs. Technology for punch needle embroidery ranges from the popular handheld manual tools to high-cost industrial tufting machines. Computer-controlled punch needle fabrication tools remain out-of-reach for most practitioners. In this work, we describe how a low-cost X-Y plotter can be repurposed to support punch needle embroidery fabrication. By adding easy-to-make physical accessories coupled with a novel software toolkit, we support the production of delicate and precise punch needle embroideries with minimal manual labor. After examining and evaluating the potential and challenges of converting X-Y plotters into punch needle embroidery fabricators, we propose guidelines that are specific to plotter-based punch needle embroideries. We demonstrate how this novel fabrication approach enables the production of a wide range of artifacts and textures.2020SHMegh Marathe et al.Circuit Making & Hardware PrototypingCustomizable & Personalized ObjectsTextile Art & Craft DigitizationDIS
texSketch: Active Diagramming through Pen-and-Ink AnnotationsLearning from text is a constructive activity in which sentence-level information is combined by the reader to build coherent mental models. With increasingly complex texts, forming a mental model becomes challenging due to a lack of background knowledge, and limits in working memory and attention. To address this, we are taught knowledge externalization strategies such as active reading and diagramming. Unfortunately, paper-and-pencil approaches may not always be appropriate, and software solutions create friction through difficult input modalities, limited workflow support, and barriers between reading and diagramming. For all but the simplest text, building coherent diagrams can be tedious and difficult. We propose Active Diagramming, an approach extending familiar active reading strategies to the task of diagram construction. Our prototype, texSketch, combines pen-and-ink interactions with natural language processing to reduce the cost of producing diagrams while maintaining the cognitive effort necessary for comprehension. Our user study finds that readers can effectively create diagrams without disrupting reading.2020HSHariharan Subramonyam et al.University of MichiganInteractive Data VisualizationUser Research Methods (Interviews, Surveys, Observation)CHI
Discovering Natural Language Commands in Multimodal InterfacesDiscovering what to say and how to say it remains a challenge for users of multimodal interfaces supporting speech input. Users end up "guessing" commands that a system might support, often leading to interpretation errors and frustration. One solution to this problem is to display contextually relevant command examples as users interact with a system. The challenge, however, is deciding when, how, and which examples to recommend. In this work, we describe an approach for generating and ranking natural language command examples in multimodal interfaces. We demonstrate the approach using a prototype touch- and speech-based image editing tool. We experiment with augmentations of the UI to understand when and how to present command examples. Through an online user study, we evaluate these alternatives and find that in-situ command suggestions promote discovery and encourage the use of speech input.2019ASArjun Srinivasan et al.Voice User Interface (VUI) DesignIntelligent Voice Assistants (Alexa, Siri, etc.)Prototyping & User TestingIUI
Affinity Lens: Data-Assisted Affinity Diagramming with Augmented RealityDespite the availability of software to support Affinity Diagramming (AD), practitioners still largely favor physical sticky-notes. Physical notes are easy to set-up, can be moved around in space and offer flexibility when clustering un-structured data. However, when working with mixed data sources such as surveys, designers often trade off the physicality of notes for analytical power. We propose AffinityLens, a mobile-based augmented reality (AR) application for Data-Assisted Affinity Diagramming (DAAD). Our application provides just-in-time quantitative insights overlaid on physical notes. Affinity Lens uses several different types of AR overlays (called lenses) to help users find specific notes, cluster information, and summarize insights from clusters. Through a formative study of AD users, we developed design principles for data-assisted AD and an initial collection of lenses. Based on our prototype, we find that Affinity Lens supports easy switching between qualitative and quantitative 'views' of data, without surrendering the lightweight benefits of existing AD practice.2019HSHariharan Subramonyam et al.University of MichiganMixed Reality WorkspacesInteractive Data VisualizationContext-Aware ComputingCHI
Vocal Shortcuts for Creative ExpertsVocal shortcuts, short spoken phrases to control interfaces, have the potential to reduce cognitive and physical costs of interactions. They may benefit expert users of creative applications (e.g., designers, illustrators) by helping them maintain creative focus. To aid the design of vocal shortcuts and gather use cases and design guidelines for speech interaction, we interviewed ten creative experts. Based on our findings, we built VoiceCuts, a prototype implementation of vocal shortcuts in the context of an existing creative application. In contrast to other speech interfaces, VoiceCuts targets experts' unique needs by handling short and partial commands and leverages document model and application context to disambiguate user utterances. We report on the viability and limitations of our approach based on feedback from creative experts.2019YKYea-Seul Kim et al.University of WashingtonVoice User Interface (VUI) DesignMusic Composition & Sound Design ToolsCHI
TakeToons: Script-driven Performance AnimationPerformance animation is an expressive method for animating characters through human performance. However, character motion is only one part of creating animated stories. The typical workflow also involves writing a script, coordinating actors, and editing recorded performances. In most cases, these steps are done in isolation with separate tools, which introduces friction and hinders iteration. We propose TakeToons, a script-driven approach that allows authors to annotate standard scripts with relevant animation events like character actions, camera positions, and scene backgrounds. We compile this script into a story model that persists throughout the production process and provides a consistent structure for organizing and assembling recorded performances and propagating script or timing edits to existing recordings. TakeToons enables writing, performing and editing to happen in an integrated and interleaved manner that streamlines production and facilitates iteration. Informal feedback from professional animators suggests that our approach can benefit many existing workflows supporting individual authors and production teams with many different contributors.2018HSHariharan Subramonyam et al.3D Modeling & AnimationInteractive Narrative & Immersive StorytellingUIST
Identifying Misaligned Inter-Group Links and CommunitiesMany social media systems explicitly connect individuals (e.g., Facebook or Twitter); as a result, they are the targets of most research on social networks. However, many systems do not emphasize or support explicit linking between people (e.g., Wikipedia or Reddit), and even fewer explicitly link communities. Instead, network analysis is performed through inference on implicit connections, such as co-authorship or text similarity. Depending on how inference is done and what data drove it, different networks may emerge. While correlated structures often indicate stability, in this work we demonstrate that differences, or misalignment, between inferred networks also capture interesting behavioral patterns. For example, high-text but low-author similarity often reveals communities "at war'' with each other over an issue or high-author but low-text similarity can suggest community fragmentation. Because we are able to model edge direction, we also find that asymmetry in degree (in-versus-out) co-occurs with marginalized identities (subreddits related to women, people of color, LGBTQ, etc.). In this work, we provide algorithms that can identify misaligned links, network structures and communities. We then apply these techniques to Reddit to demonstrate how these algorithms can be used to decipher inter-group dynamics in social media.2018SDSrayan Datta et al.Relationships across PlatformsCSCW