Generative Trigger-Action Programming with PlyTrigger-action programming has been a success in end-user programming. Traditionally, the simplicity of links between triggers and actions limits the expressivity of such systems. LLM-based code generation promises to enable users to specify more complex behavior in natural language. However, users need appropriate ways to understand and control this added expressive power. We introduce Ply, a system that tackles this challenge through the following techniques: (1) a layer abstraction that enables decomposition into smaller building blocks; (2) generated visualizations at each layer boundary to enable interrogation of the behavior of generated code; and (3) generated customizable parameters, with associated configuration UIs, to allow users to tune each layer’s behavior. We offer a technical evaluation of Ply, demonstrating how useful, custom programs can be authored and tested incrementally using this layered trigger-action technique. Additionally, we describe a first-use study with seven participants, demonstrating where and how Ply's generative features can affect how users build programs.2025TATimothy J. Aveni et al.Human-LLM CollaborationAI-Assisted Decision-Making & AutomationUIST
Steering Semantic Data Processing With DocWranglerUnstructured text has long been difficult to analyze automatically at scale. Large language models (LLMs) now make it possible to perform “semantic data processing,” where common data operators like map, reduce, and filter are powered by LLMs instead of traditional code. However, building these pipelines is challenging: users need to understand their data in order to build effective pipelines, but at the same time, they need pipelines to extract the information that would help them understand their data. They also have to navigate the quirks and inconsistencies of LLMs. We introduce DocWrangler, a mixed-initiative development environment designed for semantic data processing. DocWrangler offers three new features to help bridge gaps between users, their data, and their pipelines: (i) In-Situ User Notes, which lets users inspect, annotate, and keep track of observations across documents and LLM outputs; (ii) LLM-Assisted Prompt Refinement, which helps turn user notes into improved data processing operations; and (iii) LLM-Assisted Operation Decomposition, which detects when tasks are too complex for the LLM and suggests how to break them down. Our evaluation combines a user study with 10 participants and a public-facing deployment (at docetl.org/playground) with over 1,500 recorded sessions. The results show that users develop systematic strategies for working with semantic data pipelines—such as turning open-ended prompts into classifiers for easier validation, and deliberately using vague prompts to explore their data or the LLM’s behavior.2025SSShreya Shankar et al.Human-LLM CollaborationExplainable AI (XAI)Computational Methods in HCIUIST
Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and OutputsAuthoring 3D scenes is a central task for spatial computing applications. Competing visions for lowering existing barriers are (1) focus on immersive, direct manipulation of 3D content or (2) leverage AI techniques that capture real scenes (3D Radiance Fields such as, NeRFs, 3D Gaussian Splatting) and modify them at a higher level of abstraction, at the cost of high latency. We unify the complementary strengths of these approaches and investigate how to integrate generative AI advances into real-time, immersive 3D Radiance Field editing. We introduce Dreamcrafter, a VR-based 3D scene editing system that: (1) provides a modular architecture to integrate generative AI algorithms; (2) combines different levels of control for creating objects, including natural language and direct manipulation; and (3) introduces proxy representations that support interaction during high-latency operations. We contribute empirical findings on control preferences and discuss how generative AI interfaces beyond text input enhance creativity in scene editing and world building.2025CVCyrus Vachha et al.University of California, Berkeley, EECSImmersion & Presence ResearchGenerative AI (Text, Image, Music, Video)3D Modeling & AnimationCHI
Rambler: Supporting Writing With Speech via LLM-Assisted Gist ManipulationDictation enables efficient text input on mobile devices. However, writing with speech can produce disfluent, wordy, and incoherent text and thus requires heavy post-processing. This paper presents Rambler, an LLM-powered graphical user interface that supports gist-level manipulation of dictated text with two main sets of functions: gist extraction and macro revision. Gist extraction generates keywords and summaries as anchors to support the review and interaction with spoken text. LLM-assisted macro revisions allow users to respeak, split, merge, and transform dictated text without specifying precise editing locations. Together they pave the way for interactive dictation and revision that help close gaps between spontaneously spoken words and well-structured writing. In a comparative study with 12 participants performing verbal composition tasks, \tool outperformed the baseline of a speech-to-text editor + ChatGPT, as it better facilitates iterative revisions with enhanced user control over the content while supporting surprisingly diverse user strategies.2024SLSusan Lin et al.UC BerkeleyVoice User Interface (VUI) DesignGenerative AI (Text, Image, Music, Video)Human-LLM CollaborationCHI
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM PromptsPre-trained large language models ("LLMs") like GPT-3 can engage in fluent, multi-turn instruction-taking out-of-the-box, making them attractive materials for designing natural language interactions. Using natural language to steer LLM outputs ("prompting") has emerged as an important design technique potentially accessible to non-AI-experts. Crafting effective prompts can be challenging, however, and prompt-based interactions are brittle. Here, we explore whether non-AI-experts can successfully engage in "end-user prompt engineering" using a design probe—a prototype LLM-based chatbot design tool supporting development and systematic evaluation of prompting strategies. Ultimately, our probe participants explored prompt designs opportunistically, not systematically, and struggled in ways echoing end-user programming systems and interactive machine learning systems. Expectations stemming from human-to-human instructional experiences, and a tendency to overgeneralize, were barriers to effective prompt design. These findings have implications for non-AI-expert-facing LLM-based tool design and for improving LLM-and-prompt literacy among programmers and the public, and present opportunities for further research.2023JZJ.D. Zamfirescu-Pereira et al.BerkeleyHuman-LLM CollaborationAI-Assisted Decision-Making & AutomationCHI
Concept-Labeled Examples for Library ComparisonProgrammers often rely on online resources—such as code examples, documentation, blogs, and Q&A forums—to compare similar libraries and select the one most suitable for their own tasks and contexts. However, this comparison task is often done in an ad-hoc manner, which may result in suboptimal choices. Inspired by Analogical Learning and Variation Theory, we hypothesize that rendering many concept-annotated code examples from different libraries side-by-side can help programmers (1) develop a more comprehensive understanding of the libraries' similarities and distinctions and (2) make more robust, appropriate library selections. We designed a novel interactive interface, ParaLib, and used it as a technical probe to explore to what extent many side-by-side concept-annotated examples can facilitate the library comparison and selection process. A within-subjects user study with 20 programmers shows that, when using ParaLib, participants made more consistent, suitable library selections and provided more comprehensive summaries of libraries' similarities and differences.2022LYLitao Yan et al.Knowledge Worker Tools & WorkflowsPrototyping & User TestingUIST
Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency AnalysisUI designers often correct false affordances and improve the discoverability of features when users have trouble determining if elements are tappable. We contribute a novel system that models the perceived tappability of mobile UI elements with a vision-based deep neural network and helps provide design insights with dataset-level and instance-level explanations of model predictions. Our system retrieves designs from similar mobile UI examples from our dataset using the latent space of our model. We also contribute a novel use of an interpretability algorithm, XRAI, to generate a heatmap of UI elements that contribute to a given tappability prediction. Through several examples, we show how our system can help automate elements of UI usability analysis and provide insights for designers to iterate their designs. In addition, we share findings from an exploratory evaluation with professional designers to learn how AI-based tools can aid UI design and evaluation for tappability issues.2022ESEldon Schoop et al.University of California, BerkeleyExplainable AI (XAI)AI-Assisted Decision-Making & AutomationCHI
UMLAUT: Debugging Deep Learning Programs using Program Structure and Model BehaviorTraining deep neural networks can generate non-descriptive error messages or produce unusual output without any explicit errors at all. While experts rely on tacit knowledge to apply debugging strategies, non-experts lack the experience required to interpret model output and correct Deep Learning (DL) programs. In this work, we identify DL debugging heuristics and strategies used by experts, and use them to guide the design of Umlaut. Umlaut checks DL program structure and model behavior against these heuristics; provides human-readable error messages to users; and annotates erroneous model output to facilitate error correction. Umlaut links code, model output, and tutorial-driven error messages in a single interface. We evaluated Umlaut in a study with 15 participants to determine its effectiveness in helping developers find and fix errors in their DL programs. Participants using Umlaut found and fixed significantly more bugs compared to a baseline condition.2021ESEldon Schoop et al.University of California, BerkeleyExplainable AI (XAI)AI-Assisted Decision-Making & AutomationCHI
TransceiVR: Bridging Asymmetrical Communication Between External and VR UsersVirtual Reality (VR) users often need to work with other users, who observe them outside of VR using an external display. Communication between them is difficult; the VR user cannot see the external user’s gestures, and the external user cannot see VR scene elements outside of the VR user’s view. We carried out formative interviews with experts to understand these asymmetrical interactions and identify their goals and challenges. From this, we identify high-level system design goals to facilitate asymmetrical interactions and a corresponding space of implementation approaches based on the level of programmatic access to a VR application. We present TransceiVR, a system that utilizes VR platform APIs to enable asymmetric communication interfaces for third-party applications without requiring source code access. TransceiVR allows external users to explore the VR scene spatially or temporally, to annotate elements in the VR scene at correct depths, and to discuss via a shared static virtual display. An initial co-located user evaluation with 10 pairs shows that our system makes asymmetric collaborations in VR more effective and successful in terms of task time, error rate, and task load index. An informal evaluation with a remote expert gives additional insight on utility of features for real world tasks.2020BKBalasaravanan Thoravi Kumaravel et al.Social & Collaborative VRMixed Reality WorkspacesUIST
Polymorphic Blocks: unifying high-level specification and low-level control for circuit board designMainstream board-level circuit design tools work at the lowest level of design — schematics and individual components. While novel tools are experimenting with higher levels of design, abstraction often comes at the expense of the fine-grained control afforded by low-level tools. In this work, we propose a hardware description language (HDL) approach that supports users at multiple levels of abstraction from broad system architecture to subcircuits and component selection. We extend the familiar hierarchical block diagram with polymorphism to include abstract-typed blocks (e.g., generic resistor supertype) and electronics modeling (i.e., currents and voltages). Such an approach brings the advantages of reusability and encapsulation from object-oriented programming, while addressing the unique needs of electronics designers such as physical correctness verification. We discuss the system design, including fundamental abstractions, the block diagram construction HDL, and user interfaces to inspect and fine-tune the design; demonstrate example designs built with our system; and present feedback from intermediate-level engineers who have worked with our system.2020RLRichard Lin et al.Circuit Making & Hardware PrototypingUIST
Composing Flexibly-Organized Step-by-Step Tutorials from Linked Source Code, Snippets, and OutputsProgramming tutorials are a pervasive, versatile medium for teaching programming. In this paper, we report on the content and structure of programming tutorials, the pain points authors experience in writing them, and a design for a tool to help improve this process. An interview study with 12 experienced tutorial authors found that they construct documents by interleaving code snippets with text and illustrative outputs. It also revealed that authors must often keep related artifacts of source programs, snippets, and outputs consistent as a program evolves. A content analysis of 200 frequently-referenced tutorials on the web also found that most tutorials contain related artifactsduplicate code and outputs generated from snippetsthat an author would need to keep consistent with each other. To address these needs, we designed a tool called Torii with novel authoring capabilities. An in-lab study showed that tutorial authors can successfully use the tool for the unique affordances identified, and provides guidance for designing future tools for tutorial authoring.2020AHAndrew Head et al.University of California, BerkeleyProgramming Education & Computational ThinkingPrototyping & User TestingCHI
Heimdall: A Remotely Controlled Inspection Workbench For Debugging Microcontroller ProjectsStudents and hobbyists build embedded systems that combine sensing, actuation and microcontrollers on solderless breadboards. To help students debug such circuits, experienced teachers apply visual inspection, targeted measurements, and circuit modifications to diagnose and localize the problem(s). However, experienced helpers may not always be available to review student projects in person. To enable remote debugging of circuit problems, we introduce Heimdall, a remote electronics workbench that allows experts to visually inspect a student's circuit; perform measurements; and to re-wire and inject test signals. These interactions are enabled by an actuated inspection camera; an augmented breadboard that enables flexible configuration of row connectivity and measurement/injection lines; and a web-based UI that teachers can use to perform measurements through interaction with the captured images. We demonstrate that common issues arising in embedded electronics classes can be successfully diagnosed remotely and report on preliminary user feedback from teaching assistants who frequently debug circuits.2019MKMitchell Karchemsky et al.University of California, BerkeleyDesktop 3D Printing & Personal FabricationCircuit Making & Hardware PrototypingKnowledge Worker Tools & WorkflowsCHI
Beyond Schematic Capture: Meaningful Abstractions for Better Electronics Design ToolsPrinted Circuit Board (PCB) design tools are critical in helping users build non-trivial electronics devices. While recent work recognizes deficiencies with current tools and explores novel methods, little has been done to understand modern designers and their needs. To gain better insight into their practices, we interview fifteen electronics designers of a variety of backgrounds. Our open-ended, semi-structured interviews examine both overarching design flows and details of individual steps. One major finding was that most creative engineering work happens during system architecture, yet current tools operate at lower abstraction levels and create significant tedious work for designers. From that insight, we conceptualize abstractions and primitives for higher-level tools and elicit feedback from our participants on clickthrough mockups of design flows through an example project. We close with our observation on opportunities for improving board design tools and discuss generalizability of our findings beyond the electronics domain.2019RLRichard Lin et al.University of California, BerkeleyCircuit Making & Hardware PrototypingCustomizable & Personalized ObjectsCHI
HindSight: Enhancing Spatial Awareness by SonifyingDetected Objects in Real-Time 360-Degree VideoOur perception of our surrounding environment is limited by the constraints of human biology. The field of augmented perception asks how our sensory capabilities can be usefully extended through computational means. We argue that spatial awareness can be enhanced by exploiting recent advances in computer vision which make high-accuracy, real-time object detection feasible in everyday settings. We introduce HindSight, a wearable system that increases spatial awareness by detecting relevant objects in live 360-degree video and sonifying their position and class through bone conduction headphones. HindSight uses a deep neural network to locate and attribute semantic information to objects surrounding a user through a head-worn panoramic camera. It then uses bone conduction headphones, which preserve natural auditory acuity, to transmit audio notifications for detected objects of interest. We develop an application using HindSight to warn cyclists of approaching vehicles outside their field of view and evaluate it in an exploratory study with 15 users. Participants reported increases in perceived safety and awareness of approaching vehicles when using HindSight.2018ESEldon Schoop et al.BerkeleyVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Biosensors & Physiological MonitoringCHI
WiFröst: Bridging the Information Gap for Debugging of Networked Embedded SystemsThe rise in prevalence of Internet of Things (IoT) technologies has encouraged more people to prototype and build custom internet connected devices based on low power microcontrollers. While well-developed tools exist for debugging network communication for desktop and web applications, it can be difficult for developers of networked embedded systems to figure out why their network code is failing due to the limited output affordances of embedded devices. This paper presents WiFröst , a new approach for debugging these systems using instrumentation that spans from the device itself, to its communication API, to the wireless router and back-end server. WiFröst automatically collects this data, displays it in a web-based visualization, and highlights likely issues with an extensible suite of checks based on analysis of recorded execution traces.2018WMWilliam McGrath et al.IoT Device PrivacyContext-Aware ComputingCircuit Making & Hardware PrototypingUIST