Meridian: A Design Framework for Malleable Overview-Detail InterfacesOverview-detail interfaces (ODIs), which present an overview of multiple items alongside a detailed view of a selected item, are ubiquitously implemented in software interfaces. However, the current design and development pipeline lacks the infrastructure to easily support end-user customization, limiting its ability to support diverse information needs. This research envisions a development cycle for building malleable interfaces—one where designers, developers, and end-users alike can create, modify, and use the interface equally. To establish a foundation for this infrastructure, we introduce Meridian, a design framework for guiding and facilitating the creation of malleable ODIs. The framework consists of a high-level declarative specification language for ODIs as well as its tools, including a UI development package and a no-code web builder to facilitate the development and design of malleable ODIs. We demonstrate how Meridian supports designers, developers, and end-users alike in designing, implementing, and interacting with ODIs in novel ways using their respective familiar tools and platforms. Finally, we discuss technical tradeoffs, potential solutions, and opportunities for enabling malleability for interfaces by default.2025BMBryan Min et al.Interactive Data VisualizationKnowledge Worker Tools & WorkflowsUIST
Less or More: Towards Glanceable Explanations for LLM Recommendations Using Ultra-Small DevicesLarge Language Models (LLMs) have shown remarkable potential in recommending everyday actions as personal AI assistants, while Explainable AI (XAI) techniques are being increasingly utilized to help users understand why a recommendation is given. Personal AI assistants today are often located on ultra-small devices such as smartwatches, which have limited screen space. The verbosity of LLM-generated explanations, however, makes it challenging to deliver glanceable LLM explanations on such ultra-small devices. To address this, we explored 1) spatially structuring an LLM’s explanation text using defined contextual components during prompting and 2) presenting temporally adaptive explanations to users based on confidence levels. We conducted a user study to understand how these approaches impacted user experiences when interacting with LLM recommendations and explanations on ultra-small devices. The results showed that structured explanations reduced users’ time to action and cognitive load when reading an explanation. Always-on structured explanations increased users’ acceptance of AI recommendations. However, users were less satisfied with structured explanations compared to unstructured ones due to their lack of sufficient, readable details. Additionally, adaptively presenting structured explanations was less effective at improving user perceptions of the AI compared to the always-on structured explanations. Together with users' interview feedback, the results led to design implications to be mindful of when personalizing the content and timing of LLM explanations that are displayed on ultra-small devices.2025XWXinru Wang et al.Human-LLM CollaborationExplainable AI (XAI)IUI
Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming SupportAI programming tools enable powerful code generation, and recent prototypes attempt to reduce user effort with proactive AI agents, but their impact on programming workflows remains unexplored. We introduce and evaluate Codellaborator, a design probe LLM agent that initiates programming assistance based on editor activities and task context. We explored three interface variants to assess trade-offs between increasingly salient AI support: prompt-only, proactive agent, and proactive agent with presence and context (Codellaborator). In a within-subject study (N=18), we find that proactive agents increase efficiency compared to prompt-only paradigm, but also incur workflow disruptions. However, presence indicators and interaction context support alleviated disruptions and improved users' awareness of AI processes. We underscore trade-offs of Codellaborator on user control, ownership, and code understanding, emphasizing the need to adapt proactivity to programming processes. Our research contributes to the design exploration and evaluation of proactive AI systems, presenting design implications on AI-integrated programming workflow.2025KPKevin Pu et al.University of Toronto, Department of Computer ScienceHuman-LLM CollaborationAI-Assisted Decision-Making & AutomationPrototyping & User TestingCHI
The Shapes of Abstraction in Data Structure DiagramsTools to inspect runtime state, like print statements and debuggers, are an essential part of programming. Yet, a major limitation is that they present data at a fixed, low level of abstraction which can overload the user with irrelevant details. In contrast, human drawings of data structures use many illustrative visual abstractions to show the most useful information. We attempt to bridge the gap by surveying 80 programmer-produced diagrams to develop a mechanical approach for capturing visual abstraction, termed abstraction moves. An abstraction move selects data objects of interest, and then revisualizes, simplifies, or annotates them. We implement these moves as a diagramming language for JavaScript code, named Chisel, and show that it can effectively reproduce 78 out of the 80 surveyed diagrams. In a preliminary study with four CS educators, we evaluate its usage and discover potential contexts of use. Our approach of mechanically moving between levels of abstraction in data displays opens the doors to new tools and workflows in programming education and software development.2025DHDevamardeep Hayatpur et al.University of California, San Diego, Department of Cognitive Science and Design LabInteractive Data VisualizationProgramming Education & Computational ThinkingPrototyping & User TestingCHI
Compositional Structures as Substrates for Human-AI Co-creation Environment: A Design Approach and A Case StudyIt has been increasingly recognized that effective human-AI co-creation requires more than prompts and results, but an environment with empowering structures that facilitate exploration, planning, iteration, as well as control and inspection of AI generation. Yet, a concrete design approach to such an environment has not been established. Our literature analysis highlights that compositional structures—which organize and visualize individual elements into meaningful wholes—are highly effective in granting creators control over the essential aspects of their content. However, efficiently aggregating and connecting these structures to support the full creation process remains challenging. We, therefore, propose a design approach of leveraging compositional structures as the substrates and infusing AI within and across these structures to enable controlled and fluid creation process. We evaluate this approach through a case study of developing a video co-creation environment using this approach. User evaluation shows that such an environment allowed users to stay oriented in their creation activity, remain aware and in control of AI’s generation, and enable flexible human-AI collaborative workflows.2025YCYining Cao et al.University of California, San DiegoGenerative AI (Text, Image, Music, Video)Creative Collaboration & Feedback SystemsCHI
Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data ModelUnlike static and rigid user interfaces, generative and malleable user interfaces offer the potential to respond to diverse users’ goals and tasks. However, current approaches primarily rely on generating code, making it difficult for end-users to iteratively tailor the generated interface to their evolving needs. We propose employing task-driven data models—representing the essential information entities, relationships, and data within information tasks—as the foundation for UI generation. We leverage AI to interpret users’ prompts and generate the data models that describe users’ intended tasks, and by mapping the data models with UI specifications, we can create generative user interfaces. End-users can easily modify and extend the interfaces via natural language and direct manipulation, with these interactions translated into changes in the underlying model. The technical evaluation of our approach and user evaluation of the developed system demonstrate the feasibility and effectiveness of generative and malleable user interfaces.2025YCYining Cao et al.University of California, San DiegoGenerative AI (Text, Image, Music, Video)Human-LLM CollaborationCHI
CoLadder: Manipulating Code Generation via Multi-Level BlocksThis paper adopted an iterative design process to gain insights into programmers' strategies when using LLMs for programming. We proposed CoLadder, a novel system that supports programmers by facilitating hierarchical task decomposition, direct code segment manipulation, and result evaluation during prompt authoring. A user study with 12 experienced programmers showed that CoLadder is effective in helping programmers externalize their problem-solving intentions flexibly, improving their ability to evaluate and modify code across various abstraction levels, from their task's goal to final code implementation.2024RYRyan Yen et al.Human-LLM CollaborationComputational Methods in HCIUIST
CrossTalk: Intelligent Substrates for Language-Oriented Interaction in Video-Based Communication and CollaborationDespite the advances and ubiquity of digital communication media such as videoconferencing and virtual reality, they remain oblivious to the rich intentions expressed by users. Beyond transmitting audio, videos, and messages, we envision digital communication media as proactive facilitators that can provide unobtrusive assistance to enhance communication and collaboration. Informed by the results of a formative study, we propose three key design concepts to explore the systematic integration of intelligence into communication and collaboration, including the panel substrate, language-based intent recognition, and lightweight interaction techniques. We developed CrossTalk, a videoconferencing system that instantiates these concepts, which was found to enable a more fluid and flexible communication and collaboration experience.2023HXHaijun Xia et al.Conversational ChatbotsRemote Work Tools & ExperienceUIST
Color Field: Developing Professional Vision by Visualizing the Effects of Color FiltersColor filters are ubiquitous across visual digital media due to their transformative effect. However, it can be difficult to understand how a color filter will affect an image, especially for novices. In order to become experts, we argue that novices need to develop Goodwin’s notion of Professional Vision. Then, they can "see" and interpret their work in terms of their domain knowledge like experts. Using the theory of Professional Vision, we present two design objectives for systems that aim to help users develop expertise. These goals were used to develop Color Field, an interactive visualization of color filters as a vector field over the Hue-Saturation-Lightness color space. We conducted an exploratory user study in which five color grading novices and four experts were asked to analyze color filters. We found that Color Field enabled multiple strategies to make sense of filters (e.g. reviewing the overall shape of the vector field) and discuss them (e.g. using spatial language). We conclude with other applications of Color Field and future work to leverages Professional Vision in HCI.2023MBMatthew T Beaudouin-Lafon et al.Visualization Perception & CognitionGraphic Design & Typography ToolsUIST
Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language ModelsPeople are increasingly turning to large language models (LLMs) for complex information tasks like academic research or planning a move to another city. However, while they often require working in a nonlinear manner --- e.g., to arrange information spatially to organize and make sense of it, current interfaces for interacting with LLMs are generally linear to support conversational interaction. To address this limitation and explore how we can support LLM-powered exploration and sensemaking, we developed Sensecape, an interactive system designed to support complex information tasks with an LLM by enabling users to (1) manage the complexity of information through multilevel abstraction and (2) switch seamlessly between foraging and sensemaking. Our within-subject user study reveals that Sensecape empowers users to explore more topics and structure their knowledge hierarchically, thanks to the externalization of levels of abstraction. We contribute implications for LLM-based workflows and interfaces for information tasks.2023SSSangho Suh et al.Human-LLM CollaborationUser Research Methods (Interviews, Surveys, Observation)UIST
DataParticles: Block-based and Language-oriented Authoring of Animated Unit VisualizationUnit visualizations have been widely used in data storytelling within interactive articles and videos. However, authoring data stories that contain animated unit visualizations is challenging due to the tedious, time-consuming process of switching back and forth between writing a narrative and configuring the accompanying visualizations and animations. To streamline this process, we present DataParticles, a block-based story editor that leverages the latent connections between text, data, and visualizations to help creators flexibly prototype, explore, and iterate on a story narrative and its corresponding visualizations. To inform the design of DataParticles, we interviewed 6 domain experts and studied a dataset of 44 existing animated unit visualizations to identify the narrative patterns and congruence principles they employed. A user study with 9 experts showed that DataParticles can significantly simplify the process of authoring data stories with animated unit visualizations by encouraging exploration and supporting fast prototyping.2023YCYining Cao et al.University of California, San DiegoInteractive Data VisualizationData Storytelling3D Modeling & AnimationCHI
Log-it: Supporting Programming with Interactive, Contextual, Structured, and Visual LogsLogging is a widely used technique for inspecting and understanding programs. However, the presentation of logs still often takes its ancient form of a linear stream of text that resides in a terminal, console, or log file. Despite its simplicity, interpreting log output is often challenging due to the large number of textual logs that lack structure and context. We conducted content analysis and expert interviews to understand the practices and challenges inherent in logging. These activities demonstrated that the current representation of logs does not provide the rich structures programmers need to interpret them or the program's behavior. We present Log-it, a logging interface that enables programmers to interactively structure and visualize logs in situ. A user study with novices and experts showed that Log-it's syntax and interface have a minimal learning curve, and the interactive representations and organizations of logs help programmers easily locate, synthesize, and understand logs.2023PJPeiling Jiang et al.University of California San DiegoInteractive Data VisualizationNotification & Interruption ManagementCHI
CrossCode: Multi-level Visual Representations of Computer Program ExecutionProgram visualizations help to form useful mental models of how programs work, and to reason and debug code. But these visualizations exist at a fixed level of abstraction, e.g., line-by-line. In contrast, programmers switch between many levels of abstraction when inspecting program behavior. Based on results from a formative study of hand-designed program visualizations, we designed CrossCode, a web-based program visualization system for JavaScript that leverages structural cues in syntax, control flow, and data flow to aggregate and navigate program execution across multiple levels of abstraction. In an exploratory qualitative study with experts, we found that CrossCode enabled participants to maintain a strong sense of place in program execution, was conducive to explaining program behavior, and helped track changes and updates to the program state.2023DHDevamardeep Hayatpur et al.University of California, San DiegoInteractive Data VisualizationProgramming Education & Computational ThinkingCHI
CrossData: Leveraging Text-Data Connections for Authoring Data DocumentsData documents play a central role in recording, presenting, and disseminating data. Despite the proliferation of applications and systems designed to support the analysis, visualization, and communication of data, writing data documents remains a laborious process, requiring a constant back-and-forth between data processing and writing tools. Interviews with eight professionals revealed that their workflows contained numerous tedious, repetitive, and error-prone operations. The key issue that we identified is the lack of persistent connection between text and data. Thus, we developed CrossData, a prototype that treats text-data connections as persistent, interactive, first-class objects. By automatically identifying, establishing, and leveraging text-data connections, CrossData enables rich interactions to assist in the authoring of data documents. An expert evaluation with eight users demonstrated the usefulness of CrossData, showing that it not only reduced the manual effort in writing data documents but also opened new possibilities to bridge the gap between data exploration and writing.2022ZCZhutian Chen et al.University of California San DiegoInteractive Data VisualizationComputational Methods in HCICHI
Beyond Show of Hands: Engaging Viewers via Expressive and Scalable Visual Communication in Live StreamingLive streaming is gaining popularity across diverse application domains in recent years. A core part of the experience is streamer-viewer interaction, which has been mainly text-based. Recent systems explored extending viewer interaction to include visual elements with richer expression and increased engagement. However, understanding expressive visual inputs becomes challenging with many viewers, primarily due to the relative lack of structure in visual input. On the other hand, adding rigid structures can limit viewer interactions to narrow use cases or decrease the expressiveness of viewer inputs. To facilitate the sensemaking of many visual inputs while retaining the expressiveness or versatility of viewer interactions, we introduce a visual input management framework(VIMF) and a system, VisPoll, that help streamers specify, aggregate, and visualize many visual inputs. A pilot evaluation indicated that VisPoll can expand the types of viewer interactions. Our framework provides insights for designing scalable and expressive visual communication for live streaming.2021JCJohn Joon Young Chung et al.University of MichiganLive Streaming & Spectating ExperienceSocial Platform Design & User BehaviorCHI
Crosscast: Adding Visuals to Audio Travel PodcastsAudio travel podcasts are a valuable source of information for travelers. Yet, travel is, in many ways, a visual experience and the lack of visuals in travel podcasts can make it difficult for listeners to fully understand the places being discussed. We present Crosscast: a system for automatically adding visuals to audio travel podcasts. Given an audio travel podcast as input, Crosscast uses natural language processing and text mining to identify geographic locations and descriptive keywords within the podcast transcript. Crosscast then uses these locations and keywords to automatically select relevant photos from online repositories and synchronizes their display to align with the audio narration. In a user evaluation, we find that 85.7% of the participants preferred Crosscast generated audio-visual travel podcasts compared to audio-only travel podcasts.2020HXHaijun Xia et al.Museum & Cultural Heritage DigitizationInteractive Narrative & Immersive StorytellingUIST