ImaginationVellum: Generative-AI Ideation Canvas with Spatial Prompts, Generative Strokes, and Ideation HistoryWe introduce ImaginationVellum, a multi-modal spatial canvas for early-stage visual ideation and concept sketching with generative AI. The resulting system supports a unique style of human-AI co-creation where the canvas is the prompt. This means that ImaginationVellum employs the entire 2D canvas as an active prompt space, where spatial arrangement, proximity, and composition of diverse content elements - inking, text, images, and intermediate results - steer generative visual outcomes. As a technical probe, ImaginationVellum contributes a set of spatially-grounded direct manipulation tools for iterative visual ideation. In particular, we introduce Generative Strokes - freeform strokes that spatially modulate generation and prompt-parameters (articulated along multiple latent semantic or stylistic dimensions). These techniques afford rapid traversal of design spaces via convergence, divergence, re-composition, blending, and remixing of concepts. We detail the system architecture, design rationale, proximity-dependent intent tags for localized control, and methods for spatial prompting and varying output along spatial gradients. Temporal replay and visualization of provenance make ideation trajectories actionable, turning the design process itself into an artifact that supports reflection-in-action and revisitation of design decisions. We report insights from a preliminary study of how users construct, steer, and revisit ideas using spatial prompts, and discuss tradeoffs in modulating spatially-dependent content generation.2025NMNicolai Marquardt et al.Generative AI (Text, Image, Music, Video)Creative Collaboration & Feedback SystemsUIST
Data Formulator 2: Iterative Creation of Data Visualizations, with AI Transforming Data Along the WayData analysts often need to iterate between data transformations and chart designs to create rich visualizations for exploratory data analysis. Although many AI-powered systems have been introduced to reduce the effort of visualization authoring, existing systems are not well suited for iterative authoring. They typically require analysts to provide, in a single turn, a text-only prompt that fully describe a complex visualization. We introduce Data Formulator 2 (DF2 for short), an AI-powered visualization system designed to overcome this limitation. DF2 blends graphical user interfaces and natural language inputs to enable users to convey their intent more effectively, while delegating data transformation to AI. Furthermore, to support efficient iteration, DF2 lets users navigate their iteration history and reuse previous designs, eliminating the need to start from scratch each time. A user study with eight participants demonstrated that DF2 allowed participants to develop their own iteration styles to complete challenging data exploration sessions.2025CWChenglong Wang et al.Microsoft ResearchAI-Assisted Decision-Making & AutomationInteractive Data VisualizationComputational Methods in HCICHI
How Do Analysts Understand and Verify AI-Assisted Data Analyses?Data analysis is challenging as it requires synthesizing domain knowledge, statistical expertise, and programming skills. Assistants powered by large language models (LLMs), such as ChatGPT, can assist analysts by translating natural language instructions into code. However, AI-assistant responses and analysis code can be misaligned with the analyst's intent or be seemingly correct but lead to incorrect conclusions. Therefore, validating AI assistance is crucial and challenging. Here, we explore how analysts understand and verify the correctness of AI-generated analyses. To observe analysts in diverse verification approaches, we develop a design probe equipped with natural language explanations, code, visualizations, and interactive data tables with common data operations. Through a qualitative user study (n=22) using this probe, we uncover common behaviors within verification workflows and how analysts' programming, analysis, and tool backgrounds reflect these behaviors. Additionally, we provide recommendations for analysts and highlight opportunities for designers to improve future AI-assistant experiences.2024KGKen Gu et al.Paul G. Allen School of Computer Science & Engineering, University of WashingtonHuman-LLM CollaborationExplainable AI (XAI)Interactive Data VisualizationCHI
On the Design of AI-powered Code Assistants for NotebooksAI-powered code assistants, such as Copilot, are quickly becoming a ubiquitous component of contemporary coding contexts. Among these environments, computational notebooks, such as Jupyter, are of particular interest as they provide rich interface affordances that interleave code and output in a manner that allows for both exploratory and presentational work. Despite their popularity, little is known about the appropriate design of code assistants in notebooks. We investigate the potential of code assistants in computational notebooks by creating a design space (reified from a survey of extant tools) and through an interview-design study (with 15 practicing data scientists). Through this work, we identify challenges and opportunities for future systems in this space, such as the value of disambiguation for tasks like data visualization, the potential of tightly scoped domain-specific tools (like linters), and the importance of polite assistants.2023AMAndrew McNutt et al.University of ChicagoHuman-LLM CollaborationRecommender System UXKnowledge Worker Tools & WorkflowsCHI
Diff in the Loop: Supporting Data Comparison in Exploratory Data AnalysisData science is characterized by evolution: since data science is exploratory, results evolve from moment to moment; since it can be collaborative, results evolve as the work changes hands. While existing tools help data scientists track changes in code, they provide less support for understanding the iterative changes that the code produces in the data. We explore the idea of visualizing differences in datasets as a core feature of exploratory data analysis, a concept we call Diff in the Loop (DITL). We evaluated DITL in a user study with 16 professional data scientists and found it helped them understand the implications of their actions when manipulating data. We summarize these findings and discuss how the approach can be generalized to different data science workflows.2022AWApril Yi Wang et al.University of MichiganInteractive Data VisualizationVisualization Perception & CognitionComputational Methods in HCICHI
Collecting and Characterizing Natural Language Utterances for Specifying Data VisualizationsNatural language interfaces (NLIs) for data visualization are becoming increasingly popular both in academic research and in commercial software. Yet, there is a lack of empirical understanding of how people specify visualizations through natural language. We conducted an online study (N = 102), showing participants a series of visualizations and asking them to provide utterances they would pose to generate the displayed charts. From the responses, we curated a dataset of 893 utterances and characterized the utterances according to (1) their phrasing (e.g., commands, queries, questions) and (2) the information they contained (e.g., chart types, data aggregations). To help guide future research and development, we contribute this utterance dataset and discuss its applications toward the creation and benchmarking of NLIs for visualization.2021ASArjun Srinivasan et al.Tableau ResearchVoice User Interface (VUI) DesignInteractive Data VisualizationCHI
Fork It: Supporting Stateful Alternatives in Computational NotebooksComputational notebooks, which seamlessly interleave code with results, have become a popular tool for data scientists due to the iterative nature of exploratory tasks. However, notebooks provide a single execution state for users to manipulate through creating and manipulating variables. When exploring alternatives, data scientists must carefully create many-step manipulations in visually distant cells. We conducted formative interviews with 6 professional data scientists, motivating design principles behind exposing multiple states. We introduce forking --- creating a new interpreter session --- and backtracking --- navigating through previous states. We implement these interactions as an extension to notebooks that help data scientists more directly express and navigate through decision points a single notebook. In a qualitative evaluation, 11 professional data scientists found the tool would be useful for exploring alternatives and debugging code to create a predictive model. Their insights highlight further challenges to scaling this functionality.2021NWNathaniel Weinman et al.University of California, BerkeleyPrototyping & User TestingComputational Methods in HCICHI
Affinity Lens: Data-Assisted Affinity Diagramming with Augmented RealityDespite the availability of software to support Affinity Diagramming (AD), practitioners still largely favor physical sticky-notes. Physical notes are easy to set-up, can be moved around in space and offer flexibility when clustering un-structured data. However, when working with mixed data sources such as surveys, designers often trade off the physicality of notes for analytical power. We propose AffinityLens, a mobile-based augmented reality (AR) application for Data-Assisted Affinity Diagramming (DAAD). Our application provides just-in-time quantitative insights overlaid on physical notes. Affinity Lens uses several different types of AR overlays (called lenses) to help users find specific notes, cluster information, and summarize insights from clusters. Through a formative study of AD users, we developed design principles for data-assisted AD and an initial collection of lenses. Based on our prototype, we find that Affinity Lens supports easy switching between qualitative and quantitative 'views' of data, without surrendering the lightweight benefits of existing AD practice.2019HSHariharan Subramonyam et al.University of MichiganMixed Reality WorkspacesInteractive Data VisualizationContext-Aware ComputingCHI
Gamut: A Design Probe to Understand How Data Scientists Understand Machine Learning ModelsWithout good models and the right tools to interpret them, data scientists risk making decisions based on hidden biases, spurious correlations, and false generalizations. This has led to a rallying cry for model interpretability. Yet the concept of interpretability remains nebulous, such that researchers and tool designers lack actionable guidelines for how to incorporate interpretability into models and accompanying tools. Through an iterative design process with expert machine learning researchers and practitioners, we designed a visual analytics system, Gamut, to explore how interactive interfaces could better support model interpretation. Using Gamut as a probe, we investigated why and how professional data scientists interpret models, and how interface affordances can support data scientists in answering questions about model interpretability. Our investigation showed that interpretability is not a monolithic concept: data scientists have different reasons to interpret models and tailor explanations for specific audiences, often balancing competing concerns of simplicity and completeness. Participants also asked to use Gamut in their work, highlighting its potential to help data scientists understand their own data.2019FHFred Hohman et al.Georgia Institute of TechnologyExplainable AI (XAI)AI-Assisted Decision-Making & AutomationInteractive Data VisualizationCHI
AnchorViz: Facilitating Classifier Error Discovery through Interactive Semantic Data ExplorationIn supervised interactive machine learning, human knowledge about the target concept can be a powerful reference to build a concept classifier that is robust to unseen items in the real world. The main challenge lies in finding unlabeled items that can either help discover or refine subconcepts for which the current classifier has no corresponding features (i.e., it has feature blindness). Yet it is unrealistic to ask humans to come up with an exhaustive list of items, especially for rare subconcepts that are hard to recall. This paper presents AnchorViz, an interactive visualization that facilitates error discovery through semantic data exploration. By creating example-based anchors, users create a topology to spread data based on their similarity to the anchors and examine the inconsistencies between data points that are semantically related. The results from our user study show that AnchorViz helps users discover more prediction errors than stratified random and uncertainty sampling methods.2018NCNan-Chen Chen et al.Explainable AI (XAI)Interactive Data VisualizationVisualization Perception & CognitionIUI
What's the Difference?: Evaluating Variations of Multi-Series Bar Charts for Visual Comparison TasksAn increasingly common approach to data analysis involves using information dashboards to visually compare changing data. However, layout constraints coupled with varying levels of visualization literacy among dashboard users make facilitating visual comparison in dashboards a challenging task. In this paper, we evaluate variants of bar charts, one of the most prevalent class of charts used in dashboards. We report an online experiment (N = 74) conducted to evaluate four alternative designs: 1) grouped bar chart, 2) grouped bar chart with difference overlays, 3) bar chart with difference overlays, and 4) difference bar chart. Results show that charts with difference overlays facilitate a wider range of comparison tasks while performing comparably to charts without them on individual tasks. Finally, we discuss the implications of our findings, with a focus on supporting visual comparison in dashboards.2018ASArjun Srinivasan et al.Georgia Institute of TechnologyInteractive Data VisualizationVisualization Perception & CognitionCHI