Seeing Eye to AI? Applying Deep-Feature-Based Similarity Metrics to Information VisualizationJudging the similarity of visualizations is crucial to various applications, such as visualization-based search and visualization recommendation systems. Recent studies show deep-feature-based similarity metrics correlate well with perceptual judgments of image similarity and serve as effective loss functions for tasks like image super-resolution and style transfer. We explore the application of such metrics to judgments of visualization similarity. We extend a similarity metric using five ML architectures and three pre-trained weight sets. We replicate results from previous crowdsourced studies on scatterplot and visual channel similarity perception. Notably, our metric using pre-trained ImageNet weights outperformed gradient-descent tuned MS-SSIM, a multi-scale similarity metric based on luminance, contrast, and structure. Our work contributes to understanding how deep-feature-based metrics can enhance similarity assessments in visualization, potentially improving visual analysis tools and techniques. Supplementary materials are available at https://osf.io/dj2ms/.2025SLSheng Long et al.Northwestern University, Computer ScienceRecommender System UXInteractive Data VisualizationVisualization Perception & CognitionCHI
More Forecasts, More (Decision) Problems: How Uncertainty Representations for Multiple Forecasts Impact Decision MakingUsers often have access to multiple forecasts regarding an event. Different forecasts incorporate different assumptions and epistemic information. A growing body of work argues against decision-making solely based on expected utility maximisation strategies in multiple forecasts scenarios, in favour of other strategies such as the maximin expected utility. In this work, we compare two different approaches for depicting epistemic uncertainty—ensembles (a direct representation of multiple forecasts) and p-boxes (a representation which only communicates the bounds of epistemic uncertainty)—in plots where individual distributions are represented as cumulative distribution plots (CDFs). We conduct three experiments to investigate the impact of the visual representation on the decision-making strategies that people adopt. Our results suggest that participants adopt conservative decision-making strategies (i.e. place greater weight on the worst-case forecast than the best-case forecast) for both p-boxes and ensembles if the set of forecasts are uniformly distributed. However, if a majority of the forecasts are clustered near one of the bounds, participants may discount the forecast which appears as a visual outlier.2025ASAbhraneel Sarma et al.Northwestern University, Computer ScienceAI-Assisted Decision-Making & AutomationUncertainty VisualizationVisualization Perception & CognitionCHI
AVEC: An Assessment of Visual Encoding Ability in Visualization ConstructionVisualization literacy is the ability to both interpret and construct visualizations. Yet existing assessments focus solely on visualization interpretation. A lack of construction-related measurements hinders efforts in understanding and improving literacy in visualizations. We design and develop AVEC, an assessment of a person's visual encoding ability—a core component of the larger process of visualization construction—by: (1) creating an initial item bank using a design space of visualization tasks and chart types, (2) designing an assessment tool to support the combinatorial nature of selecting appropriate visual encodings, (3) building an autograder from expert scores of answers to our items, and (4) refining and validating the item bank and autograder through an analysis of test tryout data with 95 participants and feedback from the expert panel. We discuss recommendations for using AVEC, potential alternative scoring strategies, and the challenges in assessing higher-level visualization skills using constructed-response tests. Supplemental materials are available at: https://osf.io/hg7kx/.2025LGLily W. Ge et al.Northwestern University, Computer ScienceInteractive Data VisualizationVisualization Perception & CognitionCHI
Opportunities in Mental Health Support for Informal Dementia Caregivers Suffering from Verbal AgitationPeople with dementia (PwD) often present verbal agitation such as cursing, screaming, and persistently complaining. Verbal agitation can impose mental distress on informal caregivers (e.g., family, friends), which may cause severe mental illnesses, such as depression and anxiety disorders. To improve informal caregivers' mental health, we explore design opportunities by interviewing 11 informal caregivers suffering from verbal agitation of PwD. In particular, we first characterize how the predictability of verbal agitation impacts informal caregivers' mental health and how caregivers' coping strategies vary before, during, and after verbal agitation. Based on our findings, we propose design opportunities to improve the mental health of informal caregivers suffering from verbal agitation: distracting PwD (in-situ support; before), prompting just-in-time maneuvers (information support; during), and comfort and education (social & information support; after). We discuss our reflections on cultural disparities between participants. Our work envisions a broader design space for supporting informal caregivers' well-being and describes when and how that support could be provided.2024TKTaewook Kim et al.Session 1c: Care for the CaregiversCSCW
To Cut or Not To Cut? A Systematic Exploration of Y-Axis TruncationY-axis truncation is a well-known, much-debated visualization practice. Our work complements existing empirical work by providing a systematic analysis of y-axis truncation on grouped bar charts. Drawing upon theoretical frameworks such as Algebraic Visualization Design, we examine how structure-preserving modifications to visualization affect user performance by systematically dividing the space of possible truncations according to their monotonicity and the type of relations in the underlying data. Our results demonstrate that for comparing and estimating the difference between the lengths of two bars, truncating the y-axis does not affect task performance. For comparing or estimating the relative growth between two bars, truncating monotonically has similar performance to no truncation, while truncating non-monotonically is very likely to impair performance. We discuss possible extensions of our work and recommendations for y-axis truncation. All supplementary materials are available at https://osf.io/k4hjd/?view_only=008b087fc3d94be7ba0ce7aea95012a7.2024SLSheng Long et al.Northwestern UniversityUncertainty VisualizationVisualization Perception & CognitionCHI
In Dice We Trust: Uncertainty Displays for Maintaining Trust in Election Forecasts Over TimeTrust in high-profile election forecasts influences the public’s confidence in democratic processes and electoral integrity. Yet, maintaining trust after unexpected outcomes like the 2016 U.S. presidential election is a significant challenge. Our work confronts this challenge through three experiments that gauge trust in election forecasts. We generate simulated U.S. presidential election forecasts, vary win probabilities and outcomes, and present them to participants in a professional-looking website interface. In this website interface, we explore (1) four different uncertainty displays, (2) a technique for subjective probability correction, and (3) visual calibration that depicts an outcome with its forecast distribution. Our quantitative results suggest that text summaries and quantile dotplots engender the highest trust over time, with observable partisan differences. The probability correction and calibration show small-to-null effects on average. Complemented by our qualitative results, we provide design recommendations for conveying U.S. presidential election forecasts and discuss long-term trust in uncertainty communication. We provide preregistration, code, data, model files, and videos at https://doi.org/10.17605/OSF.IO/923E7.2024FYFumeng Yang et al.Northwestern UniversityInteractive Data VisualizationUncertainty VisualizationCHI
V-FRAMER: Visualization Framework for Mitigating Reasoning Errors in Public PolicyExisting data visualization design guidelines focus primarily on constructing grammatically-correct visualizations that faithfully convey the values and relationships in the underlying data. However, a designer may create a grammatically-correct visualization that still leaves audiences susceptible to reasoning misleaders, e.g. by failing to normalize data or using unrepresentative samples. Reasoning misleaders are especially pernicious when presenting public policy data, where data-driven decisions can affect public health, safety, and economic development. Through textual analysis, a formative evaluation, and iterative design with 19 policy communicators, we construct an actionable visualization design framework, V-FRAMER, that effectively synthesizes ways of mitigating reasoning misleaders. We discuss important design considerations for frameworks like V-FRAMER, including using concrete examples to help designers understand reasoning misleaders, and using a hierarchical structure to support example-based accessing. We further describe V-FRAMER's congruence with current practice and how practitioners might integrate the framework into their existing workflows. Related materials available at: https://osf.io/q3uta/.2024LGLily W. Ge et al.Northwestern UniversityExplainable AI (XAI)Uncertainty VisualizationCHI
Watching the Election Sausage Get Made: How Data Journalists Visualize the Vote Counting Process in U.S. ElectionsElection results in the United States are visualized online in real time by news outlets as vote counting persists over days or weeks. They are a massive public-facing exercise in managing audience understanding of uncertainty in partial data, breaking news web traffic records as the public seeks information about winners. We categorize designs of real-time election results from 19 U.S. news outlets and election results providers for the 2020 and 2022 general elections to create a visual vocabulary of live results. We then use this vocabulary to guide interviews with data journalists who worked on these designs to understand their design goals and challenges. Tying these conversations back to our visual vocabulary, we map out how communication goals like balancing certainty and uncertainty in the journey towards finding out winners, alongside challenges like determining thresholds at which information is shown, manifest in the designs displayed.2024MCMandi Cai et al.Northwestern University, Northwestern UniversityInteractive Data VisualizationUncertainty VisualizationMisinformation & Fact-CheckingCHI
Odds and Insights: Decision Quality in Exploratory Data Analysis Under UncertaintyRecent studies have shown that users of visual analytics tools can have difficulty distinguishing robust findings in the data from statistical noise, but the true extent of this problem is likely dependent on both the incentive structure motivating their decisions, and the ways that uncertainty and variability are (or are not) represented in visualisations. In this work, we perform a crowd-sourced study measuring decision-making quality in visual analytics, testing both an explicit structure of incentives designed to reward cautious decision-making as well as a variety of designs for communicating uncertainty. We find that, while participants are unable to perfectly control for false discoveries as well as idealised statistical models such as the Benjamini-Hochberg, certain forms of uncertainty visualisations can improve the quality of participants’ decisions and lead to fewer false discoveries than not correcting for multiple comparisons. We conclude with a call for researchers to further explore visual analytics decision quality under different decision-making contexts, and for designers to directly present uncertainty and reliability information to users of visual analytics tools. The supplementary materials are available at: https://osf.io/xtsfz/.2024ASAbhraneel Sarma et al.Northwestern UniversityUncertainty VisualizationVisualization Perception & CognitionCHI
Milliways: Taming Multiverses through Principled Evaluation of Data Analysis PathsMultiverse analyses involve conducting all combinations of reasonable choices in a data analysis process. A reader of a study containing a multiverse analysis might question—are all the choices included in the multiverse reasonable and equally justifiable? How much do results vary if we make different choices in the analysis process? In this work, we identify principles for validating the composition of, and interpreting the uncertainty in, the results of a multiverse analysis. We present Milliways, a novel interactive visualisation system to support principled evaluation of multiverse analyses. Milliways provides interlinked panels presenting result distributions, individual analysis composition, multiverse code specification, and data summaries. Milliways supports interactions to sort, filter and aggregate results based on the analysis specification to identify decisions in the analysis process to which the results are sensitive. To represent the two qualitatively different types of uncertainty that arise in multiverse analyses—probabilistic uncertainty from estimating unknown quantities of interest such as regression coefficients, and possibilistic uncertainty from choices in the data analysis—Milliways uses consonance curves and probability boxes. Through an evaluative study with five users familiar with multiverse analysis, we demonstrate how Milliways can support multiverse analysis tasks, including a principled assessment of the results of a multiverse analysis.2024ASAbhraneel Sarma et al.Northwestern UniversityInteractive Data VisualizationUncertainty VisualizationCHI
multiverse: Multiplexing Alternative Data Analyses in R NotebooksThere are myriad ways to analyse a dataset. But which one to trust? In the face of such uncertainty, analysts may adopt multiverse analysis: running all reasonable analyses on the dataset. Yet this is cognitively and technically difficult with existing tools—how does one specify and execute all combinations of reasonable analyses of a dataset?—and often requires discarding existing workflows. We present multiverse, a tool for implementing multiverse analyses in R with expressive syntax supporting existing computational notebook workflows. multiverse supports building up a multiverse through local changes to a single analysis and optimises execution by pruning redundant computations. We evaluate how multiverse supports programming multiverse analyses using (a) principles of cognitive ergonomics to compare with two existing multiverse tools; and (b) case studies based on semi-structured interviews with researchers who have successfully implemented an end-to-end analysis using multiverse. We identify design tradeoffs (e.g. increased flexibility versus learnability), and suggest future directions for multiverse tool design.2023ASAbhraneel Sarma et al.Northwestern UniversityInteractive Data VisualizationComputational Methods in HCICHI
CALVI: Critical Thinking Assessment for Literacy in VisualizationsVisualization misinformation is a prevalent problem, and combating it requires understanding people’s ability to read, interpret, and reason about erroneous or potentially misleading visualizations, which lacks a reliable measurement: existing visualization literacy tests focus on well-formed visualizations. We systematically develop an assessment for this ability by: (1) developing a precise definition of misleaders (decisions made in the construction of visualizations that can lead to conclusions not supported by the data), (2) constructing initial test items using a design space of misleaders and chart types, (3) trying out the provisional test on 497 participants, and (4) analyzing the test tryout results and refining the items using Item Response Theory, qualitative analysis, a wrong-due-to-misleader score, and the content validity index. Our final bank of 45 items shows high reliability, and we provide item bank usage recommendations for future tests and different use cases. Related materials are available at: https://osf.io/pv67z/.2023LGLily W. Ge et al.Northwestern UniversityInteractive Data VisualizationUncertainty VisualizationVisualization Perception & CognitionCHI
How Data Analysts Use a Visualization Grammar in PracticeVisualization grammars, often based on the Grammar of Graphics (GoG), have much potential for augmenting data analysis in a programming environment. However, we do not know how analysts conceptualize grammar abstractions, or how a visualization grammar works with data analysis in practice. Therefore, we qualitatively analyzed how experienced analysts (N=6) from TidyTuesday, a social data project, wrangled and visualized data using GoG-based ggplot2 without given tasks in R Markdown. Though participants' analysis and customization needs could mismatch with GoG component design, their analysis processes aligned with the goal of GoG to expedite visualization iteration. We also found a feedback loop and tight coupling between visualization and data transformation code, explaining both participants' productivity and their errors. From these results, we discuss how future visualization grammars can become more practical for analysts and how visualization grammar and analysis tools can better integrate within a programming (i.e., computational notebook) environment.2023XPXiaoying Pu et al.University of California, MercedInteractive Data VisualizationPrototyping & User TestingComputational Methods in HCICHI
“It can bring you in the right direction”: Episode-Driven Data Narratives to Help Patients Navigate Multidimensional Diabetes Data to Make Care DecisionsEngaging with multiple streams of personal health data to inform self-care of chronic health conditions remains a challenge. Existing informatics tools provide limited support for patients to make data actionable. To design better tools, we conducted two studies with Type 1 diabetes patients and their clinicians. In the first study, we observed data review sessions between patients and clinicians to articulate the tasks involved in assessing different types of data from diabetes devices to make care decisions. Drawing upon these tasks, we designed novel data interfaces called episode-driven data narratives and performed a task-driven evaluation. We found that as compared to the commercially available diabetes data reports, episode-driven data narratives improved engagement and decision-making with data. We discuss implications for designing data interfaces to support interaction with multidimensional health data to inform self-care.2023SRShriti Raj et al.University of Michigan, San Francisco State UniversityData StorytellingChronic Disease Self-Management (Diabetes, Hypertension, etc.)CHI
An Aligned Rank Transform Procedure for Multifactor Contrast TestsData from multifactor HCI experiments often violates the assumptions of parametric tests (i.e., nonconforming data). The Aligned Rank Transform (ART) has become a popular nonparametric analysis in HCI that can find main and interaction effects in nonconforming data, but leads to incorrect results when used to conduct post hoc contrast tests. We created a new algorithm called ART-C for conducting contrast tests within the ART paradigm and validated it on 72,000 synthetic data sets. Our results indicate that ART-C does not inflate Type I error rates, unlike contrasts based on ART, and that ART-C has more statistical power than a t-test, Mann-Whitney U test, Wilcoxon signed-rank test, and ART. We also extended an open-source tool called ARTool with our ART-C algorithm for both Windows and R. Our validation had some limitations (e.g., only six distribution types, no mixed factorial designs, no random slopes), and data drawn from Cauchy distributions should not be analyzed with ART-C.2021LELisa A. Elkin et al.User Research Methods (Interviews, Surveys, Observation)Computational Methods in HCIUIST