Key Considerations for Domain Expert Involvement in LLM Design and Evaluation: An Ethnographic StudyLarge Language Models (LLMs) are increasingly developed for use in complex professional domains, yet little is known about how teams design and evaluate these systems in practice. This paper examines the challenges and trade-offs in LLM development through a 12-week ethnographic study of a team building a pedagogical chatbot. The researcher observed design and evaluation activities and conducted interviews with both developers and domain experts. Analysis revealed four key practices: creating workarounds for data collection, turning to augmentation when expert input was limited, co-developing evaluation criteria with experts, and adopting hybrid expert–developer–LLM evaluation strategies. These practices show how teams made strategic decisions under constraints and demonstrate the central role of domain expertise in shaping the system. Challenges included expert motivation and trust, difficulties structuring participatory design, and questions around ownership and integration of expert knowledge. We propose design opportunities for future LLM development workflows that emphasize AI literacy, transparent consent, and frameworks recognizing evolving expert roles.2026ASAnnalisa Szymanski et al.University of Notre DameHuman-LLM CollaborationParticipatory DesignUser Research Methods (Interviews, Surveys, Observation)IUI
Designing Staged Evaluation Workflows for LLMs: Integrating Domain Experts, Lay Users, and Model-Generated Evaluation CriteriaLarge Language Models (LLMs) are increasingly utilized for domain-specific tasks, yet evaluating their outputs remains challenging. A common strategy is to apply evaluation criteria to assess alignment with domain-specific standards, yet little is understood about how criteria differ across sources or where each type is most useful in the evaluation process. This study investigates criteria developed by domain experts, lay users, and LLMs to identify their complementary roles within an evaluation workflow. Results show that experts produce fact-based criteria with long-term value, lay users emphasize usability with a shorter-term focus, and LLMs target procedural checks for immediate task requirements. We also examine how criteria evolve between a priori and a posteriori phases, noting drift across stages as well as convergence in the a posteriori phase. Based on our observations, we propose design guidelines for a staged evaluation workflow combining the complementary strengths of these sources to balance quality, cost, and scalability.2026ASAnnalisa Szymanski et al.University of Notre DameHuman-LLM CollaborationAI-Assisted Decision-Making & AutomationUser Research Methods (Interviews, Surveys, Observation)CHI
Nonvisual Support for Understanding and Reasoning about Data Structures Blind and visually impaired (BVI) computer science students face systematic barriers when learning data structures: current accessibility approaches typically translate diagrams into alternative text, focusing on visual appearance rather than preserving the underlying structure essential for conceptual understanding. More accessible alternatives often do not scale in complexity, cost to produce, or both. Motivated by a recent shift to tools for creating visual diagrams from code, we propose a solution that automatically creates accessible representations from structural information about diagrams. Based on a Wizard-of-Oz study, we derive design requirements for an automated system, Arboretum, that compiles text-based diagram specifications into three synchronized nonvisual formats—tabular, navigable, and tactile. Our evaluation with BVI users highlights the strength of tactile graphics for complex tasks such as binary search; the benefits of offering multiple, complementary nonvisual representations; and limitations of existing digital navigation patterns for structural reasoning. This work reframes access to data structures by preserving their structural properties. The solution is a practical system to advance accessible CS education.2026BWBrianna L Wimer et al.University of Notre DameVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Motor Impairment Assistive Input TechnologiesSpecial Education TechnologyCHI
Balancing Goals, Health, and Cost: A Food Information System for Managing Complex Choices and Fostering Sustained Food AgencyTechnology offers new opportunities to support healthier food choices, particularly for individuals in low-income communities who face systemic barriers to obtaining nutritious, affordable groceries. We introduce a novel conceptual model of grocery planning that frames food purchasing as a multi-objective optimization problem that considers cost, nutrition components, and a consumer's personal dietary goals. Guided by Zimmerman’s model of Self-Regulated Learning and prior research on food agency, we designed the Food Information System, a planning tool that provides optimized product recommendations aligned with users’ goals by integrating store inventory, prices, and nutritional data. We evaluated our system in an eight-week within-subjects intervention with 55 participants from a food-insecure community, followed by focus group sessions. While overall Healthy Eating Index scores remained largely stable, participants reported improved nutritional awareness and greater perceived agency in planning and purchasing groceries. We discuss design implications to support food agency by promoting long-term food literacy and by enhancing autonomy in making food choices.2026ASAnnalisa Szymanski et al.University of Notre DameDiet Tracking & Nutrition ManagementBehavior Change & Reflection TechnologyData-Driven Personal Decision-MakingCHI
Understanding Parents’ Perspectives on Responsible AI for Children’s Self-Directed LearningGenerative AI is increasingly present in children’s learning environments, yet little is known about how families navigate this technology in middle childhood (ages 7–13), when parental guidance remains strong but children seek independence. \rev{Drawing on self-directed learning (SDL), we explore how parents in our exploratory sample perceived children’s emerging self-directness and agency.} Through focus groups with 13 parent–child pairs, we examine parents’ views on children’s AI literacy development, readiness factors, and mediation strategies. Parents described emergent pathways shaped by screen time, self-directness, and knowledge growth. They often confined AI to learning-only contexts, positioning it as a tutor while overlooking non-learning uses and risks such as privacy and infrastructural embedding. Many acknowledged limited AI literacy and turned to joint engagement as opportunities for co-learning. Our findings surface possible parental pathways of children’s AI literacy, highlight gaps between pragmatic expectations and critical literacies, and offer situated design considerations for AI systems that scaffold SDL while balancing oversight with autonomy.2026JXJingyi Xie et al.San José State UniversityChildren's AI Literacy & Data LiteracyHuman-LLM CollaborationCognitive Impairment & Neurodiversity (Autism, ADHD, Dyslexia)CHI
``We aren't very sophisticated'': An Ethnographic Study of Knowledge Management in Community Social ServicesNavigating the complexities of knowledge management (KM) and Knowledge Transfer (KT) in community social services is a challenging task, as workers must handle a fast-paced, resource-constrained environment while supporting the urgent and multifaceted needs of vulnerable populations. Despite research into the use of technology in non-profit and community settings, little attention has been given to how community social service workers (CSWs) capture, share, and manage knowledge in practice. This study addresses this gap by conducting a three-month ethnographic study of two community organizations in \textbf{[ANON LOCATION]}, complemented by semi-structured interviews, revealing the informal, ad-hoc KM methods CSWs use to manage information flows. Using a human-socio-technical KM framework, we compare these practices with the more structured KM systems found in large corporations, identifying key differences in formality and technological integration. Our findings highlight the need for accessible, sustainable KM solutions that fit the informal knowledge-sharing practices of CSWs while enhancing knowledge retention, knowledge transfer, and collaboration. This work contributes to HCI and CSCW by providing design considerations for developing technology that better supports KM in community social services.2025OAOghenemaro Anuyah et al.Working together (with other people)CSCW
Integrating Expertise in LLMs: Crafting a Customized Nutrition Assistant with Refined Template InstructionsLarge Language Models (LLMs) have the potential to contribute to the fields of nutrition and dietetics in generating food product explanations that facilitate informed food selections. However, the extent to which these models offer effective and accurate information remains unverified. In collaboration with registered dietitians (RDs), we evaluate the strengths and weaknesses of LLMs in providing accurate and personalized nutrition information. Through a mixed-methods approach, RDs validated GPT-4 outputs at various levels of prompt specificity, which led to the development of design guidelines used to prompt LLMs for nutrition information. We tested these guidelines by creating a GPT prototype, The Food Product Nutrition Assistant, tailored for food product explanations. This prototype was refined and evaluated in focus groups with RDs. We find that the implementation of these dietitian-reviewed template instructions enhance the generation of detailed food product descriptions and tailored nutrition information.2024ASAnnalisa Szymanski et al.University of Notre DameGenerative AI (Text, Image, Music, Video)Human-LLM CollaborationDiet Tracking & Nutrition ManagementCHI
Characterizing the Technology Needs of Vulnerable Populations for Participation in Research and Design by Adopting Maslow's Hierarchy of NeedsWhile various frameworks and heuristics exist within the HCI community to guide research and design for vulnerable populations, most are centered on the researcher's involvement. In this work, we developed a conceptual framework for supporting the participation of vulnerable populations in the research and design of technologies. Building upon Maslow's hierarchy of needs, we synthesized 84 research articles that focus on vulnerable populations and technology to develop our framework. This framework conceptualizes both the barriers, such as lack of technology access and digital literacy, and assets, like social relationships, that impact effective participation in research and design. Using our framework can guide researchers in identifying and fulfilling the technology-related needs of vulnerable populations, leading to more empowering research participation for these groups. The framework's guiding questions offer researchers the opportunity to reflect on their approach prior to and during their collaboration with vulnerable populations in technology research and design.2023OAOghenemaro Anuyah et al.University of Notre DameEmpowerment of Marginalized GroupsDeveloping Countries & HCI for Development (HCI4D)Participatory DesignCHI
Understanding Gender Transition Tracking Habits and TechnologyPersonal health tracking has long been a topic of investigation in the HCI community. There is an emerging class of apps that support gender transition, which we term transition-tracking apps. However, little work has been done examining the use and impact of such apps. We aimed to address this gap by conducting an interview study with sixteen participants who are currently undergoing different forms of gender transition. We provide an understanding of transition tracking habits, the usage and potential of transition-tracking apps in the context of transition support technologies, and provide design suggestions and open areas of research.2023TCTee Chuanromanee et al.University of Notre DameCognitive Impairment & Neurodiversity (Autism, ADHD, Dyslexia)Empowerment of Marginalized GroupsCHI
Transgender People's Technology Needs to Support Health and TransitionHealth and well-being are integral parts of the human experience, and yet due to numerous factors are inaccessible for many communities. The transgender community is no exception and faces increased risk for both physical and mental illness. This population faces many unique challenges before, during, and after transition. To gain a deeper understanding of the trans community's health and well-being needs, we conducted twenty-one interviews with transgender individuals to determine how they navigated their identities and transitions. From our interviews, we examine and highlight the unique needs of the trans population with respect to health, well-being, identity, and transition. We discuss how designers can better understand and accommodate the diversity of this community, give suggestions for the design of technologies for trans health and well-being, and contribute open areas of research.2021TCTya Chuanromanee et al.University of Notre DameInclusive DesignEmpowerment of Marginalized GroupsCHI
Supporting storytelling with evidence in holistic review processes: a participatory design approachReview processes are complex and often subjective decision-making tasks in which individual reviewers must read and rate submissions, such as a college application, along many relevant dimensions and typically with a rubric in mind. A common part of the work is committee review, where individual reviewers meet to discuss the merits of a particular submission in order to recommend an accept or reject decision. Prior work indicates that visualization and sensemaking support may be beneficial in such processes where reviewers must present the "story" of the applicant under question. We conducted a series of participatory design workshops with reviewers in the domain of holistic college admissions to better understand the challenges and opportunities regarding storytelling. Based on these workshops, we contribute a characterization for how reviewers in this domain construct visual stories, we provide guidance for designing for evidence capture and storytelling, and we draw parallels and distinctions between this domain and other reviewing domains.2020RMRonald Metoyer et al.Storytelling / Research Method ReflectionsCSCW
GameViews: Understanding and Supporting Data-driven Sports StorytellingVarious stakeholders in the sports domain rely on the analysis and presentation of sports data to derive insights. In particular, sportswriters construct game stories using statistical information; fans share their viewpoints based on the real-time stats while watching the game. In this paper, we explore how these stakeholders construct data-driven sports stories. We began by observing a sportswriter, then analyzed published sports stories, and characterized 1500 fan comments about particular sporting events. We found that their story needs were similar in some respects while quite different in others. Based on the findings, we implemented two exploratory prototypes: GameViews-Writers for sportswriters to quickly extract key game information and GameViews-Fans to support a real-time data-driven game-viewing experience for fans. We report insights from two user studies conducted with four professional sportswriters and eight sports fans, respectively. We discuss the results of these studies and present several avenues for future work.2019QZQiyu Zhi et al.University of Notre DameInteractive Data VisualizationData StorytellingCHI