Talking About the Assumption in the Room

Research Background and Issues

• What problems or challenges did the authors identify?
  This paper focuses on the conceptualization, identification, and handling of "assumptions" in the fields of machine learning (ML) and human-computer interaction (HCI). Although assumptions significantly influence ML workflows, they are often marginalized, leading to confusion among practitioners regarding their scope and role. This confusion is reflected in the independence of assumptions, their reactive handling within workflows, and the lack of clear documentation.

• Why is this issue important?
  Assumptions are a core component of any ML workflow, driving institutional motivations, influencing model design, and enabling project execution. However, the ambiguity and unreflective handling of assumptions can lead to potential technical and social consequences. If ignored, this exacerbates issues of transparency, ethics, and fairness in ML systems, ultimately affecting trust and reliability.

• Research Motivation and Related Work
  While prior HCI and ML literature occasionally mentions "assumptions," most discussions focus on their peripheral role or negative impact on goals, lacking a centralized discussion and explicit analysis of assumptions. Existing tools such as Model Cards and Datasheets often require listing assumptions but do not incorporate methods for deep reflection. The motivation for this research lies in uncovering the underlying issues in handling assumptions and proposing a systematic analytical approach.

Solutions

• What methods or solutions did the authors propose?
  The authors introduce the theory of "informal logic," treating assumptions as premises for achieving a specific goal. Using this perspective, they analyze and address the confusion surrounding assumptions in ML workflows. Based on this concept, they conducted semi-structured interviews with 22 ML practitioners to explore how they conceptualize, identify, and handle assumptions, and proposed an assumption articulation framework to help practitioners manage assumptions more effectively.

• What are the innovative aspects of this solution?

  1. Proposed a theoretical framework that treats assumptions as "argument premises," enabling a more systematic analysis of assumptions.
  2. Differentiated two types of assumptions based on the psychological construction of workflows: independent construction and relative construction.
  3. Introduced the premise-target-differentiator framework to help practitioners clearly document the logical relationships of assumptions.

• What are the implementation steps and key techniques used?

  1. Theoretical Framework Construction: Introduced informal logic (e.g., deconstruction of premises and targets) as an analytical tool.
  2. Data Collection: Conducted semi-structured interviews to gather participants' experiences in identifying and handling assumptions.
  3. Data Analysis: Used the theoretical framework to perform open coding and thematic classification of interview transcripts, identifying the main sources of confusion regarding assumptions.
  4. Tool Support: Designed an assumption documentation and analysis framework that categorizes assumptions into targets (Target), premises (Premise), and their dimensions of understanding, such as behavior and goal alignment.

Research Findings

• What specific findings were achieved?

  1. Clarified two ontological perspectives on assumptions among ML practitioners:
     • Independent Construction: Assumptions as independent entities, separate from the ML process, with unclear connections to foundational goals.
     • Relative Construction: Assumptions embedded in specific stages of the development process (e.g., data quality or model specifications), making goals more explicit but potentially rationalized.
  2. Identified three major procedural confusions related to assumptions:
     • Reactive Handling of Assumptions: Assumptions are often identified late in the workflow, with no systematic mechanisms to address them.
     • Lack of Reflective Quantification: Assumptions are often simplified through quantification, neglecting deeper reflection.
     • Unstructured Documentation: Assumption documentation lacks standardization and detail, leading to collaboration challenges.

• What advantages does this solution have compared to existing ones?
  This study goes beyond the traditional ML tools (e.g., Model Cards) that merely list assumptions, offering theoretical and practical methods for classifying, linking, and supporting decision-making around assumptions. This enhanced assumption analysis can improve the ethicality and transparency of models and workflows.

• What are the experimental or evaluation results?
  Through the analysis of practitioner interview cases, the study revealed:

  1. The internal complexity of assumptions, including how they link different goals and chains of assumptions.
  2. Significant differences in how assumptions are documented and understood by various roles (e.g., ML engineers, technical managers) across organizations.
  3. The use of the informal logic framework helps practitioners better examine assumptions and their logical relationships to goals.

• Limitations and Future Directions

  1. The study's participants were primarily concentrated in the Global North, with an imbalanced gender ratio, which may affect the generalizability of the findings.
  2. The proposed framework requires further empirical evaluation, particularly in collaborative environments involving diverse roles.
  3. Future research should integrate dynamic tools, such as software-based argument mapping tools, to enhance the visualization and interactivity of assumption handling.

This study fills a gap in current research on handling ML assumptions and provides a novel analytical approach by integrating philosophical and informal logic perspectives. Future work should focus on developing corresponding tools to promote practical application and examining the universality of assumption handling across diverse cultural contexts.