What Do Hackathons Do? Understanding Participation in Hackathons Through Program Theory Analysis

Title of the Paper

What Do Hackathons Do? Understanding Participation in Hackathons Through Program Theory Analysis

Bibliographic Information

• Subject Areas: Human-Computer Interaction (HCI), Participatory Design, Research Methodology Evaluation
• Keywords: hackathon, participatory design, research methods, program theory, sustainable outcomes, education, technology democratization, data visualization, learning

Research Background and Issues

• Existing Problems and Challenges:

  • Although hackathons are applied in various fields, their effectiveness and design formats have faced widespread criticism, including:
    • Lack of sustainability in outputs.
    • Insufficient participant diversity.
    • Inability to effectively achieve the goal of technology democratization.
  • While there is extensive research on hackathons, the methodological validity of how hackathons are organized to meet different objectives has not been systematically studied.

• Significance of the Research:

  • Hackathons are regarded as an important tool for technology democratization, capable of engaging a wide audience in technology development and design.
  • Exploring adaptable organizational forms of hackathons and understanding their input-process-output mechanisms is of great significance for advancing HCI research.

• Research Motivation and Related Work:

  • Current literature focuses on general descriptions of hackathons, lacking in-depth analysis of specific cases and theoretical perspectives.
  • The introduction of "Program Theory" (PT) to understand the processes and effects of hackathons in specific contexts and to trace their causal relationships is being applied to HCI research for the first time.

Proposed Solution

• Proposed Method or Solution:

  • The authors analyzed 16 hackathon cases using "Program Theory" (PT) to reveal the causal chains of inputs, mechanisms, processes, and various outputs.
  • PT is introduced as an evaluation tool to help organizers design goal-oriented hackathons from both theoretical and practical perspectives.

• Innovative Contributions:

  • Unlike previous broad literature reviews, the authors adopted a case study approach to deeply analyze various forms of hackathons in different contexts, emphasizing the dynamic conditions in specific applications.
  • They provided analytical tables and diagrams under the PT framework, offering practical guidelines for organizers.

• Implementation Steps and Key Techniques:

  1. Case Selection:
     • Searched the ACM digital library for hackathon-related literature from 2010 to 2020, filtered 41 papers with detailed descriptions of hackathon organization, and ultimately selected 16 cases.
  2. Analysis Process:
     • Classified inputs (e.g., design materials, participant selection), processes (e.g., core activities and technical guidance), and outcomes (e.g., short-term results and long-term impacts) into three dimensions.
  3. Research Classification:
     • Categorized the 16 cases into three main types: Participation, Sustainable Outcomes, and Learning.

Research Findings

• Specific Findings:

  • Provided a catalog of cases for hackathon organization and evaluation methods.
  • Summarized the input-process-output mechanisms of different hackathon formats from three PT perspectives:
    • Participation: Enhanced participation through broader participant invitations and appropriate material selection.
    • Sustainable Outcomes: Case studies proposed improving the shareability of hackathon outputs through workbooks, documentation, and monetary incentives.
    • Learning: Created challenge-based learning and theme-driven course designs, such as short-term data analysis activities (Datathons) and code comprehension training.

• Advantages:

  • Unlike traditional literature reviews and standardized organizational methods, this study emphasizes how hackathon formats operate under different goals and conditions to adapt to new demands.
  • By formalizing hackathons through PT, the study clarifies implicit assumptions and causal mechanisms in projects.

• Experimental or Evaluation Results:

  • The authors noted that the efficiency of learning outcomes and participant diversity depends on pre-design, while achieving sustainable outcomes requires addressing post-execution issues, such as follow-up interactions with stakeholders.

• Limitations and Future Directions:

  • Limitations:
    • Only analyzed literature from the ACM digital library, potentially excluding hackathon research in other fields such as healthcare.
    • The evaluated literature may not fully cover all information related to hackathon organization.
  • Future Directions:
    • Encourages researchers to view hackathons as methodological tools for democratization and to explore their broader methodological potential.
    • Promotes research from the perspective of hackathon participants, investigating how to enhance their contributions to reflect broader and more consensus-driven knowledge production.