How Do Hackathons Foster Creativity? Towards Automated Evaluation of Creativity at Scale

Research Background and Problem

• Identified Issues or Challenges: Hackathons are regarded as activities for rapidly generating creative ideas and developing prototypes. However, most existing research on Hackathons focuses on specific cases or singular contexts, lacking large-scale quantitative analysis to explore how Hackathons inspire creativity. Evaluating creativity effectively also remains a significant challenge.
• Importance: Understanding how the organizational structure of Hackathons influences creative outcomes is crucial for improving education, corporate innovation, and research activities. Additionally, automated large-scale methods for assessing creativity can be widely applied to other fields, such as patents and creative writing.
• Research Motivation and Related Work: The authors aim to address existing research gaps by developing a methodology for large-scale analysis of creative outputs and exploring the use of large language models (LLMs) to assist in creativity evaluation.

Solution

• Proposed Solution:

  1. Operationalize creativity as novelty and usefulness, leveraging objective data to conduct large-scale analysis of 193,353 Hackathon projects.
  2. Employ LLMs (e.g., GPT) as supplementary evaluation tools to validate their performance in large-scale data assessments.

• Innovations:

  1. Introduced a dual approach combining statistical modeling and LLMs for evaluation—using data analysis to identify creative projects while incorporating AI methods for further refinement of innovative projects.
  2. Explored creativity assessment methods previously tested only in laboratory settings within real-world Hackathon data.

• Implementation Steps and Key Techniques:

  1. Data Processing: Collected 193,353 project datasets from the Devpost database and preprocessed them, narrowing down to 10,363 projects with GitHub repositories for analysis.
  2. Defining Novelty: Following related research, defined novelty based on anomalous programming package combinations introduced in the code.
  3. Defining Usefulness: Used proxies such as whether a project won a competition (winner tag) and its openness on GitHub.
  4. Statistical Analysis: Applied mixed-effects regression models to analyze the impact of variables like team size, competition intensity, and participant experience on creativity.
  5. LLM-Assisted Evaluation: Incorporated four large language models (e.g., Llama and Prometheus) to score selected project descriptions for novelty and usefulness, comparing these scores with human evaluations.

Research Findings

• Specific Findings:

  1. Smaller Hackathons (60–80 participants) are positively correlated with higher creative output.
  2. Larger team sizes (4–5 members) and high competition pressure foster creative outcomes, though competition may have negative effects on individuals.
  3. Long-term collaboration history within project teams significantly enhances creative output.
  4. Projects that receive more community recognition (e.g., "likes") suggest that participant insights can be predictive of creativity.

• Advantages Over Existing Methods:

  1. Provides a comprehensive analysis from process (team dynamics, themes, etc.) to outcomes (specific project performance), surpassing the limitations of case-specific studies.
  2. Integrates LLMs into the evaluation framework, exploring promising "human-machine collaboration" models.

• Experimental or Evaluation Results:

  1. Combined LLM and human evaluations demonstrate significant potential for AI to supplement human judgment, though models tend to give overly optimistic scores.
  2. Outputs generated by innovative models (e.g., Prometheus) scored higher in representativeness and professional assessments.

• Limitations and Future Directions:

  1. As the study relies on publicly available Devpost data, the definition of "creativity" in Hackathons may vary across contexts.
  2. Current LLM evaluations face consistency issues, necessitating broader customized training and optimization of human-machine collaboration.
  3. Future research could extend to other domains (e.g., patents, grant proposals) to test the applicability of the methods in different contexts.

Through these findings and contributions, this study makes significant advancements in large-scale creativity evaluation and methodology exploration for Hackathons, while also outlining directions for future research.