CodeToon: Story Ideation, Auto Comic Generation, and Structure Mapping for Code-Driven Storytelling

Document Title

CodeToon: Story Ideation, Auto Comic Generation, and Structure Mapping for Code-Driven Storytelling

Document Information

• Domain: Programming Education and Visualization Tools
• Keywords: Code-driven storytelling, comics, coding strips, authoring tools, programming education, computer science, automatic comic generation, structure mapping, visual programming
• Conference: The 35th Annual ACM Symposium on User Interface Software and Technology (UIST '22)
• Year: 2022

Research Background and Problem

• Problems and Challenges:

  • The abstract nature of programming languages makes them difficult for beginners to learn, especially text-based programming languages, which have complex symbolic expressions and cumbersome rules.
  • While comics (e.g., "coding strips") used in teaching can help explain code execution and underlying computational concepts, their connection to code and the labor-intensive creation process hinder widespread adoption.
  • The process of creating "coding strips" requires creators to generate stories from code and then transform these stories into comics, which is time-consuming, tedious, and demands creativity.

• Importance:

  • Reducing barriers and abstraction in learning programming to enhance learners' intuitive understanding.
  • Using vivid visual presentations (comics) to engage students and improve interactivity in the teaching process.

• Research Motivation:

  • Existing solutions for efficiently generating code-related stories and comics remain inadequate.
  • There is an urgent need to develop automated, creator-friendly tools that enable educators to quickly produce high-quality teaching materials.

Solution

• Approach:

  • Developed a comic creation tool named CodeToon, with core functionalities including:
    1. Story Generation: Using metaphor recommendation mechanisms to generate story templates aligned with code.
    2. Automatic Comic Generation: Automatically generating visualized comics from user-input story templates.
    3. Structure Mapping: Ensuring 1-to-1 mapping between code, story, and comic, maintaining semantic consistency.

• Innovations:

  1. Based on Gentner's "Structure Mapping Theory," the tool establishes clear mapping relationships between code-to-story and story-to-comic.
  2. Introduced visual narrative syntax for designing and generating comic templates.
  3. Significantly improved the creation efficiency of "coding strips" by automating the story ideation and comic design processes.

• Implementation Steps and Key Technologies:

  • In the tool's story panel, users can input their code, and the system generates corresponding story templates.
  • While filling in the templates, users receive metaphor recommendations from CodeToon (e.g., equating x=10 to "time is 10 o'clock").
  • The tool parses this content and automatically generates comics representing the logic of code execution.
  • Through 1-to-1 mapping, the tool ensures clear semantic associations between code, story, and comic.

Research Outcomes

• Specific Results:

  • CodeToon enables users to quickly create accurate, informative, and educational "coding strips" in significantly less time.
  • The design ensures clear structural mapping between code, story, and comics, enhancing understanding of programming semantics and logic.

• Advantages Comparison:

  • Compared to traditional fully manual creation methods, CodeToon simplifies the comic creation process.
  • It demonstrates how abstract-to-concrete visual transformation can improve teaching effectiveness and reduce learners' cognitive load.

• Experiment and Evaluation Results:

  • An experiment with 24 participants showed:
    • Users working with CodeToon created comics faster (saving an average of 6 minutes) and completed tasks more efficiently.
    • Participants generally found CodeToon-generated comics highly accurate and valuable for teaching purposes.
    • CodeToon scored significantly higher on the Creativity Support Index (CSI), particularly in terms of tool enjoyment and user experience.

• Limitations and Future Directions:

  • Limitations:
    • The visual design quality and thematic diversity of generated comics still require improvement.
    • Current user base primarily consists of individuals with programming knowledge; further testing is needed for non-professional learners.
  • Future Directions:
    • Integrate generative models (e.g., GPT-3 or Codex) to enhance the flexibility and creativity of story generation.
    • Conduct extensive testing in educational settings to explore its effectiveness across diverse learning scenarios and audiences (e.g., beginners and non-programming students).
    • Expand support for other programming paradigms and languages (e.g., object-oriented programming).

Overall, CodeToon demonstrates a novel and efficient approach to integrating code, stories, and visual comics to support computer science education, showcasing significant potential for practical applications.