Visual StoryCoder: A Multimodal Programming Environment for Children's Creation of Stories

Title of the Paper

Visual StoryCoder: A Multimodal Programming Environment for Children's Creation of Stories

Paper Information

• Research Domain: Computational thinking education for children and design of programming tools
• Keywords: children, computational thinking, programming education, multimodal interface, digital storytelling, block-based programming, voice interaction

Research Background and Problem Statement

• Identified Problems or Challenges:

  • Current computational thinking education tools face adaptation issues for preschool children, such as requiring high literacy levels or expensive hardware.
  • Symbolic block-based programming environments can lower learning barriers, but without teacher guidance, children struggle to independently advance or create structured programming projects.
  • Voice-based tools can bypass literacy obstacles and directly teach computational thinking concepts but may lead to difficulties in memory retention and interface navigation.

• Significance:

  • Early education in computer science fosters computational literacy and interest, particularly benefiting women and underrepresented groups.
  • Creative programming activities that embed personal meaning enhance children's learning experiences and project quality.

• Research Motivation and Related Work:

  • Previous tools like ScratchJr reduce literacy demands through symbolic programming, while voice systems like StoryCoder directly teach concepts. Each has strengths and limitations, such as ScratchJr relying on visual symbols rather than voice interaction, and StoryCoder facing challenges in navigation complexity and customization.
  • This study aims to explore the integration of voice interface teaching advantages with the navigational simplicity of block-based programming to develop a multimodal programming system for children.

Solution

• Method or Solution:

  • Introduced Visual StoryCoder, a multimodal tablet application designed to introduce computational thinking to children aged 5-8 through creative storytelling.
  • Combines the strengths of voice and graphical interfaces, featuring a guiding voice agent for real-time instruction and a modular graphical interface for memory aids and simplified navigation.

• Innovations:

  • Integrates voice teaching and block-based programming to reduce reliance on parents and teachers, enabling independent learning for children.
  • Incorporates storytelling, allowing users to draw graphics and add personal voice recordings to enhance the personal significance and creativity of projects.
  • Developed features for children to abstract and modularize projects, including chapter segmentation, event triggers, and variable switching.

• Implementation Steps and Key Technologies:

  1. Story Planning: Children use voice input and drawings to define story characters, settings, and problems.
  2. Modular Design: Users create decomposable story structures using graphical components like pages and chapters, incorporating programming concepts such as loops and event triggers.
  3. Voice Navigation Agent: A virtual voice character provides real-time guidance, using natural language input to guide users step-by-step.
  4. System Optimization and Testing: Iterative testing with adults and children to refine interface design and operational smoothness.

Research Outcomes

• Specific Outcomes:

  • Visual StoryCoder supports children's understanding of the system in independent learning environments without excessive reliance on external instruction.
  • Once familiar with the system, the tool enhances users' ability to create high-quality code, particularly excelling in code organization (sequence, modularization).
  • The system encourages children to create projects with personal significance, including creations based on their interests, experiences, or social themes.

• Comparative Advantages Over Existing Solutions:

  • Compared to existing block-based programming platforms (e.g., ScratchJr), Visual StoryCoder significantly reduces dependence on adult guidance.
  • In code quality assessments, users scored higher than control groups, especially in aspects like parallelism and data usage.
  • Compared to StoryCoder, which relies solely on voice navigation, Visual StoryCoder combines open canvas and voice input, expanding the scope of children's creative content.

• Experimental or Evaluation Results:

  • System Understanding: Controlled comparative experiments revealed that built-in voice guidance significantly improved users' ability to understand the system without external guidance.
  • Code Quality: During initial programming tasks, all groups experienced a learning curve, but supportive voice guidance effectively improved Visual StoryCoder users' code quality in subsequent tasks.
  • Engagement and Creativity: Both systems generated numerous projects related to participants' interests, but Visual StoryCoder particularly supported creative projects based on open-ended interaction.

• Limitations and Future Directions:

  • Limitations: Voice guidance created fixed design frameworks for specific learning tasks, potentially restricting creative freedom for advanced users.
  • Future Directions:
    • Explore adaptive voice guidance systems to enhance flexibility.
    • Conduct long-term studies on the potential learning effects and transferability of multimodal interfaces in computational thinking education.
    • Expand applications to include robotics programming and more diverse interaction modes.

This study demonstrates the potential of multimodal learning tools in promoting children's programming education, enabling the creation of complex, high-quality projects, and achieving interdisciplinary teaching objectives.