EvalMe: Exploring the Value of New Technologies for In Situ Evaluation of Learning Experiences

Title of the Paper

EvalMe: Exploring the Value of New Technologies for In Situ Evaluation of Learning Experiences

Paper Information

• Thematic Area: In situ evaluation of learning experiences based on touch interaction technologies
• Keywords: Education, learning, touch interface, schools, educational environments, children, parents

Research Background and Issues

• Problems and Challenges:

  • Traditional methods of learning reflection, such as post-class questionnaires or surveys, often miss students' real-time feelings during specific moments of the course.
  • Current classroom response systems (CRS) are primarily used to check students' understanding of questions but lack the ability to promote reflection on learning experiences or track changes in responses over time.
  • Students who lack reflective practices often wait for teacher guidance, with reflection being mostly an individual activity, failing to fully utilize opportunities for shared reflection among peers.

• Significance:

  • Reflection deepens students' understanding of learning content, helps them evaluate learning outcomes, and improves learning strategies.
  • Shared reflection fosters collaborative learning among peers and assists teachers in adjusting instructional design.

• Research Motivation and Related Work:

  • Existing studies show that touch interaction devices and gamified designs have significant effects in promoting reflection among children.
  • Innovative educational tools can enhance the diversity of learning feedback and increase students' engagement in the feedback process.

Solution

• Proposed Solution:

  • Developed a flexible touch-based tool named "EvalMe" for real-time collection of learning experience data, supporting both individual and shared reflection.
  • EvalMe consists of three components:
    1. EvalMe Sender: A handheld device for students to answer questions;
    2. EvalMe Receiver: A receiving device that transmits students' responses via wireless communication;
    3. EvalMe Desktop: A desktop application for data configuration, collection, and visualization.

• Innovative Features:

  • Utilizes physical knobs and LED-triggered displays for gamified and dynamic interaction during the response process.
  • Supports both anonymous and pseudo-anonymous data collection (via RFID tags), protecting privacy while enabling tracking of students' response changes over time.
  • Emphasizes shared reflection by using data visualization (e.g., bar charts) to facilitate interactive discussions between students and teachers.

• Implementation Steps and Key Technologies:

  1. The design phase employed the "Research through Design" method, collaborating with educational practitioners to generate initial design requirements;
  2. After device development, EvalMe was deployed in various educational workshops for testing;
  3. Data was transmitted wirelessly (via LoRa network) from the Sender to the Receiver, then visualized and analyzed through the desktop application;
  4. Technologies such as knob adjustment, LED feedback, and pseudo-anonymous identity authentication were used to enhance user interaction.

Research Outcomes

• Specific Outcomes:

  • During workshop deployments, EvalMe successfully collected real-time feedback from students, promoting reflection and enhancing interaction between students and teachers.
  • The physical design of the device (knobs and LED displays) improved the data input experience, making tasks more intuitive and engaging.
  • Aggregated data visualization enabled workshop leaders to quickly adjust teaching strategies and share learning outcomes with external evaluation agencies.

• Advantages Over Existing Solutions:

  • Compared to traditional classroom response systems or post-class surveys, EvalMe focuses more on real-time feedback and participant experience, capturing more instantaneous reactions and dynamic changes.
  • Its touch-based design and gamified features significantly increased students' engagement.

• Experimental or Evaluation Results:

  • In group usage, a single device was sufficient for 15-16 students. However, for workshops with 30 participants, the time taken to pass the device was too long, leading to efficiency issues.
  • In some scenarios, students using the device engaged in public discussions, promoting reflection; in other scenarios, it became a distraction for some students.

• Limitations and Future Directions:

  • Limitations:
    • Currently tested only in educational workshops, its effectiveness in traditional classroom settings remains unverified.
    • Student engagement with data visualization was relatively low, possibly due to difficulty in associating themselves with the displayed results.
  • Future Directions:
    • Adapt the system for classroom environments, such as modifying interaction design to align with traditional classroom discipline norms.
    • Enhance data presentation methods to make them more appealing and comprehensible (e.g., incorporating personalization or narrative-based displays).
    • Expand testing and optimization to other educational contexts, such as virtual education or hybrid learning environments.