AVEC: An Assessment of Visual Encoding Ability in Visualization Construction

Research Background and Problem

• Identified Issues or Challenges: Current assessments of visualization abilities predominantly focus on visualization interpretation skills, while neglecting the evaluation of skills related to visualization construction (e.g., selecting appropriate visual encodings). This omission limits the understanding of overall visualization literacy and makes teaching and skill improvement more challenging.
• Significance: With the widespread availability of data and the increasing public engagement in visualization construction (e.g., creating charts via Excel), assessing construction abilities can help improve visualization literacy among non-experts (general public and beginners) and is crucial for designing effective interventions.
• Research Motivation and Related Work: This work is driven by the limitations of existing assessments. The authors aim to develop a systematic method to evaluate individuals' abilities in mapping data to visual channels, addressing gaps in existing studies (e.g., VLAT and CALVI).

Solution

• Method or Solution: The authors propose an assessment tool called AVEC (Assessment of Visual Encoding Ability in Visualization Construction) to measure individuals' ability to correctly select visual encodings during visualization construction.
  1. Create an initial question bank consisting of nine tasks that cover various visualization tasks and chart types.
  2. Develop a user-friendly online assessment tool that supports the combination selection of visual encodings.
  3. Design and validate an automated scoring system to evaluate response quality based on expert scoring results.
  4. Validate and optimize the question bank and scoring system using test data from 95 participants.
• Innovations:
  1. Unlike existing assessment methods that use multiple-choice formats, AVEC employs a constructive response format requiring participants to generate visualization answers.
  2. Provides a highly flexible online tool capable of supporting diverse chart creation while addressing the impact of user interface complexity on scoring reliability.
  3. Develops an automated scoring system that uses clustering rules derived from expert scores to evaluate answers, significantly reducing manual scoring time.
• Implementation Steps:
  1. Define the design space and task performance requirements to generate the initial question bank.
  2. Validate question content and scoring rules through expert panels.
  3. Use final test results for item analysis (IRT model) to optimize the question bank and identify high-quality items reflecting varying difficulty and performance levels.

Research Outcomes

• Specific Results:
  1. The authors successfully developed a final question bank consisting of eight items, covering different difficulty levels and accurately distinguishing participants' skill differences.
  2. Provided a novel automated scoring mechanism to quickly evaluate the quality of constructive responses.
  3. The online assessment tool received positive user experience feedback (SUS score average: 77.76).
• Advantages Over Existing Solutions:
  1. Enhanced the ability to assess visualization construction skills, making evaluation and teaching more comprehensive.
  2. Offered a scalable question design and scoring method that can be extended to assess other construction-related skills.
  3. The automated scoring system significantly reduced manual scoring workload while improving scoring consistency and flexibility.
• Experimental or Evaluation Results:
  1. Expert scoring validated the content validity of the question design (CVI score above 78%).
  2. Item analysis results showed reliable internal consistency within the final question bank (McDonald's Omega coefficient: 0.73).
  3. IRT model analysis evaluated the difficulty and discrimination ability of each item.
• Limitations and Future Directions:
  1. The impact of individuals' prior knowledge (e.g., familiarity with visualization) on assessment results has not been explored; future research could investigate the relationships between different skills.
  2. The current assessment primarily focuses on visual encoding abilities; future work could expand to include comprehensive evaluations of data processing and advanced construction techniques.
  3. As the skill requirements of assessed individuals increase, more complex yet user-friendly assessment tools need to be developed, along with further optimization of scoring methods (e.g., integrating analytical scoring).

Conclusion

AVEC successfully defined and measured the visual encoding ability within core visualization construction skills, laying the foundation for future research while offering broad application potential in educational interventions, skill improvement, and ability evaluation. Its development process demonstrates how expert knowledge and automated strategies can be combined to effectively assess complex skills, paving the way for higher-level construction abilities and related educational practices.