Writing out the Storm: Designing and Evaluating Tools for Weather Risk Messaging

Title of the Paper

Writing out the Storm: Designing and Evaluating Tools for Weather Risk Messaging

Paper Information

• Field of Study: Human-Computer Interaction (HCI), Crisis Informatics, Risk Communication
• Keywords: Risk Communication, Disaster Management, Writing Support Tools, Climate Change Adaptation, Extreme Weather, Human-Computer Interaction, Text Analysis, Social Inclusivity, Machine Learning

Research Background and Problem Statement

• Problems and Challenges:

  • Climate change has led to an increase in extreme weather events, necessitating more effective risk communication to mitigate loss of life and property.
  • Crafting effective weather risk messages requires specific themes and presentation styles. However, under high-pressure environments, professional risk communicators may face uncertainty and competing priorities, leading to gaps in information flow and public risk perception.
  • Current research on risk communication tools and technologies is limited, and scalable evaluation of risk messages and the development of writing support tools remain emerging areas.

• Significance:

  • Effective risk information can encourage the public to take protective measures, forming the foundation for disaster mitigation and climate change adaptation.
  • As the frequency and intensity of extreme weather events increase, the demand for efficient risk communication tools becomes more urgent.

• Research Motivation and Related Work:

  • Previous studies have focused on risk message dissemination via social media or online risk communication during health crises, but tools specifically addressing climate and extreme weather risks are relatively scarce.
  • Existing creative writing support tools are primarily limited to specific tasks in narrative or scientific writing, offering potential inspiration for developing risk communication writing tools.

Solution

• Proposed Approach:

  • Design a string-matching algorithm to semi-automatically evaluate the alignment of weather risk messages with best practices.
  • Conduct interviews with professional risk communicators to explore the design space for writing support tools in risk communication.

• Innovations:

  • Developed a highly interpretable and computationally efficient string-matching method to automatically identify key themes (e.g., hazards, time, location) in risk messages and provide quantifiable evaluations.
  • Incorporated best practices in risk communication into tool design, including automatic identification of missing themes and offering suggestions.
  • Designed a "customized risk information" approach to address the needs of different populations, such as vulnerable groups.

• Implementation Steps and Key Techniques:

  1. Data Analysis: Selected 9,181 weather alerts issued by Environment and Climate Change Canada (ECCC) and analyzed whether risk messages adhered to best practices using techniques such as text readability scoring (FRE) and string matching.
  2. Theme Analysis: Annotated key themes in messages, such as location, time, and impact, using manual and semi-automated methods.
  3. Interview Study: Conducted interviews with 10 professional risk communicators to validate preliminary findings and explore the design space for writing support tools from a user perspective.

Research Outcomes

• Specific Findings:

  • Quantified the alignment of Canadian weather risk messages with best practices, revealing inconsistent coverage of certain themes (e.g., affected populations, recommended actions).
  • Developed a prototype writing support tool capable of identifying missing themes and supplementing relevant content through recommendations from historical message texts.
  • Validated the tool's potential value, particularly for less experienced risk communicators, in improving the standardization and quality of risk messages.

• Advantages Over Existing Solutions:

  • Provides automated analysis capabilities for datasets ranging from small to large scales, overcoming sample limitations of traditional manual methods.
  • Integrates best practices for writing support, effectively highlighting areas for improvement.

• Experimental and Evaluation Results:

  • The average readability score of risk messages was below the recommended level (80–100), potentially affecting public comprehension.
  • The semi-automated classification method achieved over 90% accuracy in text annotation.
  • Simplified language examples did not fully meet risk communication needs, indicating the need for more refined design strategies.

• Limitations and Future Directions:

  • The current algorithm's test dataset is relatively small, potentially limiting its ability to capture data diversity and anomalies comprehensively.
  • The generation of simplified language results is constrained by prompt design; future work could explore optimized prompt engineering.
  • Further research is needed to combine AI tools to meet the customized needs of different groups and enhance public trust in risk information.

Conclusion

This study highlights a string-matching-based method for evaluating weather risk messages and proposes design ideas for risk communication writing support tools, including customized information for specific groups and optimized theme coverage in risk communication. Future work will focus on expanding the scope of data analysis, exploring practical technology integration, and advancing participatory design-based development of risk communication tools to enhance their service capabilities for diverse users.