It's About Time: A View of Crowdsourced Data Before and During the Pandemic

Title of the Paper

It’s About Time: A View of Crowdsourced Data Before and During the Pandemic

Paper Information

• Research Area: Human-Computer Interaction, Computer Vision, Data Bias and Reproducibility
• Keywords: Crowdsourcing, Data Reproducibility, Image Annotation, Reporting Bias, Temporal Sensitivity

Research Background and Questions

• What issues or challenges did the authors identify?

  • Crowdsourced data is widely used to train computer vision algorithms, but it may contain reporting bias. Labels provided by human annotators reflect not only image content but also social information and personal cognition.
  • Major events in 2020 (e.g., the COVID-19 pandemic and social unrest surrounding racial discrimination) may have influenced annotators' labels, leading to temporal sensitivity in the labeled data.

• Why is this issue important?

  • Social biases can be amplified through crowdsourced data and embedded into computer vision algorithms, potentially exacerbating societal negative impacts (e.g., racial and gender stereotypes).
  • Exploring the temporal sensitivity of data helps understand the influence of social context on data and improves data quality.

• Research Motivation and Related Work

  • The authors aim to reveal the impact of temporal sensitivity on image annotation tasks by comparing data collected for the same task in 2018 and 2020, and to verify whether temporal changes in the data exist.
  • Unlike traditional studies focusing on data bias, this research investigates the dynamic influence of major social events on data outcomes.

Solutions

• What methods or solutions did the authors propose?

  • The authors designed and repeatedly executed an open-ended image annotation task using standardized human images (from the Chicago Face Database), collecting data in both 2018 and 2020.
  • They analyzed the themes of labels used by annotators (health status and racial identity) and labeling styles (specific vs. abstract) to explore the impact of temporal changes on data.

• What is innovative about this solution?

  • This study systematically analyzes crowdsourced data from the perspective of temporal sensitivity for the first time, linking major social events to significant changes in labeling data.
  • It introduces quantitative methods to evaluate data changes through the unique occurrence of labels, supporting social science theories on bias propagation.

• What are the implementation steps? What key technologies were used?

  1. Reproducing the open-ended image annotation task on the Appen platform.
  2. Data preprocessing: spell checking and language translation (translating Spanish labels into English).
  3. Data classification and thematic clustering: categorizing labels into health-related themes (e.g., weight, symptoms) and identity-related themes (e.g., race, ethnicity), further distinguishing specific and abstract labels.
  4. Statistical analysis: using chi-square tests and two-proportion z-tests to evaluate changes in label usage frequency and distribution.

Research Findings

• What specific findings were achieved?

  • The use of health-related labels significantly increased in 2020, particularly weight-related labels (e.g., "fat," "thin").
  • Regarding identity-related labels, annotators in 2020 more frequently specified a person’s racial identity (e.g., "Black skin," "White") rather than generalized labels (e.g., "dark," "light").
  • Annotators in 2020 were more likely to use abstract labels when describing minority groups (e.g., Black and Asian individuals), reflecting heightened sensitivity to racial issues.

• How does it compare to existing solutions?

  • Provides empirical evidence on how major social events significantly influence data outcomes, filling a research gap in the field of crowdsourced data bias.
  • The analysis goes beyond label content to include labeling styles, offering a more comprehensive understanding.

• What were the experimental or evaluation results?

  • Statistical tests demonstrated the association between major events in 2020 (e.g., the pandemic and the "Black Lives Matter" movement) and data bias.
  • Significant changes in label frequency and type indicate that social events influence annotators' attention and cognitive styles.

• Limitations and Future Directions

  • Limitations:

    • Due to the anonymity of crowdsourced sampling, it was not possible to invite the same annotators to participate in the task, making strict causal validation difficult.
    • The study focused solely on a single task (image annotation) and did not extend to other crowdsourcing domains.

  • Future Directions:

    • Explore other types of open-ended tasks (e.g., audio or video descriptions) to verify the dynamic impact of social events on crowdsourced data.
    • Investigate methods to capture and mitigate the impact of temporal changes on data quality, such as designing questionnaires to assess annotators' emotional states.