What Makes Creators Engage with Online Critiques? Understanding the Role of Artifacts’ Creation Stage, Characteristics of Community Comments, and their Interactions

Title of the Paper

What Makes Creators Engage with Online Critiques? Understanding the Role of Artifacts’ Creation Stage, Characteristics of Community Comments, and their Interactions

Paper Information

• Research Domain: Study of creator engagement behavior in Online Critique Communities (OCCs)
• Keywords: Online Critique Communities, behavioral engagement, emotional engagement, cognitive engagement, creation stage, community feedback characteristics

Research Background and Problem

• Identified Problems or Challenges:

  • In Online Critique Communities, creators' behavioral, emotional, and cognitive engagement with feedback comments directly impacts their skill improvement. However, existing studies lack comprehensive data-driven analysis and fail to fully understand creators' varied engagement responses influenced by creation stages and feedback characteristics.
  • Creators at different creation stages may have different expectations and acceptance mechanisms for feedback. For instance, creators of in-progress works may require timely and specific improvement suggestions, while creators of completed works may focus more on recognition or minor improvement suggestions.
  • Previous research often relies on small-scale experiential methods, lacking quantitative studies supported by large-scale data.

• Significance:

  • Understanding creators' responses to feedback is crucial for helping them improve their artistic works and creative skills.
  • These insights can assist in optimizing online community design, providing creators with more targeted technical support and feedback guidance.

• Research Motivation and Related Work:

  • The study combines qualitative and quantitative methods to expand understanding of how creators respond to feedback at different creation stages and what feedback characteristics effectively enhance creator engagement.
  • Through long-term OCC data analysis, it explores optimal methods for tailoring comments to creation stages and feedback characteristics to promote creator engagement.

Solution

• Proposed Solution:

  • Modeling creators' engagement behavior across three dimensions: behavioral engagement, expressed emotional engagement, and expressed cognitive engagement.
  • Using deep learning techniques to automatically classify the creation stage of target works ("in-progress" or "completed") and extract feedback characteristics (e.g., specificity, actionability, justification, emotional tone), combined with regression modeling to analyze the impact of these factors on creator engagement levels.

• Innovations:

  • Proposed a deep learning model based on large-scale OCC comment data to predict creation stages and quantify feedback characteristics.
  • Systematically analyzed the interaction between creation stages and feedback characteristics and their impact on creators' engagement behavior.
  • Introduced new metrics for feedback characteristics, such as actionability, specificity, and comment delay, providing novel dimensions for OCC feedback research.

• Implementation Steps and Key Technologies:

  1. Data Collection and Processing:
     • Collected five years of data from four visual art-related OCCs on Reddit, including posts, comments, and creator replies, totaling 81,346 posts, 312,437 comments, and 125,333 replies.
  2. Engagement Behavior Modeling:
     • Used the VADER tool to analyze creators' emotional engagement.
     • Developed a BERT-based deep learning model to predict creators' cognitive engagement levels while measuring behavioral engagement (whether comments were replied to).
  3. Creation Stage Classification:
     • Classified target works as "in-progress" or "completed" based on post text information using deep learning models (including BERT and enhanced BERT).
  4. Quantitative Regression Analysis:
     • Built three sets of regression models to evaluate the impact of creation stages, feedback characteristics, and their interactions on behavioral, emotional, and cognitive engagement.

Research Findings

• Specific Findings:

  • Behavioral Engagement: Compared to completed works, creators sharing in-progress works exhibited lower behavioral engagement but showed a relatively higher probability of responding to highly specific feedback.
  • Emotional Engagement: Creators of in-progress works tended to express more negative emotional reactions but demonstrated more positive emotional engagement with highly specific feedback.
  • Cognitive Engagement:
    • Creators of in-progress works generally exhibited higher levels of cognitive engagement.
    • For creators of completed works, higher feedback specificity and well-justified feedback significantly enhanced cognitive engagement.

• Advantages Over Existing Solutions:

  • Provides a quantitative perspective based on large-scale data, overcoming limitations of existing small-sample qualitative studies.
  • Considers the interaction between creation stages and feedback characteristics, offering practical recommendations for improving OCCs.

• Experimental and Evaluation Results:

  • Regression analysis results indicated that the interaction between different creation stages and feedback characteristics significantly influenced creators' engagement behavior and patterns.
  • The deep learning model achieved an accuracy of 0.87 on the test set.

• Limitations and Future Directions:

  1. The current study focuses on art-related communities, but future work could extend to other types of OCCs, such as writing, music, and dance.
  2. Measurement of creator engagement behavior is primarily based on publicly visible actions, which may differ from actual behavior.
  3. Causal experiments and in-depth interviews are needed to better explain the mechanisms of influence.
  4. Future research could explore the long-term impact of creators' historical posts on their interaction behavior and further improve the model's ability to predict "post-action behavior."

This study highlights the critical importance of providing personalized support tailored to creators' creation stages and precisely enhancing their engagement levels for the development of online communities.