What Social Media Use Do People Regret? An Analysis of 34K Smartphone Screenshots with Multimodal LLM

Research Background and Issues

What problems or challenges did the authors identify?

• Smartphone users often regret their use of social media, particularly the time spent on recommended content or unconscious browsing. However, the specific relationship between users' feelings of regret, their device operation behaviors, and the design of social applications remains unclear.
• The complexity of social media applications (e.g., incorporating multiple functional modules such as recommended content, messaging, discussion threads) makes it challenging to precisely trace the design factors that lead to regret.
• Conducting a fine-grained analysis of different user intentions and behaviors is a crucial prerequisite for understanding such regretful emotions, yet existing research lacks a granular analytical framework.

Why is this issue important?

• The "engagement-driven" design of social media often traps users in unnecessary and unwanted time consumption, posing a threat to individual digital well-being.
• Understanding the reasons behind users' regret over social media use can not only provide recommendations for improving social media design but also inform policy-making to reduce design elements that trigger negative user emotions.

Research Motivation and Related Work

• Motivation: To fill the gap in fine-grained analysis of the relationship between regretful emotions and usage behaviors, thereby aiding the development of designs that respect user intentions.
• Related Work: Previous studies have indicated that users sometimes feel regret due to non-goal-oriented app usage or consuming content dominated by recommendation algorithms. However, there is currently no systematic analysis based on visual data (e.g., screenshots).

Solution

What methods or solutions did the authors propose?

• The authors proposed a novel method that utilizes multimodal large language models (MLLM) to analyze smartphone screenshots, combined with experience sampling methods, surveys, and interviews for a mixed-methods study of user behavior.
• They automatically collected and analyzed 34,313 smartphone screenshots, using models such as GPT-4o to classify visual content.

What are the innovative aspects of this solution?

1. Automated Screenshot Analysis:

   • Leveraging MLLM to analyze smartphone screenshots addresses the challenge of manually coding large datasets.
   • MLLM can understand user interfaces and classify behaviors represented in screenshots, enabling efficient behavior visualization.

2. Fine-Grained Behavior Analysis:

   • Social media user behaviors are categorized into various fine-grained activities (e.g., active communication, active search, recommended content consumption) rather than treating social media use as a monolithic activity.

3. Three-Dimensional Cross-Analysis of Regret, Intentions, and Behaviors:

   • By comparing user intentions with behaviors, the study reveals the critical impact of intention deviation on feelings of regret.

Implementation Steps

1. Data Collection: Each participant installed an app that captured screenshots every 5 seconds and recorded their intentions at the time.
2. Behavior Classification:
   • GPT was used to analyze screenshot content and classify behaviors (e.g., browsing recommended content, viewing comment sections).
   • Chain-of-Thought Prompting was applied to enhance the model's accuracy in visual inference.
3. Comparison of Intentions and Actual Behaviors:
   • Users' intentions upon entering an app were matched with subsequent activities reflected in the screenshots.
4. Experimental Methods: Daily experience questionnaires and interviews were used to collect users' regret scores and reflections on specific behaviors.

Research Findings

What specific findings were achieved?

1. Impact of Intentions on Regret:

   • Users experienced stronger feelings of regret when opening social media without a clear intention.
   • Activities related to communication and productivity were associated with the lowest levels of regret.

2. Regret Characteristics of Different Social Media Activities:

   • Browsing recommended content and reading comments were the most regret-inducing social media activities.
   • The least regret-inducing activities were direct communication and active searching.

3. Relationship Between Intention Deviation and Regret:

   • Over 60% of communication-related intentions deviated into browsing social media, with such deviations associated with slight increases in regret.
   • However, browsing that stemmed from communication deviations was still less regret-inducing than intentional browsing.

What advantages does this solution have compared to existing ones?

• Achieves a higher level of behavioral granularity through multimodal analysis, rather than evaluating app usage as a whole.
• Provides a fine-grained analytical tool for addressing smartphone overuse through screenshot analysis, as opposed to coarse-grained app-level monitoring.

What were the experimental or evaluation results?

• The agreement between MLLM and human classification in behavior categorization achieved an accuracy of 79.5% (Cohen’s Kappa = 0.736).
• Quantitative model analysis revealed that the proportion of recommended content, duration of usage, and user intentions were significant predictors of regret scores.

Limitations and Future Directions

1. Research Limitations:

   • The study was limited to 17 participants in the United States, lacking validation across a larger and more diverse population.
   • Screenshot analysis encountered some errors in classifying specific activities (e.g., content shared by friends).

2. Future Directions:

   • Explore broader application scenarios (e.g., shopping apps, online video platforms).
   • Develop privacy-preserving, device-based screenshot analysis models.
   • Apply this method to real-time intervention systems to mitigate smartphone overuse.

Through this study, the authors not only uncovered behavioral causes of user regret but also provided data-driven insights for designing interfaces that respect user intentions. This has practical implications for improving digital well-being and addressing issues caused by the attention economy.