“I Got Some Free Time”: Investigating Task-execution and Task-effort Metrics in Mobile Crowdsourcing Tasks

Title of the Paper

“I Got Some Free Time”: Investigating Task-execution and Task-effort Metrics in Mobile Crowdsourcing Tasks

Paper Information

• Field of Study: Task execution and user response in mobile crowdsourcing
• Keywords: Mobile crowdsourcing, interruption, notification, ESM (Experience Sampling Method), mixed-effects logistic regression, qualitative analysis

Research Background and Issues

• Problems and Challenges:

  • It remains unclear how to identify the optimal moments for users to engage with crowdsourcing tasks on mobile platforms.
  • The timing and type of tasks users choose, and how these choices are influenced by contextual factors and personal conditions (e.g., stress and energy levels), have not been thoroughly studied.
  • Mechanisms driving task execution in commercial mobile crowdsourcing platforms need further optimization.

• Significance of the Problem:

  • Mobile crowdsourcing is an efficient method for data collection, with potential applications in navigation improvement, event monitoring, and machine learning model development.
  • Understanding when users are most likely to complete tasks is critical for increasing task completion rates, particularly for commercial crowdsourcing platforms.

• Research Motivation and Related Work:

  • Inspired by previous studies on receptivity, which found that users are more receptive during activity transitions.
  • Current research has yet to explore the specific impact of task context (e.g., activity characteristics, transition types) and personal conditions on task selection in commercial mobile crowdsourcing.

Solution

• Methods and Approach:

  • A six-week mixed-method study combining quantitative data, experience sampling questionnaires, and qualitative interviews to analyze user behavior on two commercial mobile crowdsourcing platforms.
  • Data on task behavior was collected using screen recordings and random task notifications.
  • Mixed-effects logistic regression models were employed to analyze the impact of activity context and personal states on task selection and execution.

• Innovations:

  • Activity transitions were subdivided into multiple sub-contexts (e.g., within current activity, between activities, after an activity but before the next activity).
  • Stress levels and energy levels were integrated to analyze their influence on task behavior.
  • A combination of quantitative data and qualitative interviews provided comprehensive insights into how various factors affect task selection and execution.

• Implementation Steps and Key Techniques:

  1. Develop an Android-compatible research application to track task execution and interruptions during crowdsourcing activities.
  2. Use the Experience Sampling Method to collect contextual and psychological state data via immediate post-task questionnaires.
  3. Perform data cleaning and coding, followed by analysis of task execution and effort metrics using mixed regression models.
  4. Conduct qualitative interviews with participants and use visual grouping analysis to summarize key insights.

Research Findings

• Key Discoveries:

  1. Users are more likely to actively execute crowdsourcing tasks during activity transitions (e.g., between activities) rather than during prolonged idle periods or within ongoing activities.
  2. Stress levels are significantly negatively correlated with proactive task completion, while energy levels more prominently affect task effort (e.g., duration and number of operations).
  3. Task selection is influenced by the primary attributes of current and preceding activities (e.g., complexity, attention requirements), with users preferring tasks similar to the attributes of their current activity.

• Comparison with Existing Solutions:

  • The granular categorization of activity transitions (e.g., five specific contexts) surpasses traditional studies.
  • The focus extends beyond analyzing interruption receptivity to include detailed examination of task type selection and effort metrics.

• Experimental or Evaluation Results:

  • Quantitative: 3,603 valid experience sampling questionnaire responses, with 68.8% of cases involving at least one completed mobile task. Average task session duration was 113.1 seconds, with an average of 35.13 user operations per session.
  • Qualitative: Synthesized reasons for task selection, including factors such as relaxed mood and short task duration.

• Limitations:

  • The intensity of task notifications may have artificially increased proactive task completion rates, failing to fully simulate real-world scenarios.
  • The study did not directly measure the quality of user-contributed tasks.
  • Data samples were primarily collected from Taiwanese users, which may limit the global applicability of findings.

• Future Directions:

  1. Investigate the impact of task context on task completion quality.
  2. Validate the generalizability of task context and personal condition effects across broader cultural backgrounds.
  3. Optimize personalized task notification designs to create more precise matching mechanisms for different tasks.

Design Implications

• Task platforms should prioritize sending notifications during activity transitions rather than prolonged idle periods.
• Notification designs should minimize perceived interruptions, such as silent push notifications that allow users to initiate tasks voluntarily.
• Dynamically adjust notification content and task recommendations based on users' stress and energy levels.
• Develop machine learning models for task selection that integrate users' current activities and psychological states.

Summary: This study investigates the optimal timing for users to execute mobile crowdsourcing tasks and examines how task context and personal conditions influence task selection. It offers design recommendations for improving task notification mechanisms on commercial platforms, contributing theoretical and practical insights to mobile HCI and crowdsourcing research.