Mobilizing Crowdwork: A Systematic Assessment of the Mobile Usability of HITs

Title of the Paper

Mobilizing Crowdwork: A Systematic Assessment of the Mobile Usability of HITs

Paper Information

• Research Domain: Study on the use of mobile crowdsourcing platforms
• Keywords: Crowdwork, Mobile Usability, Taxonomy, Human Intelligence Tasks, Mechanical Turk, HIT Usability, Crowdworker Preferences, Task Design, Microtask, Interaction Characteristics

Research Background and Issues

• Identified Problems or Challenges:

  • Current task designs for crowdsourcing work (e.g., Amazon Mechanical Turk, MTurk) are primarily desktop-centric, with insufficient support for mobile devices like smartphones, resulting in poor user experience.
  • Although smartphones have a vast user base, the mobile usability characteristics of Human Intelligence Tasks (HITs) remain unclear, limiting the effectiveness of mobile devices in crowdsourcing work.

• Significance:

  • Crowdsourcing work has become a significant form of digital employment in the 21st century, attracting thousands of new users daily to crowdsourcing platforms. Moreover, smartphone users outnumber desktop and laptop users, and many crowdworkers rely on smartphones for work, making the optimization of task design for mobile devices critically important.
  • If tasks can be efficiently utilized across devices, it would not only enhance the productivity of crowdworkers but also provide more employment opportunities to low-income individuals and marginalized groups.

• Research Motivation and Related Work:

  • Existing research indicates that mobile devices differ from traditional workstations in terms of screen size, computational capabilities, etc., but there has been no systematic analysis of how task interfaces can be adapted for mobile use.
  • Previous studies on performing certain crowdsourcing tasks (e.g., image classification, voice transcription) via smartphones have demonstrated the potential of mobile devices, but research on task design and usability remains fragmented and lacks a systematic framework.

Solution

• Research Methods:

  • The authors developed a new taxonomy to define the usability characteristics of HITs on mobile devices. The research process involved three stages:
    1. Literature Review: The research team summarized preliminary mobile usability characteristics of HITs through relevant literature.
    2. Validation and Expansion: Feedback from MTurk workers was collected via online surveys to further validate and expand the taxonomy.
    3. Practical Demonstration: The taxonomy was applied to analyze 519 tasks scraped from MTurk to evaluate their mobile usability.

• Innovative Contributions:

  • Proposed a taxonomy for mobile usability of microtasks, encompassing two dimensions—task characteristics and interaction characteristics—with seven specific features.
  • Systematically linked the characteristics of mobile tasks with workers' actual experiences and conducted the first mobile usability analysis on a real HIT dataset.

• Implementation Steps and Key Techniques:

  1. Development of Classification Standards: Task-related characteristics were extracted from literature using Nominal Group Technique (NGT) and optimized through expert team discussions.
  2. Worker Surveys: Questionnaires were designed for crowdworkers to understand their current practices and improvement needs for completing tasks on smartphones.
  3. Task Scraping and Quality Evaluation: Web crawlers were developed to extract MTurk task data, and screenshots were assessed to evaluate the usability characteristics of task interfaces.

Research Outcomes

• Specific Results:

  • Proposed a taxonomy divided into task characteristics (e.g., task segmentation, task size, off-site requirements) and interaction characteristics (e.g., multi-device requirements, interaction methods).
  • Surveys revealed that crowdworkers prefer certain types of tasks on mobile devices, such as surveys or simple multiple-choice questions.
  • Analysis of 519 tasks showed that over 50% of tasks exhibited significant usability issues on mobile devices (e.g., interface incompatibility, difficulties with small-screen scrolling), with mobile usability varying based on task characteristics and design.

• Comparison with Existing Solutions:

  • The study goes beyond fragmented research on specific mobile crowdsourcing scenarios, providing a comprehensive and evaluable framework.
  • Emphasized the importance of task design for mobile devices and demonstrated that certain task types can significantly improve mobile usability through simple adjustments.

• Experimental or Evaluation Results:

  • Task types such as image classification and surveys were judged as most suitable for completion on mobile devices.
  • Over 70% of tasks did not require workers to leave the main task interface to find information, indicating that meeting the "off-site-free requirement" significantly enhances mobile usability.

• Limitations and Future Directions:

  • Limitations:
    • The study was limited to the MTurk platform and did not include tasks from other crowdsourcing platforms (e.g., Upwork).
    • The taxonomy's adaptability to devices other than mobile phones (e.g., tablets, voice assistants) was not thoroughly explored.
    • The sample size of scraped tasks (519) was relatively small and did not encompass all task types on MTurk.
  • Future Directions:
    • Expand the scope of research to cover more device types and task types.
    • Develop automated tools to evaluate task design's mobile usability in real-time based on the taxonomy and provide optimization suggestions to task designers.
    • Explore frameworks for cross-device task allocation and management to maximize the potential of mobilizing work across different devices.