Crepe: A Mobile Screen Data Collector Using Graph Query

Paper Title

Crepe: A Mobile Screen Data Collector Using Graph Query

Publication Info

• Topic area: Mobile screen data collection for academic research.
• Keywords: Mobile data collection, Graph Query, Android Accessibility Service, programming by demonstration, user privacy, screen UI data, algorithm auditing, HCI, data transparency, academic research.

Background and Problem

• Problem / challenge: Academic researchers face significant challenges in collecting mobile screen data due to restricted access controlled by platforms and app developers, leading to a "data monopoly." Existing tools focus on mobile sensing data rather than screen content, and current methods are either inefficient, intrusive, or lack specificity.
• Significance: Screen data provides critical insights into user interactions, preferences, and behaviors, enabling research in areas like algorithm auditing, user experience, and social media analysis. Democratizing access to screen data can empower researchers and users while addressing privacy concerns.
• Motivation and related work: Prior tools like Aware and ShiptCalculator focus on sensor data or manual screenshot collection, while others like XPath lack flexibility for screen-specific data. Existing solutions often fail to address privacy, specificity, or ease of use, leaving a gap for a tool like Crepe.

Solution

• Proposed approach: Crepe, a no-code Android app, enables researchers to collect specific screen data using a programming-by-demonstration paradigm and a novel Graph Query language.
• Novelty:
  1. Introduction of Graph Query, a deterministic query language for identifying, locating, and collecting target UI elements.
  2. A low-code, customizable mobile screen data collection tool for researchers, emphasizing participant privacy and transparency.
  3. Comprehensive evaluation through user studies on usability, accuracy, and real-world performance.
• Procedure and key techniques:
  • Researchers specify target data by tapping on screen elements, which Crepe translates into Graph Queries.
  • Graph Queries use semantic, hierarchical, and spatial relations to identify target data.
  • Crepe operates via Android Accessibility Service, ensuring compatibility across diverse apps and frameworks.
  • Transparency features include visual highlights during data collection and a dashboard for participants to review collected data.

Results

• Concrete findings:
  • Study 1: Researchers successfully created data collectors with 100% accuracy in generating Graph Queries.
  • Study 2: Crepe achieved an overall F1 score of 96.0% (precision: 96.0%, recall: 95.8%) across three apps (Instagram, Uber, Chrome).
  • Study 3: Minimal impact on device performance, with battery usage ranging from 0% to 14% (most ≤2%), and no reported user interruptions.
• Advantage over baselines:
  • Higher specificity and flexibility compared to XPath and other tools.
  • Deterministic and reliable data collection without requiring deep learning or OCR.
  • Enhanced user privacy and transparency compared to tools that collect all screen data indiscriminately.
• Experiments / evaluation:
  • Study 1: Usability testing with 5 researchers creating collectors for Instagram, Uber, and Chrome.
  • Study 2: In-lab accuracy testing with 5 participants, generating 528 labeled data points.
  • Study 3: Field study with 7 participants over 24–72 hours, collecting 358 Instagram Story ads.
• Limitations and future work:
  • Challenges with app updates, localization, and complex UI screens.
  • Need for larger-scale deployments and support for richer media types (e.g., images, videos).
  • Future enhancements include multi-query support, real-time dashboards, and privacy-centric features like data hashing.

Summary

Crepe introduces a novel approach to mobile screen data collection by leveraging a programming-by-demonstration paradigm and the deterministic Graph Query language. It empowers researchers to collect specific UI data while ensuring participant privacy and transparency. Evaluation studies demonstrate its usability, high accuracy (96.0% F1 score), and minimal impact on device performance. Crepe addresses the "data monopoly" by democratizing access to screen data, enabling diverse research applications. Future work will focus on scalability, richer data types, and enhanced privacy features. The tool will be open-sourced to support the academic community.