A Tool for Capturing Smartphone Screen Text

Document Title

"A Tool for Capturing Smartphone Screen Text"

Document Information

• Research Domain: Smartphone screen text capture and behavior analysis
• Keywords: Screen text capture, smartphone sensing, context awareness, user behavior, digital ecology, privacy protection

Research Background and Issues

• Identified Problems or Challenges:

  1. The vast amount of textual data on smartphone screens is a valuable resource for analyzing user behavior, but existing methods face numerous challenges. Previous techniques are limited to capturing keyboard input or indirectly obtaining screen text through screenshots and optical character recognition (OCR).
  2. OCR methods have limitations, such as high computational cost, poor scalability, and recognition errors.
  3. There is currently no low-cost, high-accuracy solution for directly and comprehensively capturing all text on screens.

• Significance:

  1. Capturing screen text can provide more precise "ground truth" data for behavioral research.
  2. Combining screen text with other sensor data (e.g., location, accelerometer) enables a comprehensive understanding of smartphone user behavior.

• Research Motivation and Related Work:

  1. Current smartphone sensing technologies primarily utilize location information, app usage, or accelerometer data to infer user behavior, but their indirect nature affects the accuracy of behavior inference.
  2. This study aims to develop a novel sensor for real-time and continuous data collection based on screen content capture and analysis, overcoming the bottlenecks of traditional technologies.

Solution

• Proposed Method: A novel software sensor was developed to non-invasively and continuously capture and process all textual content on smartphone screens.

• Innovations:

  1. Utilizing Android's accessibility service to capture screen text in real-time without relying on OCR technology, thereby reducing computational demands.
  2. Integrating the sensor into the AWARE-Light framework to enable synchronized recording and analysis of screen text and multiple sensor data.
  3. Introducing a data filtering mechanism to avoid recording duplicate screen content, improving data quality and capture efficiency.

• Implementation Steps and Key Technologies:

  1. Using accessibility services to monitor screen content updates and generate a tree structure of text.
  2. Extracting text content, screen coordinates, and timestamps from UI elements, annotating and storing this data in a database.
  3. Implementing a filtering algorithm to record only screen data that has changed, avoiding duplicate content.
  4. Proposing new terminology and feature extraction methods (e.g., "screen" concept, phrase segmentation, and position coordinates) to standardize data analysis.

Research Outcomes

• Specific Achievements:

  1. During a two-week experiment, 21 participants used the sensor to generate 7,004,867 screen instances, 135,414,272 phrases, and over 4.2 billion characters of data.
  2. The data revealed daily behavioral patterns during smartphone usage (e.g., screen activity by time periods), interaction characteristics between app categories, and emotional features of content.
  3. Developed statistical analysis metrics (e.g., information density, information change variance, emotional sentiment scores) to evaluate the dynamic characteristics of screen data in different contexts.

• Advantages Compared to Existing Solutions:

  1. Improved data collection accuracy and efficiency, avoiding word recognition errors and incomplete data issues.
  2. Compared to existing "screenomes" technology, the capture interval is significantly shorter (average 0.87 seconds), enhancing temporal resolution.
  3. The developed tool is open-source, allowing researchers to freely customize and extend its functionality.

• Experimental or Evaluation Results:

  • The data supported the application of NLP techniques (e.g., sentiment analysis and named entity recognition) in behavior analysis.
  • The data revealed user habits in different times and locations (e.g., differences in screen usage between business districts and park areas).
  • Experiments validated the sensor's efficiency and reliability.

• Limitations and Future Directions:

  1. The current sensor cannot capture text within images or information in videos, only textual data from screens.
  2. Privacy concerns may influence participant behavior, and further research is needed on power and storage optimization.
  3. Future work should enhance data visibility filtering, improve participant trust in privacy protection, and expand the scope to broader user groups and usage scenarios.