QButterfly: Lightweight Survey Extension for Online User-Interaction Studies for Non-Tech-Savvy Researchers

Title of the Paper

QButterfly: Lightweight Survey Extension for Online User Interaction Studies for Non-Tech-Savvy Researchers

Bibliographic Information

• Subject Area: Human-Computer Interaction (HCI)
• Keywords: Online user interaction studies, online experiments, Qualtrics, LimeSurvey, open source, HCI toolkit

Research Background and Problem Statement

• Identified Problems or Challenges:

  1. Online user interaction studies, while popular, face technical challenges such as synchronizing multiple tools and matching data across platforms.
  2. Researchers with limited technical expertise encounter significant difficulties in designing large-scale user studies, particularly those involving experimental condition assignments and user behavior tracking.
  3. Existing tools require programming skills and are difficult to integrate seamlessly with standard survey platforms like Qualtrics and LimeSurvey.

• Importance:

  • Large-scale online experiments (e.g., crowdsourced participant studies) provide authentic user behavior data, supporting theory-driven research and meeting the growing demand in the HCI field.
  • Effectively integrating existing tools and lowering technical barriers can accelerate research progress and enhance the efficiency of user studies.

• Research Motivation and Related Work:

  • Many current tools (e.g., psiTurk, oTree) theoretically support online studies but require strong programming skills, creating high barriers for non-technical researchers to design user interaction studies.
  • Tool development in the HCI field (e.g., AWARE and QRTEngine) demonstrates that efficient tools can make profound contributions to academic research, but further optimization tailored to practical needs is still required.

Solution

• Method or Solution:

  • QButterfly, a lightweight, open-source HCI toolkit, is proposed to assist researchers with limited technical skills in designing online user interaction studies using widely adopted survey platforms (Qualtrics and LimeSurvey).
  • The toolkit includes:
    1. Pre-designed survey templates with embedded JavaScript for managing web displays and data collection.
    2. The QButterfly JavaScript library embedded in web pages to track user clicks and record data on survey platforms.
    3. Data analysis Excel templates supporting data processing with regular expressions.

• Innovations:

  1. Enables the creation of complex online user studies without programming skills.
  2. Supports user behavior tracking (e.g., clickstreams) and real-time data storage on survey platforms.
  3. Eliminates the need for complex data integration, streamlining data analysis and experimental configuration processes.

• Implementation Steps and Key Technologies:

  1. Experiment Design: Introduce QButterfly survey templates and embed the JavaScript library into target websites and HTML elements.
  2. Data Collection:
     • QButterfly fully records user clicks and timestamps, transmitting data in real time to survey platforms.
     • Uses iframes to seamlessly integrate web pages with survey platforms.
  3. Data Analysis: Standard Excel templates call analysis functions to process user behavior data.

Research Outcomes

• Specific Results:

  1. Two validation studies (a lab study and a field study with crowdsourced participants) demonstrated the reliability of QButterfly:
     • The lab study showed accurate data recording with acceptable time delays within 20 milliseconds.
     • The field study validated the tool's cross-platform compatibility and consistency in data collection in real-world settings.
  2. Compared to other tools, QButterfly significantly reduced the technical complexity of user studies, eliminating the need for cross-platform data matching and external dependencies.

• Experimental or Evaluation Results:

  • Lab Study: In two major browsers (Chrome and Safari), all user events were recorded correctly, with millisecond-level data delays meeting the timeliness requirements of online user studies.
  • Field Study:
    • A total of 6,045 participants, with device-related issues accounting for only 1.3% of recorded data.
    • Partial data loss (e.g., due to JavaScript blocking) was attributed to individual users' browser settings.

• Advantages Over Existing Solutions:

  1. Integrates mainstream survey platforms with user behavior tracking tools, simplifying data management.
  2. Fully open-source, allowing for customization to other platforms and use cases.
  3. Requires no complex programming, offering a beginner-friendly user guide.

• Limitations and Future Directions:

  1. Limitations:
     • Some data anomalies caused by device differences persist in user experiments.
     • Does not fully support outdated browsers like Internet Explorer.
     • The tool heavily relies on survey platform updates; significant changes to these platforms would require tool updates.
  2. Future Directions:
     • Expand analytical functionalities (e.g., compatibility with Python and R).
     • Enhance support for more interaction events (e.g., mouse trajectory tracking).
     • Strengthen pre-experiment device environment testing to improve data collection reliability.

Conclusion

• Significance of QButterfly: This study introduces a user-friendly, open-source tool that provides a new mechanism for non-technical users to design online user interaction studies. The research demonstrates its ability to collect reliable user data in both lab and real-world environments, with potential for supporting large-scale and complex studies in the future.

• Research Potential: The release and validation of QButterfly not only lower the technical barriers to research but also provide faster and broader feasibility for scientific studies, opening new opportunities for HCI and other research fields.

• Additional Support: Development and application were funded by the Hasler Foundation and the DIZH project.