Assessing User Apprehensions About Mixed Reality Artifacts and Applications: The Mixed Reality Concerns (MRC) Questionnaire

Title of the Paper

Assessing User Apprehensions About Mixed Reality Artifacts and Applications: The Mixed Reality Concerns (MRC) Questionnaire

Paper Information

• Subject Area: Computer Science and Human-Computer Interaction (HCI), User Experience Evaluation of Mixed Reality (MR) Systems
• Keywords: Mixed Reality (MR), User Concerns, Privacy, Security, Social Acceptance, Trust, Questionnaire Development, User Experience
• Conference Source: CHI Conference on Human Factors in Computing Systems (CHI '24), May 11–16, 2024, Honolulu, HI, USA

Research Background and Problem

• Findings and Challenges:
  • The rapid development of mixed reality technology and its application in various fields (e.g., education, manufacturing, work) has raised user concerns regarding privacy, security, and social acceptance.
  • Existing evaluation tools often focus on usability and acceptance but rarely conduct systematic studies on user concerns about emerging technologies.
  • Mixed reality integrates virtual elements with physical reality, inevitably leading to social and psychological impacts (e.g., blurred boundaries between virtual and real-world behaviors, privacy invasion, trust issues).
• Importance:
  • Mixed reality is expected to become more widespread in the coming years, making it crucial to understand users' potential concerns, which directly affect acceptance and broad adoption of the technology.
• Research Motivation and Related Work:
  • Current user measurement tools for emerging technologies (e.g., TAM model, UEQ, SUS) fail to adequately address the complex concerns associated with mixed reality systems.
  • Research related to social acceptance, privacy, and psychological safety is increasing, necessitating a specialized questionnaire tool to systematically guide and assess user concerns in these areas.

Solution

• Methods and Solution:
  • Proposed the "Mixed Reality Concerns (MRC) Questionnaire," a tool for systematically evaluating user concerns about mixed reality devices and applications.
• Innovations:
  • Categorized user concerns into four key areas (security, privacy, social impact, and public acceptance), forming the conceptual model's foundation.
  • Integrated domain literature and expert opinions, developing a concise questionnaire with nine items through multiple iterations and validations.
  • Unlike existing tools, the questionnaire focuses on capturing users' subjective concerns about privacy and trust rather than merely evaluating technical functionality or pleasurable experiences.
• Implementation Steps:
  • Initial Stage: Drafted 120 items based on literature and expert input.
  • Refinement: Reduced items to 48 through multiple rounds of expert feedback.
  • Questionnaire Validation: Further refined to nine items through exploratory factor analysis (EFA) and confirmatory factor analysis (CFA), covering three dimensions (security and privacy, social impact, trust).
  • Reliability Assessment:
    • Internal consistency: Cronbach's α > 0.8.
    • Temporal stability: Tested using a repeat-test method.

Research Findings

• Specific Findings:
  • Finalized the questionnaire with three factors (security and privacy, social impact, trust) and their corresponding nine items.
  • The questionnaire demonstrated high internal consistency (Cronbach's α=0.85), good structural validity, and temporal stability (r=0.85).
  • Can be used for rapid assessment of users' key concerns regarding mixed reality technology.
• Advantages Over Existing Solutions:
  • Provides a comprehensive quantitative tool for concerns specific to mixed reality technology, addressing the limitations of traditional evaluation tools (e.g., TAM, SUS, UEQ) in capturing negative emotions and psychological concerns.
  • Focuses on subjective perceptions, assisting developers in optimizing product design and reducing users' defensive attitudes.
• Experimental or Validation Results:
  • Exploratory factor analysis showed that the three-factor model of the questionnaire has a good structure.
  • Scores from the new questionnaire exhibited moderate correlations with the "Perceived Creepiness of Technology Scale" (PCTS) and certain UEQ dimensions (attractiveness, reliability).
  • Effectively distinguishes between high-concern and low-concern mixed reality scenarios.
• Limitations and Future Directions:
  • Limitations:
    • The primary test sample was from Western cultural backgrounds, and applicability across cultural differences remains unverified.
    • The questionnaire only assesses concerns during initial experiences, not changes over long-term use.
    • No testing in real-world scenarios, which may be influenced by design expectations.
  • Future Directions:
    • Investigate the questionnaire's robustness across different cultural and familiarity groups.
    • Integrate real-world scenarios and long-term tracking data for validation.
    • Combine results with other cognitive and social psychology models to deepen understanding of technological concerns.