Creepy Assistant: Development and Validation of a Scale to Measure the Perceived Creepiness of Voice Assistants

Title of the Paper

Creepy Assistant: Development and Validation of a Scale to Measure the Perceived Creepiness of Voice Assistants

Paper Information

• Research Area: Human-Computer Interaction, Voice Assistant Evaluation, Quantitative User Experience Research
• Keywords: Voice Assistant, User Experience, Psychological Response, Creepiness, Scale Development, Privacy Issues, Social Norms, Behavioral Anomalies

Research Background and Questions

• Research Questions:

  • The "uncanny valley" effect of voice assistants often triggers user discomfort and a sense of "creepiness," reducing people's willingness to adopt such technologies.
  • Existing evaluation tools (e.g., Creepy of Technology Scale, CRoSS) fail to fully capture the unique attributes of voice assistants and users' psychological responses.
  • This study focuses on how factors specific to voice assistants, such as "anthropomorphism," privacy threats, and behavioral anomalies, contribute to the perception of "creepiness."

• Significance:

  • Voice assistants are increasingly used in homes, workplaces, and entertainment, but emotional responses like fear and distrust may undermine their potential value.
  • Identifying and quantifying the "creepiness" of voice assistants is a critical step toward improving design, user acceptance, and market competitiveness.

• Motivation and Related Work:

  • Existing research has quantified the general "creepiness" of technology (e.g., PCTS, CRoSS) but lacks tools specific to voice assistants.
  • Privacy concerns and violations of social norms by voice assistants are particularly likely to evoke user "creepiness," impacting functionality usage and user loyalty.

Proposed Solution

• Proposed Solution:

  • Develop and validate the Perceived Creepiness of Voice Assistants Scale (PCAS), which quantifies users' initial sense of "creepiness" toward voice assistants through seven core items.
  • Identify four key dimensions influencing the "creepiness" of voice assistants: Control, Privacy, Behavior, and Value.

• Innovations:

  • A purpose-built scale that captures attributes unique to voice assistant platforms (e.g., always-on state, anthropomorphic features).
  • Introduction of new metrics, including user control and privacy perception, which are not addressed by existing tools.

• Implementation Steps and Key Techniques:

  1. Problem Definition and Initial Scale Generation: Indicators were collected through a literature review and expert interviews, resulting in an initial set of 51 items.
  2. Scale Development:
     • Conducted Exploratory Factor Analysis (EFA) with 198 users to refine and reduce the scale to seven items.
     • Assessed factor interpretability and internal consistency (Cronbach’s Alpha = 0.903).
  3. Scale Validation:
     • Conducted Confirmatory Factor Analysis (CFA) with 100 users to test the scale's structural model.
     • Compared convergent validity with existing scales (PCTS, VUS) to quantify associations with related psychological variables.
     • Performed Test-Retest evaluation to confirm the scale's long-term reliability (ICC = 0.90).

Research Findings

• Key Findings:

  • The final scale includes seven items that effectively evaluate the "creepiness" of voice assistants in terms of "degree of control," "excessive privacy collection," "behavioral anomalies," and "limited value."
  • The scale successfully differentiates between "creepy" and "non-creepy" voice assistant cases, achieving significant classification accuracy.

• Advantages Over Existing Solutions:

  • Compared to CRoSS or PCTS, the PCAS focuses more on the specific characteristics of voice assistants, offering actionable insights for industry designers and researchers to improve voice technology.
  • Emphasizes user privacy perception and social norm awareness, extending into scenarios unique to voice assistant design.

• Experimental Evaluation Results:

  • Confirmatory Factor Analysis demonstrated good model fit (TLI = 0.97) and strong convergent validity with other scales (correlation coefficient with PCTS ρ = 0.676).
  • PCAS effectively measures the dimensions of "creepiness" in initial user experiences, such as privacy overload and loss of control.

• Limitations and Future Directions:

  • Limitations:
    • The scale was developed and validated only within North American and European cultural contexts, potentially limiting its applicability to regions like Asia.
    • Initial "creepiness" scores for non-users of voice assistants were relatively high, suggesting the need for a supplementary version tailored to non-users.
    • The study focuses on short-term evaluations of "creepiness" and does not address changes in user attitudes over time.
  • Future Directions:
    • Explore how different cultural contexts and social norms influence user perceptions.
    • Conduct longitudinal studies on the long-term dynamic effects of "creepiness" in voice assistants.
    • Develop a secondary version of the scale for non-user groups and investigate potential "ethical" dimensions of future intelligent voice technologies.