Hey Alexa, Who Am I Talking to?: Analyzing Users’ Perception and Awareness Regarding Third-party Alexa Skills

Title of the Paper

Hey Alexa, Who Am I Talking to?: Analyzing Users’ Perception and Awareness Regarding Third-party Alexa Skills

Paper Information

• Domain: Human-Computer Interaction, Privacy and Security, Voice Assistant Platforms
• Keywords: Voice Assistant, Third-party Skills, User Perception, Security Indicators, Privacy Risks

Research Background and Issues

• Identified Problems or Challenges:

  • The Amazon Alexa voice assistant allows third-party developers to create skills, but these skills may request sensitive user data or share the same invocation phrases, leading to privacy and security risks.
  • Users may struggle to distinguish between third-party skills and Alexa’s native skills, making them prone to mistakenly activating or trusting the wrong skills.
  • Users lack sufficient awareness and expectations about which skills will be automatically enabled.

• Necessity of the Research:

  • Third-party skills could potentially become sources of privacy breaches or data theft.
  • Providing effective security indicators is crucial to help users understand and manage their interactions with these skills, thereby safeguarding the ecosystem’s security and user experience.

• Research Motivation and Related Work:

  • Existing studies have made progress in revealing risks and privacy issues within the Alexa skills ecosystem, but there is limited evaluation of the effectiveness of security and privacy indicators in current voice interfaces.
  • Research on visual indicators in browsers and mobile applications has shown that well-designed security indicators can significantly enhance users’ risk perception, but voice interfaces lack similar designs.

Solution

Methods or Solutions

• The authors proposed an interactive user study:
  • Investigating users' ability to distinguish between third-party and native skills through predefined tasks.
  • Testing users’ accuracy in predicting and verifying the skill selection process.
  • Prototyping and evaluating several voice-based skill type indicators to enhance users’ awareness of skill developer identities.

Innovations

• Proposed and tested three voice-based skill identification models, exploring the feasibility of marking different skill types through warning phrases, developer names, and changes in voice tone.
• Used interaction recordings with real skills and official Alexa application scenarios to enhance the ecological validity of the experiment.

Implementation Steps and Key Techniques

1. User Task Design

   • Distinguishing skill types using both visual interfaces (skill information pages) and voice interfaces (skill interactions).
   • Searching for and predicting skill selection, followed by voice invocation verification.
   • Comparing different voice indicator models and collecting user preferences.

2. Participant Recruitment

   • Recruited 52 participants via Alexa-related online forums and university mailing lists.

3. Data Analysis

   • Analyzed participants’ response accuracy and confidence using statistical methods (e.g., Fisher’s exact test, Chi-Square test).
   • Conducted textual content analysis of user suggestions and measured annotation consistency using Cohen’s Kappa.

Experimental Results and Observations

Specific Findings

• Skill Type Differentiation in Visual Interfaces:

  • Users were relatively accurate in identifying third-party skills on the skill information page (precision = 96.63%, recall = 82.69%), primarily due to developer names serving as clear distinguishing indicators.

• Skill Type Differentiation in Voice Interfaces:

  • Users were unable to effectively distinguish skill types in voice interfaces (precision = 42.85%, recall = 31.73%) due to the lack of explicit voice cues.

• Accuracy of Skill Selection Predictions:

  • Most users failed to accurately predict which skill Alexa would activate, revealing a mismatch between users’ mental models of the skill selection process and the system’s actual operation.

• Preferences for Voice-based Indicators:

  • Most users preferred voice models that included warning phrases and developer names, while changes in voice tone were considered unnatural.

Comparative Advantages

• Compared to the default design of the current Alexa ecosystem, Model 1 (warning phrases + developer names) significantly improved transparency in identifying skill developers.

Limitations and Future Directions

• Research Limitations:

  • Small sample size, with participants predominantly being young women and university students, which may not represent the general user population.
  • Only three voice models were explored, leaving other potential indicator designs unexamined.
  • The long-term impact of voice indicators on user behavior remains to be analyzed.

• Future Research Directions:

  • Testing other interaction models and their adaptability across different cultural contexts.
  • Evaluating the trade-offs between privacy indicators, user experience, and transparency.
  • Designing more precise skill indicator strategies for shared devices.

Conclusion

This study highlights the lack of transparency in the voice interface of the Alexa skills ecosystem and derives recommendations for improved indicators based on user preferences. Preliminary tests suggest that warning phrases and developer name identifiers are the most effective designs for reducing user misidentification of third-party skills. The study recommends further exploration of the long-term adaptability and user acceptance of various voice models.