Beyond Precision: Understanding the Impact of Algorithmic Accuracy and Transparency on User Perceptions in Keyword-Driven Contextual Advertising

Paper Title

Beyond Precision: Understanding the Impact of Algorithmic Accuracy and Transparency on User Perceptions in Keyword-Driven Contextual Advertising

Publication Info

• Topic area: User perceptions and behavioral outcomes in algorithmic contextual advertising.
• Keywords: contextual advertising, keyword extraction, user perception, transparency, behavioral intention, TF-IDF, KeyBERT, DeepSeek, algorithmic explanations, ad relevance.

Background and Problem

• Problem / challenge: Existing evaluations of keyword extraction algorithms focus on precision and accuracy benchmarks, but it is unclear how these metrics translate into user perceptions, ad effectiveness, and behavioral intentions. Additionally, the role of transparency in influencing user attitudes toward contextual advertising remains underexplored.
• Significance: Understanding user perceptions is critical for designing effective and ethical contextual advertising systems, especially in privacy-sensitive and regulated environments.
• Motivation and related work: Prior research has focused on algorithmic performance relative to gold-standard datasets, neglecting user-centered evaluations. Transparency mandates, such as the Digital Services Act, require platforms to disclose targeting logic, but the effects of such disclosures on user perceptions and engagement are not well understood. This paper addresses these gaps by examining the interplay between algorithmic methods, transparency, and user responses.

Solution

• Proposed approach: The study investigates user perceptions and behavioral intentions in keyword-based contextual advertising by comparing three keyword extraction methods (TF-IDF, KeyBERT, DeepSeek) and analyzing the impact of transparency through keyword-based explanations.
• Novelty:
  1. Demonstrates a misalignment between algorithmic benchmarking performance and user perceptions of relevance.
  2. Explores the dual impact of transparency, showing that explanations increase perceived relevance but reduce ad interest and behavioral intentions.
  3. Provides practical insights into optimizing algorithm selection and explanation design for user-centered outcomes.
• Procedure and key techniques:
  • Conducted an online experiment with 498 participants evaluating ad-article pairings generated by different keyword extraction methods.
  • Manipulated transparency by displaying or hiding extracted keywords.
  • Measured perceived relevance, behavioral intentions, and ad/article interest using a survey-based approach.
  • Analyzed results using Structural Equation Modeling (SEM) to assess direct, mediated, and moderated effects.

Results

• Concrete findings:
  • Without explanations, simpler methods (TF-IDF, KeyBERT) achieved higher perceived relevance than the more complex DeepSeek, despite DeepSeek's superior benchmark performance.
  • Providing explanations increased perceived relevance (e.g., β = 0.789, p < 0.001 for the Gold Standard) but reduced behavioral intentions (β = −0.303).
  • DeepSeek showed the largest increase in perceived relevance when explanations were provided, outperforming simpler methods in this condition.
• Advantage over baselines:
  • All algorithmic methods (TF-IDF, KeyBERT, DeepSeek) outperformed a random baseline in perceived relevance.
  • DeepSeek consistently outperformed TF-IDF and KeyBERT in benchmark metrics (e.g., F1@10: DeepSeek = 0.147, TF-IDF = 0.092, KeyBERT = 0.014).
• Experiments / evaluation:
  • Participants evaluated ad-article pairs generated by five methods (Gold Standard, TF-IDF, KeyBERT, DeepSeek, Random).
  • Dependent variables included perceived relevance, behavioral intention, and ad/article interest, measured on a 7-point Likert scale.
  • Explanations were manipulated between-subjects, with 227 participants seeing keywords and 278 not seeing them.
• Limitations and future work:
  • Focused on three keyword extraction methods and five ads, limiting generalizability.
  • Did not control for participants’ prior attitudes toward ads.
  • Future work could explore additional algorithms, explanation designs, and cross-cultural samples.

Summary

This study examines the effects of keyword extraction methods and transparency on user perceptions and behavioral intentions in contextual advertising. Simpler methods (TF-IDF, KeyBERT) were found to outperform the more complex DeepSeek in perceived relevance when explanations were absent, despite DeepSeek's superior benchmark performance. Transparency through keyword-based explanations increased perceived relevance but reduced ad interest and behavioral intentions, highlighting the complex role of explanations. The findings underscore the importance of aligning algorithmic performance with user-centered evaluations and suggest that explanation design should balance transparency with user engagement. These insights have implications for optimizing contextual advertising systems in privacy-sensitive and regulated environments.