Certified But Imperfect: Investigating The Role of AI Certifications And System Performance on Trust in And Reliance on AI Systems

Paper Title

Certified But Imperfect: Investigating The Role of AI Certifications And System Performance on Trust in And Reliance on AI Systems

Publication Info

• Topic area: The impact of AI certifications and system performance on user trust and reliance.
• Keywords: AI certifications, trust in AI, reliance on AI, system reliability, expectation violations, human-computer interaction, governance tools, user behavior, AI regulation, appropriate reliance.

Background and Problem

• Problem / challenge: Despite the growing policy emphasis on AI certifications as governance tools, there is limited empirical understanding of how these certifications influence user trust, reliance, and expectations, particularly in practical, interactive contexts.
• Significance: Understanding the role of certifications is critical for fostering appropriate trust and reliance on AI systems, especially in high-stakes domains like healthcare and finance, where errors can have severe consequences.
• Motivation and related work: Prior research has shown mixed results regarding the effectiveness of AI certifications, with some studies indicating increased trust and others finding no effect. Additionally, the role of certifications in shaping user expectations and their interplay with system reliability remains underexplored.

Solution

• Proposed approach: A 2x2 experimental study investigating the effects of AI certifications (certified vs. uncertified) and system reliability (high vs. low) on user trust, vigilance, reliance, and expectation violations.
• Novelty:
  1. Differentiation between pre- and post-interaction trust and reliance on AI systems.
  2. Examination of how certifications interact with system reliability to shape user behavior and expectations.
  3. Introduction of measures for appropriate reliance, such as Relative AI Reliance (RAIR) and Relative Self-Reliance (RSR).
  4. Mediation analysis to explore the role of expectation violations in trust dynamics.
• Procedure and key techniques:
  • Participants (N = 644) completed a two-step estimation task with the assistance of a fictional AI system (QuantifierAI).
  • Experimental manipulations included certification status (certified vs. uncertified) and system reliability (high vs. low).
  • Measures included trust, vigilance, reliance (Weight of Advice), expectation violations (direct and indirect), and appropriate reliance (RAIR and RSR).
  • Statistical analyses included ANOVAs and mediation analyses to assess the effects of certifications and reliability.

Results

• Concrete findings:
  • Pre-interaction, certified systems were trusted more (M = 3.53 vs. 3.19, p < .001) and elicited lower vigilance (M = 3.67 vs. 3.87, p < .001).
  • Post-interaction, trust and vigilance were primarily influenced by system reliability, not certification (p > .05 for certification effects).
  • Certified systems increased reliance (M = 0.60 vs. 0.55, p = .015) and supported more appropriate reliance (RAIR: M = 0.541 vs. 0.423, p < .001), especially for less reliable systems.
  • Certifications amplified the effect of reliability on trust, with greater trust differences between high- and low-reliability systems when certified.
  • Certifications induced more negative expectation violations for low-reliability systems (indirect EV: M = 1.42 vs. 0.96, p < .001).
  • Mediation analysis confirmed that negative expectation violations reduced trust in certified, low-reliability systems.
• Advantage over baselines:
  • Certifications improved reliance and appropriate reliance, particularly in low-reliability conditions.
  • Certifications acted as initial trust cues but were less impactful than reliability during interaction.
• Experiments / evaluation:
  • Design: 2x2 between-subjects experiment.
  • Participants: N = 644, recruited via Prolific, diverse demographics.
  • Metrics: Trust (TAIS scale), vigilance (TAIS scale), reliance (Weight of Advice), expectation violations (direct and indirect measures), RAIR, and RSR.
• Limitations and future work:
  • Lack of measures for appropriate reliance on certifications themselves.
  • Limited exploration of psychological mechanisms underlying certification effects (e.g., cognitive load, authority heuristics).
  • Focus on reliability; other trust drivers like fairness and explainability were not examined.
  • Static certification model; future work should explore dynamic, ongoing certification processes.
  • Results may differ under mandatory certification regimes.

Summary

This study investigated how AI certifications and system reliability influence user trust, reliance, and expectation violations. Certifications boosted initial trust and reliance but had no effect on trust post-interaction, where reliability became the dominant factor. Certifications supported appropriate reliance, particularly in low-reliability conditions, but also amplified negative expectation violations when performance was poor. These findings highlight the dual role of certifications as both trust cues and potential sources of expectation misalignment. Policymakers and HCI researchers should consider these dynamics when designing and implementing AI governance mechanisms.