“It’s Just a Wild, Wild West”: Harnessing Public Procurement as an AI Governance Mechanism

Paper Title

“It’s Just a Wild, Wild West”: Harnessing Public Procurement as an AI Governance Mechanism

Publication Info

• Topic area: Public procurement as a mechanism for AI governance in the public sector.
• Keywords: Public procurement, AI governance, responsible AI, public sector AI, procurement practices, EU AI Act, socio-technical systems, AI policy, vendor lock-in, public interest.

Background and Problem

• Problem / challenge: Public procurement processes for AI in the public sector are underdeveloped, fragmented, and lack AI-specific oversight, leading to risks of harm, insufficient scrutiny, and misalignment with public interests.
• Significance: Public procurement represents a significant opportunity to govern AI systems in the public sector, leveraging purchasing power to ensure alignment with public values and mitigate risks such as discrimination, bias, and lack of accountability.
• Motivation and related work: Prior work highlights the potential of procurement as a governance tool but lacks empirical insights into how this can be operationalized. Current practices often prioritize cost-effectiveness over public interest, with private vendors exerting disproportionate influence.

Solution

• Proposed approach: The study identifies six promising procurement practices and actionable mechanisms to align public sector AI with public interests, based on interviews with experts in the UK and EU.
• Novelty:
  1. Mapping current modes of AI acquisition in the public sector, including informal channels with reduced scrutiny.
  2. Identifying four core challenges to public interest-aligned AI procurement.
  3. Proposing six actionable procurement practices and mechanisms for implementation.
  4. Establishing a research agenda for integrating HCI methods into AI procurement.
• Procedure and key techniques: Conducted 16 semi-structured interviews with procurement experts, analyzed using thematic analysis to identify challenges, practices, and mechanisms. Focused on the EU and UK contexts, with insights into procurement processes, stakeholder roles, and socio-technical dynamics.

Results

• Concrete findings:
  • AI-specific procurement is nascent and often folded into standard IT processes.
  • Framework contracts dominate but lack AI-specific safeguards.
  • AI frequently enters through informal channels (e.g., system updates, hidden AI in products, pilot projects) with reduced oversight.
  • Six promising practices identified, including providing clear guidance, fostering knowledge sharing, focusing on outcomes, and implementing quality control measures.
• Advantage over baselines: The proposed practices address gaps in current procurement processes, such as lack of AI-specific assessments, vendor lock-in, and insufficient public interest alignment.
• Experiments / evaluation: Insights derived from interviews with 16 experts across roles (buyers, vendors, advisors) and regions (UK, EU), analyzed for thematic patterns.
• Limitations and future work: Findings are exploratory and not statistically generalizable. Future research should examine contextual nuances, expand geographical focus, and develop tools to support procurement practices.

Summary

This study explores how public procurement can serve as a governance mechanism to align AI systems in the public sector with public interests. Through interviews with experts, it identifies challenges such as fragmented procurement processes, vendor dominance, and lack of AI-specific guidance. The authors propose six promising practices, including clear guidance, knowledge sharing, and quality control measures, to address these issues. The study highlights opportunities for HCI scholarship to contribute to effective AI procurement and outlines a research agenda for advancing public interest-aligned practices. These findings underscore the potential of procurement to shape AI markets and supplier behavior in alignment with societal needs.