By Hugo Huang
Introduction: The Dawn of the AI Security Era
The AI wave is here, and it’s only getting bigger. According to a recent report from McKinsey, “over the next three years, 92 percent of companies plan to increase their AI investments.” As this AI wave washes over almost every industry and is integrated deeply and extensively into critical and non-critical operations, it ushers in a pivotal new cybersecurity battleground: securing AI.
There’s a lot to be concerned about. My research published in Harvard Business Review showed that “GenAI projects inherently pose a heightened risk of compromise, necessitating a well-planned security strategy from the outset.” AI systems present unique and significant security risks compared to traditional software, from data poisoning and model inference attacks to hallucination attacks, prompt injections, and more. Our latest security survey, conducted by IDC and jointly sponsored by Canonical and Google Cloud, delves into the top challenges that enterprise and open source practitioners face in securing AI infrastructure in 2025. This blog post will outline these critical challenges and provide actionable guidelines for building stronger, more resilient AI systems, leveraging the power of open source.
The landscape of AI security
AI’s rising prominence is fundamentally embedded in cybersecurity. I would echo the sentiments of Vivian Schiller and Heather Adkins that AI and cybersecurity are just different sides of the same coin. On the one side, Large Language Models (LLMs) and other AI capabilities are poised to revolutionize cyber defense, empowering defenders to process vast datasets, automate vulnerability detection, streamline incident response, and even help address the cybersecurity skills gap; On the other side of that coin is the great potential that AI has to empower malicious actors, enable highly automated attacks, and create extremely convincing synthetic content like deepfakes and advanced phishing. Effectively leveraging AI for defense while mitigating its misuse by adversaries presents distinct and urgent challenges. Our recent security survey illuminates the most pressing hurdles organizations face in securing their AI infrastructure in 2025. Let’s explore those challenges in more detail.
The Top 3 Challenges of Securing AI Infrastructure in 2025
Challenge 1: Lack of Standardized Frameworks and Best Practices for AI Security and Risk Management
This emerged as the most significant challenge overall, cited by 48.20% of respondents globally. Unlike traditional software development, AI lacks mature, widely adopted security guidelines for its unique lifecycle. The AI development lifecycle – from data collection and model training to deployment and monitoring – differs significantly from conventional software. Existing security frameworks often don’t adequately address issues like model integrity, data poisoning, adversarial attacks, or the risks associated with AI-driven decision-making. This leaves organizations without clear blueprints for managing AI-specific risks.
As anyone in the regulatory space will tell you, compliance and audit trails for ML components are underdeveloped and hard to standardize. Without these frameworks, it’s difficult to implement robust quality assurance, ensure regulatory compliance, or prove the integrity of ML components, especially in critical sectors like medtech. Frameworks like SLSA would help, if they could be applied to model pipelines – but that’s easier said than done. To address this gap and build more resilient systems, we recommend integrating security from the outset of your ML pipeline. For practical guidance, you can refer to the OpenSSF’s practical guide for building robust AI/ML pipeline security.
Addressing this challenge demands tangible steps towards building resilient AI systems. This includes implementing concrete measures across the AI lifecycle, such as securing model training pipelines (protecting data integrity), ensuring reproducible builds (versioning everything for provenance), and actively protecting against data poisoning and model tampering with continuous monitoring. Crucially, the open source community is pivotal in forging these standards, offering transparent, community-driven frameworks like SLSA for ML workflows and providing essential tooling such as Sigstore (for cryptographic signing) and the development of ML-BOMs (for comprehensive component listing). This collaborative, auditable approach is foundational to building trust and best practices in AI security.
Challenge 2: Lack of Visibility of AI Solutions Employed Across the Organization (Shadow AI)
44.8% percent of respondents identified lack of visibility as their second most pressing challenge. The rapid, often decentralized adoption of AI means many organizations simply don’t know what AI models are in use, where they are deployed, or what data they are processing. This creates significant security blind spots, making it impossible to apply consistent policies or monitor for threats effectively. If you don’t know it’s there, you can’t secure it.
Our OpenSSF members expressed a strong concern about “verifying model provenance and integrity across environments” and many of them struggle with integrating “model validation and adversarial testing into CI pipelines”, especially when the inventory of AI models and their dependencies is unknown or incomplete due to shadow AI. This is critical for an obvious reason: that you can’t secure what you can’t see. Interestingly, this challenge was particularly pronounced in the UK (63.33%), Canada (60%), and Australia (60%), indicating that more mature AI markets might be experiencing higher rates of unmanaged AI deployments.
To address the challenges of “shadow AI”, decision-makers (e.g., CIO, CISO, CTO) need to build resilience policies and embrace open source tools. Such resilience policies include implementing guardrails around the use of AI, rigorous inventory and discovery processes, and access controls to limit non-approved software. Open source tools can significantly enhance visibility and mitigate shadow AI risks. Transparent open source AI platforms and frameworks naturally encourage documented usage and make it easier to audit deployments. Furthermore, open source AI projects, such as Kubeflow, contribute to tools for AI inventory, dependency scanning, and integrated MLOps platforms that provide clearer insight into the AI landscape within an organization, contributing to “transparency and auditability.”
Challenge 3: Lack of Expertise and Skilled Personnel in AI Security
Our survey prominently identified a critical talent gap, with 40.8% of organizations struggling due to a “Lack of expertise and skilled personnel in AI security.” AI security is a distinct and difficult challenge because it requires technical personnel who not only understand cybersecurity as a core discipline but who also possess a deep understanding of machine learning nuances, model vulnerabilities, and data science intricacies. This unique skill set is currently in short supply, leaving many organizations unprepared to defend against evolving threats.
This talent shortage appears acutely felt in Brazil (63.33%), while surprisingly less so in Spain (23.33%) and the UAE (30%), which might suggest different stages of AI adoption or local talent pools. Larger enterprises (e.g., 47.73% for 2,500-4,999 employees) reported this challenge more frequently than smaller businesses, indicating the scale of the problem as AI adoption matures.
Addressing this talent gap should be a business priority for every organization. As I highlighted in HBR, CEOs must embed Talent Costs into their strategic vision, recognizing that proactive workforce training is crucial for building resilient AI environments. CTOs must invest in upskilling existing security and AI teams, fostering cross-functional collaboration, and leveraging open source AI tools to accelerate the development of a robust AI security workforce. The open source community is vital here, democratizing knowledge through transparent projects and collaborative communities, offering practical learning and mentorship. CHROs, in partnership with CEOs, must develop comprehensive talent plans and harness AI as an enabler for Talent Scale. As noted by Heather Adkins at Google in her interview, AI tools “empower existing talent, make the field more accessible, and enhance operational efficiency”. An example includes AI-assisted vulnerability analysis or automated security assessments. This holistic investment in talent is fundamental for a secure and sustainable AI future.
Building Resilient AI Systems: An Open Source Blueprint for the Future
We need to build resilient AI systems to address our top three challenges: the lack of standards, shadow AI, and the talent gap. It involves implementing concrete security measures across the AI lifecycle—from securing training pipelines and ensuring reproducible builds based on clear provenance, to actively detecting data poisoning and combating shadow AI through rigorous inventory, access controls, and transparent practices. Open source emerges as the pivotal enabler, fostering standardized frameworks like SLSA, providing essential tooling like Sigstore and ML-BOMs for transparency and auditability, and enhancing visibility with widely adopted and supported distributions of platforms like Kubeflow. At the same time, it’s vitally important that we address the AI security talent gap: CEOs must strategically invest in talent development, CTOs should prioritize upskilling and cross-functional collaboration leveraging AI-assisted tools, and CHROs have to develop comprehensive talent acquisition and development plans. Open source significantly aids this by democratizing knowledge and fostering collaborative learning, critical for a secure and sustainable AI future.
To dive deeper into these critical challenges and discover comprehensive strategies for securing your AI infrastructure, we urge you to read the full State of software supply chains report from IDC, Canonical, and Google. The future of secure AI relies on collective action; we invite you to contribute your expertise, discuss these findings, and actively shape the solutions by joining the OpenSSF community and our working groups.
About the Author
Hugo Huang is an expert in Cloud Computing and Business Models, spearheading joint innovation between Canonical and Google. With 18 years of experience in Digital Transformation, he possesses deep expertise across open source, cloud computing, 5G, AI, cyber security, and remote working. Hugo is a passionate leader of global, multi-culture, and cross-function teams, encompassing Product Management, Engineering, Strategy, Business Development, and Leadership Management. He holds an MBA from MIT Sloan.