AI-Generated Code: A New Link in the Software Supply Chain

Companies must apply the lessons learned from securing the open source software supply chain to the code they’re generating with AI.

The software supply chain comprises everything that touches an application or plays a role in its assembly, development, or deployment. This includes proprietary and open source code, APIs and cloud services, and the infrastructure used to build and deliver that software to the end user. And now, AI-generated code has rapidly emerged as the newest vector in the software supply chain. According to Gartner, 75% of enterprise software engineers will use artificial intelligence (AI) coding assistants within the next four years.

The opportunities posed by AI are vast, but they should be balanced with reasonable caution; the repositories used to train generative AI tools contain vulnerabilities, copyrighted code, and code with restrictive licensing. AI-generated code, like any other code — proprietary or open source — is likely to expose companies to software risk if left unchecked. And like any other code, it needs to be thoroughly evaluated for security vulnerabilities and weaknesses.

For many, the software supply chain often presents a complex security challenge. To cut through this confusion, security teams should approach software supply chain security programmatically.

In its nascent years, application security’s focus was on the need to secure the code you write. Now it has shifted to the need to secure the code you use. The annual Synopsys “Open Source Security Risk and Analysis” (OSSRA) report underscores this shift toward the need to secure the code you use. The 2024 analysis found that 96% of total codebases scanned contained open source. With the addition of AI-generated code, there is yet another external source of software to consider.

Organizations should rely on the tried-and-true methods of the traditional software supply chain, centering efforts on understanding the provenance, risk, and trust of all components in their software. The seeming complexities of supply chain security can be distilled into answering three questions:

How can I identify and mitigate security risks associated with each source?

This is the disciplined programmatic approach. By building a security program around answering these questions, an organization effectively adopts an approach, ideology, and toolset that enables simple and effective software supply chain security.

Although this programmatic approach helps simplify how an organization tackles software supply chain security, AI-generated code is bringing new complexity.

AI is promising to deliver efficiency and productivity gains — developers can create more code than ever before. But teams are grappling with how to adopt and manage this fourth source of software in the software supply chain (proprietary, open source, third party, and now AI-generated code).

AI-generated code is essentially copying code off the Internet without proper attribution, making it nearly impossible for the developer to understand its origin. Consequently, this exposes users of AI-generated code to risk that is difficult to quantify or even understand. With development increasing exponentially and code flowing from unvalidated sources, left-unchecked, AI-generated code stands to outpace existing security efforts.

There seems to be a general and pervasive belief in the market that large language models (LLMs) produce secure code. This engenders the false implication that AI-generated code is trustworthy.

Today, we see misleading statements about LLMs producing secure code, and recent academic research shows that developers are more likely to accept insecure or low-quality code if it is from an AI-generated tool. There is pressure for developers to trust AI-generated code as inherently more secure than open source, when in fact it should be regarded as having the same risk profile as open source software (OSS).

The risk of this false sense of security is a lack of vigilance about what sorts of quality and security issues are introduced into codebases. From open source security risks and licensing conflicts to additional risks buried in AI-generated code, lack of effective security measures could expose an organization to significant legal risk.

Companies are slowly introducing AI-generated code into development pipelines, with the most successful adopters viewing it through lens of OSS security lessons of the past. They are pulling learnings and practices from OSS governance programs and applying them to new AI strategies. Existing software composition analysis (SCA) solutions and open source governance programs are a great fit for securing AI-generated code, and customers are making efforts to modify existing programs to be “AI-code aware.”

With increases in the sheer volume of code, application security programs must be capable of performing system-level analysis. Only with a “defense-in-depth” approach capable of finding system-level defects can security efforts hope to keep pace with the speed AI promises to deliver.

AI-generated code should not be categorized as a threat; it promises to unlock massive amounts of innovation and, when handled appropriately, is safe to adopt. It is conceivable that in the not-too-distant future, AI could have all the capabilities needed to assemble complex components and even an entire application, unlocking massive innovation but also demanding security that can keep pace and scale.

As organizations evolve their software supply chain security programs, they should be tailored to enable AI coding adoption, rather than inhibit it. AI promises to transform the development landscape, and with the proper mindset and strategies, it can become productive and invigorating.

Jason Schmitt is a seasoned leader with a proven track record of deep technical knowledge, product development, insight into emerging and rapidly changing cybersecurity challenges, and go-to-market strategy and implementation. He brings more than 20 years of experience in security and enterprise product development and management. Jason most recently served as CEO of cloud security startup Aporeto, where he led the company from pre-revenue through a successful acquisition by Palo Alto Networks. He has a deep background in software development and application security– leading Enterprise Security Products at Hewlett Packard as Vice President and General Manager of Fortify and ArcSight. Jason combines security domain expertise with strong experience delivering SaaS/cloud-based solutions. Jason is a Louisiana native, who completed his Bachelor’s in Mechanical Engineering and Master’s in Computer Science at the Georgia Institute of Technology, and his MBA at Georgia State University’s J. Mack Robinson College of Business.

Where and Why Threat Intelligence Makes Sense for Your Enterprise Security Strategy

Safeguarding Political Campaigns: Defending Against Mass Phishing Attacks

EMA: AI at your fingertips: How Elastic AI Assistant simplifies cybersecurity

Shining a light in the dark: observability and security, a SANS profile

The Cloud Threat Landscape: Security learnings from analyzing 500+ cloud environments