Tag

Open Source Security

What’s in the SOSS? Podcast #69 – S3E21 Watering the Community Garden: Navigating the EU CRA for Open Source with Roman Zhukov

By EU Cyber Resilience Act, Podcast

Summary

The clock is ticking toward the European Union’s Cyber Resilience Act (CRA) deadlines, yet a staggering 66% of organizations remain completely unaware of what is coming. In this episode of What’s in the SOSS? host Sally sits down with Roman Zhukov, co-chair of the OpenSSF Global Cyber Policy Working Group and Security Communities Lead at Red Hat, to demystify this sweeping regulation. Using a brilliant “community garden” analogy, Roman breaks down the distinct roles of maintainers, stewards, and manufacturers under the law, illustrating why the traditional “consume and forget” model of open source is officially dead. They dive deep into the newly released 2026 CRA Awareness and Readiness Report, exposing the staggering $250,000+ engineering tax of maintaining private forks and detailing how active upstream collaboration is no longer just good citizenship—it’s a business and legal necessity. Tune in to discover actionable strategies, free educational resources, and how we can collectively bake “compliance as code” into the open source ecosystem.

This episode is part 2 of a four-part series on the CRA:

1. CRA Readiness: Practical Strategies for Open Source Communities with Megan Knight

3. Private Forks, CRA Deadlines, and the True Cost of Open Source Compliance with Dave Russo

4. Navigating the New Era: The EU Cyber Resilience Act Explained with Madalin Neag

Conversation Highlights

00:23 – Introduction: The CRA Countdown is On
02:04 – Tomatoes, Gardens, and Restaurants: Defining the CRA Personas
06:40 – Reality Check: Shocking Findings from the 2026 Readiness Report
10:12 – The Awareness Gap: Why Are We Ignoring the Warning Signs?
15:01 – The End of “Consume and Forget”
17:49 – The Private Fork Tax: A $250K Engineering Trap
23:34 – Red Hat’s Blueprint & Free Community Security Tools
28:44 – Taming the AI Vulnerability Tsunami
31:27 – Build Your Program Now: Action Steps for Manufacturers
36:12 – Supporting SMEs & Navigating Free Resources
40:30 – Carrying the Torch as an OpenSSF Ambassador
43:35 – Rapid Fire & How to Get Involved

Transcript

Sally (00:23)
Hello, hello, and welcome to What’s in the SOSS, an OpenSSF podcast, where we get to talk to some amazing people who make up this great open source security ecosystem. And today I have a very special guest. But before we get into that, here’s what I want you to know. The clock is ticking towards a massive cybersecurity deadline.

Research shows that 66% of organizations don’t even know what it is. Love it or hate it. It’s the European Cyber Resilience Act or the CRA. This is important because how do we navigate this shift? Stick around because today we’re going to talk to someone who has a CRA plan.

Roman Zhukov. Roman, thank you so much for being here. Why don’t you introduce yourself to everybody?

Roman Zhukov (01:21)
Certainly, thanks for having me here. Hi everyone, my name is Roman and I work for Red Hat. I do some cybersecurity things on the internet. But approximately two years ago I joined this NICE effort, as we all now call the EU Cyber Resiliency Act. So it is quite a journey, but having fun.

Sally (01:44)
Well, I love that you say it is fun. And yes, it has been quite a journey. And we are going to unpack some of that today and try to help the audience so they can better be prepared. So I guess let us just start with the basics, right? The European Cyber Resilience Act, CRA. How do you define that? And how does it fit in with the various personas who show up to meetings with you or maybe are not aware but need to be aware. I know there are manufacturers, open source stewards, and individual maintainers in this persona group. Can you define those?

Roman Zhukov (02:23)
Yeah, absolutely. and you are right, that is a lot, a lot of roles, a lot of ambiguity. But let us start with the basics. The CRA is the regulation set up to essentially safeguard European customers of digital products by establishing mandatory cybersecurity requirements for companies that operate in the European Union. and suddenly, if not compliant, those hardware and software vendors will be unable to sell their products in the EU market after December 2027. I think it is worth to mention that the first requirement actually kicks in even earlier, a few months from now, September 2026, all of those organizations need to report vulnerability and severe incidents to the authorities. I would use the analogy for the CRA.

And I mean that is so complex, but I will try to break it down to a few things that should be familiar to almost everyone to understand how the series defines the role or roles that you just mentioned. Let us use the analogy of a community garden. Think of the maintainer as the hobbyist gardener. These are developers who cultivate a beautiful, let us say, patch of tomatoes just because they love gardening and they want to share the harvest with the neighbors and the CRA recognizes those actors and the CRA recognizes that forcing them to pay for legal compliance would kill the garden eventually. Therefore, if you are an independent maintainer doing some non-commercial work, you are completely exempt. So you are out of scope.

Contributor or maintainer, if you don’t charge any money for anything for your open source project, just keep planting the code, right? Just keep doing this, you’re out of scope. Now there is an essential role of the open source software steward or simply steward that we call it in the community. You can think of them as a guard garden association, right? There are foundations like OpenSSF or Linux Foundation, they could be big companies, they do not own the veggies themselves, but they provide the land, the fencing, the water supply, you name it, and you know they protect the ecosystem. So under the CRA, a steward is responsible for setting up the baseline policies and basic health checks for the projects they host, acting, you know, as a trusted buffer, I would say, between the community, those planting veggies and the corporate interests.

Now, finally, the most important actor under the CRA that actually has the most obligations and most liability is the manufacturer. We can think in our little example of them as a commercial restaurant, right? This is like basically any company that walks into the free community garden, scoops up the tomatoes, puts them into the premium pasta sauce.

And sell it to paying customers. So the CRA is kind of crystal clear here. If you make money out of software and you know you just sell the software to the European market, you bear the full legal liability for these things. You are the manufacturer and you must run the safety gates to ensure that your product does not give your customers the let’s say the digital equivalent of food poisoning, if it would happen for community garden. So that’s those are the roles under the law.

Sally (06:12)
Love it, Roman. That really makes it make sense for me. I used to love the farm to table restaurants. That’s what I kept thinking of when you were using that metaphor. That’s such a great metaphor. Thank you for explaining it. Yeah, I want to get out into the garden now. Okay, so we have the 2026 CRA Awareness and Readiness Report, which launched on June 8th. Roman, you wrote the introduction for this report, what was the most surprising finding for you?

Roman Zhukov (06:52)
Right. I would start off with writing the introduction to this report was an incredible honor to me. And it was at the same time the sovereign experience because the data what really matters and the data gave us a massive reality check. There are a lot of really mind-blowing findings, I would say. So I encourage everyone to dive into the whole thing. It is hard to pick up.

The one thing, but the biggest finding to me personally was like really the contradictory shocking nature of findings when you combine them all together. Not single number, but when you look at them and you see the full picture, that is fascinating.

while critical enforcement milestones are rushing really towards all of us, as I mentioned, the September 2026, the first vulnerability reporting obligations will be enforceable. At the same time, the global awareness has completely stalled, right? Like over 60% of respondents still entirely unfamiliar with this CRA, which is fascinating.

Compounding this the organizations that are aware on the other hand are attempting to what they think like ease compliance by burying code inside the private forks. Unfortunately I have been hearing this kind of perception or understanding by some companies that you just can simply fork and kind of escape some of the obligations, right?

But the reality check, backed up by numbers in this report is the report explicitly provides that maintaining these isolated codebases can be really a disaster from economical perspective to a company, draining an average of more than $250k dollars per release cycle in pure engineering depth or kind of simply the l the losses if you would.

But I am glad to see this new report came out actually, as it sends an undeniable message, as far as I see it, to executives and to all the companies worldwide, because the traditional checkbox compliance exercise, as well as that we used to call “consume and forget” for open source, right? Those models are coming away actually. And we should stop treating the open source like a free vending machine.

That is going to simply give you some candies for free without any collaboration, anything else, and therefore now for all the companies that are affected by the law, it would be really hard to deny active upstream collaboration, because it is now something baked in the law and not optional.

Sally (09:50)
Thanks for that, Roman. Yeah, there’s a lot to unpack here. The one thing that stood out to me is you said over 60% of respondents are still unfamiliar with the CRA. Why do you think this awareness gap persists even as the deadlines approach?

Roman Zhukov (10:07)
Yeah, first of all, to me as a co-chair of OpenSSF Global Cyber Policy Working Group, which is a home for collaboration between all of those actors that I mentioned, manufacturers, maintainers and other community members, to me that awareness gap means that we as a working group have so many things to do. I feel we kind of do some work on the awareness front, but there is so much more to be done on that front.

But to be serious, I think this persistent awareness gap comes down to a few typical behaviors that we have seen every day in the software world specifically. First of all, that is like somebody calls it the compiler warning syndrome. That means like in software development, I think

A lot of us are famous for ignoring warnings like compiler flags, all of this that is not that red but yellow, something like that. I will fix that later, and along these lines. I think many global companies are treating the CRA compiler warnings to that extent, right? Because the active enforcement and fines and everything will be starting in

2027 and still of companies saying okay, this is not a this year thing, right? We will come to it as it goes. So they are basically waiting for the build. You know, yeah, they are simply waiting for maybe others to start off and pave the way, but also waiting for the actual lag enforcement date, which is wrong of course.

Yeah, that is unacceptable. The another thing that I think is hidden here is the problem to parse the law actually and to convert the CRA to some of the actionable items. And the lack of the clearly lack of the implementation guidance and the standards. Now the EU authorities are working on the standards and some of the other clarifications that would help manufacturers to navigate this the CRA. But so far a lot of them are still in draft mode.

That means all of these companies are simply left in front of the CRA text itself and it could be ambiguous, it is not concrete. I often got these questions like, okay, we read the CRA one hundred times, but we do not understand what we should do exactly because there are a lack of the technical details on it. So this also adds, I think, to this kind of

Awareness gap, I mean we have heard about this CI, but we don’t know what exactly that means and how exactly we would approach this. And the final piece is the geographic kind of illusion. at least previously I have heard a lot of misunderstanding about this, and I think it’s backed by the report itself.

over seventy percent of North American software producers aren’t familiar with the CRA, right? And this is a belief that okay, I you know, I’m in America, I don’t deal maybe with the EU and it doesn’t affect me, but the CRA is so complex it affects the whole supply chain and there is a non zero probability that even if you don’t sell anything directly to the European Union, you are maybe one of the suppliers of those who do, or you simply maybe the contributor to open source or open source steward for instance. Those also have some obligations. So that also adds to the lack of awareness, I think, generally.

Sally (14:02)
Yeah, it does. Thank you for that, Roman. you brought up a critical point that the report mentions the traditional consume and forget model of using open source and that it’s no longer viable under the CRA. How does this regulation change how companies must interact with upstream projects, if you will?

Roman Zhukov (14:23)
Yeah, absolutely. I mentioned it already. I like the analogy between the consumer forget and kind of free vending machine, right? You push a button and something pops out and you just drop it into your commercial application and that is it. You never thought about the maintainers.

And I personally talk to a lot of companies out there and I frequently ask the question, hey, who is consuming open source? Everybody, right? Because open source is everywhere. But when I ask the question who contributes back, significantly less hands, right? But now under the CRA if you sell a product containing the code, you know, you’re kind of legally responsible for your products, that means effectively for all kind of all of the components.

And if something breaks, even if it is third party to you, if it is an open source component, you hold the whole liability to customers, to your consumers. And I think the CRA has a good chance to end this consume and forget paradigm by transforming open source security into nto the obligations, actually, right?

And yes, historically you are right, commercial consumers operated under the assumption that because open source components work fine for years, someone else would always take care. Now there are certain obligations, like for example, if vulnerability is found in the open source, you need to report back to upstream.

If you produce a fix as a manufacturer of the product, then you actually have to upstream this fix back to the maintainers. And this is a huge push and I think a huge paradigm shift from pure consumption to actually collaborating with upstream and with open source projects.

Sally (16:37)
You talked about the cost of private forks. I was wondering if we could dive deeper into that. According to the data, maintaining private forks cost organizations an average of $250,000 per release cycle. Or actually more than that, $258,000. I’m looking at the research. What is the business case for moving that work to upstream contribution instead? Can you talk a little bit deeper on that?

Roman Zhukov (17:03)
Yeah, that is a perfect finding. That actually to me is one of the most compelling findings in this report: the hidden tax of the private forks. As a long-term open source supporter, I am personally convinced that working upstream, as I do and as we do here at Red Hat, for example, is the only path to go, right?

But now the CRA report backs it up with the numbers and provides the actual business reasons for everybody else besides me and the companies that already contribute to upstream instead of forking things. Yeah, the maintaining private forks may cost organizations more than $250,000 US dollars in labor, which is a big amount of money actually. I would justify the business case for why you should actually contribute upstream instead. I can start with the basics, right? And the basics is that maintaining a private fork forces typically engineering teams to pay a continuous rebase tax.

That means that every time that upstream project releases a new version, right, your developers must manually port your customer patches over, resolve conflicts, and et cetera, and et cetera. On the flip side, by actually merging your features or your or whatever you do new for the project directly into the upstream.community eventually inherits the maintenance and etc. And then you subsequently consume back the newer version where everything is integrated and your code is updated automatically and you don’t spend this additional money to maintain all of these forks. The other business reason could be the engineering velocity that we call in the industry.

That means when developers burdened under the technical depth of the private fork, that significantly affects the product roadmaps. Right? You just have limited amount of resources typically, so you want to spend them wisely and by contributing upstream instead, it shifts your engineering resources from defense like fixing broken mergers to more

Building revenue generating features, if you will, for your products to focus on the core innovations and etc. Those are the basics I think that are under the core incentives why you should contribute upstream. But now the icing on the cake is the CRA factor. Because under the CRA maintaining a private fork would

Increase dramatically, I would say legal risk exposure for you as a product manufacturer. Because pretend if you integrate an open source library into a product, it is third party for you in the meaning of the CRA. That means like the biggest obligation that you would need to form against this dependency, this third party is to perform due diligence, which is

Kind of the one obligation. As long as upstreaming fix things, and hopefully you help also upstream to do these fixes. So you are not responsible for vulnerability management, you are not responsible for secure by design and by default, and etc. Now imagine with a fork, right? Your organization would hold hundred percent legal accountability for this library.

That is now part of your product because you forked it, it is now kind of your property, your responsibility. And nobody cares if you actually do not know the architecture, you may not know the design of this component or even you write the product within a different framework, right? Now you should still do secure by design

Risk assessment, vulnerability management, and full conformity assessment because of the CRA. And now the scale factor, imagine now you have 1000 of these open source libraries you are ingesting every day. And you need to assess it, if it would be even feasible to ensure full engineering compliance support for all of them. So that is the massive shift and the massive push towards better upstream collaboration

backed up by the real kind of business case and business considerations from that point.

Sally (22:04)
Thanks, Roman. Well, you and Emily and me worked on an OpenSSF case study based off of Red Hat’s framework for depending the open source supply chain in this new era of regulations, what is the most important lesson other enterprise organizations can learn from Red Hat’s approach?

Roman Zhukov (22:28)
I would encourage everyone to read this case study which is published under OpenSSF now. But if I picked the only one thing that enterprise organization need to take away, it would be something like this: true software supply chain security requires enterprises to transition from passive consumers into active open source contributors, supporters with the focus on practical security.

If it seems like overhead from the you know as it looks like but it is actually not really overhead if you think it is from occasionally watering your community garden coming back to our lovely example in if you think of occasionally taking care of this community garden in your town so that yourself and others can enjoy these lovely tomatoes at the end, that is a different explanation. Yes, companies like Red Hat take up their responsible stewardship approach, for example, because we naturally support and are committed to do so, some of the open source projects that run the entire world, right.

But we understand that voluntary maintainers do not have compliance team, they do not have legal departments, and quite often the case they lack security background, right, to actually deal with complex mandates and policy and things like CRA. So for other companies, I would suggest to actually break one of the myths.

That I constantly hear say, we are not able to contribute to open source because we are not subject matter experts. Or we use the different programming language. It is kind of an excuse, I would say. Instead, think of there is a lot more you can help with actually, starting just from your questions, right? Just asking your questions is good.

Or raising your CRA concerns, or going down to the frameworks, to the white papers. So we have in the communities we have so much work to be done, both from like collaboration perspective and also on the on the practical security front. So you know, that is kind of natural that the companies that of all sides can actually contribute meaningfully to all of this work.

And I would also add that enterprise compliance has historically relied on manual forms to some extent, right? All these GRC questionnaires, legal declarations, spreadsheets. And this approach is not working for the CRA right now. The lessons from our own journey on the CRA is to focus on the tools, on the practical security things.

Because by implementing automated machine readable security tools and specs and frameworks, organizations can shift the burden of evidence away from voluntary communities to themselves, which is kind of the goal of the CRA. By the way, you cannot offload the liability to the community. So you as a company need to figure out this thing by yourself eventually.

Good news here, there are many open source security standards and tools that can help you to do so. Just to name a few under OpenSSF, we have Open Source Project Security Baseline (OSPS), we have SLSA framework, we have Gemara compliance framework, we have GUAC, and a number of other projects that actually can help both maintainers of the projects to do security better, but also can help companies as they as they adopt them.

Sally (26:43)
Love that. I’d love to hear the good news and all of the areas where we can help make this feel easier for the enterprises and for the whole ecosystem. And you know, I don’t think it would be a conversation in 2026 under open source security if we didn’t talk about AI, right?

There is a massive spike in reported vulnerabilities due to AI-driven tools. How can open source projects realistically manage this influx of disclosures?

Roman Zhukov (27:24)
Yeah, you are right to the point. To handle the massive tsunami of AI driven vulnerability reports and disclosures hitting the open source ecosystem now actually even harder than the enterprises, projects have to adjust their vulnerability management practices and of course tooling to adopt what I call the “post Mythos era” we are entering right now.

A few practical recommendations that I would stress out is first, you need to have a robust intake filter to do the basic sanity checks. It is clear that now a lot of AI agents are operating out there, and not all of them produce actually high-quality reports or high-quality pull requests, etc.

And as a maintainer, you need to introduce some of the filters and checking for things like okay, I do require the proof of concept, I do require affected versions, I do require reproduced steps and a couple of other things. If there are ants in the report, they just simply reject them. So as a maintainer I do not have capacity to deal with these reports.

Now of course automation would help. Automation of looking into some metrics even for the reporters themselves, like reputation, maximum number of emojis even in the report and in the pull requests. I have noticed that a lot of AI assisted contributions are flooding with the emojis, right? Maybe that is because some AI tools just like emojis, something like that.

Checking some of these things would actually help to reject anything that is suspicious bluntly for you as the reporter. So those would be my recommendations.

Sally (29:23)
Yeah, I have to tell my AI assistant, no emojis, no more emojis. They flood it. That’s interesting. Okay, so for the fewer than half of the manufacturers that were surveyed are expected to be fully compliant by the December 2027 deadline. What should these uncertain organizations focus on right now?

Roman Zhukov (29:48)
That is a hard question. I would say I think organizations need to stop looking at the CRA as just a 100-page legal monster something, or perceive it as a next GDPR with this cookie pop-up thingy that you need to implement, which CRA is far from actually. And organizations need to start treating it as a standards software engineering refactoring project. I would be that ambitious.

The EU commission itself actually keeps saying that the intention of the CRA is to make sure that companies invest in the real security improvements. And then compliance shall follow naturally. So start exploring the CRA in great detail right now.

Including all of the implementing acts that we have right now. And you will realize that scope is broader than you eventually think. There are a lot of terms, a lot of complexities in the intersection between the software as a service that are technically out of scope, but there are some of the components that we call remote data processing solution that could potentially make some of the cloud solutions be in scope, right?

And the open source is technically out of scope, but there are many cases when open source can be in scope. So understanding your role, whether you are a manufacturer, distributor, importer, maintainer, steward, contributor, is crucial, right? Because obligations differ dramatically.

But I would suggest to remember the one simple thing, liability always flows downstream. That means you cannot really offload the liability of anything to open source, for example. As a manufacturer, you are ultimately responsible for anything. So one of the advice that I can give to all the companies is to just stop harassing maintainers and asking them how they are doing about the CRA, right? It is not their responsibility to make sure they are CRA compliant.

To draw kind of a bottom line here, do not wait. Build your CRA program now. It is because it is definitely a cross-functional effort. I can speak from our experience. Yes, we are a big company, right, but buy-in from across the different functions inside the company is really vital to set a CRA program up for success. Right. And let us be honest, CRA did not invent brand new security practices. Some of the timelines like 24-hour reporting obligations of actively exploited vulnerabilities and severe incidents are hard. But in essence we all should be implementing all of the things like security by design or security by default, and doing proper vulnerability management as well as SBOMs for years now, right? So apply what is available right now to your products, to your practices. And again is good news. There are plenty of mature frameworks and a lot of open source tools actually that can help you, right?

If you do not know what to do the simple thing is just cooperate upstream. Join me, join my company and others, other peers at initiatives like OpenSSF, Global Cyber Policy Working Group, to learn more and to learn how to start. But yeah, there are plenty of things that you should do right now to prepare yourself to be compliant. 2027 is around the corner, really.

Sally (34:06)
I do feel more hopeful now hearing from your perspective and the things that people can do. One last thing that struck me from the report that I wasn’t expecting was that over half of the European small and medium enterprises, those SMEs, remain unfamiliar with the CRA. So how can the open source community best support these smaller type of organizations that lack a large legal team?

Roman Zhukov (34:30)
That is exactly where the community can actually help and help significantly. For example, as part of our OpenSSF Global Cyber Policy Working Group effort, we are creating digestible white papers, checklists, simple guidelines dedicated to different roles under the Cyber Resiliency Act, and they are meant to be easily consumable by those people in those organizations that are not necessarily familiar with the complex legislations or even unfamiliar with the sophisticated security concepts.

I would say one of the biggest and successful assets that we collaborated on and produced under this effort is the free CRA class that is now hitting way over 5,000 of enrollments, and this is really a good starting point specifically for small and medium organizations because it is again short, concise, sticks to the points and is free completely for everybody to take. It teaches you about the basics of the Cyber Resilience Act.

Also in our group we do regular briefings on what is new in the CRA development. Which is again super important to small and medium organizations because they simply do not have time and capability and desire to learn what is happening in the legislation every day. Therefore, we also do the workshops and brainstorming sessions, both offline and online, to handle cases that the members of our community actually bring to us. And this is a real thing. I would encourage all the SMEs that have questions to just come to our group and ask your questions. Bring up your case and we will do our best to provide our community advice for what you should do about this.

In addition, we also collaborate across multiple OpenSSF projects and broader industry initiatives to work on tools that would help companies, specifically smaller companies, because they are meant to handle or automate the large portion of the CRA compliance, at least we believe so. By standardizing and integrating community tools like the open source project security baseline or OSPS and now actively developed due diligence solution for manufacturers directly into the templates, into the development practices.

We actually as a community can bake compliance into the code, right? And finally reach the point when we can tell, okay, this is compliance as code, and we have all the evidence produced automatically. And this transforms to my personal understanding, a huge portion of complex CRA requirements into free, off the shelf, to some point click to deploy guardrails that can allow smaller companies to stay compliant with the CRA and more importantly actually implement the good security practices like secure by default and etc without needing the huge security teams or the army of lawyers or something.

So that is what can really help SMEs to navigate the Cyber Resiliency Act.

Sally (38:16)
That’s really helpful, Roman. Thank you. All right, let’s shift gears here. I want to talk about something that first off, just want to give you a huge congratulations for that you were just recently became part of our OpenSSF ambassador program, named to the first cohort. A quick question, I guess. What are you thinking you’re gonna do with this new role?

Roman Zhukov (30:40)
Thank you. Well, it is so natural to me as I feel I have been an ambassador in OpenSSF for quite a few years now already. I joined an OpenSSF fan club with the introduction of the first they call it beta or alpha version of the scorecards, a tool that analyzes the security health of open source projects in an automated fashion.

Years later I helped to roll out OpenSSF Scorecard across a few companies of different sizes. Right? So that is how my joining naturally started with OpenSSF. So now that is how I get into it. And you know the drill, right? Once you are in, it is really hard to escape from doing these good things.

With the community and for the community essentially. But to be serious, yeah, I am very proud to be named as an official OpenSSF ambassador. It is an incredible honor to me and also a huge responsibility. In this role I hope to provide what we call the open source way. And educate everybody that the open source way is not just a methodology for writing code, but also it is the most effective framework that we have now for solving global security and regulatory challenges, which we do not have a shortage of at the moment. As you mentioned, all of these AI vulnerabilities and regulatory compliance challenges across the world, they add up to the whole picture.

So by bringing a pragmatic engineering centric voice and approach that I personally really try to follow every day by bringing this to the table. I hope we can ensure that the security becomes an active feature of the ecosystem we all depend on so heavily, right? Rather than a blocking thing that is still perceived by some organizations.

So I have done security for twenty plus years now and it is so close to my heart and I just wanted to carry this torch and I want to spread the word about the beauty of security and all of the open source tools and collaborative nature of it so that it can be easily digestible and implemented across the other companies.

Sally (41:32)
Love that. Okay, that wraps wraps up our main deep dive. But before I let you go, Roman, we have a little thing called rapid fire on the OpenSSF What’s in the sauce podcast. So just first thing that comes to your mind, I’m gonna ask you a series of questions. This is fun stuff, not work stuff. All right, are you ready?

Roman Zhukov (41:52)
Mm-hmm. Yes, love it!

Sally (41:54)
Okay. Tea or coffee.

Roman Zhukov (41:56)
Coffee.

Sally (41:57)
Oh, I wasn’t expecting that. Cheers. I love it.

Roman Zhukov (41:59)
Oh really?

Sally (42:00)
Yeah, I was thinking tea. You’re in Ireland, right?

Roman Zhukov (42:07)
But I love coffee as well. I can drink it every day actually. Yes.

Sally (42:111)
I love coffee too. Really? Okay. I love that. Favorite open source mascot.

Roman Zhukov (42:16)
CRA-fish

Sally (42:17)
Yeah, good one. Star Trek or Star Wars?

Roman Zhukov (42:21)
it is hard but Star Wars, I think.

Sally (42:26)
The tracks. All right. So we talked a lot about your metaphor with the food. I loved that and with growing in the garden. If you’re gonna use one of those tomatoes and make salsa, would it be mild or spicy?

Roman Zhukov (42:39)
Spicy, one hundred percent.

Sally (42:41)
Love it. Okay. well, thank you so much, Roman. To wrap it up for listeners who want to leverage OpenSSF resources, like what you mentioned, the cyber global cyber policy working group, or maybe the new awareness sig, the training course. What’s the best place for them to start today?

Roman Zhukov (43:02)
Right. I would I I would end nearly where I started with again going back to my community garden analogy. with a community garden typically you don’t need any approvals or special permissions, you just show up, learn and see where you can actually help as like everyone else with the tomatoes to make sure they they are good then to eat and to enjoy.

Right, that’s essentially what we do in the community. And despite the scary policy compliance angle, we are super friendly at the OpenSSF Global Cyber Policy Working Group. and you have to trust me to here. just join one of our calls to learn what’s going on. It is free, open to everyone. You don’t have to be a member of anything at all. Just just show up, join.

To name a few specific entry points that could can help you depending on what you need or what you’re building. We have the one stop shop hub that we call it policy.openssf.org/CRA. This is the one stop shop that can get a handle on the landscape of the CRA, what’s happening around with the news without the getting dive into the legal text and all of these complexities.

Of course, explore our GitHub repository. Under OpenSSF, we have Global Cyber Policy Working Group. If you want to dive into the governance, check out what we’re working on, like go to this repo. There are a lot of materials published there. As I mentioned, checklists, guidelines and links and all that you need to know how to further participate in the community. Like we have the bi weekly calls of our main working group. We have also calls for our SIGs, which are awareness SIG and also standardization SIG. They all do a great job in their remit. So please consider joining these efforts if you wanted to dive a little bit deeper into either awareness things.

One of the examples I would as an opportunity I would show you is right now at the very moment we’re looking for collaborators for the blog post series that we plan for the CRA. If you would like to be the co-author, please join us and help to spread the word over there. Finally, we also do a lot of meetups across the globe and throughout the year.

But we also have the workshops and online webinars like EU CRA Monthly Tech Talks, when we actually have the guests speakers speak about everything that can be related and can be helpful on the CRA implementation, again from the different companies, from the different entities and different actors on this space. So just keep an eye out on the announcements.

We always have a lot of stuff to share, but also we have a lot of work to be done. Please come join us and collaborate on the overall effort of making this CRA meaningful and actionable for everyone.

Sally (46:42)
Brilliant, Roman. I feel much better about CRA awareness. Thank you. And until next time, stay secure and happy open sourcing. That’s a wrap.

What’s in the SOSS? Podcast #67 – S3E19 Funding the Future: Community Collaboration and the Spirit of Open Source with Mila Zhou

By Podcast

Summary

Join host Yesenia as she sits down with Mila Zhou, Open Source Program Manager at AWS, to explore the fascinating intersection of finance, strategy, and security in the open source ecosystem. Mila shares her unique journey from forensic auditing to spearheading AWS funding initiatives, breaking down how strategic financial backing transforms vulnerable “long tail” projects and empowers dedicated security champions. Discover how full-time security engineers at foundations are securing critical repositories like PyPI, why community-driven forks like Valkey represent the true spirit of collaboration, and how the OpenSSF Ambassador Program is helping close the gap between developers and security experts.

Conversation Highlights

00:25 – Welcome & Introductions
01:03 – From Accounting to AWS OSPO
06:26 – A Day in the Life of an OSPO Program Manager
09:53 – Navigating Critical Funding & The Long Tail
13:25 – Valkey and Community-Driven Innovation
15:29 – The Invisible Power of Dedicated Security Engineers
28:46 – Marketing, Non-Code Contributions, and the Ambassador Program
36:25 – Rapid Fire Fun
37:05 – Final Thoughts & Closing

Transcript

Intro Music & Promotional Soundbyte (00:00)
Free open source software, they are not free as beer, but free as puppy, right? So like, how can you make sure that they are sustained, they are secured, they have the resource to continue to build? That’s critical. Those crises like Log4j, like those are also like a wake up call to everyone that like to see how much you rely on open source. So, but in how much you are underfund it. So take action.

Yesenia (00:25)
Hello and welcome to What’s in the SOSS, OpenSSF’s podcast where we talk to interesting people throughout the open source ecosystem, sharing their journey, experiences and wisdom. So Yesenia, one of your hosts, and today I have the utmost pleasure of having Mila here. I first met Mila when we had worked on the Alpha-Omega project together and truly very excited about today’s call, knowing her background, which I won’t spoil much.

It’s really nice to see the intersection of strategy, security, and sustainability. Welcome, Mila. Please introduce yourself to the audience.

Mila Zhou (01:03)
Hello, Yesenia. Thank you for having me. And it’s a great joy. It’s a great joy to talk to the audiences here. And my journey is kind of not typical, kind of everywhere. I actually was trained as an accountant. So that’s budget planning, that’s forensic floor auditing. But then, like, there’s always a voice in my mind that’s like, you’re not going to do what you started.

So when I started to work, my first job actually is a pricing analyst. I look at the ticket price. I sold ticket for Packers, for MADS, for Bus Cable 2. And it’s from there that I got used to look at data to forecast the market trend.

And from there, I moved to AWS, become open source program manager. That was very unexpected. Probably like the focus is very different as a program manager comparing to pricing analyst or accountant, because I would say as program manager, the key there is communication. Like you talk to..the community, talk to service teams to understand the importance of open source, to understand how it’s connected with your business. And then what would allow me to leverage my background in business in accounting is that I mostly manage the funding program, the credit program and security program, and both of them have the budget management mindset inside. So that’s where like I can really use what I studied before.

Yesenia (03:04)
And I love that going from accounting to now helping us in the open source space. And from what I know is you wear a lot of hats at AWS and within OpenSSF. Can you share your origin story of how you moved into the world of OSPO and open source security?

Mila Zhou (03:30)
Uh, I feel like it kind of, I kind of spoiled a lot in my introduction. Hah! Uh, well, I think really is the mindset is the mindset really is like, I want to look for things I’m interested in. So like I was keep on switching positions, looking for areas that I found like I can keep on learning and open source is the field like ever since I took this job as open source program manager, I never got, I never got bored, but always I feel like, okay, I’m learning something new. That’s incredible. And get into this field, I think open source is like an OSPO is solving a lot of, good challenges because open source projects, a lot of time we’ll talk to them as like they’re probably goods.

So most people rely on them, use them, even without notice that. like free open source software, they are not free as beer, but free as puppy, right? So like, how can you make sure that they are sustained, they are secured, they have the resource to continue to build? That’s critical.

And that’s also a problem a lot of time people and companies don’t think about. But like I think a lot of I would say challenges, crisis, but there’s also opportunities for open source to get noticed. It’s like those crises like Log4j,

Yesenia (05:07)
Mm-hmm.

Mila Zhou (05:08)
like those are also like a wake up call to everyone that like to see how much you rely on open source. So, but in how much you are underfund it. So take action to make sure that all those things are in the right place before the next crisis come. I think all those interesting problems that we’re trying to solve is fascinating to me, what inspired me to continuously work on that and persuade me that this work is very meaningful.

Yesenia (05:42)
I love that. the open source ecosystem I know when I first got in here, it was just, like you said, it’s ever changing.

Mila Zhou (05:49)
Yes.

Yesenia (05:50)
It’s ever changing, like per week with the latest breaches, with the latest technology and everything. And you really never get bored and there’s always a challenge and there’s, you know, lot of it’s a technical challenge, but there’s also the people challenge and then there’s the financial challenges…

Mila Zhou (06:06)
Yes.

Yesenia (06:07)
Regulatory challenges. It just continues and…I love that we have you here in this space and I get a first eye view at some of the work that you do. For those that don’t know, you know, the role of OSPO and what’s involved, could you guide us through what a typical day looks like?

Mila Zhou (06:26)
I think it depends on which part of OSPO you are looking at. Mine is less about license, but more about collaboration. So a huge focus on me is to connect with the open source communities through events, through our programs. Like, I think my job, the core part is to make sure that we connect with open source communities and at the same time, make sure that we work with service teams to participate in open source in the right way, in the best way, in a way that fits into the community. Because we notice that open source communities, like they are very different.

Like you talk to Kubernetes and you talk to Python, you talk to PostgreSQL in very different ways. So a typical day of me is like we talk with service teams, understand, OK, what open source activities, upstream activities they are participating in, and we will suggest them like what else we can do. And at the same time, like we do talk with open source communities, ensure that

Um, like how we can help you to ensure that they have the resource they need to continuously to build, to innovate. And then another part of me is like, uh, talking about all those open source stories, uh, like through giving talks at events, through podcasts, through blogs and others, because especially security, I do find that like open source security oftentimes it’s less a tool issue, but more an adoption issue. And that takes you to just keep on talking about it, like nudging people. But that’s amazing, because that’s actually a low hanging fruit. And of course, a very big part is how to streamline our process, our operation, to make sure that we’d the resources we have, can maximize the impact of that and have a strategic plan that we can tie back all the impacts we have in open source to business so that we can really bridge the gap of sustainability.

Yesenia (09:01)
I love that. It’s a lot to do in a day, which I know doesn’t take a day.

Mila Zhou (09:06)
Well, like, I will probably be doing a lot more every day.

Yesenia (09:11)
Yeah, but it just goes through the multi-tier thought process that you have to flow through in your day. And I’m sure the context switching from, you know, project X to program Y to foundation W, right?

Mila Zhou (09:26)
Yeah.

Yesenia (09:27)
And really seeing it from the different layers and levels that is open source, which people don’t think about. Let’s shift gears into AWS’s OSPO. I, from my understanding, you spearhead funding programs at AWS that provide resources to open source projects. When you’re thinking about this, what are like key criteria as you’re looking for when you’re deciding which communities to support?

Mila Zhou (9:53)
Yeah. Well, I think actually Alpha-Omega kind of best describes how we look at that. Like Alpha is that we look at all those obvious ones, critical ones, like Python, Rust, No-Brainer, that you definitely need to support them. Definitely need to make sure that they have the resources to build, to fix, to sustain and secure, right?

And then the other side is you look at all those long tails because you only add secure and sustain as your shortest piece. then, so look at all those who are like, I think there are a lot of critical open source projects that rely on single maintainers or a very small group of people. Then we need to…

Also love to support those projects to ensure that like they do have the resources so we can relieve some weight or burdens on the maintainers. With the credit program, we actually take credits publicly as well as internally. We definitely prioritize those projects. They are not CV backed.

They are critically important to the ecosystem. And they do have a community. If you are a single maintainer project, that’s critical, that’s widely used, of course we are going to support. But if you are a single developer who are trying to build your brand new open source projects, then probably it’ll be hard for us to prioritize you just because there is a long tail of open source projects that we want to make sure that they can get the help.

Yesenia (11:50)
Yeah, and I know from our conversations within the Alpha-Omega group, like, there’s like a thousand that’s considered critical and there’s definitely more.

Mila Zhou (12:01)
I think the number they mentioned is like 10k or 100k something like that. So I was like, wow, that’s a lot.

Yesenia (12:10)
Yeah, every year that’s a lot. And I know there was a lot of debate in even creating that list. So it’s interesting because there’s projects that even though it’s like 10,000 projects, there’s projects that we’re not even considering or thinking about or know about really. And it’s not because we’re not thinking about it. It’s just we are thinking about the 10,000 that we already have.

Mila Zhou (12:29)
Yeah, I mean, almost with Alpha-Omega, I think a lot of conversation you also heard is like the Omega part, we actually have kind of have real challenge to connect with them, identify them and meaningfully improve their position. And I felt like we are trying to use AI or like trying to leverage all those security tools and help maintainers to use those tools. And then that comes to the adoption challenge we talked about.

Yesenia (13:10)
Yeah. So it’s interesting because you’re both in AWS and the Alpha-Omega, but I’m just curious, how do you balance AWS’s strategic goals with the need to foster neutral, community-driven innovations within open source?

Mila Zhou (13:25)
I think sitting in my role, I do have the privilege that our goal actually align pretty well. Fostering community-driven innovation wherever we can is my team’s strategic goal. So we partner with service teams to persuade them to invest on Upstream and drive the community building. That definitely put me in a kind of position where I can just focus on that, like foster the community-driven innovation. And if we take a look at all the work we have done, I would say Valkey is a very good example. Valkey is a public fork of Redis, but Valkey has been like keeping open source and community building as its core value. And I really looking forward…

I’m super excited about Valkey and I have been helping them to organize events, especially connect the community in China. I found my work is super rewarding in that part.

Yesenia (14:40)
I love that. And especially one of my favorite things about open source is, you know, it’s not just within whatever country or city or state you’re in and you know, it’s impacting different areas in the world. Like you just mentioned China…

Mila Zhou (14:55)
Yeah.

Yesenia (14:57)
So it’s pretty, I love that about it. So let’s, let’s shift gears a little bit because I think between you and me as Alpha-Omega leaders, this is probably something we can talk about all day. But AO, Alpha-Omega, invested over $7 million last year to secure open source projects. And a lot of it was beyond just patching vulnerabilities. What is the most significant, not obvious outcome that you’ve seen from your perspective from these investments in your AWS work?

Mila Zhou (15:29)
I love talking most about our investment on staffing. Like we staff Python software security for Seth, for Mike. One is the Python language security engineer in residence, and Mike is the PyPI security and safety engineer. We also sponsored before Samuel in Ruby Central.

And we sponsor a few other staff in Rust, Linux kernel, things like that. I found those work actually super meaningful. A lot of time, there is a lot of invisible work they did. This year, I have a talk that I’m giving. I talked at FOSS Backstage and OCX about the power of dedicated security engineer that’s mostly taking Seth and Mike’s work as a use case talking about the return of like staff security engineer at foundations.

I talk with Mike a lot and Seth, but Mike is easier because he’s in New York. We can just meet anytime. So Mike told me that like, example, for PyPI, they have about 300 malware every month. That’s 300.

Yesenia (16:58)
Wow.

Mila Zhou (16:59)
Yeah. And then ever since he started, added a feature to PyPI that’s quarantine packages. PyPI is one of the only major repositories that has this feature. So that allows him to drop the handle time, respond to malware reports time from days to less than a minute. And then,

Yesenia (17:31)
Oh wow.

Mila Zhou (17:32)
Yeah, and all the, because he can just quarantine them. Like it does no damage, but like people will not be able to download it. And then he investigates it and the report completely resolved before it was days, but now it just takes like within hours they will resolve the report. So, I think with the staff, you have a full-time eye on your environment. That’s unbelievable. And another very funny story he shared with me, I really love that. always, PyBI doesn’t allow you to upload package that contains obfuscated code.

But they didn’t enforce it because they didn’t have the tool for that. And Mike built a tool to detect that, to investigate. And then he found a package that’s been like that. And that’s actually someone’s side business. They were just taking PyPI’s resource to store their stuff and then they will direct their customers to come to PyPI to download the package.

Yesenia (18:51)
Oh, wow!

Mila Zhou (18:53)
Yeah. And Mike just gave them a few months to migrate. And they did that a few months later, the business was gone because they were kind of like using PyPI to compensate themselves for the cost.

Yesenia (19:10)
Of storage and transfer. Wow!

Mila Zhou (19:)
Yeah. But if you don’t have someone working full time on that, you will never see that. I feel like we actually definitely should talk more about how much the power of a dedicated security engineer on that. And adoption problem is also something they can really help a lot. Like with PyPI, like they have the, like the 2FA mediation part. was really like 2FA was there since 2019.

But in the past few years, really like no one cared about adopting it. And then Mike started. They had like email campaign. They went to podcast, write blogs, and then roll out all those programs that make sure that people are aware of that, encouraged to do that and make it as easy as possible for people to just change their habits, adopting to that.

That’s also invisible work, but very critical.

Yesenia (20:13)
Yeah, it’s a lot of the work that the Alpha-Omega does is very, I would say invisible, you from the audits, from the staffing, I think is one of the biggest. So I agree with that because the efforts and the work that the staff that has been funded through Alpha-Omega, just the impacts, like you said, it’s just a dedicated focus on security, whatever they see, whatever they find – has made one of the biggest changes, especially Seth’s work in the Python Foundation. And a lot of the information that they do, a lot of the things that they find and they fix within their own ecosystem, my favorite part is that they share it with the community. It’s not just like they do the work in silence and it’s like magic…

but they’re like – It’s like, we go: this is how I did it. These are the steps. This is the improvement. This was the impact. Go forth and conquer. And I think I agree with you on that. The staffing is one of the…

Mila Zhou (21:09)
Yeah. I think that’s really a beautiful part. Like with Python built trust publishing, and then we see Ruby started to also have the trust publishing process. And then like all those sharing, they’re actually really… That’s the spirit of open source. Like you stop to reinvent the wheel. But just learn from each other that allows us to innovate fast, that allows us to help each other, especially with security. There are too many news of attack.

Yesenia (21:43)
Yeah.

Mila Zhou (21:45)
And everyone feel that we need help. Especially like when you are the one who is attacked. It’s not that you don’t have the ability to respond, but like there is a lot on you, like emotionally, mentally and physically, and like you have to respond to all those. And it’s amazing if you can have a support. I think with Alpha Omega, we also see that we provide a space for like security experts in different ecosystems to collaborate together to solve the problem. So that’s actually amazing.

Yesenia (22:22)
Yeah, community collaboration, know, not many, not many people, maybe by the time this podcast is released, we’ll, we’ll be sharing about the corp of security engineers. It’s something I’m working on right now to share insights of that, but just that’s one of my favorites really is that corp of security engineers of folks from these different foundations and these different ecosystems coming together and trying to solve one of the bigger open source issues of like, how do we patch these vulnerabilities? How do we help a single maintainer or a volunteer maintainer with a critical project that has, you know, a massive log4j for example, or, you know, the onyx access that happened, the NPM attacks. it’s, know, it’s job security as a security professional. We’ll never, we’ll never get bored because every week there’s at least three that show up.

Mila Zhou (23:20)
Yeah, that problem sounds super exciting. Very, very interesting. is it a program that you are leading or a conference or a paper you are working on?

Yesenia (23:32)
It’s an article that we’re going to be releasing, but the group has cadence conversations about once a month, once a month every two weeks. We’re meeting to try to really solve and put together that process. And this is folks from industry, I’m not going to call out companies, from the industry and then from the foundations that are coming together.

And we’re looking at how do we manage these vulnerabilities? How do we manage these security fires that happened and how do we put things together so open source can be a little bit more proactive rather reactive.

Mila Zhou (24:09)
Well, let us know. Let us know when it’s published.

Yesenia (24:13)
Well, we’ll probably have a podcast episode. We’ll see. So moving forward, I know we talked about the power of a dedicated open source security engineer. In context of Alpha Omega, like how do we ensure security found funding, you know, goes to those that are going to help maintainers and manage the human infrastructure and not just like code fixers. Like the difference between our dedicated open source security engineers versus our volunteers.

Mila Zhou (24:45)
Yeah. I actually, I like to go back to the Python model, because in the Python model, you actually see, a set that might as the community champion, like, they are the one build a playbook. They are the one like provide suggestions and help to enable volunteers. Seth drove PAPE, I think that’s A10 – That actually make the process to become a Python security response team member process become transparent. In Seth words: before the member on the team actually are like the, like it’s more trusted group on a mail list, a private mail list.

So the same group of people doing everything like triage, fix bad patch, and then disclaim. But like that model is not sustainable because you put all the burden on a few people and you don’t have the process for people to raise hand to join you to help you. And that PEP actually built a standard process and made it transparent. So ever since that, it was approved August last year. And ever since then, they have more than four people join the team. yeah, so I think open source on one hand, we always say that, okay, we don’t have people to help. On the other hand, actually, there is a lot of people who are willing to help, but like there is not a very clear way. I think like that actually like with security staffing, you actually enable the volunteers to join you as long as like they are providing the clear process and they are doing the…cheerleader work, to encourage people to join us.

Yesenia (26:42)
I love that because it shows it more as a role model. I know when I started, there was several folks in open source that for me were role models. I was like, I want to be just like them, or I love the work that they do, or how they present themselves. So I agree with the security champion becomes that like that advocate, that visual, that role model, the inspiration for people to like, I really like the work that you do. I like how you do the work. Teach me how you work or how you do.

Mila Zhou (27:15)
Mm-hmm. Mm-hmm. Yeah, yeah, yeah. And I think they represent the spirit of like, yeah, just do it. Because a lot of time in community, people will also ask, can I, could I, do I have the permission? But they are the ones who just do it and then tell you, you can, and I can help you. I think last year when I went to…like last year when I attended the Spring Days at PyCon and then that was the first time I attended the spring today. I don’t know if you know Eric in Read the Doc. Yeah, it was super nice. It was super nice encouraging me to like it. you can commit. You can help to like improve read the doc. I was like, wow, I want to try.

So like that kind of encouraging environment and like especially security expert like Seth and Mike, they can really help you to solve your problem, understand how security works and improve your position. With them being there, that’s incredible.

Yesenia (28:13)
Yeah, it’s such a great space and the same, and it’s not just Seth and Mike, but a lot of the community members are like, you want to join? Let me show you, come. Let’s be best friends. So moving over into marketing security, I think that’s a good topic that we’ve kind of clicked on in different areas in today.

Mila Zhou (28:23)
Yes.

Yesenia (28:28)
All right, so moving over into the marketing security section of today’s talk, I know we kind of touched on it in different areas today. But why does clear conversation and clear communication matter as much as the technical work that’s within the open source space?

Mila Zhou (28:46)
Yeah, I think that’s super interesting and like a super interesting topic because in open source a lot of time our focus actually is on the code. Last year, my coworker and I, Lahari and I, we did a talk about how to attract or get more non-code contribution to a project. Because if you really think about how a open source project can grow, can become a successful community, there is a lot more work beyond code. There is documentation. There is marketing. There is community building and all those things. So if we really want to have a sustainable and secure open source project, we should focus on all the other elements. That’s not just code.

And marketing, clear messaging, communication, it’s critical for you to send your message out to let the world know who you are, what you are doing, and why they should care about it, and how can they be a part of it. Because really, open source, a core value of it is the community. If you cannot grow a successful community and enable the community to grow and to sustain itself, then probably the project will not last that long.

Yesenia (30:31)
Mmm hmm

Mila Zhou (30:32)
So then it comes back to marketing. In marketing, what’s a powerful message? It’s usually simple, short, brief, and cut to the core. When you think about Nike, you think about its slogan right away. So that’s why I definitely believe that as we are promoting the open source security work, we should pay attention on what’s the message we’re delivering and how we’re delivering it, how it can resonate with our audiences so that we can drive adoption, we can encourage funding, we can make sure that whatever we’re doing will continue to get the support.

Yesenia (31:19)
Yeah, like that. You know, it’s just, it’s interesting when I got into the open source space, it was a lot of like the technical piece. And I’m like, my technical brain’s tired from work. Like what else can I do? which is why one of the reasons I co-lead one of the BEAR working group, which is, you know, that, how do we get the word out? How do we get people in? How do we get them to stay? And a lot of it has been just marketing from public speaking engagements, getting people to post on social, that’s not just OpenSSF, you know, and really advocate for the community. It’s one of the bigger things that I drive folks towards. I’m like, you can just start with that. Just start, get on LinkedIn and talk about the projects. And hey, this week I learned about Rust and Rust is doing XYZ and I think it’s pretty cool. And you can join us on this call. It’s free, it’s public.

Mila Zhou (32:01)
Yes!

Yesenia (32:15)
So I really, I dived in to that side, even though my background’s all engineering and cybersecurity. I was like, I can go out in public stage and talk about the great work that other people are doing. Sure. You know, I’ll advocate. And then, you know, as you get in and you just get comfortable, then you find your project and dive into it further. But I think it’s great to know that the marketing piece within the open source has been growing because of folks like yourselves and others really bringing that advocacy and voice to the maintainers that are already tired. And the last thing I’m sure they want to do is like talk about their work in a marketing standpoint. Like we’ll leave you to the technical pieces.

Mila Zhou (32:54)
Yeah, definitely. That’s why we come together, to work together. You’re not alone.

Yesenia (33:01)
Yes, you’re not alone. Let me be the face and help kind of just drive people towards the great work that you’re doing.

And with that, OpenSSF recently launched an ambassador program. So for you, I would like for you to share your general thoughts about how individuals, contributors can shift a security culture with something like the ambassador program, which is somebody that’s really putting time into these projects and whatever kind of contributions that they’re doing.

Mila Zhou (33:31)
This ambassador program, I think that actually echoes what we just talked about. We need people to get on the stage to share the work. if you look at the criteria for you to apply for the ambassador program, we are looking for people who have been contributing, participating in the work group.

or who have the experience to public speaking, share the word, or organizing events like meetup, community management. And all those actually are critical criteria for an open source project to be adopted. We talked that in open source security, a huge part really is adoption.

a lot of challenges, attacks we got, it’s not too fancy. It’s just like, because you have a weakest link and the weakest link actually is kind of easy to break because you didn’t do your security hygiene because like all those basic tools available there, you’re not using them. So I think the ambassador program, like from personal perspective, like…

they will provide you the cohort that where you can have a more systematic understanding of best practices, security positions. And at the same time, like you can be the one who share the word to amplify the impact of the tools of the knowledge and help one more developers, maintainers to improve their packages, their software.

I remember last time, GitHub has a open source security, cohort. And then like they did a video record with the log4j contributors. And he talked about like when log4j come, actually they didn’t, they were not, they were not security experts. So they, they did patches after patches to fix things.

Mila Zhou (40:25.534)
because they were not expert. They don’t know the best way to solve the problem. They were just trying their best. Well, they also found the security cohort was super helpful. So that tells us there is a gap between developers and the security experts. And that ambassador program can help us.

Yesenia (40:37.997)
Mm-hmm.

Mila Zhou (40:54.506)
hopefully can help us to really close the gap or at least make the gap smaller.

Yesenia (41:34.349)
All right, well, thank you for that. Let’s move over to the rapid fire part of the interview. I’m gonna ask you serious questions. First thing that kind of pops into your mind and we’ll take it from there. What’s your favorite off of the computer activity?

Mila Zhou (41:56.383)
Reading.

Yesenia (41:58.563)
What books?

Mila Zhou (42:00.52)
I’m reading the dungeon Cruel or Carl? It’s a sci-fi,

Yesenia (42:07.244)
Nice, nice. I’ll put that one on my to read list. Sweet or sour?

Mila Zhou (42:14.859)
Sour.

Yesenia (42:17.15)
Vim or Mac or Emacs.

Mila Zhou (42:19.635)
Emacs.

Yesenia (42:21.528)
favorite open source mascot?

Mila Zhou (42:24.466)
PG elephant, “Slonik”

Yesenia (42:28.11)
Favorite nighttime treat.

Mila Zhou (42:36.126)
Fruit!

Yesenia (42:39.15)
And then best way to grow a project, social media conferences or contributors.

Mila Zhou (42:44.874)
conferences.

Yesenia (42:47.864)
There you have it folks. That is our rapid fire section. Mila, any last minute advice or thoughts for our audience?

Mila Zhou (42:58.058)
I definitely encourage everyone to be a part of open source community like find something that’s Find a project that’s like interesting or find a group of people that you like just be together with them because like Open source is so

Like we use it everywhere, but we really don’t know that we’re using that. But like it takes, it takes people to care about the thing that we’re using and make sure that it can continuously to exist. And then participating in it, to invest in it is the best way for us to…

Mila Zhou (43:44.714)
to be a part of it and to ensure that what we like will continue to exist.

Yesenia (43:50.936)
Yeah, it’s definitely needed. It’s across so many things I’ve used. Mila, thank you so much for joining us today, for all the knowledge that you shared and all the contributions that you do to the open source community. It’s greatly appreciated, seen, and admired. So thank you so much to our listeners, and we’ll catch you on the next episode.

Mila Zhou (43:55.059)
Yeah.

Mila Zhou (44:13.522)
Yeah, talk to you soon. Thank you for having me. Bye bye.

What’s in the SOSS? Podcast #64 – S3E16 The Heartbeat of the Kernel: Why Upstream is the Ultimate Security Strategy with Greg Kroah-Hartman

By Podcast

Summary

What does it feel like to wake up and realize your weekend passion project is now the critical infrastructure powering the planet? In this episode of What’s in the SOSS?, CRob sits down with Linux kernel maintainer and open source icon Greg Kroah-Hartman. Greg takes us on a journey from his early days writing firmware for printer and hospital ATMs to managing the relentless, everyday engineering task of maintaining the Linux kernel over decades. He breaks down the realities of modern kernel security, dismantles the myth that maintainers know every single vulnerability exploit , and details how looming global regulations like the EU’s Cyber Resilience Act (CRA) are shifting the compliance burden onto vendors. If your organization uses Linux, Greg has a simple, urgent message for you: stop fearing change, build your testing infrastructure, and update your systems.

Conversation Highlights

00:03 Welcome: Host CRob introduces Linux kernel legend Greg Kroah-Hartman.
01:04 From Printers to Richard Stallman: Greg shares his origin story in embedded engineering and his introduction to free software.
02:00 The Weekend Driver and the Dopamine Hit: How a quick weekend project turned Greg into a lifelong kernel contributor.
04:20 Realizing Linux is Critical Infrastructure: The moment the telcos and banks moved in, signaling that Linux was everywhere.
05:28 2005: The Year the Kernel Grew Up: Implementing stable releases, the security team, and the rule to never break user space.
07:05 What People Get Wrong About Kernel Security: Linus’s mantra that “a bug is a bug,” and the reality of handling 35 fixes a day.
09:32 The Historic Fear of Updating: Why lagging behind for “stability” is actually incurring massive risk.
12:17 Global Regulation and the Cyber Resilience Act (CRA): How upcoming laws are changing vendor responsibility and why the open source community is exempt.
13:51 How OpenSSF is Reducing the Burden: Applauding the cross-ecosystem collaboration that protected maintainers from onerous drafts.
17:05 Pay Your Employees to Contribute: Greg’s best piece of advice for downstream enterprises relying on open source.
18:39 Inside the Kernel Security Alias: How the ad-hoc security team handles triage and assigns CVEs.
21:11 The CVE Numbers Game: Why the kernel ranks #2 in CVE creation and the trouble with automated severity scores.
25:39 We are Already There: AI Slop and Static Analysis: Greg addresses the recent surge of AI-generated bug reports and patches.
31:20 Rapid Fire Round: Spicy food, coffee, Star Wars, and the ultimate call to action.

Transcript

CRob (00:23)
Welcome, welcome, welcome to What’s in the SOSS, the OpenSSF podcast where I talk to maintainers Contributors security researchers and just anyone in and around the amazing world of open source and open source software security Today we have an amazing treat One of my open source idols somebody that I get the opportunity to work with pretty frequently now I’m quite excited today. We have Greg Kroah-Hartman one of the kernel maintainers with us today. So Greg, welcome to the show.

Greg Kroah-Hartman (00:55)
Hey, thanks for having me.

CRob (00:58)
Yeah, it’s amazing that we get to interact with folks such as yourself. This is a real treat for us.

Greg Kroah-Hartman (01:04)
I’ve been interacting with you quite a bit over the years.

CRob (01:07)
I know. So thinking about it, how did you get into open source? Back in days of yore when you were, I’m assuming you started off as a software engineer of some type. How did you get into open source proper in the kernel?

Greg Kroah-Hartman (01:23)
Yeah, I was embedded. I did embedded work. So I wrote firmware for printers and barcode scanners and ATMs that dispense drugs in hospitals, fun things like that. And we had to use Linux for some of our devices, some of the companies I worked for. But even earlier than that, when I was in university, my girlfriend at the time, who’s now my wife, had this class that would have a visiting speaker every class.

Greg Kroah-Hartman (01:50)
And she was a political science major. was computer science. And she came back. was like, I heard this guy came and spoke about something called free software. His name was Richard Stallman. This was the early 90s. And I was like, oh, that sounds very strange. And then I ran into the GCC developers at a conference.

Greg Kroah-Hartman (02:10)
And GCC was putting out a compiler for embedded systems. That was free. And I was like, how in the world is this business model working? This is crazy. I told them that, and they’re like, well, we’re selling it pretty well. And then I actually used it on some of my products that I had worked on. And I’m like, this compiler is better than the stuff we pay a lot of money for. So free software turned out to be working pretty well. But then years forward, I worked on the USB standard. And I was writing firmware for USB devices.

I had a barcode scanner that showed up as a keyboard. And my Linux had a little bit of support for USB, not much. So started testing my firmware and all sorts of different operating systems to try and make sure it worked. Under Windows, it worked fine because it had USB support. Linux, it showed up as 102 key mouse. I was like, what? It turns out that was my fault in the firmware because the kernel was being very strict on what accepted Windows would be very loose on what they accepted. So I was like, I’ve fixed some bugs on the stuff and then it was fun. I got the driver book and played around with it some more and then played with it on the weekends. One day my wife was going away with my daughter at the time to visit some family and she said, write that driver for that device you’ve always wanted to do. I’m like, okay. So I spent the weekend, a driver, submitted it. And then I instantly got back responses like, this is wrong, this is wrong, this is wrong. And if you heard of this thing called multiprocessors, I was like, what?

Greg Kroah-Hartman (03:34)
What do mean locking? Two processors at once? This is crazy. But it was great. It was like a dopamine hit of like, these are really good engineers telling me what to do and how to make my code better. And I was hooked at that. And then over the, I started writing drivers and eventually realized that more people were using the stuff I gave away for free on the weekends than were getting, using the product of the companies I worked for. Not that they weren’t that successful, but they weren’t that good.

CRob (03:45)
Yeah. That’s awesome.

Greg Kroah-Hartman (04:00)
Soon then I got a job in the early 90s, or late 90s, doing Linux full time, working on the security model for Linux. Funnily enough, I was one of the architects of that. And I’ve been paid ever since for that.

CRob (04:14)
Great. So you’ve been working on the kernel for how many years now?

Greg Kroah-Hartman (04:21)
I think 25, 24, that’s long time. Yeah, long time.

CRob (04:22)
20, yeah. So kind of reflecting back over that long time within this ecosystem, was there a specific moment where you kind of realized that, holy crap, what I’m doing is critical infrastructure for the planet?

Greg Kroah-Hartman (04:40)
Way too early. In the late 90s, the telcos adopted Linux first. They’re seen as like the stodgy companies, but they were the leading edge and they adopted Linux and they had this big telco initiatives through what was called OSDL at the time. And we’re butting heads with the telco people and we’re like, what are you using Linux for? They’re like, oh, it’s running our digital infrastructure. We’re like, what? So they were one of the first users and really showed that it could really be used for backbones and networking and stuff like that. And I was like, oh, wow. And then you hear things like, banks for throwing out all their old sun systems and using it and then the stock exchange and then air traffic control but uniquely it showed up like in a programmable like thermostat in my mother’s house once and I was like this is weird so then I realized yeah okay we’re really everywhere this is gonna stick around for a

CRob (05:36)
Yeah.

CRob (05:41)
Did that change how you approach security decisions as you were doing your work?

Greg Kroah-Hartman (05:48)
Yes and no. mean, we took, it made us take things more seriously. So once the banks and other people got involved, we we started taking things a lot more seriously. Like 2005 is like the year we grew up. We got instituted a security team. We instituted the stable way we did releases. We instituted the rule, maybe the year before that, we will never break user space again. We instituted time-based releases before that so people could rely on us. So we kind of grew up between 2003 to 2005 in that way. there’s a big tipping point at that point in time that we realized we got to do this right. People are relying on us and let’s come up with an engineering process that people can rely on. And that’s why I like Linux, because it’s one thing to write an operating system once or software once. It’s another thing to maintain it over decades so that people can rely on it and it will survive and grow and evolve over time because the world evolves. And that’s a fun engineering project and a fun engineering task. And I don’t think people… give us enough credit for Linux being something like that. It’s pretty unique as far as software systems go.

CRob (06:52)
Yeah, I would agree. So thinking about this kind of sense of responsibility that so many derpy things like thermostats and whatnot depend on you, but so many critical things are dependent on the Linux and the kernel. And securing things at this scale is very different than just securing a web server or a point of sale system.

From your perspective, what do most people get wrong about kernel security?

Greg Kroah-Hartman (07:22)
Two things to get wrong. They got wrong. One that they think we know what all the security bugs are. It turns out a bug in our area is a security bug. So Linus’s old mantra of a bug is a bug is a bug, let’s fix it and move on is more true than ever. I don’t, so open source, you listen to a few different things.

Open source is unique in that we can’t dictate use, or used in satellites to cow milking machines, right? Some are more critical to others, but those cows would be pretty upset if you got that wrong in there too. bugs at our level should be fixed and pushed out to users and they should update. The trick is I don’t know if this bug is applicable to you or not.

Greg Kroah-Hartman (08:09)
And that’s the thing. You use this, I’m not forcing you to use Linux, so you should just take our bug fixes and move on. do that. I remember sitting in a meeting once with some big company, they’re like, insisting, well, you need to tell us everything. All the bugs that you fixed that are applicable to us. I’m like, I don’t know what you do. I know you must take them all. You must take them all, or you must tell us. I’m like, I’m giving them to you for free. Somebody else corrected, like, the community is giving you what? 30 bug fixes a day for free and you want them to do more work for you? It’s like, this is a very odd thing to be demanding. That’s the way it is. So take our bug fixes because we know they fix something. And if they don’t fix something or if they break something, because people are always worried about regressions, we’ll fix that. But it’s much better to take a fix for a known issue than worry about a potential for an unknown issue. That’s the thing that people need to get the whole bitch to take.

And also look and test. mean, we are 40 million lines of code, but you only run a very small portion of it.

Your laptop runs two to maybe three million. Your server runs two million, maybe one and a half servers are the simplest, dumbest things ever. Your phone runs five and a half to six million. They’re the most complex beasts out there. So phones have tons and tons of issues, or tons of crazy hardware and stuff like that. So just take all our fixes and move on. And the phone companies, Android’s proven that you can update to every single RC release for the past four or five years on Linus’s tree and have a stable phone.

CRob (09:23)
Eesh.

Greg Kroah-Hartman (09:24)
So you can update these things and you can keep them up to date and you have to rely on that and luckily people are finally realizing that they have to update their systems and take the bug fixes and move on.

CRob (09:51)
And where do you think some of the hesitation is from being closer to head that organizations have? it kind of historically there wasn’t a lot of cross testing or?

Greg Kroah-Hartman (10:01)
Well, the old historic model is change is bad, right? And if it works for me now, great, I’ll just stick with it. But the world changes, and the world changes their ideas of their threat models and how bugs are found and things like that. Once a bug is found and is public, could be trivial to reproduce. But the trick is if it’s not known, then that’s another issue.

So just because you aren’t changing doesn’t mean the world isn’t changing. So you have to update and go on. So it’s an old model of that way. And also people will work. So you should have a testing infrastructure in place to accept this.

Famously, everyone’s like, we’ll just take tiny fixes. Well, Dijkstra famously said a one bit change could radically alter a program. Software is unique in that. It is not proportional to the size of the change, it’s to the size of the result. So very tiny changes can have major, major impacts. Giant changes can have minor impacts. So it all depends on your use. So you better have a pipeline in order to this stuff. Also, people used to have huge kernel forks out of the tree and they weren’t working upstream. Embedded devices were famously like that. Some companies made that decision. It’s a business decision. It costs money to keep code out of the kernel, which is fine. They had money to waste. Now they’ve realized it and they’re working upstream. IBM and Intel publicly said it saves us time and money to work upstream. Why not do that? So it takes time for companies to move their code forward over time.

Pixel, that Pixel device I mentioned, it still has 300 out of tree drivers, which is crazy how many drivers the phone has. And it still was able to update to each one along the way. Engineers adapted the APIs and kept it up to date. It worked pretty well. So you can do things out of tree.

And you can work that way, but you just have to have the testing infrastructure in place. And I’ll point at Android. I mean, it’s a few billion, many billions of devices, but they have a testing infrastructure in place that they can update, take the latest releases, take the latest state releases, run it through their testing system. I got a report back pretty fast, whether everything’s good or bad, and they push out to devices over time. They stage them and roll them out and things like that. But you need that testing infrastructure in place. But you need that testing infrastructure in place no matter what. So create it. Build it for your systems and then you’re good. You’re good to go.

CRob (12:24)
Do you have any other advice for organizations that might be hesitant for staying current and, you know, afraid to update anything else beyond having kind of a very thorough testing plan and infrastructure?

Greg Kroah-Hartman (12:38)
If you’re afraid to update, then you better partition and firewall the heck out of those devices. I’m just by virtue of also it’s going to be against the law in many companies now not to update. So it’s nice that the governments are finally realizing the US government, EU government, Japan. It’s all a new initiative of China for the way they’re treating their software and automotive. This is actually really good. So they are going to require updates and that’s a good thing. And so you should be updating things over time.

CRob (13:05)
So basically lagging behind for stability is actually incurring more risk now, right?

Greg Kroah-Hartman (13:10)
it’s incurring more instability because you have known issues that your systems have. So that is a known problem and that means your system has a known vulnerability, right? That’s, mean, on your risk management scale, that’s very high, right? As far as I think the way risk management scales go. So be aware of that.

CRob (13:35)
So then you’ve touched on it. I won’t say the magic three letters, but we’ve seen a lot of increasing pressure globally on enterprises from regulators. So how do you see organizations like the OpenSSF or other open source foundations, how do you see them helping kind of reduce that compliance burden on the upstream maintainers?

Greg Kroah-Hartman (13:56)
I’ll say the magic word, three magic letters, CRA, Cyber Resilience Act from the EU, which I actually think is a really good thing. And I might be biased, I live here in Europe, I’m also on the expert committee to do this stuff.

Greg Kroah-Hartman (14:10)
It is mandating that vendors update their software and report vulnerabilities and report bugs upstream to open source developers and maintainers. And that’s a good thing overall. There’s also other things with SBOMs and all, whole life cycle and risk management, which manufacturers will have some additional work, but it’s good. It’s just a list of ingredients, right? Like you have to do for food or for electrical devices. This is nothing out of the ordinary that good software engineering firms shouldn’t be doing.

It’s going to be interesting to see. But the openness to self, will call out. You guys have done a very good job in handling the paperwork, a lot of documentation you have, like here’s a one-pager on what an open source steward means and how to handle it. There’s really just two things you got to do. It’s pretty simple. Who is an open source steward? There’s a flowchart. So you guys have documentation. I’ve been giving talks for the past year and a half about this stuff.

So there’s tons and tons of resources. I think you’re doing stuff for actually manufacturers too, aren’t you? Okay.

CRob (15:07)
Yep, the manufacturers of we’re in full swing there. So we should have some guidance out for those types of folks who are the consumers of a lot of the work of all of you upstream. So hopefully we can coach them to treat and interact with upstream maintainers in a much more professional manner.

Greg Kroah-Hartman (15:28)
Yeah, and I’ve worked with a lot of companies over the years. That’s a lot of work I do. And some companies are great at this, right? And some companies are not. So it’s a learning experience on their half. And I want to see that happen. These companies rely on open source and they rely on the fact that open source works. So why not become part of the community to ensure it continues to work for them? And that’s my biggest pitch to them is like, great, if you want to use Linux as is or other open source projects, just take, and that’s wonderful. But if you want to make sure that it will continue to work properly for you in the future, become part of the community and work with the guidance and work with the developers to ensure that your use case still is valid. Because all we are are tools for you to do your business, right? And so you have your value added, whatever you’re building or making. So work with us to make sure that still works properly. And open SSF again, yeah, you’re doing stuff for manufacturers. That’s good. Open source developer. Open source developers and CRA and laws do not apply to it all. So I was happy and happy. don’t have to worry about that.

But a lot of the work that OpenSSF did with the CRA was to ensure that that was going to be the case because the way the original laws were being drafted were very onerous when it came to actually developing it. And I will call out there’s a really a lot of good job work and jobs you guys have done for that to make sure it didn’t harm us.

CRob (16:42)
That was a really nice kind of cross ecosystem effort. I’m pretty proud of that, that a lot of the foundations and big projects came together to express the really negative consequences of the first draft of the law.

Greg Kroah-Hartman (16:53)
Yeah, yeah, and I’ll call out Apache has done an amazing job also with us continuing to do that. Eclipse has done some work as well. So we’re not alone there by any means, but all the open source stuff is.

CRob (17:07)
So from your perspective, kind of looking at all this looming regulation, what types of support would you try to suggest for downstream to support maintainers and projects in their communities?

Greg Kroah-Hartman (17:23)
Kind of support, well just send us patches that work. So, I mean, or just become developers, or just, or better yet, have your employees work on open source projects and company time. That’s the best thing to do.

CRob (17:28)
Patches welcome.

Greg Kroah-Hartman (17:37)
Maintainers need to be able to do their work of these open source projects on company time. The kernel has this document that shows how open source friendly is your company with different levels, like level zero is they’re not allowed to work on anything upstream to level five as they pay you to be a maintainer and work on whatever you want. There’s a little gradient type thing there. And that’s good. When I’ve worked at other companies, we had in our employee reviews of, we mandated that you had to work X percentage of time on upstream development. And that was good, and that was an instant payback. But if you’re not actually measuring that at the end of the year for their employee review, they’re not going to do that. that always comes, that’s the first thing to give.

Smart companies know that this is actually, it pays off in huge, huge benefits. I think there was actually a report by the LF recently about that. showed dollar amounts. Yeah, Frank did good job with that.

CRob (18:31)
Yep. Yeah, Frank Nagel did it. Yeah. I agree. So thinking about your day job is helping lead security for the Colonel. When there’s a vulnerability discovered from your perspective, what’s the hardest part of getting that addressed? The technical patch writing and testing or the coordination to getting that fixed once it’s ready available to the consumers?

Greg Kroah-Hartman (18:57)
So I’m one of the members of the security team. I won’t say we lead it, it’s an ad hoc group. But our kernel team is, it’s an alias. You send us a bug report. And if the people on that alias isn’t, we don’t know how to fix it, the kernel is huge. We will grab the right maintainer first thing and grab them in and join them to the emails for added.

CRob (19:20)
Mm-hmm.

Greg Kroah-Hartman (19:21)
Go from there. And ideally, the submitter has submitted a proposed patch. If they have a proposed patch, even if it’s wrong, don’t be afraid to send us something that you think might fix it. Because then that’s something you can work off of. I’d much rather have a broken patch, because it’s like, this is the area I think this might work with. Let’s go from there. If we drag you in enough times, the security team, we then gently ask you, do you wish to be part of this alias? So it’s not really a good thing to be part of this alias. So that kind of means your subsystem has been little bit problems over time. That being said, yeah, just work with that. So the majority of bugs we have are not a big deal. Like, oh, we expose some uninitialized stack memory here, or here’s a way to bypass random number or random address of stuff, which is, as a local user, that’s not a really security issue or et cetera. We have lots of minor issues reported all the time.

We work on the fix. We usually post the fix publicly and don’t declare that, this is a security fix, it’s just a bug fix, and away it goes. Because if you look on our mailing list, we’re fixing bugs that end up being vulnerability fixes because they can crash the machine. Every day we have 35 bug fixes a day we seem to average. That goes into the stable kernels, that goes into the lenin-systery first. And about one-third of those are deemed of CVE. So we average about 10 to 13 CVEs a day.

which seems like a high number. But so the small number that we get sent to the security list are not very much, not very big. Very rarely do we have anything that’s major. And even then we just work on fixing it and get pushups to the tree and go. And we don’t assign a CVE until things are in a stable kernel release. So we give people who follow the stable kernels a chance to be up to date and secure before things are publicized to the world, which is good.

And we don’t do any disclosure. It’s up to the submitter if they want to do some disclosure or they want to write a report or have some paper or whatever they do. We just fix bugs and move on. And that’s all we do.

CRob (21:29)
We had chatted earlier in an earlier call today. You folks lost a pretty impressive title that you had for several years. You want to talk about your rankings within the CNA ecosystem? I thought you were number one for a while. Okay, my mistake.

Greg Kroah-Hartman (21:40)
No, we’ve never been number one. We’ve always been number two. No, well, we’re the number two creators of CBEs. Number one is WordPress. But it’s WordPress and all the different plugins somehow. I don’t know how those people keep up to date with all that stuff.

And then if you look at GitHub, we alternate between us and GitHub, but GitHub is a conglomeration of all open source projects that don’t really have the CNA that’s assigned bugs to. So it’s from a single entity, we are kind of still the largest one. But we don’t declare severity to bugs for the most part. I’ll talk about that in a minute. So other companies, if you look at Microsoft and Apple, they only report severe vulnerabilities. They do not report low or minor or medium or high or things like

So if you look at just quantities, you can’t go off of quantities. And that’s an issue that a number of people have realized slowly over the years. it’d be nice if the CBE group would require all vulnerabilities to be released as a CBE.

CRob (22:42)
I would like that quite a lot.

Greg Kroah-Hartman (22:44)
Yeah, so the fact that we have a lot is unusual. But the scoring of CVEs is tough because I don’t know your use case, as I mentioned before. So I don’t know if this is an issue or not. And there’s a famous researcher that said, even if I know a vulnerability or a bug, determining if I can actually exploit that is almost impossible. just depends on your use case, depends on this, depends on how your system’s set up, depends on all these random different things which is both good and bad, right? It’s good in that these things are usually very hard to exploit, it’s bad in that we have no idea if they are exploitable or not. So you see some groups like NVD from NIST will go through and give a random number for a severity score and that random number for the next kernel was 5.5 for everything last year.

That number we was like, why did they pick that number? And that number is just below the threshold that triggers some automatic tools for which you must upgrade. This isn’t a good thing. We’ve asked NVD to stop this. We asked CISA to stop this. They said they would stop this, and they did that. And we are starting to roll out. We just actually scored some CVEs the other day and pushed out updates.

CRob (23:55)
yeah?

Greg Kroah-Hartman (23:58)
We picked some that were like, okay, if you do have this network type of connection, this is a bad bug. So we went in that, scored that. There’s a group of companies that got together, like, take, started looking at different use cases. So like the cloud server use case. And they were like, here’s some rules and let’s try and score them all this way. That kind of fell down and they kind of fell apart. And we’re trying to drag them back because it’s hard to add this metadata to a CVE of what is the use case and what is the score for that use case. CVE wants to have, what is it, supplier? They call it something. They want to add.

Something starts with an A. They want to add different people can do things to a record.

CRob (24:36)
The ADP, the authoritative data provider.

Greg Kroah-Hartman (24:39)
ADP. Yes. Right. They want to do that. But that’s great. But you have to be a CNA to do that. And that’s not going to really work well for larger projects. So we’re thinking with the kernel, we’ll take groups of people, like the cloud people, and have a way to add to our record that we publish. Because we publish everything in the Git repo that is an addition to the CVE record that information.

And then we might be able to push into something else like OSV or things like that where we can define the data format better. And then maybe get the embedded people to add their own record and we’ll just keep adding these different use cases as time goes by. And I think that’s the only really way forward we have to do that. It’s like, OK, for this use case, this category of people are claiming that it affects it this way. And sadly, LLMs can actually grade these things based on the use case.

It actually kind of works kind of well if you say, here’s my use case. Is this bug applicable to me or not if I run this code? And they come back with a pretty good score. And then there’s just a pattern matching. It’s a pattern matching of use case to code. And OK, yeah, here’s a problem. So people can some LLMs at it. And I think one reason the cloud group fell down is everybody wanted to use their own specific LLM model. They didn’t trust anybody else’s LLM model, which is.

We’ll just run it on my local machine and we’ll take this as one and go from there. Local models are actually working really well these days.

CRob (25:57)
Good to hear. Well, since you said the other magic word, thinking ahead, are we prepared for AI generated vulnerabilities at the scale of AI?

Greg Kroah-Hartman (26:11)
We are already there. So something happened about a month or two ago and we used to get AI slop and I’ve said this before and those are easy to notice. About two months ago, we started getting real reports that were written nicely, actually reported a bug and sometimes came with a patch. But it’s real. I don’t know what triggered it and I don’t know what seem to do things well and so something happened. So in talking to the other open source security people because we have some meetings at times and we see each other, we’ve all hit it. GitHub’s hit it for their number of things, Curl’s hit it. All the different open source projects have hit it, HAProxy, the kernel, everybody. Kernel, we can handle this because we’re a big group and we can partition our farm it out to different maintainers.

But other smaller projects, I am actually worried about it. But it’s kind of like when fuzzers hit us, right? Fuzzers hit us and fuzzers were like, okay, the world is on fire. Here’s a thousand bugs. We’re like, And then we just grind through them. And we also say, hey, fuzzing companies, once you made this tool, you provide the gun, once you provide the resources to hold the gun properly. And to be fair, Google did. Google provided the resources to triage the bugs and say, okay, there’s serious, and they still do this. They run the fuzzing tools and they find the severe bugs first.

They fix them, send out the fixes before they actually let the rest of the fuzzing tools, reports go public. And the other reporting tools from the public fuzzing results are we have what, 400 open bugs right now that interns are really good to work on because they’re tiny bugs.

Hard to hit, and so let’s let the interns learn things, and that’s a good learning community.

CRob (27:48)
Yeah, cut their teeth. Yeah.

Greg Kroah-Hartman (27:50)
Yeah, same thing with these AI steps. So the AI code reporting tools, I mean, I’ve run a bunch of these over the past couple of months, and some of them are good. They’re good static analysis, because it’s pattern matching, right? And that’s what we actually have been using in the kernel for over a decade. Famously, for the stable kernels, we knew that when we went to backport patches, developers would say, this is a bug fix, and we backported.

Well, Julia Lowell, a professor in Paris, said, hey, why don’t we compare all the patches that we humans say are good with all the patches that we don’t know if we’re good or not and see if they match? Because LLMs are just a fuzzy statistical matching model. And it turned out to work really well. And she’s written a bunch of papers. And her and Lowell or Sasha, has come up with some great tools. they backport stuff. And they do some wonderful things. And we’ve been using this over a decade.

So this pattern matching of is this a bug and is this not a bug is there and that’s what Anthropic published like not so long ago they said we found 500 bugs and there basically was like we looked at all these old CVEs and saw where they weren’t fixed in the current two bases – code bases.

And I’ve run those same types of scripts. And it’s like, hey, this actually matches pretty well. Like, OK, it catches some things. And it out we forgot to cut a bunch of things. I was like, I thought we had tests for that, and we forgot to run our tests. So even as kernel developers, our infrastructure isn’t always working properly. So we have to go back and turn on a few tests that we had stopped running. Which is good. It cut it. So it’s just going to be another layer. It’s another static code that will grind through and fix up all the bugs and keep on moving. But it’s going to affect a lot of people for a while because fuzzers were kind of hard to run and set up.

These AI tools are a little bit easier for other people to hit and run. But the fun thing is some of these AI tools can spit out patches. We’ve taken security bugs sent to us and I run it through a local model and here’s a dumb, stupid patch to fix this problem. And it kind of works about two thirds at a time. But it gives you that base to go off of. And then there’s something to go for there. A few of those patches have actually been merged. So because they actually do fix the bug, which is kind of sad and funny in the same way.

CRob (29:52)
Yeah, exactly.

Greg Kroah-Hartman (30:04)
Yeah, I mean the old adage of all input is evil. Sometimes we forget to check the input. these things actually catch that well.

CRob (30:08)
Well, and it’s interesting. I’m not worried about teams like you or Apache Kubernetes, the folks that have a large group of security experts. It’s the single maintainer projects that might not have the expertise or the time or the tools. That does worry me quite a lot.

Greg Kroah-Hartman (30:26)
Yeah, it does work for me too. And they could get overwhelmed. But ideally, can also, I tell these people, just push back and say, come with a patch. And these tools can make a patch. And it does work that. But a lot of those tools and infrastructure aren’t security like.

facing, right? So say my LSUSB, which is like, it scans the number, the USB devices and prints out a little graph of what devices you have. People will try to report security bugs in that before. I’m like, yeah, that’s wonderful. It’ll crash. But that’s not a problem. So you physically told it to run as a user. It’s going to crash. OK. And if you look at the way Rust does memory safety, if it encounters problems in its code, Rust, by default, will just crash because that’s safe. It’s causing a safe way. Now using Rust in the kernel, you don’t want the crash when you do other things in there. But memory safetiness and crashing is kind of a very common thing when security issues come up, you just want to panic and go away. And it’s very good for users to do that. Because normally somebody can notice it, kick the box or kick the node and away you go, you start it up again.

CRob (31:37)
Well, this has been, I think, very illuminating, but let’s move on to the rapid fire part of the conversation. I have a couple of crazy questions. Just give me the first thing that comes off the top of your head. Mild or spicy food?

Greg Kroah-Hartman (31:57)
Spicy.

CRob (32:02)
So what’s one security myth you’d like to eliminate?

Greg Kroah-Hartman (32:06)
One security myth that I know if a bug is a security issue for you or not.

CRob (31:)
There you go. Very nice. What’s one thing that every company using Linux should do tomorrow?

Greg Kroah-Hartman (32:02.376)
Update to the latest version.

CRob (32:21)
When you’re doing your patch releases, what’s your favorite beverage of choice?

Greg Kroah-Hartman (32:27)
Coffee

CRob (32:29)
Yummy! And finally and most importantly, Star Trek or Star Wars?

Greg Kroah-Hartman (32:38)
Star Wars, sorry.

CRob (32:40)
There are no wrong answers. But Star Wars is very excellent. I appreciate that. So Greg, as we wind down, do you have a particular call to action you want to share with our audience outside of the amazing little snippets of advice you’ve given us during our time here?

Greg Kroah-Hartman (32:54)
Report bugs and report bugs to maintainers and developers. If something isn’t working, tell us because otherwise we don’t know. Linux has been working for me for 20 years, but obviously it’s been working for everybody. tell me and do that.

But also on the flip side I only see bugs, which is funny. I’m amazed that this stuff actually works. So my daughter, who is now a doctor, she’s like, it’s just like a doctor. Doctors only see problems. How does the human body ever work? The majority of the time, the software does work, and it is all good.

But if you do have a problem, please let me know. I’m here to fix it and help us. We want to know how to fix it. And tell us how you’re using it. We’re kind of curious sometimes. I know, like the cow milking machine, satellites, the Mars Rover, fun things like that. It’s kind of interesting to see where our code is at. It’s fun.

CRob (33:46)
Excellent. So amazing advice, incredible storied career, sir. Thank you for your time. And with that, we’re going to call this a wrap. I want everyone to stay Stiber safe and sound and have a great day and happy open sourcing.

What’s in the SOSS? Podcast #63 – S3E15 Big Thoughts, Open Sources: Driving Enterprise Security and Career Growth Through Open Source with Jamie Thomas (IBM)

By Podcast

Summary

In this episode of Big Thoughts, Open Sources, host CRob sits down with Jamie Thomas, IBM Enterprise Security Executive and OpenSSF Governing Board Member (former Chair!), to tackle the vital shifting dynamics of enterprise open source engagement. From IBM’s historical “billion-dollar bet” on Linux to modern supply chain wake-up calls like SolarWinds and Log4j, Jamie pulls back the curtain on what it truly means to move from accidental consumption to intentional stewardship. Tune in to discover how active participation in neutral foundations like the OpenSSF acts as a fast track for engineering career trajectories, why soft skills like “the art of influence” are critical for upstream collaboration, and how organizations can protect their crown jewels while implementing a powerful “give-back strategy.”

Conversation Highlights

00:00 – Intro Music + Promo Clip
00:21 – Introduction & Welcoming Luminary Jamie Thomas
01:32 – Wearing the Enterprise Security Hat at IBM
02:10 – Supply Chain Wake-up Calls: From SolarWinds to Log4j
03:14 – Unlocking Open Ecosystems: IBM’s Early History with Java and Linux
05:21 – Mainframe Debates and Portability: The Evolution of Open Source Adoption
06:24 – The Red Hat Acquisition and Monetizing the Developer Ecosystem
08:20 – The Myth of “Free” Software: Securing Regulated Enterprise Deployment
10:15 – Why a Seat at the Table Matters: The Value of Neutral Foundations
11:29 – The Art of Influence: Upstream Contributions as a Career Catalyst
13:50 – Moving Innovation from Open Source Kernels to Commercial Value
16:12 – Storming, Norming, and Conversation: Lessons from the Kubernetes Era
17:38 – Pitching Upstream Time: Helping Developers Sell Open Source to Management
19:30 – Beyond Code: Bringing Domain Expertise and Soft Skills Upstream
21:40 – Conquering the Chasm: Automating CI/CD Pipelines and Testing at Scale
22:56 – Consuming with Intent: Active Stewardship and the OpenSSF Scorecard
25:21 – Rapid Fire Round: Mainframes, AI-Generated Code, and Star Trek nostalgia
27:53 – Call to Action: Crafting Your Organization’s “Give-Back Strategy”

Transcript

Music & Intro/Promo clip (00:00)
If you’re a direct consumer of open source, do it with intent. And intent means that you’re responsible for what that means to your organization, both from a productivity perspective, but also from a security perspective and what you can expect from an investment point as well. You have to be an active steward in the maintenance of your open source strategy, just like you would have to be a steward of the maintenance of your own home.

CRob (00:26)
Welcome, welcome, welcome to Big Thoughts Open Sources. My name’s CRob. I’m your host today. This is a special video series where we’re talking to some of the leaders within open source. And today we have an amazing treat. We have Jamie Thomas from IBM. It’s a little company you might’ve heard of. And she’s here today to kind of talk about maintaining a career and influence within open source. Welcome, Jamie.

Jamie Thomas (00:54)
Thanks, Crob. Thanks for having me.

CRob (00:56)
I think we’re going to have a pretty amazing conversation today. For those of you that might be unfamiliar with kind of the luminaries that exist within our little slice of open source security heaven, Jamie has been alongtime contributor and member to the OpenSSF, predominantly through influence through our Governing Board. You actually were our Governing Board Chair for a period of time. So for those that may be unfamiliar with you, could you maybe talk about what your current role is at IBM and kind of what you do within the OpenSSF?

Jamie Thomas (01:32)
Absolutely, and thanks for having me again. I think this is a really interesting topic. At IBM, I have a couple of hats, but the main hat that is important to this discussion today is IBM Enterprise Security Executive. And I’m therefore responsible for the protection of the IBM company. It includes our CISO office, our Cybersecurity Operations, as well as our Product Security, which has to span a number of our units, including our software, hardware, our consulting unit. So it’s an interesting job.

CRob (02:05)
Really big job.

Jamie Thomas (02:10)
Yeah, and you know, as you know, CRob, I got involved in OpenSSF from the very beginning of the Governing Board. And I remember when I got involved in enterprise security, someone told me, by the way, it’s all quiet, there’s not much going on right now. And, and the first thing that I recall happening not long after that was SolarWinds, which was one of the first interesting supply chain attacks. And then of course, we had Log4j, which is another whole realm of fun.

But certainly I felt that joining OpenSSF has been important for us, the IBM company, to stay abreast of what’s happening in the industry and to be a participant in this open source security realm. It continues to be an ever changing landscape.

CRob (02:53)
Absolutely, a blink and it totally changes. It’s a pretty wild space we get to live in. Thinking back through your career here, was there a moment or something that really sparked your interest that kind of drew you towards more open source perspectives, participation?

Jamie Thomas (03:14)
Yeah, yeah, absolutely. I mean, when I started IBM, I did start as a programmer in the software organization. And at that time, of course, we worked on, like all of the industry, probably at that time, closed source systems. But at some point in IBM, we got very involved in something called Java. As part of…

CRob (03:32)
I’ve heard of it.

Jamie Thomas (03:34)
Haha, yeah you’ve heard of Java…As part of Java, we wanted to engender an open ecosystem around the Java programming model.
And to that end, we realize we’re not going to attract developers at scale without a different approach.

And so we made some concerted decisions at that time. One was that we would invest in Linux. There was something called Linux at that time. And we made a decision to put into the corporation Linux Technology Center to invest in Linux, be open source contributors to the Linux operating system. And the other big decision we made is that we would outsource some of our Java development tooling to something called the Eclipse Foundation.

CRob (04:16.691)
Mm-hmm, oh yeah.

Jamie Thomas (04:17)
So it seems like, you know, tribal knowledge at this point, but to say the least, embarking on some of these endeavors was quite unique for the IBM company at the time. And getting people to understand their role within an open source community as opposed to what we had always traditionally done in terms of how we provided software to clients was very, different.

And I had the pleasure of being involved in both the Java effort and the Eclipse Foundation effort through our Rational Software Division. And at that time, I actually owned all of the folks that were contributing to Eclipse and got to see that evolve for a period of time. You know, that was very, very fascinating times for IBM and also, I think, important for the evolution of open source that we have today.

CRob (05:09)
Well, in thinking further downstream, the folks, the organizations that would be your customers, Java was probably one of their, also their entry points into the open source ecosystem.

Jamie Thomas (05:21)
Mm-hmm. It absolutely was because you found through the eyes of lot of our enterprise clients as they adopted Java, they were naturally then adopting Eclipse. And over a period of time, they certainly became very strong supporters of the Linux operating system. And in fact, I remember one of the first meetings we had where we actually were in a room with all these clients and there became this huge argument, this huge argument about whether we should put on the IBM mainframe.

And you can imagine, there were a variety of opinions in that room. But what made sense, if you thought about it, is that Linux would give us portability. And if folks developed on Linux, of course, then ultimately portability to different hardware platforms, of course, would be gained through Linux. And that’s exactly what did happen.

But at that time it was a very, very novel concept that we should embrace this new operating system. And so it was fun, fun stories.

CRob (06:24)
Speaking of fun stories, several years back, IBM made the air quotes, billion dollar bet on Linux and kind of looking back at kind of IBM’s initial involvement with the ecosystem and Java, and then through the acquisition of Red Hat, kind of thinking about what was the most difficult part kind of moving these massive enterprises towards an open source first mindset.

Jamie Thomas (06:53)
Well, I think that by the time we made that multi billion dollar bet on something called Red Hat, our perspective of open source had matured quite a bit. We had then already discovered and learned that we did.

CRob (07:02)
Mm-hmm.

Jamie Thomas (07:03)
We had then already discovered and learned that we did. monetize quite well the Java ecosystem and part and parcel to what we did around Eclipse and Linux. We’d also adopted scale Linux on our platforms on both the Z platform in particular and Power and also we were stewards of Linux on x86 for our clients. So we had a different perspective. And I think at the time we acquired Red Hat though, we did understand the needle had moved quite a bit on the developer ecosystem. In that period of time since we had first started some of these efforts and that Red Hat would enable us to have a bigger stake in the open source community as well as the ability to reach developers at scale.

And that was part of the motivation for the acquisition and I think that it definitely served us well. I believe that what Red Hat does, curated open source is something that’s really important for the industry to have a competent enterprise level open source provider, but that once again,has their finger in the pie around the open source community because that’s where everything happens. It happens in the community and then it has to bubble upstream and be effective for the enterprises that are consuming it.

CRob (08:20)
I know we’ve talked many times that you get the opportunity to talk to a lot of business and cyber leaders at firms that you partner with. Kind of when you’re providing advice and counsel to these leaders, how do you kind of reassure them that they’re not going to be contributing to open source isn’t going to be giving away their secret sauce or their crown jewels? How do you encourage them in engaging more and more effectively with open source?

Jamie Thomas (08:50)
Well, I think there’s two different perspectives. And when you talk when I talked to a lot of the client organizations, I think they have pretty much accepted that open source is a valuable asset to them. What they’re then looking for is who can help me ensure that I’ll have operational fidelity, security around this open source. And that’s when they then start to say, should I have a trusted vendor to work with on that point? Or am I going to consume it directly?

And, and certainly, Isee that early on in the acquisition of technology that many enterprises just take the open source and try it out. I mean, that’s one of the benefits, right? You can try it out. You can test drive it, you can decide if it works for you. But then normally, if you go into enterprise deployment, you do want someone that’s going to be there. In many cases I’m going to provide the care and feeding for you, particularly if you’re in a regulated industry. And of course, we deal with a lot of regulated firms and in our environment.

I also also make sure that I remind everybody involved, including ourselves and our ecosystem, that open source is really not totally free. I mean, it’s something that we all have an obligation to support. So if we’re consuming it, we need to understand what that means and we need to be stewards of ensuring that the open source communities are successful going forward.

CRob (10:15)
That’s an interesting point that leads me to my next question. How vital is it for enterprises to participate in neutral bodies like the OpenSSF or Eclipse or CNCF, rather than just trying to take all these amazing tools and kind of assemble it themselves, how does that neutral collaboration help them?

Jamie Thomas (10:36)
Well, I think it’s very important for you to, as an organization to have a seat at the table, if you will, realizing that in today’s world, you know, software doesn’t stay in the boundaries of one company. I mean, there’s an extensive ecosystem out there, we’re all somewhat interconnected. And if you’re at the table, you have an ability to influence the direction of a lot of these projects, you had the ability to contribute uniquely, but most importantly, to make sure that the projects understand your unique point of view. Certainly there’s a point of view of technology organizations like IBM. There’s a there’s a the the point of view of the startups out there that are so critical to the ecosystem. And there’s the point of view of the consuming enterprises, the downstream enterprise, I think all of those play a critical role in having that seat at the table.

CRob (11:29)
I absolutely agree. So as we continue our conversation here and getting more to, think, is one of your true areas of passion.

From your perspective, you’ve seen a lot of engineers in your career and you’ve seen them become kind of strong speakers and potentially global influencers through this open source ecosystem. From your perspective, do you see active participation in like open source projects as some type of a fast track towards leadership for engineers?

Jamie Thomas (12:04)
I think it’s one of those many facets that can really improve your career trajectory. The reason I think this is that for any accomplished computer science engineer today, you really have to have the ability to influence others. You have to have the ability to influence networks. Maybe that’s outside of your organization or inside of your organization. You have to have the ability to influence your customers.

And so how do you build that kit bag of resources and skills that allow you to do that? And I think participation in an open source community gives you a lot of industry perspective.

You know, when I go to the OpenSSF, I don’t just go there and understand, of course, what my point of view is, I’m understanding the collective point of view. And that collective point of view is very interesting and fascinating. Learning from others and then having that ability to share your unique perspective with clients and even in your internal organization is a skill that many people often underestimate.

The skill of influence is something that’s really important in today’s world. And I think that you can build that. You can also strengthen your own personal presentation skills.

I’ve seen many people presenting in a lot of these forums, whether they’re presenting at one of the conferences or presenting in a governing board or presenting at one of the breakouts. And it really is, once again, an opportunity for you to build your own personal presentation skills, understand how you can represent your ideas more effectively to another organization or set of organizations. And that is something I think is critical for a lot of individuals that are pursuing a different career trajectory.

CRob (13:50)
Mm-hmm. I really appreciate that insight. If thinking about this, how important is it for an engineer to have kind of this open source pedigree, whether it’s like a recognition, like your IBM Open Innovation Awards or other types of awards? How does participating in those things help that individual stand out, especially in this age where organizations are rapidly changing and pivoting?

Jamie Thomas (14:19)
Well, thanks for bringing that up IBM does have an open innovation award that we give to individuals every year who are standouts in the participation of open communities. And I think that’s really important. And we have both executive contributions as well as non-executive contributions and I always recognize those individuals in my all hands meetings to make sure they get their name in lights a bit internally. But I think that this is particularly important for individuals who once again want to take the learning from these kind of communities and use that to more broadly influence the direction of organizations. In many cases and I was just on one of these today in fact a lot of the innovation does start in an open source format because once again it’s very easy to get your ideas out there to start seeing them scale understand what really appeals to clients and what is not then how do you take the open source kernel and then create something that is of commercial value right.

You either have to have the concept like we did with Eclipse that it’s going to enable developers to use other products more effectively, or you have to take the kernel and then become an enterprise open class support organization like what Red Hat did eventually…

CRob (15:41)
Mm-hmm.

Jamie Thomas (15:42)
Very accomplished open source curation organization. So I think for individuals to help companies like IBM or to help their team to help their little asset that they created become more successful, then this participation is absolutely critical. And without a few individuals along the way that really understand how to do this, then I think many, many rewarding projects will not get from point A to point B and be as successful as they could have been.

CRob (16:12)
Right, Yeah, I definitely agree with that. So how…

Jamie Thomas (16:17)
I mean, look at things like probably, you know, Kubernetes or many of the things that we recognize today is just we take for granted without that kind of vested participation support would they have become what they are today? I mean, Linux is the best example, of course.

CRob (16:36)
I absolutely agree. Kubernetes is another example of where we had an organization that had an idea and there were some competing ideas at the same time. And then that project was donated to a neutral foundation where all these fierce competitors could get together and work together in that space to help make the technology itself far more impactful than it ever would have been.

Jamie Thomas (17:00)
Yes. And I hear I hear behind the scenes, even though I was not part of all of that directly, that there was a little bit of storming and norming and maybe a few disagreements along the way and everything.

CRob (17:10)
Yep.

Jamie Thomas (17:11)
But eventually, you know, some really good things came out of that. And, you know, it’s like anything when you nothing, nothing really good is achieved in my mind without conversation, you know, without really having a human dialogue to understand what’s working and what’s not. Certainly big things are not achieved typically through subterfuge and so I think that is the value of a lot of these communities.

CRob (17:38)
I agree. So when you’re thinking about, again, from the developer maintainer perspective, what’s advice you would give to give a developer to pitch to their manager about spending some part of their time working upstream, hopefully in security, but participating in an upstream project? How do you help them sell that to their management?

Jamie Thomas (18:01)
Well, I think in many cases, the individual needs to take a point of view of what’s in it for my manager, right? I mean, it’s kind like when you’re trying to sell anything, right? There are certainly clear attributes that help you as an individual, right? Can improve your leadership skill, your ability to help the manager from that perspective, I think is quite valuable. Problem solving is something we often take for granted today, but it’s not always there, right? But then the other thing is what what particular.

But will the participation aid from a business perspective? And I think there’s a lot of different aspects of that. If someone’s working in the security domain, I think there’s really clear outcomes.

where their participation will benefit for the enterprise you learn so much right in terms of what you need to do. If you’re working in a particular project that is going to have downstream impacts possibly to the organization I think there’s a clear linkage there. But first of all never take for granted that everyone does understand open source.

I find that even in today’s environment, there’s a lot of upline management or senior executives perhaps don’t understand the importance of open source and open source participation. It’s kind of like the water that’s running in our house and we just think it’s gonna always be there. It’s always gonna be on, right? So building that cell package is important. It’s important.

CRob (19:30)
So from, again, your perspective, outside of having some coding skills or some other technical ability, what other types of specific things should people think about when they’re going to go engage upstream? Are there other abilities or skills they might be able to bring to bear to help upstream?

Jamie Thomas (19:52)
Well, I think that coding skills are one but don’t underestimate just basic ability to influence basic ability to bring technical perspectives together and drive a conclusion. I mean, we’ve certainly seen a lot of that in the OpenSSF right where the technical committees have really had to come together and you’ve been a big part of that to render a recommended outcome right that’s influence and getting people to agreements really important back to your Kubernetes point. Somebody had – a group of people had to get some level of agreement to move forward. So influence is really important. Having domain knowledge of things like security, of secure CICD practices, sharing that with organizations I think is really, important. That’s one of the things you know that we’re really focused on is not how do we just create best practices and tools, but how do we help other projects consume those tools at a faster rate and pace.

So individuals that have that passion, who have the ability to help teams be more successful, those are maybe some soft skills that I think a lot of projects really need.

CRob (21:04)
And I think that that’s, hear this a lot from non-developer folks that are interested in contributing is, you what could I possibly bring to the table? I’m not a coder. I don’t understand C or Rust or Go. And I tell them, you have value, you have domain expertise, you understand networking or like CI systems far better than most software engineers do because software engineers are studying the language or a particular community. They don’t necessarily have a lot of these additional skills like program management and communication.

Jamie Thomas (21:40)
Yeah, and one of the things I remember the most about, you know, one of the IBM projects I worked on WebSphere is one of the most fundamental investments we made was automating the CI-CD pipeline in a really cool way. And of course, this was 20 years ago. But I’ll remember that point, like it was yesterday, because our lives change when we achieve this automation at scale. And so I think that that kind of thing can have a huge impact on the projects today. One of things we’ve been talking about as you know with all the AI mania that’s out there these days is one thing to find the defects but how do you fix the defects and actually test them at scale?

So we all know the secret of many software projects is while we have a lot of software out there, testing them in an automated fashion even is quite a challenge for many organizations and many projects. So those people that are able to conquer those leaps across the chasm help with that CI/CD automation, help infuse security, or help create a different perspective of how to automate the test associated with not only remediating but making sure that we’re not breaking everything that’s out there that uses that particular asset. Small thing.

CRob (22:58)
Yeah, very small. So again, let’s pivot back to your conversations with enterprises and leaders broadly. How do we help move the industry from thinking about open source is something that’s consumed to something that is more of your steward? Your participant in this shared infrastructure?

Jamie Thomas (23:21)
Well, I think we have to continue what we’re doing here in this session. Really, we have to continue to provide a lot of education and perspective about what happens when you don’t do that. Right. This is, you know, it’s like, you know, my house, I do wake up every day and I expect the plumbing, the water and electricity to run. But on the other hand, if I’m not a good steward of the basic underpinnings of some of that capability, it might not always be that way. And I think there’s the same can be said about open source.

So when you consume it, you have to consume it, I think, with a recognition that it’s a critical part of your infrastructure investment. You should understand what you’re consuming. should typically I don’t think it should be accidental consumption unless you’re depending on perhaps a packaged app vendor to provide it to you. And then that could be something that you’re unaware of, right? S bombs and things like that are particularly making that more apparent, of course.

But I think if you’re a direct consumer of open source, do it with intent. And intent means that you’re responsible for what that means to your organization, both from a productivity perspective, but also from a security perspective and what you can expect from an investment point as well. One of the things that we’ve learned through the OpenSSF is not every project maintains a healthy state over a period of time. And so that’s why we’ve created this scorecard that gives organizations a perspective of whether the project is healthy.

And I think today, if you’re consuming open source, you need to be using those kind of assets to understand, are you consuming healthy projects? If not, what do you want to do about that? Right? So it’s you have to be an active steward in the maintenance of your open source strategy, just like you would have to be a steward of the maintenance of your own home.

CRob (25:21)
I love that. participating with intent and having a strategy. That’s excellent advice.

Jamie Thomas (25:25)
Yeah, it shouldn’t be an accidental consumption approach. It should be with intent.

CRob (25:36)
Right. Well, let’s move on to the rapid fire part of our talk. I have a couple wacky questions. I would just like the first answer off the top of your head, please. Blue suits or blue jeans?

Jamie Thomas (25:53)
I think both. I have a blue jacket on today. What am I supposed to say?

CRob (25:55)
Both very nice. That was kind of a leading question. Next question, mainframe or microservice?

Jamie Thomas (26:10)
Oh, both. Oh, I’m cheating on this quiz. But you know, I do have the fondness for mainframes. Here’s my little mainframe chip, you know, the chip package here. It’s like a little paperweight, right? This one is z 16. I don’t have z17. But anyway, and then microservices, I think the world does depend on microservices. Very, very important part of the architecture.

CRob (26:33)
And I heard you can run open source microservices on a mainframe.

Jamie Thomas (26:37)
That’s right, that is true.

CRob (26:40)
AI generated or hand coded?

Jamie Thomas (26:44)
Well, I think in today’s environment AI generated is becoming quite prevalent, but I do believe that developers who are standouts are gonna be those individuals that go back in there and use it to their advantage. So I think there will be humans in the loop or humans somewhere in many cases, and it’s up to the human to decide how do I use this cool innovation to my personal advantage.

CRob (27:12)
That’s amazing. And then finally, most importantly, Star Trek or Star Wars?

Jamie Thomas (27:19)
Now I have to confess that I was a Star Trek person growing up, right? I watched those cool Star Trek episodes and everything so I guess I’m more partial to Star Trek.

CRob (27:34)
There are no wrong answers, both are good, but it’s just kind of fun. I love both. I grew up on Star Wars and then Star Trek and then Star Wars changed my life in 77.

Jamie Thomas (27:43)
Well, I do like Princess Leia, of course. Being a woman, Princess Leia was a hero for all of the young ladies out there. Quite good. Good stuff.

CRob (27:53)
Absolutely. Well, and so as we wind down, Jamie, thank you for playing along. Your call to action. So if we have an enterprise that’s only consuming open source today, what’s one thing that you would suggest to them to get them to start participating?

Jamie Thomas (28:12)
I would ask everybody to think about what is your give back strategy?

Because if you’re consuming open source, but you’re not participating in open source projects, maybe with your developers, you’re not participating in an organization in terms of lending your unique perspective of your strategy and how the foundation or the open source project could help you with your strategy. You could do that even through maybe just the use of open source and beta testing, but be an active participant.

Think about consuming with intent. And what does consuming with intent mean for you as an organization?

CRob (28:54)
I love it. Thank you, Jamie Thomas, for participating in Big Thoughts Open Sources. This was a delight. Thank you very much.

Jamie Thomas (29:06)
And thank you very much, CRob. It’s always good to talk with you.

CRob (29:09)
Absolutely – and to those of you listening today, please check out the transcript, we’re gonna have some really great links and you know, stay cyber safe and sound. Bye everybody

Jamie Thomas (29:20)
Bye.

What’s in the SOSS? Podcast #62 – S3E14 The Ghost in the Dependency Tree: Navigating Open Source End-of-Life with HeroDevs

By Podcast

Summary

In this episode of What’s in the SOSS, host CRob sits down with Isaac Wuest, Product Line Leader at HeroDevs, to explore the critical and often overlooked “gray area” of the software supply chain: End-of-Life (EOL) software. While the industry heavily relies on CVEs to track vulnerabilities, Isaac explains how maintainer abandonment creates a vacuum where risks are present but remain undiscovered and unreported. From the origins of HeroDevs supporting AngularJS to the nuances of the EU Cyber Resilience Act (CRA), this conversation provides a practical framework for distinguishing between inherent hazards and actual risk in your dependency tree.

Conversation Highlights

00:04 Host CRob welcomes Isaac Wuest from HeroDevs to discuss secure open source ecosystems.
00:45 The HeroDevs origin story: How Google sunsetting AngularJS created a need for secure drop-in replacements.
02:44 Isaac’s path to open source: Transitioning from product management to supporting maintainers.
04:06 Exploring the “Gap” in CVEs: Why dictionary-based vulnerability tracking misses EOL and malicious packages.
07:03 The challenge of “Maintainer Attestation”: Why most open source projects lack a formal EOL calendar.
09:52 Compliance and Risks: How EOL dependencies create blank spots for security professionals and auditors.
11:27 The Shark in the Tank: Using a food regulation analogy to differentiate between hazard and risk.
13:22 Navigating the EU Cyber Resilience Act: Preparing for increased manufacturer accountability in software.
14:08 Maintainer Abandonment: Identifying the moment a project stops receiving patches without formal notice.
16:14 Scanning for Gaps: Why standard industry tools currently struggle to provide a complete EOL picture.
18:49 Practical Remediation: Recommendations for researching upgrade paths using tools like endoflife.date.
20:49 Analyzing SBOMs: How engineers can leverage free datasets to identify and fix deep dependency risks.
23:00 Rapid Fire: Coffee, Star Wars, spicy food, and the favorite apocalyptic robot.
25:01 Final Thoughts: A call to action for educating yourself on your application’s EOL exposure.

Transcript

CRob (00:04.053)
Welcome, welcome, welcome to What’s in the SOSS, the OpenSSF’s podcast where I talk to people that are in, around, and creating this amazing open source ecosystem that we all benefit from. Today, we have a pretty interesting guest with some pretty cool and timely topic. We have Isaac from Hero Devs. Isaac, welcome to the show.

Isaac Wuest (they/them) (00:26.474)
Awesome, thanks for having me. I’m glad to be here.

CRob (00:29.197)
Yeah, so everybody that listens to our podcast may or may not know the HeroDev story. Could you maybe explain a little bit about your company that you’re representing and then maybe give us a little peek into your open source origin story? Like what got you into this space?

Isaac Wuest (they/them) (00:45.694)
Absolutely, absolutely. So the story of Herodevs actually goes back to the story of AngularJS and when Google decided to sunset support for Angular. So Herodevs as a company started a few folks that were on the Angular team, realized that Google was sunsetting support and that there would be breaking changes and people would need a secure version that was supported and being patched in the future.

CRob (00:52.256)
Mmm.

Isaac Wuest (they/them) (01:11.754)
And that was the birth of HeroDevs. HeroDevs as a company has expanded out of AngularJS to many other types of large open source frameworks where we offer secure drop-in replacements for the end-of-life versions. And that was kind of the inception of the company. What we do today is keep those versions secure. That being said, what we’re now doing and kind of moving more into is that end-of-life space more broadly, saying how can we support not just individual frameworks,

But when things go end of life, how can we make sure that there are secure versions available across a deeper range down in the dependency tree for companies?

CRob (01:52.545)
Makes a lot of sense. I just had the opportunity to read the recent Sonotype stated supply chain report. And I think the new number of dependencies for commercial applications is around 1,100 on average. So this is a pretty, pretty big issue, huh?

Isaac Wuest (they/them) (01:59.79)
Mm.

Isaac Wuest (they/them) (02:06.754)
Yep, yep.

Isaac Wuest (they/them) (02:11.178)
It really is. We actually partnered with Sonatype on some sections of that report there came from some of our data. you’re absolutely right. It’s a growing issue, both because the velocity of open source ecosystems, like the rate of new packages and new versions, continues to accelerate. And as you know, more more CVEs are getting reported year over year. So there’s this whole volume problem that is, I’m sure we’ll talk more about aspects of that. yeah.

CRob (02:15.585)
Thanks.

CRob (02:31.965)
yeah.

CRob (02:37.109)
We sure will. But before we get to that, how did you get into open source? What brought you to this meeting today?

Isaac Wuest (they/them) (02:44.744)
Absolutely, absolutely. So my interest in open source goes back about three to four years at this point. So a previous company that I worked at, I was good friends with several developers who were in the open source space. And they said, hey, we think you might really enjoy the space and enjoy learning about open source as a product manager. And I said, absolutely, I’d love to. So I first started working in open source in that environment where you’re working directly with maintainers. So I got to know

several hundred maintainers and was like, this is a really, really robust and a really, I love the ethos of the open source community. So I feel like you kind of get a taste of it and you go, wow, this is for me and I’ve been in the open source space ever since. Not as someone doing the hard work that everyone else does, but as a PM thinking, how can I support and help with all the important work everyone else is doing?

CRob (03:38.525)
What I love about open source is not everybody has to be the developer. There’s a room for a lot of different skills. That’s pretty great. Speaking of some different skills, and this is something that a lot of developers sometimes struggle with, let’s talk about security. Broadly, the ecosystem uses a tool called CVE, which used to stand for common vulnerabilities and exposures.

Isaac Wuest (they/them) (03:46.913)
Absolutely.

CRob (04:06.889)
And that was kind of a dictionary of these are the discovered vulnerabilities and this is how you can go learn more to protect yourself. CVS is an interesting methodology in a program. Had a couple challenges over the last few years. And there are also whole categories of things that lay people may see as a problem. Like your.

the end of life you teased or malicious packages. And these are things that aren’t accounted for in the CVE methodology. So from your perspective, what does that gap in the CVE program, what kind of risks or problems does that, not ignoring, but that not accounting for these other types of problems, what does that incur to consumers of open source?

Isaac Wuest (they/them) (04:59.083)
Yeah, yeah, that’s a really good question. And I will say that the answer to that question is still evolving. Now, my angle, like my perspective on it from the end of life side, is that, know, CVEs, of course, rely on this kind of public disclosure where someone discovers it, someone who’s kind of certified, I think they’re called a CNA, this is someone who’s allowed to report CVEs in the standard way, reports them.

And that’s great. Of course, CVEs have their limitations. So we have these other scores that are evolving, your EPSS and the known exploited and that system. But the predicate for all of those are that the vulnerability has been discovered and reported. Now, the problem there is that as packages or versions within packages in the open source space go end of life, meaning they’re no longer getting

security updates from that upstream maintainer, that upstream community. As that occurs, a second thing is happening as well. They’re not going to get patches or updates for CVEs. In addition, they’re typically no longer investigated for vulnerabilities to begin with. So it creates this, we just don’t even know, right? Maybe it’s vulnerable, maybe it’s not. Oftentimes, the maintainers around those packages don’t have the time to…

CRob (06:10.783)
Right.

Isaac Wuest (they/them) (06:21.557)
retroactively investigate every CVE on their entire version range to even see if it applies or not. So that space, those unknowns, create a real concern that the broader security market and the SDA space is trying to figure out how to react to.

CRob (06:39.937)
So from your experiences, especially dealing with upstream maintainers, how much attention does upstream pay to end of life? Is this something like I know commercial vendors generally will advertise we do support between X and Y dates. So from your experiences upstream, what does end of life look like or how much do they advertise that stuff?

Isaac Wuest (they/them) (07:03.373)
Yeah, it’s a good question. it’s extremely variable. So for large frameworks, if we’re talking about your Angular’s, your Views, your Springs, you can often use tools like endoflife.date or go directly to their sites and you’ll see their schedules. When, these are the versions, this is when it goes, LTS for that long-term support. And then here’s when LTS is dropped and that’s when it goes end of life. The problem is that that covers a very thin slice of the total number.

packages that are in open source. Most maintainers don’t have time to publish a calendar for their specific packages. As a result, it’s often very ambiguous. Unless we’re talking about those big frameworks, those big libraries, outside of that, it’s kind of like the maintainers know when they’re going to generally they’ll stop supporting major release lines once it’s two or three in the past.

When you talk with maintainers, what you learn is that it’s kind of ambiguous even for many of them. They’ll say, well, if a CVE is reported, I might be able to release a security patch. I might not. It kind of depends how busy I am. They’re volunteering their time. And that ambiguity makes it really challenging to know, well, how secure is it? Is it supported? That initial problem of are we investigating and reporting and patching risk on those older versions?

CRob (08:30.645)
And so again, from your experiences, how have developers or projects articulated this or do they at all?

Isaac Wuest (they/them) (08:38.871)
Great question. So apart from those large frameworks where they publish them on their sites, most of the time it’s an array of methodology. So sometimes they’ll post in the readme’s on their projects. You’ll see information about what’s being end of life. Status is like deprecated where in certain registries, NPM and others, can mark, a maintainer can mark and say, this release line or this version is deprecated. That’s a bit of a proxy for end of life.

Sometimes though, I’ve seen Twitter or X, like social media, be a primary mode where the maintainers of packages are saying, this is when I’m gonna be no longer supporting version 1.1.1. And it’s highly variable and that makes it really brittle for any large enterprise to rely on.

CRob (09:28.769)
Mm-hmm. Well, hey, let’s switch our focus a little bit. Let’s move a little bit more downstream. So let’s say I’m a security professional at an organization, or I’m a developer that’s working on some type of internal app that’s leveraging open source components. What does it mean to them when some dependency or package that they’re using becomes end of life?

Isaac Wuest (they/them) (09:52.814)
Yeah, that’s a great question. I will say, firstly, they’re still learning what it means in terms of compliance frameworks. So when something goes into life, there’s the security implications that we just talked about. Hey, it’s not going to get security patches, and it likely won’t get reports of vulnerabilities that may exist on it. Now, if I’m a security professional working at a company that

CRob (10:03.139)
yeah, yeah.

Isaac Wuest (they/them) (10:19.915)
You know, we have some number of applications we’re building that use that dependency. I know immediately that there’s this kind of blank spot there where even if risks aren’t being flagged by, you know, my security scanning tool of choice, I know that there may still be risks present. That might be a problem for that own company’s security posture, where they have their own standards and say, Hey, we can’t rely on that. Oftentimes what I see though, is that

The main motivation is the fact that it puts them out of compliance with these frameworks that exist. Where if you’re dealing with health data, or you’re dealing with credit card data, and you’re subject to HIPAA, high trust, PCI, as a security team, you can’t tolerate dependencies that won’t get updates. That’s too much risk. And that’s created a real vacuum in the space of open source end of life.

CRob (11:17.857)
And so from your perspective, if something is end of life, does that automatically mean that package is a problem? What options do people have?

Isaac Wuest (they/them) (11:27.883)
Yeah, that’s a great question. So there’s an analogy I sometimes draw from in the food regulation space. If you ever get into how FDA or the EU thinks about risk in food. there’s a difference between risk and hazard. Hazard is just anything can kind of, anything could be a hazard. If I go to an aquarium, the fact that there’s a shark in a tank is a hazard.

Oh boy, if something were to happen, I fell in that tank, slipped, However, the risk is quite low. security teams within companies are trying to figure out the balance within the context of end of life. Is end of life is inherently hazardous. How much risk does it represent though? And based on the risk, what should we do? How much effort should we as a company invest? We can tolerate some risk, we can’t tolerate all risk.

And then the question becomes, what are my options if I’m an engineer or security person who I discover I have several end of life packages in my dependency tree for some application? What should I do? The first problem they have is, how risky are these packages? And the fact that they’re end of life means, what are my options for fixing it? Do have to migrate? Can I patch it myself? That becomes the natural.

kind of next questions and problems that these folks are facing.

CRob (13:01.087)
And we’re definitely going to see that in the coming months and years as the EU’s Cyber Resilience Act comes online, because manufacturers are held accountable for all components within their products. And they need to do that risk calculus to understand what am I going to do if there is a critical vulnerability in one these things that doesn’t have support potentially upstream.

Isaac Wuest (they/them) (13:22.626)
Yeah.

You’re absolutely right. The landscape around shipping secure software continues to get more more strict, which I think is a very healthy orientation for the industry. It’s important that we’re responsible as technologists for the code we’re shipping, even if that’s open source code. But that doesn’t mean that it’s not going to be a really challenging problem to solve as the EU cybersecurity

actor or many others continue to gain traction.

CRob (13:58.525)
And with your interactions both up and downstream, are people aware of kind of this end of life opportunity that’s before them?

Isaac Wuest (they/them) (14:08.641)
folks are becoming increasingly aware. There’s actually a distinction now in two different types of end of life that’s been emerging when talking about it. there’s end of life, similar to those large frameworks and the calendars I was mentioning, that’s often categorized as maintain or attested end of life. And that attestation is often the very compliance-y legal term, where the maintainers are attesting that it is end of life or that it is supported.

And if I’m an auditor, I can rely on that when I’m auditing some large company’s project. The category that’s emerging as a term is maintainer abandonment when it comes to end of life. Now there’s other terms you’ll hear that are similar. Really all we mean is there’s a point at which that maintainer can no longer continue to support. It could be a specific kind of major release line. They’ve moved on.

It could be even specific miners that they’re no longer going to be supporting. you think about that SIMVR, that 1.1. Or it could be the entire package gets abandoned. So they rename the package, or the maintainer has experienced burnout, and they’re no longer able to support it. They’re going to need to take a step back. No one’s filling that vacuum or that void. That maintainer abandonment category is proving to be a real opportunity.

to say how do we identify and report on this even when the maintainer doesn’t have a way to clearly tell everyone and broadcast like the big players in open source do the large frameworks.

CRob (15:41.439)
Interesting. Thinking about the tools we have today and end of life, have for good or bad, the CVE program, we have software builds materials, we have a whole fleet of scanners, AI or not. We have bug bounty programs. Now, from your perspective, do these tools and maybe ones that I didn’t mention, do they give a consumer

Kind of that full security picture or are there gaps?

Isaac Wuest (they/them) (16:14.625)
Right now, the gaps are really starting to show. Most of the available kind of industry standard tools, be it SBOMs, Cyclone DX or SPDX, even the biggest kind of formats out there, or most scanning tools as well, they’re still focused on publicly reported vulnerabilities as the top priority. And then they do talk about end of life. Most of the end of life that you see reported,

are those maintainer attested schedules, where your big frameworks tend to be tracked well, which is really important to be clear. Those big frameworks often represent a lot of risk and a lot of pain if you have to make a migration or something like that. Very few tools exist today in the security scanning space that have a robust picture, a complete picture of that maintainer abandonment side. There’s people doing kind of innovative work there trying to

and understand, but it’s a pretty open field space right now even for these large security scanning tools.

CRob (17:19.947)
Well, that’s always been an historic challenge, of that information sharing upstream is, you know, is the project active? When was the last commit? How many stars and like all that kind of normal lifecycle stuff helps, but is not that developer attestation saying on this day, I’m stopping this particular stream.

Isaac Wuest (they/them) (17:32.488)
Absolutely.

Isaac Wuest (they/them) (17:42.722)
Yep, yep, you’re absolutely right. And yeah, that’s been a challenge for some time. It’s also often a challenge because these tools need certainty in order to report on a scan. Like you said, they get that complete picture. They wanna be able to say with confidence, it’s this or it’s that. It’s supported or it’s not supported. And when you talk with maintainers, there’s a lot more gray there where they say, well, I could support it if I need to. If someone tells me I’m willing to try and do them a favor, because open source is so…

CRob (18:06.209)
Mm-hmm.

Isaac Wuest (they/them) (18:11.329)
Everyone’s so willing to pitch in and help each other. And that’s a feature of the space, but it becomes a bug for those consumers, those large enterprises that need certainty when they’re reporting on compliance, alignment, and all that sort of stuff.

CRob (18:15.595)
There it is, yep.

CRob (18:26.721)
So I’m thinking about this from the developer perspective and end of life. Are there any practical steps that you would recommend a project to consider or do to help be a little bit more communicative around this to deflect a lot of downstream questions? You know, ideally.

Isaac Wuest (they/them) (18:49.335)
Yeah, yeah, absolutely. There are some practical things that are available today. And then there’s some opportunities where I’m really hoping as an industry, security scanning focuses on solving this problem more broadly. Today, though, I the way it’s typically done when I talk with enterprises and work with developers on the front lines is they usually get a notice that something has a CVE on it. And then,

I mean, I’m sure you know well that they end up going and researching. You grab that package name and the version, and you go to Google, and you start looking at the registries. And you say, is this supported? If there’s not an obvious patch version I can just upgrade to, Like oftentimes, engineers stumble upon the reality of it being end of life if it’s not that large framework where there’s a clear calendar. That’s, of course, painful because it means

Do I need to migrate? What do I have to find a replacement, et cetera? So my recommendation today is there’s tools like endoflife.date. That’s a great tool, site and project out there. There’s other ways to go research. However, there are some free tools available that you can load an SBOM or a manifest into. There’s one that we’ve been working on called the eoldataset.com that you can.

Loads free tool that has a large database detecting maintainer abandonment. I’m hopeful many other organizations work on this problem so that engineers don’t have to spend far too much time researching everything one off and get some confident data about what is almost certainly end of life and what is likely supported even for that long tail of packages.

CRob (20:31.329)
That’s good advice. if from your perspective, how accessible, how easy are these systems to kind of get in there and get the data you need so you can make your risk decision? Do I want to continue with this particular branch or not?

Isaac Wuest (they/them) (20:49.293)
Yeah, so if for existing SEA tools, most of them, of course, they’re going to tell you about that maintainer attested. And if you load in your SBOM, there’s usually some other version or remediation path where they say, hey, here’s what you can do about it. There is a supported version. And at that point, it becomes a question of how quickly do you need to migrate? Or can you get some sort of support right now? For the other side of it.

when it comes to maintainer abandonment. I would definitely recommend using the tool I just mentioned where you can load in that SBOM. And once you get that full picture of, here’s all those other packages that are end of life or likely end of life, the next step becomes, OK, well, are there supported versions that I can upgrade or migrate to? Most of them there are. Then it becomes a question of, of course, breaking changes.

How challenging is that? Is it deep in the dependency graph such that the version is constrained and it becomes this kind of gotcha problem? And oftentimes, even knowing something is end of life, that’s not enough for engineers. They need to know what are their upgrade paths if they have to fix that one package, that one version. Our tool that I mentioned that’s free to use has some of those, but many others.

do as well. Your SEA tools often do contain migration options and engineers can often use those tools. They’ll load in data sets like ours or others about end-of-life data that then connects to the dependency graphs and says here are your migration options. You’d have to upgrade these two direct dependencies. That gets you into the versions you need that are no longer end-of-life deeper down.

CRob (22:42.657)
This is an amazing topic that I think not everyone is aware of it as they should be. So thank you for joining us today and kind of sharing a little bit about end of life. But let’s move on to the rapid fire part of our talk.

Isaac Wuest (they/them) (22:57.357)
you

CRob (23:00.405)
That’s spicy. I have a handful of questions that we’ll ask you. Just give me the first thing that comes off the top of your head. What’s your favorite beverage?

Isaac Wuest (they/them) (23:09.441)
Got it.

Ooh, I’m gonna have to go with a cappuccino. I love coffee, but I love espresso, so give me that. Give me that right there. Yes, I need that caffeine.

CRob (23:14.593)
Ooooo

CRob (23:20.193)
Yum. All right. Star Trek or Star Wars?

Isaac Wuest (they/them) (23:24.885)
I’m so sorry, but it’s gonna be Star Wars for me. It’s gonna be Star Wars. Okay, good, good, good. Well, I appreciate Star Trek. I certainly love it, but Star Wars, there’s just something about it that speaks to me.

CRob (23:27.723)
There are no wrong answers, but Star Wars is a pretty good one.

Well, do you prefer spicy or normal food?

Isaac Wuest (they/them) (23:42.024)
I love spicy. Please.

CRob (23:43.751)
ooo that’s spicy

Isaac Wuest (they/them) (23:47.853)
Yeah, I don’t understand people that don’t like spicy food. It’s just more exciting. It’s more interesting.

CRob (23:49.249)
Thank

Right? Exactly. And then, you being that we are in the a new age here, who’s your favorite apocalyptic robot?

Isaac Wuest (they/them) (24:06.029)
Ooh, my favorite apocalyptic. is this, ooh, is this existing AIs or is this fictional robots? my goodness. I think I could have one of each. could have one of each. Okay, for fictional, I’ve been reading all these dystopian books that involve robots. I love Heinlein’s The Moon is a Harsh Mistress. There’s an AI there, Mike or Michelle, not a robot that causes an apocalypse, but one that kind of helps solve kind of a crisis.

CRob (24:13.622)
Sure.

You could have one of each.

Isaac Wuest (they/them) (24:35.373)
So that’d be my favorite fictional. Ooh, for existing, I don’t know. There’s a number of AI models I’m pretty scared of right now. We’ll see. And I don’t want to badmouth any, because maybe in two or three years, I’ll be hoping that they take care of me.

CRob (24:35.489)
Nice.

CRob (24:43.297)
You

CRob (24:50.241)
That’s our dream. Well, thank you for playing along, Isaac, being a good sport. And as we wrap up, do you have any closing thoughts or a call to action for our listeners?

Isaac Wuest (they/them) (25:01.161)
Absolutely. Well, thank you so much for bringing me on. I really appreciated the conversation. This was really exciting and interesting. What I would definitely say is, for anyone listening, if you’re worried about End of Life and the potential exposure that any of your applications have, feel free to check out what we’ve been working on. We have a free tool with a data set called eoldataset.com. And we’re looking just to give away as much data that we have about the End of Life space. So if this is something that was interesting to you and you want tolearn more about what might be end of life, or you want to contribute thoughts and ideas about how to improve the data set, please let us know. We’re very interested. yeah, with that, that’s my main…

CRob (25:43.189)
Well, Isaac from HeroDevs, thank you for joining us today. And I want everyone to kind of listen to this, maybe take some action, do a little research to educate yourselves. And I’d like everyone to stay cyber safe and sound out there. Happy open source and folks. That’s a wrap.

Aligning on Machine-Readable Signals as the Foundation for Due Diligence

By Blog, EU Cyber Resilience Act

By Madalin Neag, EU Policy Advisor, OpenSSF

Introduction

The software supply chain has reached a level of complexity where manual oversight is no longer a viable strategy for security or regulatory compliance. Modern systems depend on vast, rapidly evolving networks of components, making manual, paper-based approaches to due diligence impractical. Machine-readable, continuously generated security signals are therefore the only realistic way to support Cyber Resilience Act (CRA) due diligence at scale. These signals already exist across open source ecosystems as a natural byproduct of standard development practices, though they remain fragmented across tools, repositories, and pipelines. Crucially, these signals are best understood as mechanisms for transparency rather than assurance: they expose observable characteristics of software development and operational behavior without constituting guarantees, certifications, or transfers of liability. 

This shift is driven by a need for technical accuracy. Static documentation and point-in-time attestations cannot reflect continuously evolving software systems. This limitation has been underscored by recent U.S. enforcement actions, such as Department of Justice settlements involving inaccurate cybersecurity compliance certifications, which highlight how formal attestations can later be treated as misleading when they diverge from actual system behavior, significantly increasing legal exposure.

Established approaches such as continuous compliance, evidence-based assurance, and secure-by-design all rely on the same principle: replacing subjective, point-in-time claims with dynamic, verifiable proof, automated data that reflects actual system behavior.

Within this model, roles remain clearly separated. Upstream open source projects may choose to publish security-relevant signals in machine-readable formats, while manufacturers, who bear the legal responsibility under the CRA, consume and interpret this information as part of their due diligence processes. This preserves the foundational “no warranties, no liabilities” principle of open source. Participation from upstream remains strictly voluntary and must not introduce legal obligations, certification expectations, or shifting of compliance risk and liability to the project community, ensuring that ecosystem sustainability is maintained while enabling effective downstream risk management.

This discussion builds on earlier reflections on voluntary attestation models under the Cyber Resilience Act, particularly Article 25, which enables voluntary security attestation programmes to support manufacturer due diligence for products incorporating free and open source software while preserving the separation between upstream development and downstream regulatory responsibility. From a systems perspective, however, Article 25 also exposes an important limitation of paper-based attestation approaches. Static, human-authored representations of security struggle to remain accurate within environments defined by continuous change, where rapidly evolving components and deeply nested dependencies can quickly render point-in-time attestations incomplete or outdated. This leads to a broader architectural observation: effective due diligence at scale increasingly shifts away from narrative declarations toward machine-readable, continuously updated security signals embedded directly within development and release workflows. In this framing, voluntary attestation is valuable not as a mechanism for upstream certification, but as a way to enable structured, interoperable security data that downstream systems can automatically consume and evaluate. Machine-readable signals thus become the practical substrate for operationalizing the intent of Article 25 in complex software ecosystems, preserving voluntariness, avoiding any conflation of transparency with assurance, and enabling evidence-based due diligence aligned with the dynamic nature of modern software systems.  

Due Diligence under the CRA: a continuous risk-based process

Due diligence under the CRA must be understood as a continuous, risk-based obligation rather than a procedural formality. As clarified in the European Commission’s FAQ and further complemented via the CEN PT1 standard, it is not a checklist to complete or a document to obtain, but an ongoing responsibility carried by manufacturers placing products with digital elements on the market. Its purpose is to ensure that third-party components, regardless of origin, do not compromise the cybersecurity of the final product.

At its core, due diligence requires manufacturers to make informed, traceable decisions about the software they integrate. This includes understanding the origin and role of components, evaluating their security characteristics, and determining whether their use is appropriate within the context of the product. These activities form a continuous lifecycle process covering evaluation, integration, monitoring, and remediation. The level of scrutiny applied is inherently risk-based and contextual, and depends on the role, exposure, and criticality of each component within the manufacturer’s system.

This obligation is dynamic by nature. Software components evolve, vulnerabilities are disclosed continuously, and integration contexts change over time. Due diligence therefore extends across the entire lifecycle, requiring manufacturers to revisit earlier assumptions and adjust mitigation strategies as new information becomes available. This creates a continuous feedback loop between upstream changes and downstream risk decisions.

The regulatory expectation is that this process is demonstrable through technical documentation that allows decisions and risk assessments to be traced and verified. The emphasis is not on collecting predefined assurances, but on ensuring that decision-making remains consistent, auditable, and defensible over time.

For open source components, due diligence relies on observable project characteristics rather than formal assurances. Manufacturers assess elements such as maintenance activity, responsiveness to security reports, release practices, and the availability of structured security documentation, drawing on signals that reflect how a project is actually developed and maintained. These signals can be aggregated into continuously updated, machine-readable evidence reflecting the current security posture of both the component and its dependencies. This approach does not create any dependency on upstream attestations: under the CRA, manufacturers remain solely responsible for their assessments, while any transparency provided by open source projects is entirely voluntary and does not constitute certification or liability. Machine-readable security signals therefore function primarily as decision-support inputs within downstream risk management processes. They improve the quality, consistency, and scalability of due diligence activities, but they do not replace the manufacturer’s obligation to exercise independent judgment and accountability. 

Machine-Readable Signals in Practice

A mature ecosystem of tools already generates machine-readable signals that can support due diligence under the CRA. These signals span multiple layers of the software lifecycle, from component identification to vulnerability management and build integrity. Standards such as SPDX and CycloneDX enable structured software bills of materials (SBOMs), while frameworks like SLSA define levels of build provenance and integrity. Complementary technologies such as Sigstore provide cryptographic mechanisms to verify artifacts, and formats like CSAF and VEX support the structured exchange of vulnerability and exploitability information. Tools such as SBOM CVE Check, Dependency-Track, Syft, and Grype operationalize these standards by enabling SBOM-driven component analysis and automated vulnerability scanning, while SBOMQS provides additional validation of SBOM quality and compliance against established standards.

Within the OpenSSF ecosystem, these capabilities are reinforced by a growing set of complementary tools that standardize, expose, and operationalize security-relevant signals across different layers of the software lifecycle. Some focus on repository security posture and development practices, others on supply chain integrity and provenance, while newer systems increasingly aggregate these signals into unified, queryable models suitable for large-scale risk analysis and automated due diligence workflows. At the project governance and security posture layer, OpenSSF Scorecard provides automated checks on repository hygiene and secure development practices, while the Best Practices Badge and Open Source Project Security (OSPS) Baseline initiatives offer structured indicators of project maturity and security adoption. OpenSSF Security Insights extends this approach by introducing a standardized, machine-readable format for publishing security policies, development processes, and maintenance practices, enabling more consistent interpretation of project security posture across tools and downstream consumers. Complementing these efforts, LFX Insights aggregates operational and community signals related to project activity, contributor diversity, governance, and security posture, helping organizations evaluate the long-term sustainability and operational health of dependencies over time. Together, these tools transform otherwise fragmented repository metadata into reusable signals that support risk-based evaluation without requiring additional compliance artifacts from maintainers. 

At the supply chain integrity layer, frameworks such as in-toto provide cryptographically verifiable attestations describing individual steps within software build and release pipelines, strengthening provenance visibility and artifact integrity. SBOMit builds on this model by combining SBOM generation with in-toto attestations and signed supply chain layouts, enabling verifiable component composition during the build process. Related tooling such as Protobom and Bomctl improves interoperability and operational reuse of SBOM data. Protobom provides a format-neutral intermediate representation that allows SPDX and CycloneDX documents to be transformed and consumed consistently across heterogeneous tooling ecosystems, while Bomctl enables structured manipulation, merging, and management of SBOM trees across complex dependency environments.

Increasingly, these signals are being aggregated into higher-level analytical systems capable of supporting continuous, ecosystem-scale risk analysis. GUAC (Graph for Understanding Artifact Composition) demonstrates this direction by ingesting SBOMs, provenance attestations, vulnerability reports, OpenSSF Scorecard results, and related metadata into a continuously queryable graph model. This enables dependency-aware analysis of upstream risk exposure, artifact relationships, and vulnerability propagation across software ecosystems. Architecturally, systems such as GUAC illustrate a broader shift within software supply chain security: compliance and due diligence increasingly become problems of correlating continuously generated technical evidence rather than collecting static documentation.  

Additionally, tools and frameworks such as OSS Review Toolkit (ORT), Community Health Analytics in Open Source Software (CHAOSS), and OpenChain extend this landscape by enabling deeper analysis and contextual understanding. ORT integrates dependency, license, and vulnerability analysis into reproducible workflows, while CHAOSS provides metrics on project activity, health, and sustainability. OpenChain complements these by defining standards for open source compliance and supply chain governance, helping organizations establish consistent, auditable processes for managing open source use. Together, these perspectives allow manufacturers to assess not only technical risk but also organizational maturity and long-term sustainability of the components on which they depend.

This direction is also reflected in European cybersecurity guidance. ENISA’s Security by Design and Default Playbook highlights machine-readable signals as a mechanism for making security both demonstrable and verifiable within development processes. By enabling continuously generated, machine-consumable evidence that can be automatically validated and reused, this approach reinforces the shift from static documentation to dynamic, lifecycle-integrated assurance, directly supporting scalable due diligence.

European initiatives further build on this foundation by operationalizing these signals into compliance workflows. EU-funded projects such as CRACoWi, CYBERFORT, CONFIRMATE, OSCRAT, and OCCTET ingest machine-readable inputs (including SBOMs, vulnerability data, and provenance information) and transform them into risk assessments and technical documentation. These platforms demonstrate how compliance can be implemented as a continuous, automated process, reinforcing the complementarity between upstream signal generation and downstream consumption.

Taken together, these tools form an interoperable ecosystem of continuously generated signals. They already provide most inputs required for effective due diligence, demonstrating that the necessary data exists within current development workflows. This confirms a key architectural reality: compliance at scale is fundamentally a data integration problem, not a documentation problem.

A large share of due diligence-relevant information is already present within open source repositories. Security policies (SECURITY.md, security.txt), contribution workflows (CONTRIBUTING.md), release histories (changelogs), issue templates, licensing files, and repository governance practices (branch protection, maintainer authentication) collectively describe how software is developed, maintained, and secured. Together, they provide a rich baseline for assessing development discipline, update reliability, and supply chain integrity, without requiring additional compliance artifacts.

However, these signals are often distributed across heterogeneous formats and locations, making them difficult to discover and reuse at scale. Improving their visibility through lightweight structuring or simple indexing can significantly reduce friction. This is not about adding new artifacts, but about improving the accessibility of existing ones.

The viability of this model is already demonstrated in practice. Many open source projects publish structured security information as part of their normal operations. Projects such as K3s provide comprehensive self-assessments describing architecture and security considerations, while the Argo project maintains clear documentation of its vulnerability disclosure processes. Other initiatives, including Privateer and other projects within the CNCF ecosystem, expose structured security information directly within their repositories. These projects demonstrate that “CRA readiness” is essentially an extension of existing high-quality security engineering. Documenting security posture is already a well-established practice; what is missing is not the content, but a consistent way to connect that content to automated due diligence workflows.

Guidance from OpenSSF security assessments further supports this practice by helping projects think systematically about their security posture, development processes, and risk boundaries.

This becomes particularly critical at scale. Modern systems depend on hundreds or thousands of components, making manual evaluation (and reliance on manual, individualized attestations) infeasible. Machine-readable signals enable automated collection, continuous updates, and consistent analysis across dependency graphs, transforming due diligence into a reproducible computational workflow aligned with contemporary software realities.

Voluntary Upstream Participation and Ecosystem Engagement

It is critical to start with a baseline legal protection: maintainers and stewards are not “suppliers” in a commercial sense and are not required or expected to sign contracts or guarantee compliance outcomes. Under the CRA, open source projects are under no obligation to support compliance activities. Responsibility remains exclusively with manufacturers placing products on the market. This legal boundary is intentional and reflects the regulation’s effort to preserve the open source model, particularly the “no warranties, no liabilities” foundation that enables broad participation and innovation. More broadly, the legislative intent of the CRA is to protect the “long tail” of community-driven innovation, ensuring that smaller projects, individual maintainers, and informal communities are not burdened with obligations designed for commercial actors.

At the same time, many projects already choose to publish security-relevant information such as vulnerability handling policies, release processes, or build metadata. These actions improve transparency and usability for downstream users, but they do not create legal obligations, warranties, or liability. They are best understood as voluntary engineering practices that reduce friction and make projects easier to adopt within regulated environments.

A central principle throughout this model is the clear distinction between transparency and assurance. Transparency refers to voluntarily published, descriptive information about how software is developed, maintained, and secured. Assurance, by contrast, implies some form of validated guarantee, certification, or assumption of responsibility. Under the CRA and within open source ecosystems, transparency must never be interpreted as assurance. 

Any attempt to interpret voluntary signals as certification risks undermining both the legal structure of open source licensing and the intent of the CRA. From a regulatory perspective, oversight remains focused on products placed on the market rather than upstream development processes. As a result, any security signals published by projects function as inputs into downstream evaluation, not as regulatory objects in themselves.

Within this framework, upstream security signals serve only as inputs into downstream due diligence. Manufacturers remain responsible for evaluating those inputs and making risk-based decisions. The availability of better signals can improve that process, but it does not shift accountability or create dependencies on upstream participation.

At the same time, the CRA introduces an important change in incentives. Manufacturers can no longer rely on passive consumption of open source components without understanding their security implications. When gaps in security signals are identified, the most effective response is not to request formal assurances but to improve the upstream ecosystem through tooling, documentation, funding, or engineering contributions. This includes supporting SBOM generation, provenance tooling, vulnerability disclosure processes, or automation of security pipelines.

This dynamic creates an opportunity for a more balanced and sustainable relationship between upstream and downstream actors. By investing in the security and transparency of the projects they depend on, manufacturers not only support their own compliance efforts but also strengthen the resilience of the broader ecosystem. This shift does not alter legal responsibilities, but it encourages a model of shared interest where better upstream practices benefit all participants without imposing new obligations on open source maintainers. 

Building a Scalable Model for Cyber Resilience

Modern software systems routinely incorporate thousands of independently evolving components, making static documentation obsolete almost as soon as it is produced. In this context, scalable due diligence cannot rely on manual, document-driven approaches. Machine-readable security signals provide a scalable alternative: continuously generated, verifiable, and aligned with dynamic software supply chains.

A scalable due diligence model therefore depends not only on machine-readable signals, but also on correctly interpreting their nature. They are evidence of behavior, not guarantees of outcome. Maintaining the distinction between transparency and assurance is what allows these signals to be useful without distorting responsibility or imposing unintended obligations upstream. 

The CRA establishes a clear downstream responsibility model, but its effectiveness depends on implementation. When grounded in automation, interoperability, and continuously updated evidence, due diligence becomes operational rather than procedural. This approach builds on existing engineering practices and widely adopted tooling, enabling scalable risk assessment without imposing new burdens on upstream maintainers. This direction is consistent with ENISA’s Security by Design and Default Playbook, which emphasizes machine-readable security attestations as a foundation for demonstrable and continuously verifiable security across the software lifecycle.

Crucially, this model preserves the sustainability of the open source ecosystem. It avoids shifting liability or compliance expectations onto maintainers, while still improving transparency through low-friction, machine-readable signals. Much of the required information already exists within projects today; the challenge is not creation, but integration and consistent reuse. By treating compliance as something assembled from technical evidence rather than declared through static attestations, the process remains both accurate and adaptable over time.

Ultimately, a shift toward continuous, evidence-based due diligence ensures cybersecurity can scale alongside software complexity. It enables manufacturers to manage large dependency landscapes efficiently, supports ecosystem resilience, and fosters more meaningful upstream–downstream collaboration. Compliance is not a one-time declaration but an ongoing capability that strengthens both regulatory outcomes and the integrity of the digital infrastructure.

About the Author

Madalin Neag works as an EU Policy Advisor at OpenSSF focusing on cybersecurity and open source software. He bridges OpenSSF (and its community), other technical communities, and policymakers, helping position OpenSSF as a trusted resource within the global and European policy landscape. His role is supported by a technical background in R&D, innovation, and standardization, with a focus on openness and interoperability.