Podcast – Open Source Security Foundation

Jun 16

What’s in the SOSS? Podcast #63 – S3E15 Big Thoughts, Open Sources: Driving Enterprise Security and Career Growth Through Open Source with Jamie Thomas (IBM)

By OpenSSF Podcast

Summary

In this episode of Big Thoughts, Open Sources, host CRob sits down with Jamie Thomas, IBM Enterprise Security Executive and OpenSSF Governing Board Member (former Chair!), to tackle the vital shifting dynamics of enterprise open source engagement. From IBM’s historical “billion-dollar bet” on Linux to modern supply chain wake-up calls like SolarWinds and Log4j, Jamie pulls back the curtain on what it truly means to move from accidental consumption to intentional stewardship. Tune in to discover how active participation in neutral foundations like the OpenSSF acts as a fast track for engineering career trajectories, why soft skills like “the art of influence” are critical for upstream collaboration, and how organizations can protect their crown jewels while implementing a powerful “give-back strategy.”

Listen on Apple Podcasts Listen on Spotify Listen on Overcast Listen on Pocket Casts

Conversation Highlights

00:00 – Intro Music + Promo Clip
00:21 – Introduction & Welcoming Luminary Jamie Thomas
01:32 – Wearing the Enterprise Security Hat at IBM
02:10 – Supply Chain Wake-up Calls: From SolarWinds to Log4j
03:14 – Unlocking Open Ecosystems: IBM’s Early History with Java and Linux
05:21 – Mainframe Debates and Portability: The Evolution of Open Source Adoption
06:24 – The Red Hat Acquisition and Monetizing the Developer Ecosystem
08:20 – The Myth of “Free” Software: Securing Regulated Enterprise Deployment
10:15 – Why a Seat at the Table Matters: The Value of Neutral Foundations
11:29 – The Art of Influence: Upstream Contributions as a Career Catalyst
13:50 – Moving Innovation from Open Source Kernels to Commercial Value
16:12 – Storming, Norming, and Conversation: Lessons from the Kubernetes Era
17:38 – Pitching Upstream Time: Helping Developers Sell Open Source to Management
19:30 – Beyond Code: Bringing Domain Expertise and Soft Skills Upstream
21:40 – Conquering the Chasm: Automating CI/CD Pipelines and Testing at Scale
22:56 – Consuming with Intent: Active Stewardship and the OpenSSF Scorecard
25:21 – Rapid Fire Round: Mainframes, AI-Generated Code, and Star Trek nostalgia
27:53 – Call to Action: Crafting Your Organization’s “Give-Back Strategy”

Episode Links

Transcript

Music & Intro/Promo clip (00:00)
If you’re a direct consumer of open source, do it with intent. And intent means that you’re responsible for what that means to your organization, both from a productivity perspective, but also from a security perspective and what you can expect from an investment point as well. You have to be an active steward in the maintenance of your open source strategy, just like you would have to be a steward of the maintenance of your own home.

CRob (00:26)
Welcome, welcome, welcome to Big Thoughts Open Sources. My name’s CRob. I’m your host today. This is a special video series where we’re talking to some of the leaders within open source. And today we have an amazing treat. We have Jamie Thomas from IBM. It’s a little company you might’ve heard of. And she’s here today to kind of talk about maintaining a career and influence within open source. Welcome, Jamie.

Jamie Thomas (00:54)
Thanks, Crob. Thanks for having me.

CRob (00:56)
I think we’re going to have a pretty amazing conversation today. For those of you that might be unfamiliar with kind of the luminaries that exist within our little slice of open source security heaven, Jamie has been alongtime contributor and member to the OpenSSF, predominantly through influence through our Governing Board. You actually were our Governing Board Chair for a period of time. So for those that may be unfamiliar with you, could you maybe talk about what your current role is at IBM and kind of what you do within the OpenSSF?

Jamie Thomas (01:32)
Absolutely, and thanks for having me again. I think this is a really interesting topic. At IBM, I have a couple of hats, but the main hat that is important to this discussion today is IBM Enterprise Security Executive. And I’m therefore responsible for the protection of the IBM company. It includes our CISO office, our Cybersecurity Operations, as well as our Product Security, which has to span a number of our units, including our software, hardware, our consulting unit. So it’s an interesting job.

CRob (02:05)
Really big job.

Jamie Thomas (02:10)
Yeah, and you know, as you know, CRob, I got involved in OpenSSF from the very beginning of the Governing Board. And I remember when I got involved in enterprise security, someone told me, by the way, it’s all quiet, there’s not much going on right now. And, and the first thing that I recall happening not long after that was SolarWinds, which was one of the first interesting supply chain attacks. And then of course, we had Log4j, which is another whole realm of fun.

But certainly I felt that joining OpenSSF has been important for us, the IBM company, to stay abreast of what’s happening in the industry and to be a participant in this open source security realm. It continues to be an ever changing landscape.

CRob (02:53)
Absolutely, a blink and it totally changes. It’s a pretty wild space we get to live in. Thinking back through your career here, was there a moment or something that really sparked your interest that kind of drew you towards more open source perspectives, participation?

Jamie Thomas (03:14)
Yeah, yeah, absolutely. I mean, when I started IBM, I did start as a programmer in the software organization. And at that time, of course, we worked on, like all of the industry, probably at that time, closed source systems. But at some point in IBM, we got very involved in something called Java. As part of…

CRob (03:32)
I’ve heard of it.

Jamie Thomas (03:34)
Haha, yeah you’ve heard of Java…As part of Java, we wanted to engender an open ecosystem around the Java programming model.
And to that end, we realize we’re not going to attract developers at scale without a different approach.

And so we made some concerted decisions at that time. One was that we would invest in Linux. There was something called Linux at that time. And we made a decision to put into the corporation Linux Technology Center to invest in Linux, be open source contributors to the Linux operating system. And the other big decision we made is that we would outsource some of our Java development tooling to something called the Eclipse Foundation.

CRob (04:16.691)
Mm-hmm, oh yeah.

Jamie Thomas (04:17)
So it seems like, you know, tribal knowledge at this point, but to say the least, embarking on some of these endeavors was quite unique for the IBM company at the time. And getting people to understand their role within an open source community as opposed to what we had always traditionally done in terms of how we provided software to clients was very, different.

And I had the pleasure of being involved in both the Java effort and the Eclipse Foundation effort through our Rational Software Division. And at that time, I actually owned all of the folks that were contributing to Eclipse and got to see that evolve for a period of time. You know, that was very, very fascinating times for IBM and also, I think, important for the evolution of open source that we have today.

CRob (05:09)
Well, in thinking further downstream, the folks, the organizations that would be your customers, Java was probably one of their, also their entry points into the open source ecosystem.

Jamie Thomas (05:21)
Mm-hmm. It absolutely was because you found through the eyes of lot of our enterprise clients as they adopted Java, they were naturally then adopting Eclipse. And over a period of time, they certainly became very strong supporters of the Linux operating system. And in fact, I remember one of the first meetings we had where we actually were in a room with all these clients and there became this huge argument, this huge argument about whether we should put on the IBM mainframe.

And you can imagine, there were a variety of opinions in that room. But what made sense, if you thought about it, is that Linux would give us portability. And if folks developed on Linux, of course, then ultimately portability to different hardware platforms, of course, would be gained through Linux. And that’s exactly what did happen.

But at that time it was a very, very novel concept that we should embrace this new operating system. And so it was fun, fun stories.

CRob (06:24)
Speaking of fun stories, several years back, IBM made the air quotes, billion dollar bet on Linux and kind of looking back at kind of IBM’s initial involvement with the ecosystem and Java, and then through the acquisition of Red Hat, kind of thinking about what was the most difficult part kind of moving these massive enterprises towards an open source first mindset.

Jamie Thomas (06:53)
Well, I think that by the time we made that multi billion dollar bet on something called Red Hat, our perspective of open source had matured quite a bit. We had then already discovered and learned that we did.

CRob (07:02)
Mm-hmm.

Jamie Thomas (07:03)
We had then already discovered and learned that we did. monetize quite well the Java ecosystem and part and parcel to what we did around Eclipse and Linux. We’d also adopted scale Linux on our platforms on both the Z platform in particular and Power and also we were stewards of Linux on x86 for our clients. So we had a different perspective. And I think at the time we acquired Red Hat though, we did understand the needle had moved quite a bit on the developer ecosystem. In that period of time since we had first started some of these efforts and that Red Hat would enable us to have a bigger stake in the open source community as well as the ability to reach developers at scale.

And that was part of the motivation for the acquisition and I think that it definitely served us well. I believe that what Red Hat does, curated open source is something that’s really important for the industry to have a competent enterprise level open source provider, but that once again,has their finger in the pie around the open source community because that’s where everything happens. It happens in the community and then it has to bubble upstream and be effective for the enterprises that are consuming it.

CRob (08:20)
I know we’ve talked many times that you get the opportunity to talk to a lot of business and cyber leaders at firms that you partner with. Kind of when you’re providing advice and counsel to these leaders, how do you kind of reassure them that they’re not going to be contributing to open source isn’t going to be giving away their secret sauce or their crown jewels? How do you encourage them in engaging more and more effectively with open source?

Jamie Thomas (08:50)
Well, I think there’s two different perspectives. And when you talk when I talked to a lot of the client organizations, I think they have pretty much accepted that open source is a valuable asset to them. What they’re then looking for is who can help me ensure that I’ll have operational fidelity, security around this open source. And that’s when they then start to say, should I have a trusted vendor to work with on that point? Or am I going to consume it directly?

And, and certainly, Isee that early on in the acquisition of technology that many enterprises just take the open source and try it out. I mean, that’s one of the benefits, right? You can try it out. You can test drive it, you can decide if it works for you. But then normally, if you go into enterprise deployment, you do want someone that’s going to be there. In many cases I’m going to provide the care and feeding for you, particularly if you’re in a regulated industry. And of course, we deal with a lot of regulated firms and in our environment.

I also also make sure that I remind everybody involved, including ourselves and our ecosystem, that open source is really not totally free. I mean, it’s something that we all have an obligation to support. So if we’re consuming it, we need to understand what that means and we need to be stewards of ensuring that the open source communities are successful going forward.

CRob (10:15)
That’s an interesting point that leads me to my next question. How vital is it for enterprises to participate in neutral bodies like the OpenSSF or Eclipse or CNCF, rather than just trying to take all these amazing tools and kind of assemble it themselves, how does that neutral collaboration help them?

Jamie Thomas (10:36)
Well, I think it’s very important for you to, as an organization to have a seat at the table, if you will, realizing that in today’s world, you know, software doesn’t stay in the boundaries of one company. I mean, there’s an extensive ecosystem out there, we’re all somewhat interconnected. And if you’re at the table, you have an ability to influence the direction of a lot of these projects, you had the ability to contribute uniquely, but most importantly, to make sure that the projects understand your unique point of view. Certainly there’s a point of view of technology organizations like IBM. There’s a there’s a the the point of view of the startups out there that are so critical to the ecosystem. And there’s the point of view of the consuming enterprises, the downstream enterprise, I think all of those play a critical role in having that seat at the table.

CRob (11:29)
I absolutely agree. So as we continue our conversation here and getting more to, think, is one of your true areas of passion.

From your perspective, you’ve seen a lot of engineers in your career and you’ve seen them become kind of strong speakers and potentially global influencers through this open source ecosystem. From your perspective, do you see active participation in like open source projects as some type of a fast track towards leadership for engineers?

Jamie Thomas (12:04)
I think it’s one of those many facets that can really improve your career trajectory. The reason I think this is that for any accomplished computer science engineer today, you really have to have the ability to influence others. You have to have the ability to influence networks. Maybe that’s outside of your organization or inside of your organization. You have to have the ability to influence your customers.

And so how do you build that kit bag of resources and skills that allow you to do that? And I think participation in an open source community gives you a lot of industry perspective.

You know, when I go to the OpenSSF, I don’t just go there and understand, of course, what my point of view is, I’m understanding the collective point of view. And that collective point of view is very interesting and fascinating. Learning from others and then having that ability to share your unique perspective with clients and even in your internal organization is a skill that many people often underestimate.

The skill of influence is something that’s really important in today’s world. And I think that you can build that. You can also strengthen your own personal presentation skills.

I’ve seen many people presenting in a lot of these forums, whether they’re presenting at one of the conferences or presenting in a governing board or presenting at one of the breakouts. And it really is, once again, an opportunity for you to build your own personal presentation skills, understand how you can represent your ideas more effectively to another organization or set of organizations. And that is something I think is critical for a lot of individuals that are pursuing a different career trajectory.

CRob (13:50)
Mm-hmm. I really appreciate that insight. If thinking about this, how important is it for an engineer to have kind of this open source pedigree, whether it’s like a recognition, like your IBM Open Innovation Awards or other types of awards? How does participating in those things help that individual stand out, especially in this age where organizations are rapidly changing and pivoting?

Jamie Thomas (14:19)
Well, thanks for bringing that up IBM does have an open innovation award that we give to individuals every year who are standouts in the participation of open communities. And I think that’s really important. And we have both executive contributions as well as non-executive contributions and I always recognize those individuals in my all hands meetings to make sure they get their name in lights a bit internally. But I think that this is particularly important for individuals who once again want to take the learning from these kind of communities and use that to more broadly influence the direction of organizations. In many cases and I was just on one of these today in fact a lot of the innovation does start in an open source format because once again it’s very easy to get your ideas out there to start seeing them scale understand what really appeals to clients and what is not then how do you take the open source kernel and then create something that is of commercial value right.

You either have to have the concept like we did with Eclipse that it’s going to enable developers to use other products more effectively, or you have to take the kernel and then become an enterprise open class support organization like what Red Hat did eventually…

CRob (15:41)
Mm-hmm.

Jamie Thomas (15:42)
Very accomplished open source curation organization. So I think for individuals to help companies like IBM or to help their team to help their little asset that they created become more successful, then this participation is absolutely critical. And without a few individuals along the way that really understand how to do this, then I think many, many rewarding projects will not get from point A to point B and be as successful as they could have been.

CRob (16:12)
Right, Yeah, I definitely agree with that. So how…

Jamie Thomas (16:17)
I mean, look at things like probably, you know, Kubernetes or many of the things that we recognize today is just we take for granted without that kind of vested participation support would they have become what they are today? I mean, Linux is the best example, of course.

CRob (16:36)
I absolutely agree. Kubernetes is another example of where we had an organization that had an idea and there were some competing ideas at the same time. And then that project was donated to a neutral foundation where all these fierce competitors could get together and work together in that space to help make the technology itself far more impactful than it ever would have been.

Jamie Thomas (17:00)
Yes. And I hear I hear behind the scenes, even though I was not part of all of that directly, that there was a little bit of storming and norming and maybe a few disagreements along the way and everything.

CRob (17:10)
Yep.

Jamie Thomas (17:11)
But eventually, you know, some really good things came out of that. And, you know, it’s like anything when you nothing, nothing really good is achieved in my mind without conversation, you know, without really having a human dialogue to understand what’s working and what’s not. Certainly big things are not achieved typically through subterfuge and so I think that is the value of a lot of these communities.

CRob (17:38)
I agree. So when you’re thinking about, again, from the developer maintainer perspective, what’s advice you would give to give a developer to pitch to their manager about spending some part of their time working upstream, hopefully in security, but participating in an upstream project? How do you help them sell that to their management?

Jamie Thomas (18:01)
Well, I think in many cases, the individual needs to take a point of view of what’s in it for my manager, right? I mean, it’s kind like when you’re trying to sell anything, right? There are certainly clear attributes that help you as an individual, right? Can improve your leadership skill, your ability to help the manager from that perspective, I think is quite valuable. Problem solving is something we often take for granted today, but it’s not always there, right? But then the other thing is what what particular.

But will the participation aid from a business perspective? And I think there’s a lot of different aspects of that. If someone’s working in the security domain, I think there’s really clear outcomes.

where their participation will benefit for the enterprise you learn so much right in terms of what you need to do. If you’re working in a particular project that is going to have downstream impacts possibly to the organization I think there’s a clear linkage there. But first of all never take for granted that everyone does understand open source.

I find that even in today’s environment, there’s a lot of upline management or senior executives perhaps don’t understand the importance of open source and open source participation. It’s kind of like the water that’s running in our house and we just think it’s gonna always be there. It’s always gonna be on, right? So building that cell package is important. It’s important.

CRob (19:30)
So from, again, your perspective, outside of having some coding skills or some other technical ability, what other types of specific things should people think about when they’re going to go engage upstream? Are there other abilities or skills they might be able to bring to bear to help upstream?

Jamie Thomas (19:52)
Well, I think that coding skills are one but don’t underestimate just basic ability to influence basic ability to bring technical perspectives together and drive a conclusion. I mean, we’ve certainly seen a lot of that in the OpenSSF right where the technical committees have really had to come together and you’ve been a big part of that to render a recommended outcome right that’s influence and getting people to agreements really important back to your Kubernetes point. Somebody had – a group of people had to get some level of agreement to move forward. So influence is really important. Having domain knowledge of things like security, of secure CICD practices, sharing that with organizations I think is really, important. That’s one of the things you know that we’re really focused on is not how do we just create best practices and tools, but how do we help other projects consume those tools at a faster rate and pace.

So individuals that have that passion, who have the ability to help teams be more successful, those are maybe some soft skills that I think a lot of projects really need.

CRob (21:04)
And I think that that’s, hear this a lot from non-developer folks that are interested in contributing is, you what could I possibly bring to the table? I’m not a coder. I don’t understand C or Rust or Go. And I tell them, you have value, you have domain expertise, you understand networking or like CI systems far better than most software engineers do because software engineers are studying the language or a particular community. They don’t necessarily have a lot of these additional skills like program management and communication.

Jamie Thomas (21:40)
Yeah, and one of the things I remember the most about, you know, one of the IBM projects I worked on WebSphere is one of the most fundamental investments we made was automating the CI-CD pipeline in a really cool way. And of course, this was 20 years ago. But I’ll remember that point, like it was yesterday, because our lives change when we achieve this automation at scale. And so I think that that kind of thing can have a huge impact on the projects today. One of things we’ve been talking about as you know with all the AI mania that’s out there these days is one thing to find the defects but how do you fix the defects and actually test them at scale?

So we all know the secret of many software projects is while we have a lot of software out there, testing them in an automated fashion even is quite a challenge for many organizations and many projects. So those people that are able to conquer those leaps across the chasm help with that CI/CD automation, help infuse security, or help create a different perspective of how to automate the test associated with not only remediating but making sure that we’re not breaking everything that’s out there that uses that particular asset. Small thing.

CRob (22:58)
Yeah, very small. So again, let’s pivot back to your conversations with enterprises and leaders broadly. How do we help move the industry from thinking about open source is something that’s consumed to something that is more of your steward? Your participant in this shared infrastructure?

Jamie Thomas (23:21)
Well, I think we have to continue what we’re doing here in this session. Really, we have to continue to provide a lot of education and perspective about what happens when you don’t do that. Right. This is, you know, it’s like, you know, my house, I do wake up every day and I expect the plumbing, the water and electricity to run. But on the other hand, if I’m not a good steward of the basic underpinnings of some of that capability, it might not always be that way. And I think there’s the same can be said about open source.

So when you consume it, you have to consume it, I think, with a recognition that it’s a critical part of your infrastructure investment. You should understand what you’re consuming. should typically I don’t think it should be accidental consumption unless you’re depending on perhaps a packaged app vendor to provide it to you. And then that could be something that you’re unaware of, right? S bombs and things like that are particularly making that more apparent, of course.

But I think if you’re a direct consumer of open source, do it with intent. And intent means that you’re responsible for what that means to your organization, both from a productivity perspective, but also from a security perspective and what you can expect from an investment point as well. One of the things that we’ve learned through the OpenSSF is not every project maintains a healthy state over a period of time. And so that’s why we’ve created this scorecard that gives organizations a perspective of whether the project is healthy.

And I think today, if you’re consuming open source, you need to be using those kind of assets to understand, are you consuming healthy projects? If not, what do you want to do about that? Right? So it’s you have to be an active steward in the maintenance of your open source strategy, just like you would have to be a steward of the maintenance of your own home.

CRob (25:21)
I love that. participating with intent and having a strategy. That’s excellent advice.

Jamie Thomas (25:25)
Yeah, it shouldn’t be an accidental consumption approach. It should be with intent.

CRob (25:36)
Right. Well, let’s move on to the rapid fire part of our talk. I have a couple wacky questions. I would just like the first answer off the top of your head, please. Blue suits or blue jeans?

Jamie Thomas (25:53)
I think both. I have a blue jacket on today. What am I supposed to say?

CRob (25:55)
Both very nice. That was kind of a leading question. Next question, mainframe or microservice?

Jamie Thomas (26:10)
Oh, both. Oh, I’m cheating on this quiz. But you know, I do have the fondness for mainframes. Here’s my little mainframe chip, you know, the chip package here. It’s like a little paperweight, right? This one is z 16. I don’t have z17. But anyway, and then microservices, I think the world does depend on microservices. Very, very important part of the architecture.

CRob (26:33)
And I heard you can run open source microservices on a mainframe.

Jamie Thomas (26:37)
That’s right, that is true.

CRob (26:40)
AI generated or hand coded?

Jamie Thomas (26:44)
Well, I think in today’s environment AI generated is becoming quite prevalent, but I do believe that developers who are standouts are gonna be those individuals that go back in there and use it to their advantage. So I think there will be humans in the loop or humans somewhere in many cases, and it’s up to the human to decide how do I use this cool innovation to my personal advantage.

CRob (27:12)
That’s amazing. And then finally, most importantly, Star Trek or Star Wars?

Jamie Thomas (27:19)
Now I have to confess that I was a Star Trek person growing up, right? I watched those cool Star Trek episodes and everything so I guess I’m more partial to Star Trek.

CRob (27:34)
There are no wrong answers, both are good, but it’s just kind of fun. I love both. I grew up on Star Wars and then Star Trek and then Star Wars changed my life in 77.

Jamie Thomas (27:43)
Well, I do like Princess Leia, of course. Being a woman, Princess Leia was a hero for all of the young ladies out there. Quite good. Good stuff.

CRob (27:53)
Absolutely. Well, and so as we wind down, Jamie, thank you for playing along. Your call to action. So if we have an enterprise that’s only consuming open source today, what’s one thing that you would suggest to them to get them to start participating?

Jamie Thomas (28:12)
I would ask everybody to think about what is your give back strategy?

Because if you’re consuming open source, but you’re not participating in open source projects, maybe with your developers, you’re not participating in an organization in terms of lending your unique perspective of your strategy and how the foundation or the open source project could help you with your strategy. You could do that even through maybe just the use of open source and beta testing, but be an active participant.

Think about consuming with intent. And what does consuming with intent mean for you as an organization?

CRob (28:54)
I love it. Thank you, Jamie Thomas, for participating in Big Thoughts Open Sources. This was a delight. Thank you very much.

Jamie Thomas (29:06)
And thank you very much, CRob. It’s always good to talk with you.

CRob (29:09)
Absolutely – and to those of you listening today, please check out the transcript, we’re gonna have some really great links and you know, stay cyber safe and sound. Bye everybody

Jamie Thomas (29:20)
Bye.

Jun 02

Love0

What’s in the SOSS? Podcast #62 – S3E14 The Ghost in the Dependency Tree: Navigating Open Source End-of-Life with HeroDevs

By OpenSSF Podcast

Summary

In this episode of What’s in the SOSS, host CRob sits down with Isaac Wuest, Product Line Leader at HeroDevs, to explore the critical and often overlooked “gray area” of the software supply chain: End-of-Life (EOL) software. While the industry heavily relies on CVEs to track vulnerabilities, Isaac explains how maintainer abandonment creates a vacuum where risks are present but remain undiscovered and unreported. From the origins of HeroDevs supporting AngularJS to the nuances of the EU Cyber Resilience Act (CRA), this conversation provides a practical framework for distinguishing between inherent hazards and actual risk in your dependency tree.

Listen on Apple Podcasts Listen on Spotify Listen on Overcast Listen on Pocket Casts

Conversation Highlights

00:04 Host CRob welcomes Isaac Wuest from HeroDevs to discuss secure open source ecosystems.
00:45 The HeroDevs origin story: How Google sunsetting AngularJS created a need for secure drop-in replacements.
02:44 Isaac’s path to open source: Transitioning from product management to supporting maintainers.
04:06 Exploring the “Gap” in CVEs: Why dictionary-based vulnerability tracking misses EOL and malicious packages.
07:03 The challenge of “Maintainer Attestation”: Why most open source projects lack a formal EOL calendar.
09:52 Compliance and Risks: How EOL dependencies create blank spots for security professionals and auditors.
11:27 The Shark in the Tank: Using a food regulation analogy to differentiate between hazard and risk.
13:22 Navigating the EU Cyber Resilience Act: Preparing for increased manufacturer accountability in software.
14:08 Maintainer Abandonment: Identifying the moment a project stops receiving patches without formal notice.
16:14 Scanning for Gaps: Why standard industry tools currently struggle to provide a complete EOL picture.
18:49 Practical Remediation: Recommendations for researching upgrade paths using tools like endoflife.date.
20:49 Analyzing SBOMs: How engineers can leverage free datasets to identify and fix deep dependency risks.
23:00 Rapid Fire: Coffee, Star Wars, spicy food, and the favorite apocalyptic robot.
25:01 Final Thoughts: A call to action for educating yourself on your application’s EOL exposure.

Transcript

CRob (00:04.053)
Welcome, welcome, welcome to What’s in the SOSS, the OpenSSF’s podcast where I talk to people that are in, around, and creating this amazing open source ecosystem that we all benefit from. Today, we have a pretty interesting guest with some pretty cool and timely topic. We have Isaac from Hero Devs. Isaac, welcome to the show.

Isaac Wuest (they/them) (00:26.474)
Awesome, thanks for having me. I’m glad to be here.

CRob (00:29.197)
Yeah, so everybody that listens to our podcast may or may not know the HeroDev story. Could you maybe explain a little bit about your company that you’re representing and then maybe give us a little peek into your open source origin story? Like what got you into this space?

Isaac Wuest (they/them) (00:45.694)
Absolutely, absolutely. So the story of Herodevs actually goes back to the story of AngularJS and when Google decided to sunset support for Angular. So Herodevs as a company started a few folks that were on the Angular team, realized that Google was sunsetting support and that there would be breaking changes and people would need a secure version that was supported and being patched in the future.

CRob (00:52.256)
Mmm.

Isaac Wuest (they/them) (01:11.754)
And that was the birth of HeroDevs. HeroDevs as a company has expanded out of AngularJS to many other types of large open source frameworks where we offer secure drop-in replacements for the end-of-life versions. And that was kind of the inception of the company. What we do today is keep those versions secure. That being said, what we’re now doing and kind of moving more into is that end-of-life space more broadly, saying how can we support not just individual frameworks,

But when things go end of life, how can we make sure that there are secure versions available across a deeper range down in the dependency tree for companies?

CRob (01:52.545)
Makes a lot of sense. I just had the opportunity to read the recent Sonotype stated supply chain report. And I think the new number of dependencies for commercial applications is around 1,100 on average. So this is a pretty, pretty big issue, huh?

Isaac Wuest (they/them) (01:59.79)
Mm.

Isaac Wuest (they/them) (02:06.754)
Yep, yep.

Isaac Wuest (they/them) (02:11.178)
It really is. We actually partnered with Sonatype on some sections of that report there came from some of our data. you’re absolutely right. It’s a growing issue, both because the velocity of open source ecosystems, like the rate of new packages and new versions, continues to accelerate. And as you know, more more CVEs are getting reported year over year. So there’s this whole volume problem that is, I’m sure we’ll talk more about aspects of that. yeah.

CRob (02:15.585)
Thanks.

CRob (02:31.965)
yeah.

CRob (02:37.109)
We sure will. But before we get to that, how did you get into open source? What brought you to this meeting today?

Isaac Wuest (they/them) (02:44.744)
Absolutely, absolutely. So my interest in open source goes back about three to four years at this point. So a previous company that I worked at, I was good friends with several developers who were in the open source space. And they said, hey, we think you might really enjoy the space and enjoy learning about open source as a product manager. And I said, absolutely, I’d love to. So I first started working in open source in that environment where you’re working directly with maintainers. So I got to know

several hundred maintainers and was like, this is a really, really robust and a really, I love the ethos of the open source community. So I feel like you kind of get a taste of it and you go, wow, this is for me and I’ve been in the open source space ever since. Not as someone doing the hard work that everyone else does, but as a PM thinking, how can I support and help with all the important work everyone else is doing?

CRob (03:38.525)
What I love about open source is not everybody has to be the developer. There’s a room for a lot of different skills. That’s pretty great. Speaking of some different skills, and this is something that a lot of developers sometimes struggle with, let’s talk about security. Broadly, the ecosystem uses a tool called CVE, which used to stand for common vulnerabilities and exposures.

Isaac Wuest (they/them) (03:46.913)
Absolutely.

CRob (04:06.889)
And that was kind of a dictionary of these are the discovered vulnerabilities and this is how you can go learn more to protect yourself. CVS is an interesting methodology in a program. Had a couple challenges over the last few years. And there are also whole categories of things that lay people may see as a problem. Like your.

the end of life you teased or malicious packages. And these are things that aren’t accounted for in the CVE methodology. So from your perspective, what does that gap in the CVE program, what kind of risks or problems does that, not ignoring, but that not accounting for these other types of problems, what does that incur to consumers of open source?

Isaac Wuest (they/them) (04:59.083)
Yeah, yeah, that’s a really good question. And I will say that the answer to that question is still evolving. Now, my angle, like my perspective on it from the end of life side, is that, know, CVEs, of course, rely on this kind of public disclosure where someone discovers it, someone who’s kind of certified, I think they’re called a CNA, this is someone who’s allowed to report CVEs in the standard way, reports them.

And that’s great. Of course, CVEs have their limitations. So we have these other scores that are evolving, your EPSS and the known exploited and that system. But the predicate for all of those are that the vulnerability has been discovered and reported. Now, the problem there is that as packages or versions within packages in the open source space go end of life, meaning they’re no longer getting

security updates from that upstream maintainer, that upstream community. As that occurs, a second thing is happening as well. They’re not going to get patches or updates for CVEs. In addition, they’re typically no longer investigated for vulnerabilities to begin with. So it creates this, we just don’t even know, right? Maybe it’s vulnerable, maybe it’s not. Oftentimes, the maintainers around those packages don’t have the time to…

CRob (06:10.783)
Right.

Isaac Wuest (they/them) (06:21.557)
retroactively investigate every CVE on their entire version range to even see if it applies or not. So that space, those unknowns, create a real concern that the broader security market and the SDA space is trying to figure out how to react to.

CRob (06:39.937)
So from your experiences, especially dealing with upstream maintainers, how much attention does upstream pay to end of life? Is this something like I know commercial vendors generally will advertise we do support between X and Y dates. So from your experiences upstream, what does end of life look like or how much do they advertise that stuff?

Isaac Wuest (they/them) (07:03.373)
Yeah, it’s a good question. it’s extremely variable. So for large frameworks, if we’re talking about your Angular’s, your Views, your Springs, you can often use tools like endoflife.date or go directly to their sites and you’ll see their schedules. When, these are the versions, this is when it goes, LTS for that long-term support. And then here’s when LTS is dropped and that’s when it goes end of life. The problem is that that covers a very thin slice of the total number.

packages that are in open source. Most maintainers don’t have time to publish a calendar for their specific packages. As a result, it’s often very ambiguous. Unless we’re talking about those big frameworks, those big libraries, outside of that, it’s kind of like the maintainers know when they’re going to generally they’ll stop supporting major release lines once it’s two or three in the past.

When you talk with maintainers, what you learn is that it’s kind of ambiguous even for many of them. They’ll say, well, if a CVE is reported, I might be able to release a security patch. I might not. It kind of depends how busy I am. They’re volunteering their time. And that ambiguity makes it really challenging to know, well, how secure is it? Is it supported? That initial problem of are we investigating and reporting and patching risk on those older versions?

CRob (08:30.645)
And so again, from your experiences, how have developers or projects articulated this or do they at all?

Isaac Wuest (they/them) (08:38.871)
Great question. So apart from those large frameworks where they publish them on their sites, most of the time it’s an array of methodology. So sometimes they’ll post in the readme’s on their projects. You’ll see information about what’s being end of life. Status is like deprecated where in certain registries, NPM and others, can mark, a maintainer can mark and say, this release line or this version is deprecated. That’s a bit of a proxy for end of life.

Sometimes though, I’ve seen Twitter or X, like social media, be a primary mode where the maintainers of packages are saying, this is when I’m gonna be no longer supporting version 1.1.1. And it’s highly variable and that makes it really brittle for any large enterprise to rely on.

CRob (09:28.769)
Mm-hmm. Well, hey, let’s switch our focus a little bit. Let’s move a little bit more downstream. So let’s say I’m a security professional at an organization, or I’m a developer that’s working on some type of internal app that’s leveraging open source components. What does it mean to them when some dependency or package that they’re using becomes end of life?

Isaac Wuest (they/them) (09:52.814)
Yeah, that’s a great question. I will say, firstly, they’re still learning what it means in terms of compliance frameworks. So when something goes into life, there’s the security implications that we just talked about. Hey, it’s not going to get security patches, and it likely won’t get reports of vulnerabilities that may exist on it. Now, if I’m a security professional working at a company that

CRob (10:03.139)
yeah, yeah.

Isaac Wuest (they/them) (10:19.915)
You know, we have some number of applications we’re building that use that dependency. I know immediately that there’s this kind of blank spot there where even if risks aren’t being flagged by, you know, my security scanning tool of choice, I know that there may still be risks present. That might be a problem for that own company’s security posture, where they have their own standards and say, Hey, we can’t rely on that. Oftentimes what I see though, is that

The main motivation is the fact that it puts them out of compliance with these frameworks that exist. Where if you’re dealing with health data, or you’re dealing with credit card data, and you’re subject to HIPAA, high trust, PCI, as a security team, you can’t tolerate dependencies that won’t get updates. That’s too much risk. And that’s created a real vacuum in the space of open source end of life.

CRob (11:17.857)
And so from your perspective, if something is end of life, does that automatically mean that package is a problem? What options do people have?

Isaac Wuest (they/them) (11:27.883)
Yeah, that’s a great question. So there’s an analogy I sometimes draw from in the food regulation space. If you ever get into how FDA or the EU thinks about risk in food. there’s a difference between risk and hazard. Hazard is just anything can kind of, anything could be a hazard. If I go to an aquarium, the fact that there’s a shark in a tank is a hazard.

Oh boy, if something were to happen, I fell in that tank, slipped, However, the risk is quite low. security teams within companies are trying to figure out the balance within the context of end of life. Is end of life is inherently hazardous. How much risk does it represent though? And based on the risk, what should we do? How much effort should we as a company invest? We can tolerate some risk, we can’t tolerate all risk.

And then the question becomes, what are my options if I’m an engineer or security person who I discover I have several end of life packages in my dependency tree for some application? What should I do? The first problem they have is, how risky are these packages? And the fact that they’re end of life means, what are my options for fixing it? Do have to migrate? Can I patch it myself? That becomes the natural.

kind of next questions and problems that these folks are facing.

CRob (13:01.087)
And we’re definitely going to see that in the coming months and years as the EU’s Cyber Resilience Act comes online, because manufacturers are held accountable for all components within their products. And they need to do that risk calculus to understand what am I going to do if there is a critical vulnerability in one these things that doesn’t have support potentially upstream.

Isaac Wuest (they/them) (13:22.626)
Yeah.

You’re absolutely right. The landscape around shipping secure software continues to get more more strict, which I think is a very healthy orientation for the industry. It’s important that we’re responsible as technologists for the code we’re shipping, even if that’s open source code. But that doesn’t mean that it’s not going to be a really challenging problem to solve as the EU cybersecurity

actor or many others continue to gain traction.

CRob (13:58.525)
And with your interactions both up and downstream, are people aware of kind of this end of life opportunity that’s before them?

Isaac Wuest (they/them) (14:08.641)
folks are becoming increasingly aware. There’s actually a distinction now in two different types of end of life that’s been emerging when talking about it. there’s end of life, similar to those large frameworks and the calendars I was mentioning, that’s often categorized as maintain or attested end of life. And that attestation is often the very compliance-y legal term, where the maintainers are attesting that it is end of life or that it is supported.

And if I’m an auditor, I can rely on that when I’m auditing some large company’s project. The category that’s emerging as a term is maintainer abandonment when it comes to end of life. Now there’s other terms you’ll hear that are similar. Really all we mean is there’s a point at which that maintainer can no longer continue to support. It could be a specific kind of major release line. They’ve moved on.

It could be even specific miners that they’re no longer going to be supporting. you think about that SIMVR, that 1.1. Or it could be the entire package gets abandoned. So they rename the package, or the maintainer has experienced burnout, and they’re no longer able to support it. They’re going to need to take a step back. No one’s filling that vacuum or that void. That maintainer abandonment category is proving to be a real opportunity.

to say how do we identify and report on this even when the maintainer doesn’t have a way to clearly tell everyone and broadcast like the big players in open source do the large frameworks.

CRob (15:41.439)
Interesting. Thinking about the tools we have today and end of life, have for good or bad, the CVE program, we have software builds materials, we have a whole fleet of scanners, AI or not. We have bug bounty programs. Now, from your perspective, do these tools and maybe ones that I didn’t mention, do they give a consumer

Kind of that full security picture or are there gaps?

Isaac Wuest (they/them) (16:14.625)
Right now, the gaps are really starting to show. Most of the available kind of industry standard tools, be it SBOMs, Cyclone DX or SPDX, even the biggest kind of formats out there, or most scanning tools as well, they’re still focused on publicly reported vulnerabilities as the top priority. And then they do talk about end of life. Most of the end of life that you see reported,

are those maintainer attested schedules, where your big frameworks tend to be tracked well, which is really important to be clear. Those big frameworks often represent a lot of risk and a lot of pain if you have to make a migration or something like that. Very few tools exist today in the security scanning space that have a robust picture, a complete picture of that maintainer abandonment side. There’s people doing kind of innovative work there trying to

and understand, but it’s a pretty open field space right now even for these large security scanning tools.

CRob (17:19.947)
Well, that’s always been an historic challenge, of that information sharing upstream is, you know, is the project active? When was the last commit? How many stars and like all that kind of normal lifecycle stuff helps, but is not that developer attestation saying on this day, I’m stopping this particular stream.

Isaac Wuest (they/them) (17:32.488)
Absolutely.

Isaac Wuest (they/them) (17:42.722)
Yep, yep, you’re absolutely right. And yeah, that’s been a challenge for some time. It’s also often a challenge because these tools need certainty in order to report on a scan. Like you said, they get that complete picture. They wanna be able to say with confidence, it’s this or it’s that. It’s supported or it’s not supported. And when you talk with maintainers, there’s a lot more gray there where they say, well, I could support it if I need to. If someone tells me I’m willing to try and do them a favor, because open source is so…

CRob (18:06.209)
Mm-hmm.

Isaac Wuest (they/them) (18:11.329)
Everyone’s so willing to pitch in and help each other. And that’s a feature of the space, but it becomes a bug for those consumers, those large enterprises that need certainty when they’re reporting on compliance, alignment, and all that sort of stuff.

CRob (18:15.595)
There it is, yep.

CRob (18:26.721)
So I’m thinking about this from the developer perspective and end of life. Are there any practical steps that you would recommend a project to consider or do to help be a little bit more communicative around this to deflect a lot of downstream questions? You know, ideally.

Isaac Wuest (they/them) (18:49.335)
Yeah, yeah, absolutely. There are some practical things that are available today. And then there’s some opportunities where I’m really hoping as an industry, security scanning focuses on solving this problem more broadly. Today, though, I the way it’s typically done when I talk with enterprises and work with developers on the front lines is they usually get a notice that something has a CVE on it. And then,

I mean, I’m sure you know well that they end up going and researching. You grab that package name and the version, and you go to Google, and you start looking at the registries. And you say, is this supported? If there’s not an obvious patch version I can just upgrade to, Like oftentimes, engineers stumble upon the reality of it being end of life if it’s not that large framework where there’s a clear calendar. That’s, of course, painful because it means

Do I need to migrate? What do I have to find a replacement, et cetera? So my recommendation today is there’s tools like endoflife.date. That’s a great tool, site and project out there. There’s other ways to go research. However, there are some free tools available that you can load an SBOM or a manifest into. There’s one that we’ve been working on called the eoldataset.com that you can.

Loads free tool that has a large database detecting maintainer abandonment. I’m hopeful many other organizations work on this problem so that engineers don’t have to spend far too much time researching everything one off and get some confident data about what is almost certainly end of life and what is likely supported even for that long tail of packages.

CRob (20:31.329)
That’s good advice. if from your perspective, how accessible, how easy are these systems to kind of get in there and get the data you need so you can make your risk decision? Do I want to continue with this particular branch or not?

Isaac Wuest (they/them) (20:49.293)
Yeah, so if for existing SEA tools, most of them, of course, they’re going to tell you about that maintainer attested. And if you load in your SBOM, there’s usually some other version or remediation path where they say, hey, here’s what you can do about it. There is a supported version. And at that point, it becomes a question of how quickly do you need to migrate? Or can you get some sort of support right now? For the other side of it.

when it comes to maintainer abandonment. I would definitely recommend using the tool I just mentioned where you can load in that SBOM. And once you get that full picture of, here’s all those other packages that are end of life or likely end of life, the next step becomes, OK, well, are there supported versions that I can upgrade or migrate to? Most of them there are. Then it becomes a question of, of course, breaking changes.

How challenging is that? Is it deep in the dependency graph such that the version is constrained and it becomes this kind of gotcha problem? And oftentimes, even knowing something is end of life, that’s not enough for engineers. They need to know what are their upgrade paths if they have to fix that one package, that one version. Our tool that I mentioned that’s free to use has some of those, but many others.

do as well. Your SEA tools often do contain migration options and engineers can often use those tools. They’ll load in data sets like ours or others about end-of-life data that then connects to the dependency graphs and says here are your migration options. You’d have to upgrade these two direct dependencies. That gets you into the versions you need that are no longer end-of-life deeper down.

CRob (22:42.657)
This is an amazing topic that I think not everyone is aware of it as they should be. So thank you for joining us today and kind of sharing a little bit about end of life. But let’s move on to the rapid fire part of our talk.

Isaac Wuest (they/them) (22:57.357)
you

CRob (23:00.405)
That’s spicy. I have a handful of questions that we’ll ask you. Just give me the first thing that comes off the top of your head. What’s your favorite beverage?

Isaac Wuest (they/them) (23:09.441)
Got it.

Ooh, I’m gonna have to go with a cappuccino. I love coffee, but I love espresso, so give me that. Give me that right there. Yes, I need that caffeine.

CRob (23:14.593)
Ooooo

CRob (23:20.193)
Yum. All right. Star Trek or Star Wars?

Isaac Wuest (they/them) (23:24.885)
I’m so sorry, but it’s gonna be Star Wars for me. It’s gonna be Star Wars. Okay, good, good, good. Well, I appreciate Star Trek. I certainly love it, but Star Wars, there’s just something about it that speaks to me.

CRob (23:27.723)
There are no wrong answers, but Star Wars is a pretty good one.

Well, do you prefer spicy or normal food?

Isaac Wuest (they/them) (23:42.024)
I love spicy. Please.

CRob (23:43.751)
ooo that’s spicy

Isaac Wuest (they/them) (23:47.853)
Yeah, I don’t understand people that don’t like spicy food. It’s just more exciting. It’s more interesting.

CRob (23:49.249)
Thank

Right? Exactly. And then, you being that we are in the a new age here, who’s your favorite apocalyptic robot?

Isaac Wuest (they/them) (24:06.029)
Ooh, my favorite apocalyptic. is this, ooh, is this existing AIs or is this fictional robots? my goodness. I think I could have one of each. could have one of each. Okay, for fictional, I’ve been reading all these dystopian books that involve robots. I love Heinlein’s The Moon is a Harsh Mistress. There’s an AI there, Mike or Michelle, not a robot that causes an apocalypse, but one that kind of helps solve kind of a crisis.

CRob (24:13.622)
Sure.

You could have one of each.

Isaac Wuest (they/them) (24:35.373)
So that’d be my favorite fictional. Ooh, for existing, I don’t know. There’s a number of AI models I’m pretty scared of right now. We’ll see. And I don’t want to badmouth any, because maybe in two or three years, I’ll be hoping that they take care of me.

CRob (24:35.489)
Nice.

CRob (24:43.297)
You

CRob (24:50.241)
That’s our dream. Well, thank you for playing along, Isaac, being a good sport. And as we wrap up, do you have any closing thoughts or a call to action for our listeners?

Isaac Wuest (they/them) (25:01.161)
Absolutely. Well, thank you so much for bringing me on. I really appreciated the conversation. This was really exciting and interesting. What I would definitely say is, for anyone listening, if you’re worried about End of Life and the potential exposure that any of your applications have, feel free to check out what we’ve been working on. We have a free tool with a data set called eoldataset.com. And we’re looking just to give away as much data that we have about the End of Life space. So if this is something that was interesting to you and you want tolearn more about what might be end of life, or you want to contribute thoughts and ideas about how to improve the data set, please let us know. We’re very interested. yeah, with that, that’s my main…

CRob (25:43.189)
Well, Isaac from HeroDevs, thank you for joining us today. And I want everyone to kind of listen to this, maybe take some action, do a little research to educate yourselves. And I’d like everyone to stay cyber safe and sound out there. Happy open source and folks. That’s a wrap.

May 19

Love0

What’s in the SOSS? Podcast #61 – S3E13 Beginner to Builder: Shaping the Conversation in Open Source Security

By OpenSSF Podcast

Summary

In this episode of What’s in the SOSS, Yesenia Yser interviews cybersecurity analyst Ejiro Oghenekome about her journey from UI/UX design to becoming a key contributor to the OpenSSF. Ejiro shares the inspiration behind her public “100 Days of Cybersecurity” challenge, which has helped her maintain discipline and consistency while making the field less intimidating for beginners. She discusses how connecting with the OpenSSF community led her to the BEAR Working Group, where her authorship of the “Beginner to Builder” blog series has allowed her to move from consuming content to actively shaping the open source security conversation. Ejiro also offers advice to the next generation, emphasizing that open source contribution is not just about coding but is a welcoming space for anyone to learn and grow, regardless of their current expertise.

Listen on Apple Podcasts Listen on Spotify Listen on Overcast Listen on Pocket Casts

Conversation Highlights

00:00 – Music, Promo clip, & Welcome
01:11 – Ejiro details her transition from UI/UX design to cybersecurity and connecting with OpenSSF.
03:39 – Ejiro explains her motivation for starting the 100-day challenge, including receiving advice to learn publicly and a previous rejection from an internship.
06:49 – Ejiro shares that she is currently on day 44 and expects to complete the challenge around April.
07:50 – Ejiro discusses her biggest personal lesson: understanding consistency and discipline, and learning from the community.
10:45 – Ejiro describes her authorship of the “Beginner to Builder” blog series, which shifted her from consuming content to shaping the open source conversation.
15:47 – Ejiro shares the impact of her work, noting that it has made cybersecurity feel less intimidating for beginners and helped her grow in confidence.
18:22 – Rapid Fire Questions: Ejiro shares her preferences on books, cooking, social media, and more.
21:13 – Ejiro offers advice to the next generation, emphasizing that open source is welcoming, not just about coding, and provides great opportunities for learning and growth.
24:46 – Yesenia concludes the interview, thanking Ejiro for her time and contributions

Episode Links

Ejiro (Sonia) Oghenekome LinkedIn page
Ejiro’s GitHub page
BEAR Working Group
Ejiro’s OpenSSF Beginner to Builder Blog Series:
Get involved with the OpenSSF
Subscribe to the OpenSSF newsletter
Follow the OpenSSF on LinkedIn

Transcript

Intro Music (00:00:00)

Ejiro Oghenekome (00:01.366)
So I have embarked on a 100-day cybersecurity challenge where I post whatever I learn about cybersecurity in the open. I posted both on LinkedIn and on Twitter, currently known as X. I was told to learn publicly. It has really helped me to stay consistent and it has also helped me to stay disciplined.

Yesenia Yser (00:23.662)
Hello and welcome to What’s in the SOSS, OpenSSF’s podcast where we talk to interesting people throughout the open source ecosystem, sharing their journey, experience, and wisdom. So Yesenia, one of your hosts, and today I have the utmost pleasure of interviewing Jiro, who has been such a great part of the open source community and has done a lot for us already from part of the FAIR program.

writing a few blogs that we have seen out in the wild, and much more. don’t want to share details of the upcoming podcasts, but welcome, Ejiro. Please, you know, let’s start with you for listeners that are meeting you for the first time. Can you introduce yourself and share your journey into open source cybersecurity? Like what really pulled you into this space?

Ejiro Oghenekome (01:11.822)
Thank you very much for having me here. Hello everyone. My name is Ejiro Oghenekome. I’m a cybersecurity analyst. Currently I am contributing to the OpenSSA. So for how I got into this space and where I am at right now, I’d like to give a little backstory on myself so that I would better understand how I got to this particular phase of my career. I used to be a UI UX designer a couple of years.

But I think about 2024, I started to like not see myself doing UI UX in a long time. And as at that point, I was already interested in security. I was already curious to know how data is secured and a lot of other things about security. So I decided to dive in and take it as a career to learn about security. And the first course I took was the Google Cyber Security Certification course on

Coursera and it was a very interesting course. I took that. had other little courses that I took some on YouTube and other very, very not so prominent courses that I took that helped my career helped shape my career going forward. And something I didn’t have, I didn’t mention was the fact that I’ve always known about open source, even during my UI UX design time, but I really did not partake in open source contribution as at that point.

But I really, did not want my cybersecurity journey to be that way. So I was looking for every means to get into this space, to try to contribute to open source with my cybersecurity career. So fortunately for me during that time, I think about 2025, met a friend, told me, or I saw a post from a friend where she had an interview with someone that was talking about open source and open source security. I found that very interesting and I reached out to her.

I like to connect to the person so that they would share more light on contributing to open source, especially with my focus, which is cyber security. And she actually did that to me. She actually did that for me. And I connected with the person. The person was Sal. I a couple of meetings with Sal and she got to know where I was in my career, which led her to introduce me to the OpenSSF. And yeah, I am today trying to contribute to the OpenSSF in whichever way I can.

Yesenia Yser (03:39.854)
That’s such a great story how one friend, one webinar connected you to one individual that opened up the space of open source and it’s brought you to where you’re at today. Such a great story to hear. And one of the little birdies in the open source told me that they gave you a hundred days of cybersecurity challenge that you’ve been publicly documenting on LinkedIn. Like what inspired you to start that journey and

What do you hope would come from it?

Ejiro Oghenekome (04:10.67)
So I have embarked on a 100-day cybersecurity challenge where I post whatever I learn about cybersecurity in the open. I posted both on LinkedIn and on X. Twitter, currently known as X. So my journey can be documented. What’s made me do this was the advice I got from friends and loved ones. I was told to learn publicly. And that really has shaped me over time coming out.

because looking back, it has really helped me to stay consistent and it has also helped me to stay disciplined in terms that I feel so indebted to the cause of posting because I’ve seen a lot of people grow interest in what I have posted about my career, everything I do about cyber security. It has really been an interesting journey for me. Also, another reason why I embarked on the 100 day of cyber security challenge was because

I would say I got a rejection from an internship. I really did not get a feedback from them. So I would know, I don’t know if I should say that’s a rejection, but technically it is because I didn’t get the feedback. I really wanted to get a practical knowledge of what I was already learning. I’ve learned for a while and I wanted to get into the practical space. I wanted to get into the real world space to practice what I have been learning. Applied for the internship. Unfortunately.

I did not get it, so I had to take some step back and make a curriculum for myself where maybe I would be able to create something that feels really practical. The internship I applied for was a three-month internship, which is 90 days, if I would say technically. So I just had to do it, 90 days for an internship that I did not get. So I had to make it a 100-day for myself. Looking at all I have done, what I hope

to come out from this, which I am already saying is for people to know me for what I do. For people to know me for open source, for people to know me for open source security, for people to know me for cybersecurity, and for people to know me for preaching and being an enthusiast of open source. People should come into the open source space to contribute to open source and see the opportunity it comes with. And people to also know that

Ejiro Oghenekome (06:34.734)
I’m a cyber security analyst and I also give best practices. I also give basic knowledge of what cyber security is and all of that. Yeah, that is what I hope to get my 100 day challenge. And it has really been turning out well for me.

Yesenia Yser (06:49.836)
I love that because getting online and really just sharing what you’re learning, you know, on a cadence, whatever cadence that is for you is such an important way just for your own accountability and for others to connect to you, connect with you and learn what you’re learning, especially if you’re looking for jobs. I’m just curious right now it’s, you know, mid February. What day are you in for this challenge?

Ejiro Oghenekome (07:13.774)
Yeah, I think I’m in day 44. Nice. Yeah, day 44. It’s been a great journey. Yeah.

Yesenia Yser (07:22.574)
24 days. So when do you envision, it’s 100 days, so when do you envision this challenge ending?

Ejiro Oghenekome (07:29.39)
Let me try to do a rough calculation right now in my head. So we have a couple of these. So let’s see the beginning of around April. I’m not sure the dates for April. I’m not going to give an exact estimate, but yeah, by April I should be done with the 100 day cybersecurity challenge.

Yesenia Yser (07:50.85)
Very nice. Okay, so we’ll keep watching. you’re deep into this challenge. 44 days is a great time because that’s built in that habit to get it done and share out what you’ve learned. But I’m curious, what’s your biggest lesson that you’ve learned so far? And not just like technically, but like personally, like how have you changed how you see your learning, your discipline, or just like your community growth?

Ejiro Oghenekome (08:17.614)
Okay, that’s a very interesting question. I would really say if I’m going to put it short, I’ll say it has not been an easy journey. It’s not been easy because it’s not easy to stay consistent and trying to like, remodel my expectations of what I have to post, what I have to do. It is not easy. I’ve come to see that everything cannot go on the same pace every day.

I’ve had to stay consistent. I’ve had to understand what consistency and discipline means. I’ve come to get that. Consistency does not mean I have to be in the same place every day. I do not do the same thing every day. Some days I might not even feel motivated to want to partake in that particular challenge for that day. But have to stay disciplined. I have to stay consistent, which might make me cover less than what I covered the previous day.

Other days I might feel so motivated that I might cover more every other thing I’ve covered in the past. It just happens. One thing I’ve learned is staying consistent, what consistent really mean, being very disciplined in the space. Also it has given me a very good routine. In terms of community, I’ve come to understand that community is where I learn.

This learning can come by interacting with projects and interacting with people in the community that have more experience than I do. During my 100-day challenge, I’ve been able to have the opportunity to be part of the OpenSSF. This has gone hand in hand with the 100-day challenge that I’m doing. For the fact that I’m part of the OpenSSF and doing my 100-day challenge, I’ve seen the impact that the OpenSSF community has had on me. I’ll give a very simple example.

We had a blog post or we had a blog post that talks about a lot of things that we might go over eventually. Because of one research I did for one part of one series of the blog post, I a course on the OpenSSF and the Linux Foundation Education that I took in and I benefited from. That is the LFD121, that is developing secure software.

Ejiro Oghenekome (10:34.274)
want to understand from this journey that I’ve taken that I could learn from community, I could learn from interacting with people, want to understand what consistency and discipline mean.

Yesenia Yser (10:45.678)
That’s awesome. Yeah, I when you started being involved in the Bear Working Group, you and Saul were working on a blog series and I you just lightly mentioned it. It’s beginner to builder. What has that experience been like moving from learning to actually contributing publicly? And you know, this blog, it’s a big deal. It’s a three series blog. Like what does this authorship mean to you in the aspect of open source?

Ejiro Oghenekome (11:13.208)
Again, I would try to give an example to put what I have learned and how contributing to open source has been to me. During my design career, I’ve always known about open source, but I was not involved in open source contributions because as I did, I would say I did not know where to start. I did not know what to do. I did not know how to get into the space. I also felt most of the times that I did not know enough to be able to partake in open source contributions.

all of that and that’s feeling of mine is something I feel like a lot of other persons also do have. It was a problem for me and that problem I felt that a part of the blog post was able to solve it. One thing about me is if I experience a particular challenge or problem going forward in my career I try my best to solve it so that when people come behind me and they experience such problems they would not find it difficult to solve because they are

cases or maybe they are documentations that will help them go through this problem. That is one thing that I’ve been able to do with the blog posts and that is how, that is why the blog post publication was made public. And for me, Autorship in open source is more than just putting my name on the blog post or making contribution. It represents ownership of my learning and my voice. When I started my cyber security,

I was mostly consuming content. was reading documentations, watching tutorials and following experts. questions, I did all of that. But authorship changed the dynamics of everything for me. It shifted me from being just someone that consumed information to someone that is actively shaping the conversation, even if it was in small ways. Authorship made me feel responsible. I know that something I am going to write

is going to be published and the knowledge I share is going to be put out in the ecosystem. It would make me more focused. It would make me more thoughtful. It would make me more intentional about what I’m going to post. And this led me to call back, questions, ask people from the community to give me feedback on the blog post I wrote. I think you must have experienced that because

Ejiro Oghenekome (13:35.916)
During the first part, the second part and the third part, we were always very intentional to make sure that we got feedback from the community so that the best resources can be put out there to solve the actual problem we saw that we wanted to solve. And also, Authorship for me means visibility. As someone from Nigeria and someone who transitioned from design to cyber security, Autorship allows me to exist in a public space where people like me

are not highly represented. It shows that contributions does not have to fit a specific style or character. Also, it also makes me confident. It means that I am no longer waiting until I know everything before I can speak and before I can contribute in the open source space. Comfortable contributing while I am learning.

This is very powerful in the open source space because the open source space does not work with one person’s perfection, but it works with individuals putting together their efforts and their knowledge to try to make things work. The bear walking group and generally the OpenSSF community has really been helpful in this part. I’ve been encouraging, they’ve been friendly and they have pushed me to understand things. They have guided me each step of the way.

to understand what I am doing so that whatever resources we put out there will be the best quality for people that are going to have that.

Yesenia Yser (15:08.448)
It reminds me a lot of when I started, like just grabbing whatever kind of resources I could find and just learning. And when I realized that I was able to use my voice or my penmanship, so to speak, to share out information, I realized the power and the impact that I can have, you know, just not for my own credibility, but also you never know who’s going to read it down the line. Like I have articles that I wrote years ago or that I published that people still reference.

Nowadays that they’re like, this was an amazing article that you wrote. learned so much. So big kudos to you for that.

Ejiro Oghenekome (15:45.55)
Thank you.

Yesenia Yser (15:47.17)
Before we get into the rapid fire, I would love to know what impact you’ve seen in the community, either from your 100 day posts or your bear working group, like the work you’ve done with the blog. I know you mentioned a bit in this session, but I would love to learn, know a little bit more of looking at your journey so far. Like what impact have you seen?

Ejiro Oghenekome (16:08.634)
Genuinely speaking, I really did not think about impact when I started all of this. When I started my 100-day challenge, I was not thinking about the impact it was going to have on anyone. I just wanted to learn. When I started contributing to open source, I just wanted to learn. But over time, I started to notice little impact on people. I saw that for my 100-day challenge, people would message me saying things like, they started learning because they were following my post.

Some people asked me questions on the tools that I use and if I will be able to share resources with them. Other people said that made cyber security feel less intimidating because of course, a lot of cyber security posts we see online are from experts that would tell us in cyber security knowledge and try to express things in very technical terms for us, which could be very intimidating for beginners.

for people that are beginners that could relate to what I was saying, that could relate to very basic things in cyber security. It really felt nice. It really felt welcoming. It gave them confidence to say, okay, I could learn this. I could start somewhere. I could get some of this knowledge and get to that point of expertise where I would be able to have this opposed, intimidating knowledge also to myself. Also talking about the community.

The impact has been slightly different. I’ve been able to be part of so many decision-making. I’ve connected with experts that are very kind and friendly, and they want to see me grow. From publishing the blog series, this has made me more aware of my words, that my words could guide people that are just starting up, and this makes me feel so happy. I’m growing in the community in terms of

confidence and experience and also in transferable skills, in terms of receiving feedbacks and all of that growing. And I see that when resources are put out there, it’s really encouraging to me.

Yesenia Yser (18:22.126)
That’s awesome to hear the impacts from, I think he started maybe like a year or so ago into the organization. So it’s great to see and hear what has happened within a year.

So let’s go ahead and move on to the rapid part of the interviews. You gotta have fun with some of these parts. So I’m gonna ask you a series of this or that kind of questions or what’s your favorite X and then you just go ahead and respond. So first question, books or podcasts?

Ejiro Oghenekome (18:38.776)
FIRE!

Ejiro Oghenekome (19:02.117)
I don’t really like reading. I just have to read because I need to get those informations in my head.

Yesenia Yser (19:08.258)
Yeah, I get that. A favorite off-computer activity.

Ejiro Oghenekome (19:18.51)
enjoy cooking a lot. Yeah, enjoy cooking a lot.

Yesenia Yser (19:22.158)
What’s that one meal you cook often that you enjoy?

Ejiro Oghenekome (19:27.926)
I know if you would know, but I cook fried rice. I like seafood a lot. So I cook fried rice, prawns, salad.

That’s my favorite meal. That’s my favorite. Maybe one will see one of these days, I’ll make it and you will definitely testify to its greatness.

Yesenia Yser (19:48.086)
am ready for that. Next question. Best way to grow a project. it social media, conferences, or contributors?

Ejiro Oghenekome (19:57.44)
is social media yeah if I’m going to be very honest social media can do that

Yesenia Yser (20:03.726)
I feel like social media drives the other two. Next question, sweet or sour?

Ejiro Oghenekome (20:12.014)
No, sir, I don’t like sweets like that.

Yesenia Yser (20:16.366)
We had a quick, quick, quick change there.

Ejiro Oghenekome (20:19.575)
I just had to think about suits, so I really didn’t like suits.

Yesenia Yser (20:25.166)
I know we’re meeting early morning for you, so are you an early bird or a night owl?

Ejiro Oghenekome (20:32.386)
I I’m an early bird. I really do think I’m an early bird because I wake up very early and do things. I’m an early bird and I try to sleep very early.

Yesenia Yser (20:41.77)
I’m the opposite. just, at night I’m like a week. It’s so strange.

Ejiro Oghenekome (20:46.894)
I’m really not sleeping lot. So I just try to sleep at night. I stay awake very early in the morning. I get up very early in the morning and try to go on my day.

Yesenia Yser (20:57.464)
Yeah, I’ve adapted myself to it, but naturally I could stay up all night and sleep all day. Last question is your favorite treat or dessert?

Ejiro Oghenekome (21:10.702)
I’d say cakes.

Yesenia Yser (21:13.422)
That’s a good answer. There you had it. The rapid fire interview questions focused on food. So as we wrap things up, any advice for the next generation entering tech or security? What advice would you give them about using open source as a way to launch pad their career?

Ejiro Oghenekome (21:36.494)
Okay, well, I’ll give a disclaimer. would say I’m still part of the next generation. So whatever advice I’m going to say, I’m giving that to myself also. This is something I would have told myself earlier on in my career during design. Try to understand open source and the opportunity it provides. Also, open source is not just about coding. There are different things that someone can do in the open source space.

As a designer, could contribute to the open source space. As a writer, you could contribute to the open source space. As a community manager, you could contribute to the open source space. Obviously, very obvious ones. You could write codes. You could review codes. And you could do a whole lot of other things. Even joining calls, giving your suggestions on calls and decision making during the call is also a way to be part of the open source space.

get involved in the open source space. has a lot of opportunities for people. It’s a very welcoming space. I can testify to that from the community I am part of. It’s a very lovely community with lovely people. The OpenSSF has been a great space for me to learn and grow. And I strongly believe that this is how most, if not all of the open source communities are.

It’s a place where you can learn. It’s a place where you can build your confidence. It’s a place where you can grow. also open source is not about you being an expert. are with the knowledge you have. You could be part of an open source space. You could be part of, you could contribute into the open source. So commonly try to understand open source. is not as difficult as it might look from the outside. Trust me. in, learn.

be part of it and contribute. And I promise you it’s a very welcoming space to be part of. And talking about open source and advice I’ll give to people, have an article coming up that will be talking about contributing into the open source space generally. How to work for communities that you could contribute to, how to understand the communities, and maybe how to make it a first time contribution in a community.

Ejiro Oghenekome (23:54.668)
that you’re contributing to. This is not going to just be specifically about the open access, but open up source generally, how to be part of the space, how to try to understand the space and get into the space. Something else I would have, I would love to talk about is the opportunity for open source for us in Africa. I really don’t know that we, the idea of open source is not so widespread in Africa. That is why it has to be preached. It has to be introduced to a lot of people.

And I would love us to consider that, to try to make sure we introduce people in Africa to open source and the benefits it has on us, what it can do to us and the privileges it can give to us. Yes, that is the advice I would give to the next generation, also myself, the open source space.

Yesenia Yser (24:46.478)
Thank you so much for your time today, your impact, your contributions. I love that you have another article coming out to help those, know, explore the different open source communities and how to search. Thank you so much for everything you do within our community and all the hard work you’re putting together. I really appreciate your time and to our listeners, reach out to Jiro. She’s doing great work. Find her on LinkedIn and keep tracking on that 100 day challenge. Thank you so much everyone and we’ll catch you on the episode.

May 05

Love0

What’s in the SOSS? Podcast #60 – S3E12 Packaging, Transferring, and Deploying Software in Air-Gapped Environments with Zarf

By Jeff Diecks Podcast

Summary

Host Sally Cooper is joined by Brandt Keller, a Staff Software Engineer at Defense Unicorns and Maintainer of the OpenSSF Sandbox Project, Zarf. Brandt discusses Zarf’s origins as a tool designed to reliably package, transfer, and deploy software components (like container images and Helm charts) specifically for critical, air-gapped environments that lack internet connectivity. The conversation explores Zarf’s evolution, highlighting its current role in introducing security gates, improving transparency, and consolidating various management and SBOM tools into a single, declarative workflow. Finally, Brandt explains how Zarf’s declarative manifest model is helping to secure open source software by reducing the cognitive burden on maintainers and giving integrators confidence in upstream artifacts.

Listen on Apple Podcasts Listen on Spotify Listen on Overcast Listen on Pocket Casts

Conversation Highlights

00:01: Welcome and Introduction to Brandt Keller and Defense Unicorns
02:01: What is Zarf and its history: Solving the air-gapped use case
04:33: Zarf’s critical function today: Security, transparency, and packaging
09:18: How Zarf has evolved: From niche tool to agnostic distribution and GitOps integration
12:07: Zarf’s role in OpenSSF and securing open source software
16:05: Rapid Fire and Call to Action (Zarf.dev)

Episode Links

Transcript

Sally Cooper (00:01.748)
Hello, hello, and welcome to What’s in the SOSS, where we talk to amazing people that make up the open source ecosystem. These are engineers, developers, maintainers, researchers, and all manner of contributors that make open source so great. I’m Sally, and today I’m really excited to be joined by Brandt Keller from Defense Unicorns. Brandt, thank you so much for being here.

And to get us started, can you tell our listeners a little bit about yourself, your role, and the kind of problems you focus on at Defense Unicorns?

Brandt Keller (00:35.742)
Absolutely. Thanks for having me. so yes, I’m Brandt Keller, primarily, a Staff Software Engineer at Defense Unicorns where I get to, know, I have the privilege of getting to focus on open source software, and maintaining that across both the open source software, projects that we have created as well as kind of the intersection of all of the things that we depend on as a company, we want to be able to be, you know,

appropriate stewards for not only consuming, but also trying to, you know, identify how we can contribute back. And so my role in particular is that of a maintainer for Zarf, which is an OpenSSF Sandbox Project. And outside of that, trying to be more of a, you know, kind of advocate in a few different spaces.

The software supply chain integrity group, working group under the OpenSSF. I try to be a contributor there, in the CNCF spaces. also contribute to, the security and compliance technical advisory group as a technical lead. And so kind of, you know, broad span, but have the definitive privilege of getting to kind of work with communities, build communities, and build technology.

Sally Cooper (02:01.972)
Oh, wow. This is so exciting to have you on the show, especially to talk about Zarf. So you mentioned Zarf. And I was just wondering if you could tell our listeners, I’m sure many of them know what Zarf is. But for those who don’t, what is Zarf and what’s its history?

Brandt Keller (02:19.02)
Yeah, for Zarf in particular, it’s, it’s evolved for sure. But I think kind of at the, at its cusp, it’s always been about trying to take this process of like, where does our software come from? and that the answer to that is varying and nuanced for everyone. but wherever it comes from, let’s try to find a way that we can package it up in a way that’s reliable and repeatable to make it secure. Ultimately, ultimately we really want to lean into that security posture.

and so what happens is software comes from many different places. people are running Kubernetes everywhere. and they have to collect all the disparate puzzle pieces in order for things to run. They have to collect their container images, their helm charts, every other file that they need to run an application. Maybe it’s their application. Maybe it’s an open source application, that their environment relies on. and we want to make that.

As easy as possible in such a way that it’s, you know, very transparent. You have everything you need. And when I say that, you know, you can maybe get to your environment and find out that you missed a piece and maybe that’s okay. But for Zarf in particular, we’ve always built for the air gapped use case, the most critical environments that don’t have connectivity back to any upstream internet connection. they have a pretty big problem here where there is no reaching back and grabbing that thing. You have to go back out and bring it back in. And that’s, been a lot of consternation for people who operate in these environments. so. Zarf really wants to make it. Let’s let’s package all that up into a single archive and make it so it’s easily transferable. It’s repeatable. So if somebody wants to take that same set of applications.

They can package it up and it’s declarative. The manifest really supports this. and in the end we have made it so that it’s, it’s a lot more repeatable to deploy to air-gapped environments where typically that has always been a lot of a juggling of many different artifacts and many different problems.

Sally Cooper (04:33.275)
Wow, yeah, that context is really helpful, especially for understanding where ZARF came from and what it was originally designed to solve. And from there, it kind of naturally leads to how ZARF is applied today. I was wondering if you could differentiate when trying to solve distinct critical environment problems.

Brandt Keller (04:53.442)
I think in the way that it’s kind of like well-postured to help people and its critical function today is to be more than its sole collection and transfer processes. We want to grab the things, we want to make sure that we have everything we need because we can’t easily reach back out. We have to go back out to the internet.

Grab more things if we don’t have them. so we want to make that process as easy to use. want to be able to kind of put those security constraints on the connected side so that we don’t bring anything that we shouldn’t bring with us to an environment that is, let’s say security critical. If you’re operating in those environments in every single piece of the, you know, software puzzle is scrutinized. we want to be very careful that nothing accidentally.

Sally Cooper (05:40.496)
Yeah.

Brandt Keller (05:51.936)
malicious or otherwise makes it into those environments. so there’s a, there’s a, you know, a wide opportunity here to introduce kind of like some security gates. how can we ensure that it’s going to be functional when it gets to the critical or secured environment? This air gapped environment in particular, is, know, first and foremost, where we, want to, you know, kind of help the community prosper. but on the outside of that is, one transparency.

If you’re pulling artifacts, where did you pull them from? What are they comprised of? And again, for those who may be operating this space, you can see, well, it’s like, well, I’m doing all those things today and that’s wonderful. but you’re probably doing them with a variety of tools. You have your, you know, container image management tool. have your Helm chart management tools. have your SBOM tooling that will then scan your.

Software artifacts. So you know what they’re comprised of know what that inventory looks like. And for Zarf, we really wanted to wrap all that up and more into a single process. You create this manifest. And after that manifest is created, you do as our package create, it’s going to grab all that stuff. We’re going to be creating SBOMS on the fly for those artifacts and putting them in the package so that again, we transfer this.

We still see things burned to CDs, as well as, that may seem. And, we want to make it so that when that artifact is in the environment, it’s very, it’s very transparent. You can look at it and be what, what am I about to install? and so there’s a, I think that’s the first layer of Zarf, which is what’s package and transfer it. there are other tools that do this and that’s wonderful. We like to kind of collaborate on solving this problem.

How do we make these artifacts more portable? but then Zarf’s kind of, you know, second, would say superpower is really to enable deployment of software. so how do you know what you deployed? Where did, where did, what are its pieces? how can you work with those pieces and kind of version control them so that the kind of parts and pieces can, you know,

be sustainable and sustainability is kind of a really big part of what we want to solve. but on the outside of that, how do we do all of that without that air gapped environment in particular, having to juggle a lot of disparate or possibly insecure infrastructure. in particular, the, problem that we most often see is that when you want to operate in an air, air gapped environment, you have to bring all this other infrastructure with you.

You have to kind of stand up a registry to pull images from and more often than not, those aren’t secured with TLS because we just want to get it up and running. We want to see that it works. and so for Zarf, we really took a step back and said, how can we bring the, everything we need for the environment to operate in such a way that we’re not going to be left with these disparate puzzle pieces of, infrastructure running that could be potentially insecure. That could be hard to maintain, hard to manage and again, leaning back into ultimately not very sustainable.

Sally Cooper (09:18.49)
One thing that stands out about Zarf is that as more teams use it in real world conditions, the project itself has continued to mature. I’m just wondering in your opinion, how has Zarf evolved?

Brandt Keller (09:34.062)
I think that it’s evolved in a variety of ways, um, kind of on the, on the cusp of in the kind of the very early days, was what’s all of a very niche use case to some. And we say that in such a way that it was like, it was very much a single, uh, Kubernetes distribution, um, air gap tool. And on its onset since then we’ve kind of seen that it really does.

prosper when we look into how can this be a, you know, a tool that is very agnostic of this underlying Kubernetes distribution model, right? Now we have cloud vendors that we want to operate with air gap in the cloud. Maybe not something people always think about, but it is very prevalent. firewalls as kind of the most basic use case of kind of isolating.

cloud infrastructure from the rest of the internet or the cloud. But then we’ve got, you know, the onset of many different new Kubernetes distributions being released, many different problems to solve where you just want to be able to transfer files and maintain, maintain state on those. And so we kind of see the evolution of, you know, taking what we’ve always wanted to solve, which is to make the transfer and deployment process easier, and then integrating the rest of the ecosystem around that.

people came to us and were requesting, you how do we integrate the GitOps ecosystem with Zarf? And how do we make it so that, you know, the mutation model, which I won’t go into unless people really want to hear about it, but how do we make it so that when some of the underlying magic is happening, that, you know, hey, that’s a really great thing, but that’s a cool pattern. Could we apply it to, let’s say, GitOps and Git?

And, you know, kind of stepping back and being like, yes, yes, we can, we can make it so that rather than it being, rather than it always being a process in the air gap where you have to transfer the artifacts and then change a bunch of references so that they point to the right place in your new environment. What’s, what’s kind of handle that for the user, let’s make it more of a, you know, nice and consolidated user experience. so it’s evolved in that way, as well as, know, kind of.

as it relates to the OpenSSF, trying to make open source software more secure.

Sally Cooper (12:21)
Love that. the mutation model, I do think we’re going to need a part two for that because now I’m super curious. But with the evolution, it’s especially interesting when you zoom out and you think about software supply chain security. You think about the broader ecosystem. Why is this a good fit for OpenSSF and how does Zarf help secure open source software?

Brandt Keller (12:30.52)
That is one of the most fun parts of my job is to really try to find new avenues, right? Zarf on again, stepping back a little bit to the early days, it’s been software integrators. They want to take software from its source. Usually these are open source projects and they want to package them up and take them to their environments.

But there’s a real like key opportunity that we’re seeing manifest here really within the last, I would say year and even last couple of months where like Zarf’s transparency models, Zarf’s packaging mechanism could be a key enabler for how software consumers look at open source projects and kind of deem, you know, how well are they doing software like supply chain security?

There’s going to be some of these like onset, I’d say issues where again, maintainers of projects, whether they’re sponsored by a company or whether they’re doing it out of their free time, because they love the technology. they only have so much time in the day. And, you know, I think supply chain security, it’s evolving quickly and it’s continuing to evolve. And most of the time that asks maintainers to do more things, right? Initially you were developing an application.

And you’re saying, Hey, it’s free for everyone to use. I’m not, I’m not asking anybody to pay me to do this. and that’s great. And then more people are like, Hey, for supply chain security, I really need to know what the SBOMs look like. Can you add SBOMs to your releases? Can you sign your releases? And you can kind of see each one of these layers is now a new cognitive burden for, maintainers to kind of normally manage, then

They have to manage the processes on top of like performing that activity. so, I kind of, see some like very distinct opportunities where. If the goal is to help secure open source software, we can really do that by using kind of like Zarf’s declarative manifest model in order for upstream projects to produce releases that consolidate all of that burden again into.

Brandt Keller (14:54.294)
A declarative manifest that they orchestrate in their pipeline, just one, one command, right? Hey, let’s do as our package create. And it’s going to create my, everything I need to run an application such as guac, for instance. we’re working with GUAC on kind of this problem statement. and it’s like, there’s some very fascinating opportunities where you’re going to have everything you need to run GUAC. You’re going to have the SBOMs for GUAC. You’re going to have.

You know, the ability to have that release artifacts signed. And so if you go back to that software integrator, they typically have wanted to deploy GUAC. would create their own Zarf package and pull all those artifacts. Well, now Zarf could just, the integrator could just do a Zarf package poll and they would have everything they would need. And they would know that it’s coming from the upstream source. They would have the confidence around it. They could check the signatures, review the SBOMs and kind of.

you know, kind of look at the posture of, Hey, everything I’ve needed now is kind of consolidated into a concise workflow.

Sally Cooper (16:05.62)
Alright, Brant, before we wrap, it’s time for Rapid, Rapid Fire. These are questions I’m going to ask you, you’re going to answer without any overthinking. No explanations, just your first instinct answers. Are you ready for Rapid, Rapid Fire?

Brandt Keller (16:32)
Let’s do it.

Sally Cooper (16:34)
Star Wars or Star Trek.

Brandt Keller (16:35)
Star Wars.

Sally Cooper (16:36)
Solid. Favorite retro video game?

Brandt Keller (16:41)
Oh, uh, Spyro.

Sally Cooper (16:46)
Ooh, nice. Marvel or DC?

Brandt Keller (16:49)
Marvel.

Sally Cooper (16:51)
Excellent. All right, and last one, favorite open source mascot?

Brandt Keller (16:56)
Oh, not counting Zarf?

Sally Cooper (16:59)
Zarf, of course, after Zarf?

Brandt Keller (17:03)
Um, probably Golang Gopher.

Sally Cooper (17:08)
Oh right, I have that one too. And Zarf is really cool. I love that. All right, perfect. No notes. As we wrap things up, what’s your call to action for our audience? If someone’s listening and wants to learn more about Zarf, try it out or to get involved, where should they start?

Brandt Keller (17:25)
They should start as a Zarf.dev, short and sweet and kind of take a look around and really what we’re trying to understand is, you know, what are the needs of users in different critical environments? I like to visit the public sector groups in a variety of different foundations to see kind of what problems they face, because most often than not, the most constrained regulatory environments are the hardest ones to solve for because they can’t rely on the internet or they have additional scrutiny and we really believe that Zarf is a a great avenue to evaluate and understand. And if it’s not, we’d love to hear from you. Like why, why not? And what are the things that we could do to improve?

Sally Cooper (18:10)
Brandt, thank you so much for joining us today and for all the work you’re doing to help secure how open source software is delivered into some of the most challenging environments. And to everyone listening, happy open sourcing and that’s a wrap.

Apr 06

Love0

What’s in the SOSS? Podcast #58 – S3E10 Big Thoughts, Open Sources: Beyond the Hype: Brian Fox on Securing the Agentic Future of Open Source

By OpenSSF Podcast

Summary

In this inaugural episode of Big Thoughts, Open Sources, host CRob sits down with Brian Fox, Co-founder and CTO of Sonatype, to discuss the friction between rapid AI adoption and foundational software security. Brian shares insights from the 11th annual State of the Software Supply Chain Report, revealing the emergence of “slop squatting” and the high frequency of AI models recommending non-existent or vulnerable dependencies. The conversation explores how the Model Context Protocol (MCP) could revolutionize developer compliance and why the industry must fund the critical infrastructure supporting our trillion-dollar open source ecosystem.

Listen on Apple Podcasts Listen on Spotify Listen on Overcast Listen on Pocket Casts

Conversation Highlights

00:23 – Welcome: Big Thoughts, Open Sources inaugural episode.
01:01 – Brian Fox’s journey: Apache Maven, Sonatype, and OpenSSF.
02:53 – The critical role of Maven Central in the software supply chain.
03:26 – Decades of security trends: The persistent “Log4Shell” pattern.
05:34 – The “Tribal Knowledge” problem for AI agents.
07:06 – State of the Software Supply Chain Report: AI recommending made-up code versions.
08:09 – Explaining “Slop Squatting” and AI hallucinations.
10:03 – Model Context Protocol (MCP): Turning security tools into AI expert systems.
13:42 – Do not ignore 60 years of software engineering “physics”.
15:11 – The “Vulcan Mind Meld”: Injecting governance data into AI agents.
17:19 – Risks, rewards, and the need for ML SecOps discipline.
19:30 – “Inefficient code is still inefficient code”: Lessons from cloud migrations.
21:01 – Building an “AI-native SDLC” with upfront security.
24:18 – The sustainability crisis: Secure open source builds are not free.
27:17 – Conclusion: Funding open source infrastructure (8 trillion dollars of value).

Episode Links

Transcript

Crob (00:23)
Welcome, welcome, welcome to Big Thoughts, Open Sources, the OpenSSF’s new podcast. We’re gonna dive a little more deeply in with some of the amazing community members, subject matter experts, and thought leaders within open source, cybersecurity, and high technology. Today in our inaugural episode, I’m very pleased to welcome a friend of the show, Brian Fox from Sonotype. How you doing, Brian?

Brian Fox (00:47)
I’m doing well, how are you?

Crob (00:48)
Excellent, we’re super glad to have you today. So maybe just for our audience members that are unfamiliar with your work, could you maybe talk a little bit about how you got into open source and kind of what you specialize in in this amazing ecosystem?

Brian Fox (01:01)
How I got an open source, that’s a long conversation. geez, all the way back in 2002, 2003, I suppose, is when I really, really got involved. I had done some dabbling and some other things before that, but I got involved around that time in Apache Maven. I started writing some plugins.

They’re pretty popular plugins, people still use them these days. And those ultimately got pulled into the Apache project, the official project. I kind stowawayed and came in as a committer. A few years later, I joined up with some other folks that were also working on Maven and we co-founded Sonatype. It’s been 19 years now.

CRob (1:45)
Wow, that’s awesome.

Brian Fox (1:46)
Yeah, and so then I was ultimately the…the chair of the Apache Maven project for a long time. still an Apache member of the foundation. And then more recently, even though it’s been a while, what, four or five years now, we joined the OpenSSF. I’ve been on the Governing Board with you for a while. I’m also on the Governing Board of FINOS, which is the financial open source.

And for the last couple years, also been on the Singapore Monetary Authority’s Cyber
Experts Group. Yeah, that’s fun And so, you know, I’ve spent a lot of time focused on those things. One of the things that Sonatype does for the community we run Maven Central, right? Which for people that don’t know that’s where all the world’s open source Java components come from.

CRob

Yeah, it’s kind of sounds like a big job It is a big job running critical infrastructure for all that kind of stuff And so, you know over the years that’s given us really interesting insights into what’s going on with the supply chain so, you know, that’s kind of that’s sort of what led us to the path that brought me to OpenSSF and all those other things.

CRob (2:53)
Yeah, you and your team have been amazing participants and contributors to our community and just kind of even putting aside all the work with Maven. Just your kind of participation in our working groups and our efforts has been amazing. Yeah, thank you. So today I think you wanted to talk about a topic a lot of people probably haven’t heard about. This little thing called AI. I have a hard time spelling that.

Brian Fox (3:16)
Right?

CRob (3:20)
Let’s just set the stage. What are you thinking about? What do we want to have a conversation around AI about?

Brian Fox (3:26)
Yeah, I think so if we back up a little bit, right? So it was probably around 2011, 2012, I suppose. We started looking at some of the trends that we were seeing within the Maven central downloads. We were seeing things like the most popular version of a crypto library was the one that had a level 10 vulnerability.

fixed and patched years before, but that everybody was still using the vulnerable version. The log for J, log for shell pattern has existed basically forever. It’s not actually new. And so that led us down the path to start doing different things to help our customers A, understand what open source they were using. Way back then, nobody knew. They were like, we’re not using open source. What they really meant was, I don’t think I’m using Linux in open office. They didn’t understand.

that their developers were pulling in all these components. And so the problem space back then was helping them have visibility and then providing data and controls to help them better govern their choices. So we’ve always been trying to help expedite and make it more efficient for developers to make better choices. And so it’s interesting to see this development of AI and all of the kind of things that have come along with it. So that got me thinking, you know, what?

When we started out to build some of the stuff that we built for our customers, my focus at that time was to make it possible to do the analysis in real time so that it wasn’t the case that, we’re just going to do all our stuff and then we’re going to run a compliance scan at the end of the week or end of the month or something. So we were very focused on, it needs to be able to be run every single bill all the time. We need to be able to provide guidelines so that they don’t have to ask the legal team and wait six weeks for an answer, or the security team, right?

We were trying to capture those roles, or those rules, into the system so that they could make better choices in real time. And that was a big thing that organizations needed to be able to scale and become efficient. When you start dealing with thousands of developers, tens of thousands or millions of applications, the tribal knowledge problem kind of falls apart.

CRob (5:33)
Absolutely.

Brian (5:34)
Right? And so you start thinking about what happens with AI, and if you don’t have that stuff in an automated, you know, coded kind of way, how do you feed that to an agent? The agent’s not hanging out with you at lunch. It doesn’t get an onboarding session where we say things like, you know what, we never use an LGPL dependency because we ship our code. Or, you know, we only fix vulnerabilities five and above. Or, you know, whatever the policy may be, those things sometimes can be shared among developers.

CRob (6:02)
Right. and it plays into kind of the classic problem with engineering – is most engineers I’ve met don’t like doing documentation. And with AI entering the chat room becoming this accelerant, it’s making decisions based off of knowledge or lack thereof. if you don’t have your security policy documented, it even goes back to thinking about the early days of Kubernetes.

Where it was a big mental shift for people to have that software defined network inside. And that helped, I think, a lot of organizations get better discipline and rigor where you had less mysterious outages. Because the firewall guy in the back end said, I didn’t do anything, but try test it again.

Brian Fox (6:46)
Right, right. Yeah, for sure. And that’s kind of what we’re seeing now. We’re seeing a lot of that with the, not just with agents. mean, agents are sort of like the next big step and not everybody’s
fully there yet. Some people are dabbling with it. But even just AI assisted coding, you’re seeing the same problem that you come in and you say, hey, I want a new feature. And it just grabs whatever statistically likely thing dependency is going to be in there. We’ve done some studies. We recently released the state of the software supply chain report. It’s a great report. Yeah, thank you. This was our 11th year. We just published it last month. And we did a deep dive on AI recommendations, you know, and we found that 30 % of the time the models were recommending made up versions.

CRob (7:35)
What?

Brian Fox (7:36)
Yeah, just making them up. You know, so it’s kind of shocking. In the real world, you know, if you’ve got a tool, that’s one of those things that fails fast, right? Like it picks a version that doesn’t exist, the thing goes and it immediately blows up and then, you know, Claude or whatever you’re using will go, whoops, and it’ll fix it. So it’s kind of funny, burns some tokens, but the downsides aren’t huge.

If the agent randomly picks a terrible dependency or a very old one that does in fact exist, I would argue that’s worse because there’s no fail fast in that scenario.

CRob (8:09)
Well, you also have the whole problem with slop squatting. Where the models seem to, regardless of what vendor provider you’re getting it from, they seem to fairly consistently suggest the wrong dependencies, kind of like typo squatting.

And so now the bad guys have recognized this kind of fairly consistent behavior and they’re uploading malicious packages with those bad names so that you don’t break the build because it can find what needs.

Brian Fox (8:33)
So instead of it failing fast, it fails fast by grabbing a back door or something. Exactly, that’s exactly right. That’s what slop squatting is what they call it now. Yeah, and so those are some of the challenges that we observe and you kind of take it to the extreme where now you potentially have less sophisticated developers, not classically trained developers using these tools, and they don’t know what they don’t know.

They wouldn’t necessarily stop to say, hey, I want you to now be a security expert and do an assessment of the code you just created. Like somebody who knows better will do that. But if you’ve not lived through the pain that you and I have lived, you wouldn’t think about that. And so on average, these things are going to potentially toss away a lot of the learnings that we’ve known for so many years.

CRob (09:21)
And that’s been a chronic challenge, trying to get the tribal knowledge instantiated, trying to help people make those right decisions. And the AI tools are amazing productivity and efficiency savers, but they are bringing in, as you said, classically untrained professionals that they are not a software engineer. They don’t understand how a system should be architected, or they don’t understand kind of the app sec best practices that help secure the foundation of everything and not let the world fall apart.

Brian Fox (9:59)
The interesting thing is I think they can be if prompted correctly.

CRob (10:02)
Yes.

Brian Fox (10:03)
Right? And that’s where some of the knowledge gap comes in. And I think, what was it last summer, Anthropic released the MCP model control protocol, right? Which is, I’ve spent a lot of time thinking about that pretty deeply and looking at all the tools. And I wrote about this. I think that there’s a high likelihood that we see a lot of the tools we use in software, in the SDLC today, moving more towards providing their capabilities as subject matter experts in “a thing” to an AI agent via MCP.

So I think that, for me, is pretty exciting for a number of reasons. It means, as a tool vendor, I don’t have to create a plug-in for IntelliJ and one for Eclipse and one for VS Code. As an example, MCP can be the same thing for I don’t care what tool you’re using, because it’s interacting with me via this standard API. And I’m kind of talking to it in more or less English prompts. So my ability to deliver the value that we have into whatever tool you feel like using today, and they change every week, is pretty cool.

And I would also argue that the ability to insert that information and to potentially roll out the root prompting that all of the developers are using in these capabilities is better. You’re going to get potentially better compliance than you do today. One of the things I struggled with forever was we created an IDE plugin for our capabilities that it demoed amazingly well. It showed, hey, this dependency has vulnerabilities, or license, or would make recommendations. It was great. But developers just didn’t want to install more plugins. They just weren’t using it, right?

So while it demoed well, the actual usage of it was very low for compliance reasons. That’s a thing we struggle with. Every tool vendor struggles with that. But if you were able to insert that same information into an MCP capability and the company rolls out a root prompt that says something like, hey, every time you’re choosing a new project or a new dependency or trying to assess a dependency, use this MCP server to get up-to-date real-time information, it’s more or less going to do that every time. Right.

CRob (12:13)
Yeah, and I think back to like when I was a baby cyber person going studying for my CISSP, there was a lot of talk in the exam materials about expert systems, which is exactly what I think a best case scenario with these tools can be. It’s you’re expert. I don’t have to necessarily have this expertise. That’s right. But thinking about it takes a lot of knowledge to craft these expert systems. Let’s talk about how some of these models have been trained on potentially less than expert data.

Brian Fox (12:43)
Right, and that’s just, think, the inevitable nature that the frontier models have been trained on, you know, all the stuff that they can find.

CRob (12:49)
The internet.

Brian Fox (12:49)
On the internet, good information, bad information, people talking about terrible dependencies a lot might statistically make that more of recommendation, right? And I think that can be okay as long as you’re plugging in the models that have real data. The things that we’ve seen, you know, when we assess the models is that like I said, they make up versions, they pick old versions arbitrarily, they don’t know about anything newer than when they were last trained, which means both new versions and also vulnerabilities that might be an older version.

So they’re inadvertently recommending, and it’s not even a recommendation really, it’s just using it, right? It’s putting it in there and writing the code around it. Imagine picking Spring, right? It’s just going to go, I’m going to write a Spring app and I’m going to use all Spring 5.

And then when you probe it, then it’s like, oh, sorry, I have to do two major framework updates. You almost have to throw it away and start over. And so if you’re able to plug the right data in up front, you don’t have all of that waste. And again, if you have people who don’t know to prompt it to ask about the latest versions, you can insert that underneath the hood. I think that’s what’s really cool.

Brian Fox (13:59)
But what we’re seeing currently, I kind of wrote about this a little bit too, that I feel like we’re throwing out all the lessons of the past. We’re talking about situations where whole tools, SAST is under fire right now, right? Because when all the code can be just completely generated, what’s the problem with SAST? But I do think that we’ve learned a lot of things over the years if we can figure out how to plug those capabilities into what’s being generated.

I think we can bring all of that forward with us. But the entire SDLC is going to have to adapt to that. It’s not going to be sufficient to say, I’ve got a bunch of developers over here. They’re doing AI assisted development. And then later, we’re going to run a bunch of SAS and produce legacy reports. That’s not going to work. The information has to be fed directly into the AI capabilities up front.

CRob (14:51)
And it’s the classic problem we’ve always had, where security historically is the the last thing done, addressed, it’s bolted on at the end in a lot of cases. And just this AI tooling and just the velocity it has is a huge accelerant for the sins of our past we’ve never actually addressed.

Brian Fox (15:11)
Absolutely, but it also provides the Vulcan mind meld if you want to think of it. You now have that opportunity to plug that right into what the agent is thinking about in the moment. You can’t do that with the humans, but you can do that with the agent. And that’s what I think is potentially exciting about this.

Where I described it recently at a summit, we’re sort of in a bootstrap situation, though, right? Like, we don’t have all of those capabilities. Organizations haven’t rolled them all out. And so we’re sort of in this weird situation, one foot on the boat, one foot on the dock, and it’s not going to end well as we’re going through it. And worse, there are people that are afraid of the MCP protocol. So I hear lots of organizations say, we just block it completely.

Yeah. It’s a little hard to argue that that’s not a reasonable place to start because of the nature of what’s happening. We saw just the other day the latest version of Shai-Hulud came out. Did you see this? And they used MCP capabilities as data exfiltration. And I’m like, come on, guys. There’s so much power in this, but now you’re making it like a bad thing. So I think the industry and the tools and all of that are going to have to work through governance of the MCP capabilities, sanitization inspection of the MCP capabilities just like we’ve seen. So it’s sort of one of these things like when you’ve been around long enough you can recognize the patterns. It’s new and exciting but also the pattern rhymes with a bunch of stuff we’ve done before and what frustrates me is that like everybody charges ahead so fast they just feel like it’s all new it’s all different it’s like yes but let’s not forget everything we’ve learned over the last 60 years of software engineering because the physics is still the same.

CRob (16:50)
Well, and that’s where so our AI / ML working group wrote a paper around ML SecOps. And the paper was really interesting. I recommend the audience check it out. But it was they talked about classic techniques that are assisted and are helpful with AI development. And then it talked about some gaps where we have things like are not documented policies that are kind of a hindrance and something that’s an opportunity in the future to try to get addressed.

Brian Fox (17:18)
Yeah.

CRob (17:19)
But…I’m of two minds about my friends, our new robot overlords, in that it can be extremely helpful, but I don’t see a lot of people reconsidering those lessons of the past of software engineering. To say this is all brand new and totally different, like, well, you’ve got different GPU accelerators and dedicated cores to do things.

And now with this like agentic and ADA architecture where things are more highly distributed, yeah, that’s new twists, but it’s not brand new. We’ve done networking. We’ve done composite applications for decades.

Brian Fox (18:01)
Right. It’s the same thing, you know, we saw when, you know, we were like, oh, everything should be serverless or let’s go to the micro architecture, micro architecture, micro service architecture is going to solve everything until it doesn’t. Right.

Or, you know, that’s no problem, we’ll just put it in the cloud because I can just infinitely scale my machines, right? So I see the same pattern all again, that we sort of say, yes, but this time is different because insert new technology, and then we realize, yes, but everything we know is still true. And that’s what I think we’re sort of grappling with right now as we go through this. What is absolutely true is that, you know, the AI capabilities, the agents, all these things are making everything happen so much faster. That can be good.

can also be bad. If you’ve forgotten all the lessons of the past, you’re just going to create a ton of crap much faster than you could before. And by the time you realize it, it might be too late.

CRob (18:57)
I’m familiar with a lot of enterprises that were going through a digital transformation journey, trying to update their heritage software to newer things and to the cloud to get that scalability and cost efficiency. But a lot of organizations didn’t take that journey, didn’t learn from lessons from the past.

they just crammed what they had out in the cloud, and then a month later they get this giant bill and they’re shocked and confused, or they didn’t understand that this thing wasn’t architected for zero trust, and they’re leaking data everywhere.

Brian Fox (19:30)
Right, right. Or that, or just even the performance reasons why you were excited to infinitely scale, sure, but somebody’s not excited to infinitely pay a bigger bill. Inefficient code is still inefficient code, right? And that’s what I think we’re gonna see with… with AI capabilities is just going to happen faster. And without humans in the loop, it provides less opportunities for us to course correct, which is why I’ve been taking a step back and thinking about how do we do that? How does it make sense? I think for some of the stuff that we’ve been doing as a business, it’s really exciting because we have built up really interesting, unique data sets based on being able to see everything going on with Maven Central. We’ve long had Nexus, the repository manager that’s out there.

We have hundreds of thousands of instances. Those things are proxying for enterprises, not just Maven, but NPM, Nougat, Python, all the things. And so that gives us visibility into other ecosystems so we can understand what’s going on, what’s commonly used in enterprises, these kinds of things. And so all of that data can be fed now directly, like I said, the Vulcan mind meld directly into these tools. And it makes it a lot easier.

So in some ways, when we sort this out, and people become less afraid of MCP capabilities, we can directly inject a stream of high quality data to make all of those things better. But, before businesses can really leverage that, they have to get out of the experimentation phase. They have to roll that out. And these things are kind of interrelated. What we see is that organizations are afraid to let developers just go with AI assisted development because it’s not governed, because they can’t govern it.

And those are echoes of what I saw firsthand during the early days of open source. Like I said in the beginning, people said, we’re not using it. And then I’d tell them, yes, you are. And then their reaction was like, well, just shut it all out. It’s like, right, you can’t do anything. So the reaction that some enterprises have right now of like, we’re just not going to do anything with AI, is just setting themselves up to be left behind.

The right answer is to do it thoughtfully and use tools to help them make better decisions.

CRob (21:43)
So reflecting back, mentioned in your report that you have some guidance for people around AI. What would the top two or three things, if somebody’s thinking about moving more aggressively in this AI direction, what can they take away and do immediately or start thinking about?

Brian Fox (21:01)
Yeah, I mean, think the biggest thing is humans like to…try to take the old patterns and just adopt it to the new new technologies like we were talking about take an inefficient architecture and throw it in a cloud It’s gonna fix everything. No, it’s not and I think that’s true of Let’s call it the AI SDLC right an AI native SDLC Might resemble a normal SDLC, but it should be designed differently, right?

You know trying to do the checks and balances after the fact is even worse than it was with humans You need to think about providing that information upfront so that you get the value in the creation of the code and not try to chase it out. You need to be able to think about how all of these things can be done in parallel with agents, breaking these things down. what I would say is, don’t just try to do what you’re doing today and use AI to do it. Take a step back and really assess how can you adopt this.

It’s sort of like the conversations we were having in the board today about developers, maintainers are getting overwhelmed with AI slop. It’s true. A reaction is to stop allowing that to be contributed, just dismiss everything AI. That’s not a good answer. A better answer is let’s figure out how to help them use AI tools to be able to keep up with that, right? Because that’s what it’s good for. It can review and assess the patches faster than the maintainers and then provide sort of a first pass filter, if you will, right?

But that requires thinking outside of the box. Don’t just try to keep doing what you’re doing and try to keep up with it. Think about how you judo move that into something that makes more sense for your organization.

CRob (23:42)
And this skirts along another kind of project you’re passionate about, sustainability and funding. It is one thing to try to admonish the developers, why aren’t you using AI? But there are real costs involved around this. And, just to say, well, you should use the tool that doesn’t help them when there’s no funding. They don’t have access to infrastructure to be able to do these things. And that’s like, think, it touches on your passion project around trying to help get the package repositories more sustainably funded.

Brian Fox (24:18)
That’s right. Yeah, I mean, if you take a step back and you think about open source when probably you started, certainly when I started, what that really meant was you were donating your time. And you were sharing your thoughts, and you were sharing your words via code. And that was in a time when it was perfectly acceptable. In fact, it was the only choice that you built things and you shipped them off of your laptop.

There was when the Apple MacBook Air launched, the first one. That launched with a version of Maven on it that was signed by my key, my personal key, that was built on my personal laptop. So everybody that bought the launch version of the MacBook Air had my signed code on it. That’s kind of cool.

But also kind of scary, right, when you think about it. Like, what if my laptop was compromised? And that’s the world we live in today. Fortunately, in 2009 or whatever it was, that was a little bit more remote of a chance. And so everybody thinks like, well, that’s crazy. You wouldn’t do that anymore. So what does it mean today? It means you have to have certified builds. Usually that means it’s running in the cloud, and it’s attested to, and all these kinds of things. And that’s not free.

Like I can’t donate that, I’m not a hyperscaler. Most open source gets that infrastructure donated by these big companies, but there’s a lot of opportunity for abuse, right? And these types of things. it’s just, at the end of the day, it’s not free. So the cost of producing open source is not free anymore. It’s not just donating my time with equipment and internet access I already have, right? That’s the big difference. And I think people don’t really recognize that and now fast forward to what we’re just talking about AI the obvious answer to deal with AI, you know Piled on PRS is to have AI assistance help.

Who’s gonna pay for that? It’s literally not free. It costs electricity, last time I checked we still pay for our electricity Regardless

CRob (26:12)
Electricity, water…

Brian Fox (26:14)
Right all of these things, right? These are very…they have very real implications. They’re just not free and so There’s no good answer to that. How does that get aligned? How do we…how do we continue to create open source software that can power all of these industries in a world where it’s not just somebody donating their time and thoughts? There are no good answers. But we’re working towards trying to align that. Because the bulk of open source software, certainly in our world, in these areas, is being consumed by organizations that are selling for-profit software, more or less.

There’s definitely a lot of hobbyists and stuff like that the biggest consumers from our repositories are all the giant companies. I’ve named the top 100. You would know every single one of them. And I’m sure that’s true for all the registries. So there has to be an answer in there. I don’t know the stat off the top of my head, but the Linux Foundation does the census, right? And it’s billions of dollars of economic value that open source creates. Eight billion? Nine billion?

CRob (27:16)
Trillion.

Brian Fox (27:17)
Oh, it’s a trillion now?

CRob (27:17)
It’s eight (8) trillion, I believe.

Brian Fox ()
Eight (8) Trillion dollars worth of economic value being produced by open source…1 % of that would pay for a lot of that infrastructure, and then a whole bunch more. And so I think that’s what ultimately we have to figure out how to balance. AI just makes that worse, because it moves the bar even further.

CRob (27:39)
Interesting conversation. Any final thoughts you want our listeners and viewers to take away?

Brian Fox (27:46)
Well, certainly go take a look at The State of the Software Supply Chain Report.

CRob (27:51)
Great report.

Brian Fox (27:52))
sonatype.com/SSCR Certainly, I’ve also written a number of blogs. You can find those at our website as well. That deep dive, kind of all these topics we touched on here. Yeah.

CRob (28:02)
Excellent. We’ll put some links as we do our summary. So Brian, thank you for our inaugural episode of Big thoughts, Open Sources. I think this was an amazing conversation that we’re gonna continue to be adding onto and reconsidering in the coming weeks and months.

Brian Fox (28:20)
Yeah, thanks for having me kick it off in Napa.

CRob (28:25)
Thank you. Well, I hope everybody stays cyber safe and sound. We’ll talk to you soon.

Mar 24

Love0

What’s in the SOSS? Podcast #57 – S3E9 From Noise to Signal: Security Expertise and Kusari Inspector with Mike Lieberman

By Jeff Diecks Podcast

Summary

In this episode, CRob talks with Mike Lieberman from Kusari about the current state of open source security. They discuss the growing burden on maintainers from the “deluge” of noisy, low-quality vulnerability reports, often generated by AI tools, and the vital role of “a human in the loop.” Mike introduces Kusari’s tool, Inspector, explaining how it uses codified security expertise to process data from tools like OpenSSF Scorecard and SLSA, effectively filtering out false positives and giving maintainers only high-quality, actionable reports. They also dive into the design philosophy of “don’t piss off the engineers” and share a vision for the future of security tooling that focuses on dramatically better user experience and building security primitives that are “secure by design”.

Listen on Apple Podcasts Listen on Spotify Listen on Overcast Listen on Pocket Casts

Conversation Highlights

00:06 Introduction: The Biggest Challenge in Security Tooling
01:12 Overwhelmed Maintainers: The Deluge of Low-Quality AI Reports
04:00 Introducing Kusari’s Inspector: How it Filters False Positives
08:40 The Secret Sauce: Security Expertise and the Need for Reproducible Tests
12:03 Meeting Engineers Where They Are: Design Choices to Reduce Maintainer Burden
18:16 The Future of Open Source Security Tooling: Focusing on Better UX
22:19 Call to Action: The Responsibility of Large Organizations

Episode Links

Michael Lieberman’s LinkedIn page
Learn more about Kusari Inspector (free to use for OpenSSF and CNCF projects)
Get involved with the OpenSSF
Subscribe to the OpenSSF newsletter
Follow the OpenSSF on LinkedIn

Transcript

(0:00) Intro Music

Mike Lieberman (00:06)
I think the biggest thing in security tooling is better user experience. I think that to me is one of the biggest challenges.

CRob (00:25)
Welcome, welcome, welcome to What’s in the SOSS?, the OpenSSF’s podcast where I talk to developers, maintainers, security experts, and people in and around this amazing open source ecosystem. Today, again, we have a real treat. Friend of the show, Mike Lieberman from Kusari is joining us again after – I don’t know if your podcast was toppled from its place of the most listened to before, but we’re gonna see if we can make another hit for us. But we’re here today to talk about some interesting developments that you and your crew are involved in and just things going on in open source security. So how have you been, sir?

Mike Lieberman (01:07)
Well, thank you for having me back and yeah, things are going pretty well.

CRob (01:12)
Well, let’s dive right into it. Recently, and this is a topic that I’m actually dealing with this very moment while we’re recording this podcast, that open source maintainers are just currently overwhelmed by just this deluge of noisy, low quality reports, a lot of them generated by AI tools. So kind of thinking about it with your, many hats you wear, as you know, business owner, a community member, and a long time developer, a security expert. From your perspectives, what is actually creating the most burden today? And think about it through the lens of this project you’re going to share with us in a moment.

Mike Lieberman (01:57)
Yeah, sure. So I think to kind of start, the problem has been the same problem since, know, throughout human history, it is a combination of either bad actors or just lazy people that are I would say the biggest issue here. Right. We have a lot of things like AI reports generating awful sort of vulnerability, know, fake vulnerabilities or whatnot. But if we kind of look at it through the lens of like history through tech, we saw the same thing with any sort of automation, right? When, yeah, exactly. When people could kind of create scripts, hey, let me go in spam this one project with my script. Let me spam a whole bunch of projects with my sort of automation or whatever. And, you know, the same thing sort of happened when people, when we started moving away from mailing lists to sort of GitHub and those sorts of things as well. So I think it’s really to kind of take a step back. It’s kind of how people are using the tools more so than the tools themselves. But I do think when it comes to a lot of the security reports, yeah, it is folks who are just kind of asking an LLM.

Hey, find me find me some zero day. And of course, that’s never going to work because the LLMs don’t have that information. And it’s just it kind of comes back to you need people who understand what they’re doing, using the tools in the right way in order to kind of figure out some of this stuff.

CRob (03:42)
Yeah, human in the loop. Our dear friend, Dr. David Wheeler has a saying, he says, a fool with a tool is still a fool. So again, having those experts in there, helping out is critical. So let’s…

Mike Lieberman (03:43)
Yeah.

CRob (04:00)
You’ve been in this space for a long time, focusing in on supply chain security, and you’ve written or contributed to a ton of tools. And most recently, you all helped create over at Kusari a tool called Inspector. So from just a high level TLDR, how do you see things like Inspector kind of changing this dynamic of getting more people involved or getting more expert knowledge in?

Mike Lieberman (04:26)
Sure. I think like the things. So actually to take a step back, right? There’s a lot of great tools that are being built. The challenge with those tools is, and the way I kind of think about it is like, you know, home security, right? It’s, hey, there’s a ton of tools out there that are helping out with open source security the same way that there’s a ton of tools out there for, you know, a smarter lock. A better security system.

CRob (04:59)
A doorbell that can find your dog.

Mike Lieberman (05:02)
There’s privacy concerns on that one. think, you know, we can all agree on that. But I think to that extent, when it comes to sort of these tools, it’s in how they’re used. And also, the expertise that’s required in how to use them. And also, when building the tools, what sort of expertise went into building the tools? And I think that to me is where the big gap is with just sort of some of the AI related things is you have folks using a very generic system like an LLM. And just saying, hey, LLM become a security expert and do this stuff. And of course the LLM makes a lot of mistakes and whatever. But if you were to kind of say through things like MCP and LLM skills and all these other things, if you have a way of codifying, run open SSF scorecard, run, you know, SLSA and run all of these various things and put all of this together and generate me an SBOM using these tools and whatnot. And then you can take all that, then hand the output to the LLM and say, hey, here’s everything I discovered. Here’s also the code. Help me make sense of it. And I think that to me is kind of where a lot of the benefit is. And again, what I just described is essentially inspector, right?

We’re running all of these various tools, again, that we understand because we’ve contributed to those tools, we’ve helped maintain some of those tools, we have been users of these tools for years. So we understand how they’re supposed to be used. We understand how a human who is, before the age of AI would be using these tools. And we recognize the burden of that expertise. And we’ve sort of encoded it. Had the LLM kind of come in at the last mile and then take all that information, and say, hey, if there is a finding, a vulnerability, great. Where does that vulnerability live? Is that a vulnerability in a core piece of my code, which yes, I need to address right now, or is it like, it’s in a test? Yes, it’s probably something I should fix, but maybe not the biggest issue right this second. And so I think tools like that are really helping because the thing that we found, and again, a user of inspector told us this, and I won’t call out the exact AI tool they were using, but they were using a generic LLM with some stuff. And then they were using inspector. And one of the things that they had said was, wow. Yeah. Like inspector is actually catching the issue with all of the, the, uh, it detected that a particular issue was essentially a, um, a false positive because it looked at a potential remote code execution and looked at all the stuff alongside the code and said, you are clearly have an allow list. So given that you have this allow list, we recognize it’s not a remote code execution, or rather arbitrary code sort of execution attack. And I think it’s stuff like that, that we’re seeing starting to get developed more and more. Whereas a lot of the tradition, I want to say traditional with AI, even though it’s been, you know, like in the past year, everything shifts. Yeah.

When we look at sort of how folks were using LLMs even just a year ago, a lot has shifted and we’re seeing less of these false positives coming out of AI because people are using AI the way it should be used, where it’s you’re supplementing all these other tools that are out.

CRob (08:40)
That’s awesome.

And this might lead into this next question. AI and automation are finding a lot more potential issues, but they’re not always better. And is that what you think that secret sauce of having that security expertise and that helps kind of balance out finding a vulnerability and then kind of sharing that information with the maintainer effectively?

Mike Lieberman (09:08)
Yeah, so I mean, I think when it comes to stuff like that, the way I, you know, I was actually having a conversation with a friend just a few days ago about this issue and I’m reminded of issues just even before AI. And one of the big things that maintainers would ask is, give me a way to reproduce this. If you’re not gonna give me a way to reproduce this, I’m not gonna, you know, I’m not gonna accept your report here and I’m not gonna do a ton of investigation to figure out what you intended to mean.

And I think it’s the same way with AI here, where we’re starting to see with some of the stuff coming out of AI XCC and some other places, we are starting to see tools that are being built that are actually generating the tests and whatnot that can reproduce these vulnerabilities that the LLMs are claiming, or AI tools are claiming. And I think that to me is important because when I look at Daniel from Curl or some of these other folks who are like,

I am so sick of all of these AI reports. It’s like every single one that they’re claiming is an AI report, it’s like they didn’t give me a way to reproduce it. Or even worse, the AI said, here’s a list of steps to reproduce. folks are coming out and saying, that function that you were claiming needs to get run doesn’t exist. And so I’m just thinking to myself, well, why not just write a test that does that thing, you know, and have the LLM write the test, whatever, but prove out that like, hey, an AI tool generated a test and I can run that test and I could see, yep, that is an exploit. That is actually a vulnerability. Now I can go and take that and package it up, it over to, you know, hand it over to the maintainer. And I think if I’m as a maintainer of various open source projects,

If I received something that said, hey, here is a test, you can run that test. And again, by the test, mean like an actual test, a test that makes up other code and tries to do whatever, but an actual test. If you have that, I as a maintainer would say, absolutely, that is a real vulnerability. But I think the thing that we’re seeing right now is we’re seeing all this sort of slop, which is.

Again, it’s just similar to the slop we saw years ago with other sort of automated vulnerability reporting and just generally in tech. And I think the problem here is still kind of comes back to lazy maintainer or sorry, not lazy maintainers, but lazy submitters and just other sort of bad actors who are just like, yeah, I’m just gonna throw a thing out there and hopefully one of these is right. And I’m gonna get it, make a name for myself.

CRob (11:58)
A wise man once said that knowing is half the battle.

Mike Lieberman (12:01)
Yes.

CRob (12:03)
And thinking about it from this maintainer developer perspective, almost always maintainers are volunteers first. They’re there because they have amazing idea they wanna share, they have a problem they’re trying to solve. Some people are paid to do a specific thing, but the majority of folks are volunteers first. And security experts 12th, 18th, security is not necessarily a core skill that most developers have. what design choices, kind of thinking about when you were looking at Inspector, what design choices did you make to help meet the maintainers where they are, where they are experts in languages or frameworks or kind of these techniques or algorithms? But how are you helping them where they are rather than expecting them to become a full-time securityologist like you or I?

Mike Lieberman (12:58)
So we have a mantra here at Kusari, which is essentially just don’t piss off the engineers, right? As engineers ourselves, as folks who, myself, I am a software engineer first, or really more of a dev ops, dev sec ops engineer first, became more of a software engineer over time. But one of the big sort of mantras was, one of the things that always frustrated me was you have to do all the security stuff.

And they were burdens to my daily job, right? Where I was not being, you know, again, this is me both as a maintainer of open source projects and also just, hey, I get paid as an engineer or whatever. What, but at the end of the day, I wasn’t incentivized to do secure things. I might’ve been yelled at. I might’ve been told thou shalt do this security thing, but my incentives were getting out this new feature, making my customer or my user happy, right?

And so when it comes to those sorts of things, that’s kind of how we’ve encoded all of this, where if somebody told me, hey, Mike, you put in a potential remote code execution attack or arbitrary code, whatever it is, like you put a SQL injection attack or some other, you’re not handling this off thing correctly. If you told me, yeah, that’s the thing. And you told me what I might need to look at. yeah, let me get on that. Let me fix that.

If you were to tell me, hey, you’re using a library that isn’t maintained and that everybody has mostly moved over to this other library, cool. I’ll, I’ll work on that, but don’t make the burden. Hey, there’s a, this library is unmaintained. Okay. What am I, what am I supposed to do about it? I don’t know what I’m supposed to do about it. Help me with suggestions. So when it comes to inspector, those are the sorts of things that we sort of baked in is we’re not just telling you this project. Is it maintained?

We’re telling you, hey, this project isn’t maintained, but it’s used in just one test. So maybe it’s not the immediate thing that needs to be fixed versus, hey, this thing is completely unmaintained and it’s potentially vulnerable. And this is something new you’re adding. Like this isn’t something that already exists. This is just bad practice. Like you should probably not include this new thing. Or, you know, and again, providing the suggestions to the user on what to actually do about it.

And some of those things can then be, know, know, inspector has a CLI tool that you can use. And I use it myself with Claude where, Hey, I run it kind of come in and, you know, uh, fix it. And like, it works pretty well. So I think again, it’s, it’s having, um, it’s, it’s the combination of things to sort of make sure that it, as an engineer, you know, you’re not being asked to become an expert in this thing, right? Uh, it’s okay to ask an engineer.

You are a database expert, you should be reasonable at securing databases, but securing the underlying OS and yada yada, hey, maybe you don’t need to be an expert in that. And that’s where tools like Inspector I think really help is they’re the ones who are being experts. Again, kind of going back to that, the home analogy, right? If I run a house, if I have a house, I don’t need to know the inner mechanics of know, a pin tumbler lock and yada, yada and, and, and how the various cameras, you know, that that are looking at the outside of my house, how they all interoperate. No, I just need to know, are they working if something kind of, you know, the battery died on this, I know how to change your battery, let me kind of focus on that. But if they were kind of come in and say, No, no, you need to understand the innards of the networking and you need to understand audio visual processing, I’d be like, No, just not gonna work.

So again, make sure that developers can focus just on what their experts in, and what their primary responsibility is, which is usually to the user. And yes, security is a responsibility there, but they’re not going to be generic security experts. And so what can we do to help them hold their hand and tell them what needs to be done in a way that they can kind of say, yeah, you’re asking me to do two or three small little things. Awesome. By the way, we here at Kusari have made Inspector free for open source, but not just open source, specifically for CNCF and open SSF. You have full sort of unfettered access, no rate limits, no quotas. And love to see folks sign up. The website is kusari.cloud. And yeah, yeah, I want to see folks using it.

CRob (17:45)
It’s really interesting and I love the focus on again, because you’re all you grew up through this. are in software engineer. So I love the focus on trying to how to relieve that burden from these, this army of volunteers. So let’s do something else we do often in cybersecurity. Let’s get our crystal ball out and, you know, thinking ahead from your perspective, what do you think, you know, good security tooling for open source looks like in three to five years?

Mike Lieberman (18:16)
I think the biggest thing in security tooling is better user experience. think that to me is one of the biggest challenges. And right today, and I think that’s where a lot of folks are focusing their efforts, it’s, you we need to some extent, you know, and I know, like, the first thing that came to mind, is Kubernetes, but for security, right? And I recognize that Kubernetes, depending on who you talk to, you know, YAML files,

But no, it really did democratize and make simpler the orchestrating complex container workloads, right? And I think when it comes to security, user experience is often kind of a secondary concern compared to just the, did I prevent the security, you the issue, but that’s kind of, as our world continues to get more complex and complicated and things are scaling up and we’re having AI and all these different things. The need for security continues to increase more and more every day. But with that said, if the answer is using these security tools requires, you know, tons of certifications and whatnot for just to use the security tool, right? Not to become an expert, but just to use the security tool, if you need to be an expert in all these different things, it becomes super difficult, nobody’s gonna do it. So I think we’re gonna start to see to some extent, more tools like Inspector, but also in addition to that, more tools like, and I know we’re working on this in OpenSSF, tools that make adopting of Salsa trivial for the average project. Tools that help just sort of generally with security build out that UX, make it simpler for the average engineer to do that. Similar to how we saw stuff

in that space with DevOps, right? Where you had developers and operations, those worlds kind of became more combined. And what happened was you had tools like your Terraforms or, you know, Open Tofu and Ansible and all of these great things that kind of came out of that space to kind of make it easier for both folks who are focused in operations to get a little closer to developers and then developers to actually also help out with some of the operations, infrastructure, engineering, those sorts of things. And I think we’re gonna start to see more of that as time kind of goes on where those like, I’m gonna call like security primitives are more encoded in the tools we have. So I think we’re gonna start to see a lot of tools out there become secure by design and have a lot of the security features baked in. And then also the security tools that we have just generally become a little bit simpler and where areas where they can’t be super simple, we’re gonna see tools more tools like Inspector that kind of come in and operate similar to how you might imagine the security expert to kind of come in and put the pieces together, which again, doesn’t eliminate the security engineer. I just want to be clear, like security engineers are very much still needed. The challenge is the security engineer is now being tasked. Whereas before you had to be an expert in a small set of domains. Now you’re being asked to be an expert across everything and they need to understand that they’re going to be the ones who are like taking these new security tools and given that better UX are going to be able to scale that across, you know, 10,000 projects, you know, a hundred different AI agents, all of this, like, you know, a million containers, all of those things. So I think we’re going to start seeing a lot more of the security tools working better to scale up what we’re doing.

CRob (22:04)
That is an amazing vision. look forward to observing that over the years. Hopefully your vision becomes a reality. Yeah, thank And as we’re winding down, do you have any closing thoughts or any call to action for the audience?

Mike Lieberman (22:19)
Yeah, I think the, so there’s two big ones. One is, hey, if you’re a maintainer and engineer, right? I know you care about security because even when I was not a security engineer, I cared about security. So what I want to hear from maintainers is how can the open source world help, right? How can we help you not get clobbered by a million?

letters from lawyers and other people demanding security features in your stuff. How can we, as an open source community, help out, open source security community, help out? How can we make the tools easier? How can we make sure that those tools fit your needs? And that includes whether it’s inspector or, you know, other things, hey. And on that note as well, you know, CNCF and OpenSSF projects can use inspector..

And the other call to action, I know I say this a lot, large organizations that are using open source, it is your responsibility to provide the incentives to make sure that open source is more secure. Like we can all demand, hey, we need better open source security tooling. We need this, that, and the other thing. But if nobody’s paying for it, if at the end of the day, you know, a random engineer who’s making that open source security tool, if they can’t pay the bills, they’re not going to do that. If they are getting clobbered with a million different feature requests, it’s just not going to work. So we need to make sure. And I know that there’s things like the sovereign tech fund want to see more of that. But just sort of generally, I think it needs to come from these multi billion, multi trillion dollar companies coming in and saying, hey, we are willing to foot a good deal of this bill right in order to make the world more secure for everybody.

CRob (24:17)
Those are some wise words and also I think a wonderful vision we all can work towards together. Mike Lieberman from Kusari, thank you my friend. I loved having you on. And with that, we’re gonna call this a wrap. I want everyone to stay cyber safe and sound and have a great day.

Mar 17

Love0

What’s in the SOSS? Podcast #56 – S3E8 Empowering New Maintainers: Inside the OpenSSF Mentorship Program

By Jeff Diecks Podcast

Summary

In this episode of What’s in the SOSS? host Sally Cooper sits down with Yesenia Yser, co-lead of the OpenSSF Mentorship Program and the BEAR Working Group, and Kairo De Araujo, Open Source Software Engineer and mentor for rstuf. They dive into the success of the OpenSSF Mentorship Program, which focuses on bringing underrepresented voices into software security. Kairo shares an incredible outcome from the last cycle – where two out of three mentees became project maintainers – while Yesenia discusses the evolution of the BEAR Working Group (Belonging, Empowerment, Allyship, and Representation) mentorship program. Whether you are a potential mentor or a mentee looking to break into open source, this episode provides a roadmap for the upcoming paid mentorship cycle.

Important Dates for the 2026 Mentorship Cycle:

Applications Open: March 24, 2026
Applications Close: April 12, 2026
Selection Period: April 13 – April 30, 2026
Notification Date: May 1, 2026
Onboarding: May 5 – May 29, 2026
Mentorship Period: June 1 – August 21, 2026

Listen on Apple Podcasts Listen on Spotify Listen on Overcast Listen on Pocket Casts

Conversation Highlights

00:01 – Welcome
01:43 – Kairo on his work with the Repository Service for TUF (rstuf).
02:30 – Yesenia on the BEAR Working Group and making open source accessible.
04:30 – The “Why” behind mentorship: Solving the barrier to entry for security beginners.
07:28 – Success strategies: Working as a team across time zones with multiple mentees.
09:28 – The ultimate goal: Moving mentees from learners to official project maintainers.
10:58 – Challenges and growing pains: Managing deadlines and interview chaos.
13:48 – Advice for Mentors: The importance of clear communication and flexibility.
15:02 – Advice for Mentees: Don’t be afraid to join; focus on “pre-onboarding”.
17:13 – Key Dates for the 2026 Mentorship Cycle.
20:15 – Call to Action: Get to know this year’s participating projects (gittuf, rstuf, SBOMit, Minder) and how to get involved.

Transcript

00:00 – Music & Intro clip

Sally Cooper (00:24)
Hello, hello and welcome back to What’s in the SOSS? An OpenSSF podcast where we get to talk to some amazing people who are involved in open source software and open source software security. And today we have a very special treat two repeat offenders coming back and they do some critical work in the OpenSSF community.

They have firsthand knowledge of the mentorship program, which we’re going to talk about today, which is a hands-on initiative designed to help underrepresented voices break into software security. So first, have Kairo. Hi, Kairo, an open source software engineer who served as one of the key mentors during last year’s program.

And we’re here to talk about the powerful impact of that mentorship program and also dive into the important work of the BEAR Working Group. So we have Yesenia also joining us from the perspective as a co-lead of the mentorship program in the BEAR Working Group. And I just have to say, Yesenia, it’s super nice to have you on this side of the microphone as a guest. So Kairo, Yesenia, welcome back and introduce yourselves.

Kairo De Araujo (01:43)
Yeah, thank you. Well, my name is Kairo, as you said, and I’m based in the Netherlands and I’m working as a software engineer for a few years. And the past six years, actually, I really focused on the security supply chain. And I’m an author of Repository Service for TUF (rstuf). That is a project to help the security supply chain, that’s part of OpenSSF. And I’m also maintainer of other critical open source projects in the security supply chain. Yeah, and last year, as you mentioned, and we’ll talk more about I participated in the mentorship program with the rstuf project, the repository size for tuf.

Yesenia (02:30)
Oh how the tables have turned tables. Hey everyone, soy Yesenia, not your co-host today, but a guest on today’s episode. I have a extensive background in security. I usually like to say I’ve been Jacqueline of all, cyber master of none, working in various umbrellas and have made my way into open source, love it, and do a lot of advocacy and outreach for it because of just the amazing folks that I’ve met on here that have done amazing things, as you’ve heard in many other episodes.

So, based out in the sunny state of Florida, I’ve seen snow once and then another time through a window. So I always forget that winter’s here. So if you show me snow in the background, I’m going to be so surprised. But that’s great. I love the work that we’re doing with bear. It was originally our DEI group. But since the man banned the word DEI, we chose the bear. So this is our belonging empowerment, allyship, and representation.

group in which we are making this more accessible for folks to enter into open source, giving them opportunities like these mentorships because I firsthand, have seen it from a mentorship that I’ve hosted several years ago. Seeing the folks that have come onto this mentorship enter into the field for very fancy Fortune 100 companies doing amazing work and align with their career. So, That’s a little about me and I’m excited for today’s episode.

Sally Cooper (04:02)
Wow, that’s incredible. It’s a lot to unpack. First off, the winter comet. I will get you back on that one. I’m going to send you lots of pictures of snow. But no, in all seriousness, it’s such great work that’s going on in the bear working group. for the mentorship program, maybe Kairo, can you tell me a little bit about like the why for your mentorship in open source security? What is the problem that you were trying to solve and by showing up for this?

Kairo De Araujo (04:30)
Well, besides security, it’s something really important. There are a lot of people that are a little bit afraid to step in in open source projects like that. Sometimes because they don’t have a huge security background, or they are just coming out of the university or learning coding, but they are not, let’s say, comfortable to step in in open source.

It happened with me as well. Everybody here once was like a beginner in something, and we need to trust ourselves. And I think that it’s really important for products like rstuf to get new people in, new ideas, and also looking to the future like a, keeping more contributors in the project or making, spreading the security, what we are doing, what the project is doing, because we know how it works. like spreading the knowledge, it’s not only talking about the project, but getting people involved in that. And we have some good engineers that would like to try it and they don’t have opportunity. And I think that, for example, this mentorship is a very good opportunity.

And me as a maintainer, I need more contributors for my project because we know that the success and how the product can grow, it’s based on contributions, right? People really writing code, understand how the project works. And this was our goal. Like let’s try to give opportunity to people to step in the project.

And give opportunities to people to understand how we can do security in different ways. Because rstuf is a really specific project based on the the update framework (TUF) that’s really complex. And maybe it can also give more knowledge to others to let’s contribute with a project in a way that I can participate and maybe grow, get knowledge and I don’t know, get a new job or…

Just knowing a little bit more about the language or the service or how I can help the security and participating in OpenSSF as well. Because it can be an entry point for people to get to the community as we have many different other products inside.

Sally Cooper (07:11)
Yeah, I love that. When you think about last year’s program, what would you say stands out the most? What worked better than you expected about the mentoring? And what lessons really stuck with you?

Kairo De Araujo (07:28)
Well, I have done a few mentorships before, not in OpenSSF, but in other companies that I worked at, helping junior engineers and so on. And usually we are afraid to have multiple mentees because how I will manage different people all together in one folks.

And what we experiment really last year that was really nice was to not get only one mentee, but at least three mentees in the product in the way that we can try to work as a team to not teach only about how to make the open source, but they could really feel how the open source works, how they can work from different time zones, from different projects and how, because we had people working documentation, another working on a specific feature and how these overlaps with each other and how we can work as a team.

This was something really that we did different in the project, giving a lot of freedom to them. Like we have the projects that we want to accomplish as the goal of the mentorship, but let’s try to work in the…as a team with a very flexible way that we can help each other. And really, this was very positive for the product last year. And everybody did very well.

Sally Cooper (09:06)
Yeah, it sounds like you set it up in a way with the freedom, the flexibility, you know, and the education to really do well. That’s great. I love to hear that. Just thinking back like to last year’s program, is there anything that surprised you and how did that shape the experience and how do think it will shape the experience going forward?

Kairo De Araujo (09:28)
What really surprised me was the commitment from the mentees. The process also for us to selecting the mentees with the proper like basic skill, not being really a job interview, right? Making it…understand what they can bring as a background to us was very nice. And…

But what surprised me in the end, and I think we will have another podcast about that in the future, that will be that now out of three…the two of the three mentees become maintainers of the project. It means that they started as mentees, then they jump as contributors of the project, and right now they are helping how to run the product. And this is…for me, is amazing as a maintainer. It’s really a relief for me, right? Because I have more people to help me running the product.

Sally Cooper (10:30)
Right best case scenario there. That’s an incredible outcome. I love to hear it. Okay, let’s shift gears. Yesenia, what would you say are some of the challenges or growing pains that you learned in running this program? And what did these lessons teach you? About how to build a sustainable mentorship program in an open source community that you could share with with our audience here

Yesenia (10:58)
Good question. Like first I want to say like… what Kairo just mentioned, the fact that these mentees go to maintainers is the ultimate goal. It doesn’t have to be one of the goals, but it’s one of the reasons we set up the mentorship was to allow folks to enter and come in and see these new projects that they may have never heard of or had visibility to. And really just come in and dive in, become part of a team and see…my experience with open source, it’s the same team kind of dynamic, but in a different aspect – as what you would feel and see in a corporate space.

So, hats off to you, Kairo and the maintainers. I’m very excited to interview them or at least hear the podcast once they’re on. So, future plug for when it will be released. And when we think back on last year’s mentorship, well, I had already done one with OpenSSF, the Linux Foundation.

One of the biggest challenge was the amount of time, right? So was the first time we had to do this. We had the deadlines and the maintainers had about a week to put together the project description. They had a week to like shift through all the mentees and interview them and make their selection. So there was a lot of chaos to the front. So big kudos to them to like push through and find the mentees and get that program kind of running.

I think once the program started running within a few weeks, it just kind of smoothed, and there was a lot less questions, was a lot less friction. And what we decided to do this year is just start early. So we won’t release the date just yet. You got to listen in a little bit further in the episode. But we are looking to run a next iteration of the mentorships, starting the program early, giving the mentees enough time before the official quote unquote start date to get onboarded. So that they can really take advantage of those 12 weeks. That’s kind of what we’re thinking of and just keep an eye out for later for those dates and important information.

Kairo De Araujo (13:04)
Yeah, I want to say that this year I will be participating again. And the good thing is that the mentees from last year will help me doing the mentorship. So we will distribute the tasks. So as you can see how beneficial it can be for our project and for mentors also, engaging to do that.

Sally Cooper (13:26)
Yeah, that’s really full circle. Thanks for sharing that. Well, since the mentorship cycle is on the horizon and the expectations are set or being set, if someone wanted to participate and join this year’s cycle for the first time, what would you give for a potential mentor for key advice based on what you did last time?

Kairo De Araujo (13:48)
Well, communication is the key because everybody is remote, everybody has different backgrounds. So, I advise to really making clear the communication with the mentees, understand what are the goals, understand what are their backgrounds, right? And we have preset projects within the product to be…done by the mentees.

Be flexible as well, because maybe you need to shape a little bit those projects to fit well for the mentees. These are my key advice that I can give. And focus on the communication with the folks, because they can deliver very good outcomes from that.

Sally Cooper (14:48)
Yeah, the outcomes will definitely come from communication. And that’s key. So for a potential mentee, what are expectations you’d set to help them make the most of this structured and paid opportunity?

Kairo De Araujo (15:02)
What I can say for the mentees is that don’t be afraid to join. Don’t care about your background or from where you are coming. You have something to help in the project. Work together with your mentor and now also work together if you have other mentees together with you in the mentorship program, try to work together. Because everybody is here to help. Be relaxed, try to do the best you can, get committed with what you want to deliver.

But as I said before to the mentors, communication, also communicate well. Ask questions, and try to help as well because you…And try to do what I always say, try to do a good pre-emboarding that it’s like, try to understand the project. I’m not saying that you need to know everything, just to understand what are you doing, what are the products, what you need to deliver and enjoy it because it’s really, really good.

Yesenia (16:17)
Yeah, one thing I wanted to jump in and add is for the project maintainers, if you need help with some of your onboarding documents, the BEAR working group is working through a process to create onboarding documents. So it is an added bonus that we can help you with that, especially if you’re a single maintainer or your team is just over at capacity right now. We are working through onboarding documents for OpenBao. So we could expand that process to other teams just to something on the floor to make it a lot easier.

Sally Cooper (16:51)
Wow, that’s so helpful. It’s really great to know that the bear working group’s doing that. For those listening who are excited to join the next cycle, can you walk us through those dates, Yesi, that we were talking about? So if you’ve stayed long enough for this, here’s your payoff because we’re gonna learn all about these upcoming dates.

Yesenia (17:13)
Yes, so the application will be released around March 24th and will be open for a few weeks ending April 12th. So that’s a good amount of time. Check us out at OpenSSF on our socials, on Slack, follow me on LinkedIn, me and Kairo, OpenSSF. I go ahead and will be reposting this and blasting it everywhere. Then from April 13th to April 30th, the mentors are going to be reviewing the applications. May 1st, you should expect an accepted decline notification by May 1st.

May 5th to the 29th, we’ll be working with getting you onboarded to the LF platform and onto the project. So this is when you’ll be getting your environment up, getting any documentation that could take some quiet time. So we’ll ask for you to be a part of it. And then the mentorship kicks off June 1st to August 21st. Within there, these we forgot to mention. This is a paid mentorship.

So, there are two evaluation points. July 10th will be your first evaluation. After that, you get half of the siphon. And then August 28th, that’s an important date for me, you get the final siphon from after you perform your evaluation. The cool part of this is not only do you get to do your mentorship program, but you’ll be part of a BEAR welcome call, where we would showcase your project.

So you’ll be able to get a public recording where you present the work that you’ve done over the mentorship. And as an option, as you heard previously, the mentees that became mentors will be on podcasts. So as an added option, if you’re welcome to share your voice, we will also love to interview you after the project for the OpenSSF, What’s in the SOSS Podcast.

These are great because you get to put them on your resume, you get to put them on your LinkedIn, show your parents, show your mom, show your dad, your grandma, your grandpa, your dog, whoever it is that you want to share it with. I definitely give my dogs my podcast episodes because they’re very proud of me. But those are just some key highlights and if you have more questions, find us on Slack and ask us and we’ll let you know.

Sally Cooper (19:43)
love it. And cats too. Plug to my cats.

Yesenia (19:47)
My cats are always here, so they hear everything anyway.

Sally Cooper (19:50)
Okay. Yeah, they hear it all. I love it. Well, thank you both so much. This has been a really interesting conversation. I learned a lot. Really excited for this next session. And to see all the great work that’s going to come out of it. Thank you. But before we wrap, are there any other calls to action for the audience if someone’s listening?

I know you gave the dates, what’s like the next best step for them?

Yesenia (20:15)
From the BEAR working group perspective, we have, we didn’t name the projects. We have gittuf, rstuf, thank you, SBOMit and Minder that are coming on board as mentors. So if you’re not sure what those are, take a moment, go to the openssf.org/getinvolved page, look at the working groups, check out their GitHub, get on Slack and check out the groups. Join one of the public calls. If you’re too nervous or introverted (I dropped after my first call, so don’t worry). Find different resources so you can be familiarized with the project that you might like and enjoy.

We also have a BEAR welcome call that we did in January that walks through all the working groups. So that’s also a good avenue to start. Let’s say you look at the projects, none of them really excite you. Mind you, they are paid. You can check out some of the other working groups and start getting involved in there as well.

Kairo De Araujo (21:18)
Yeah, even beside the mentorship, if you are not able to join the mentorship this summer, or if you don’t feel comfortable yet to join, our project repository service for tuf (rstuf) is really looking for new contributors. And we’ll do like in the mentorship, we’ll guide you to join the project, get the community, we’ll help you through that.

You can make a lot of difference out there if you want to collaborate with us. So everybody is welcome in our project as well.

Sally Cooper (21:54)
Fantastic. Well, Yessenia, Kairo, thank you so much for your time today and all the work that you’re doing for the mentorship program and the bear working group. We appreciate you both and to everyone listening, happy open sourcing and that’s a wrap.

Mar 10

Love0

What’s in the SOSS? Podcast #55 – S3E7 The Gemara Project: GRC Engineering Model for Automated Risk Assessment

By Jeff Diecks Podcast

Summary

Hannah Braswell and Jenn Power, Security Engineers from Red Hat and contributors to the OpenSSF, join host Sally Cooper to discuss the Gemara project. Gemara, an acronym for GRC Engineering Model for Automated Risk Assessment, is a seven-layer logical model that aims to solve the problem of incompatibility in the GRC (Governance, Risk, and Compliance) stack. By outlining a separation of concerns, the project seeks to enable engineers to build secure and compliant systems without needing to be compliance experts. The speakers explain how Gemara grew organically to seven layers and is leveraged by other open source initiatives like the OpenSSF Security Baseline and Finos Common Cloud Controls. They also touch on the ecosystem of tools being built, including Queue schemas and a Go SDK, and how new people can get involved.

Listen on Apple Podcasts Listen on Spotify Listen on Overcast Listen on Pocket Casts

Conversation Highlights

00:00 Welcome music + promo clip
00:22 Introductions
02:17 What is Gemara and what problem does it address?
03:58 Why do we need a model for GRC engineering?
05:50 The seven-layer structure of Gemara
07:40 How Gemara connects to other open source projects
10:14 Tools available to help with Gemara model adoption
11:39 How to get involved in the Gemara projects
13:59 Rapid Fire
16:03 Closing thoughts and call to action

Episode Links

Transcript

Sally Cooper (00:22)
Hello, hello, and welcome to What’s in the SOSS, where we talk to amazing people that make up the open source ecosystem. These are developers, security engineers, maintainers, researchers, and all manner of contributors that help make open source secure. I’m Sally, and today I have the pleasure of being joined by two fantastic security engineers from Red Hat. We have Hannah and Jenn.

Thank you both so much for joining me today and to get us started, can you tell us a little bit about yourselves and the work that you do at Red Hat? I’ll start with Jenn.

Jenn Power (00:58)
Sure. I am Jenn Power. I’m a principal product security engineer at Red Hat. My whole life is compliance automation, let’s say that. And outside of Red Hat, I participate in the OpenSSF Orbit Working Group, and I’m also a maintainer of the Gemara project.

Sally Cooper (01:18)
Amazing. Thank you, Jenn and Hannah. How about you? Hi.

Hannah Braswell (01:21)
Hey, Sally. Thanks for the nice introduction. I’m Hannah Braswell, and I’m an associate product security engineer at Red Hat. And I work with Jenn on the same team. And I primarily focus on compliance automation and enablement for compliance analysts to actually take advantage of that automation. Then within the OpenSSF, I’m involved in the Gemara project. I’m the community manager there. And then

I’m kind of a fly on the wall at a lot of the community meetings, whether it be the Gemara meeting or the orbit working group. I like to go to a lot of them.

Sally Cooper (02:01)
we love to hear that. I heard Orbit working group from both of you. That’s exciting. And I also really want to dive in to the project Gemara. So before we do dive into those details, let’s make sure that everyone’s starting from the same place. So for listeners who are hearing about Gemara for the first time, what is Gemara and what problem is it designed to address?

Jenn Power (02:23)
Sure, can start there. It’s actually secretly an acronym. So it stands for GRC Engineering Model for Automated Risk Assessment. So that’s kind of a mouthful, so we just shorten it to Gemara. And the official description I’ll give, and then I can go into it like a little bit more of a relatable example, is that it provides a logical model for describing categories of compliance activities, how they interact,

And it has schemas to enable automated interoperability between them. So like, what does that mean? I think a good, if we anchor this in an analogy, we could call Gemara like the OSI model for the GRC stack. In fact, that was one of the kind of primary inspirations for the categorical layers of Gemara. And Gemara also happens to have seven categorical layers, just like the OSI model.

So if you think about it in networking, if I want to send an email, I don’t have to understand like packet routing. I can just send my email. So in GRC, we can’t really do that today. We have security engineers that also have to be compliance experts to be successful. And so with Gemara, we want to outline the separation of concerns within the GRC stack to make sure that like each specialist can contain their complexity in their own layer while allowing them to exchange information with different specialists completing activities in different layers.

So like if I could give one takeaway, we want to make it so engineers can build secure and compliant systems without having to understand the nuance of every single compliance framework out

Sally Cooper (04:14)
I love that. So we have a baseline now. Let’s talk about the problem and get a little bit deeper into that. So Gemara is responding to a problem that you touched upon. Why do we need a model for GRC engineering and what incompatibility issue are you trying to solve? If you could go a little deeper.

Jenn Power (04:34)
Sure. So I think sharing resources in GRC is just really hard today. Sharing content, sharing tools, none of those tools and content, it doesn’t work together today, if I could say that. engineers are typically having to reinterpret security controls. They’re having to create a lot of glue code to make sure that a tool like a GRC system and a vulnerability scanner can actually talk to each other.

So we’re trying to solve that incompatibility issue on the technical side. But this is also a human problem. And I think that’s kind of the sneakiest part about it. A lot of times, we’re not even saying the same things when we use the same terms. And so that’s another thing that we’re trying to solve within the Gemara project.

This one comes up all the time. Take the word policy. If you say that to an engineer, you’re thinking immediately, policy as code, like a rego file or something you’re going to use with your policy engine. But if you’re talking to someone in the compliance space, they’re thinking like, this is like my 40 page document that outlines my organizational objectives. So we created definitions within the Gemara project to go along with the model to solve the human problem while we’re also trying to solve the technical problem.

Sally Cooper (06:05)
That’s interesting. Okay, I heard you say something about a seven-layer structure. Can you tell me why you chose a seven-layer structure for Gemara?

Jenn Power (06:17)
So this actually stemmed from an initiative under the CNCF called the Automated Governance Maturity Model. And that started as four concepts actually, policy, evaluation, enforcement, and audit. And that established the initial kind of lexicon that the project had been using.

And it initially got some adoption in the ecosystem, specifically in projects under the Linux Foundation, like FINOS Common Cloud Controls (CCC) and the Open Source Project Security Baseline (OSPS Baseline). And through the application of that lexicon, we found that there needed to be more granularity within that policy layer. So it expanded to two new layers called guidance and controls.

And I didn’t mention that we were creating a white paper yet, but we do have a white paper. And through the creation of that white paper, which Eddie Knight did so much work to create that initial draft there, we actually found that we were missing a layer. We had a seventh layer, and it was something that we had called sensitive activities. And it’s something kind of sandwiched in the middle of the Gemara layer. And so with that, we kind of organically grew to seven layers. So that I think is the kind of origin story on how the layers got to seven.

Sally Cooper (07:54)
I love that. And you’re really talking about how Gemara is not built in isolation, that you’re working with other open source projects. For example, you mentioned Baseline and the FINOS Common Cloud Controls. Can you tell me how Gemara connects to those projects?

Hannah Braswell (08:09)
Yeah. So in terms of Gemara connecting to the other open source projects, the first thing that comes to mind is really the CRA because of how prominent it is right now and just the future of its impact. And I really think that Gemara is going to be a catalyst for open source projects in general that are in need of some kind of mechanism to, you know, implement security controls and align with potential compliance requirements.

And the good thing about Gemara is that you don’t have to be a compliance expert to make sure that your open source project is secure. And so I would say that the OSPS Baseline is a great example of Gemara’s layer two, because it provides a set of security controls that engineers can actually implement. So in that case, other projects can reuse the baseline controls and then fit them to their needs.

And I think it also goes to say that, anyone that is actually building a tool they want to sell or distribute in the European Union market that’s using the open source components, they’re gonna have to think about what’s in scope and having something like the OSPS Baseline to understand how to effectively assess your open source components and their risks is really, really valuable and just gonna be super useful. And then in terms of the FINOS Common Cloud Controls, I think that’s

Also another great example, just in terms of the use case and implementation of Gemara, because they have their core catalog, which has its own definitions of threats and controls that’s then imported to their technology specific catalogs. And yeah, so that’s a great implementation within the financial sector.

And then where we’re trying to expand the ecosystem for Gemara, as in the Cloud Native Security Controls catalog refresh. And that’s actually an initiative that Jenn is leading. I’ve done a few contributions to it, but it’s essentially an effort to take the controls catalog that currently exists as a spreadsheet and make it available as a Gemara layer one machine readable guidance document. So Gemara is really connecting to projects that are all great to have on your radar, especially with the CRA coming up.

Sally Cooper (10:26)
Wow, that sounds great. But I’m just thinking about our listeners. They’re probably wondering, like, what does this look like in practice? And I’m curious if there are any tools available to help with the Gemara model adoption.

Jenn Power (10:39)
So we’re actually working on an ecosystem of tools. So we want to bridge that theory that we’re creating within the Gemara white paper to things that are actually implementable just to make sure that you don’t have to start from scratch if you’re trying to implement the Gemara model.

So we have a couple tools within the ecosystem. One would be our implementation of the model. We’re using queue schemas to allow users to create the models like in YAML, for instance, if you wanted to create your layer two, you would create YAML, you could use our queue schemas to validate that your document is in fact a Gemara compliant document. And then we’re also building SDKs. Right now we have a Go SDK, so you can build tooling around the programmatic access and manipulation of Gemara documents. A tool in the ecosystem that’s using this currently is a tool called Privateer that automates the layer five evaluations.

Sally Cooper (11:47)
Wow, that’s great. And of course, none of this works without the people. So I know you mentioned a few of them. How can new people get involved in the Gemara project?

Hannah Braswell (11:58)
So anyone that’s new and interested in getting involved in the Gemara project, my first piece of advice would just be to jump in a community meeting and listen in on what’s happening. I know I started out just by joining those meetings and I, you know, I didn’t necessarily have much to say, but I appreciated the culture and the open discussion, just like bouncing ideas back and forth off of one another.

And there’s also plenty of times when I joined a community meeting and still trying to understand the project if there was some kind of group opinion trying to be formed. Like I think it’s perfectly fine to say, you know, I don’t have the information right now. I don’t have an opinion. I’m still trying to learn about the project. But if something piques your interests and you want to contribute, then volunteer for it or show you’re interested because people are not going to forget about your willingness to step up and be part of the community.

But I started joining those meetings before we were rolling out the white paper. So that kind of brings me back to my first piece of advice. So I’d really suggest reading the white paper first, because it describes the problem and the trajectory of the project so well, and in a really clear way that I think is super important context for anyone that wants to start contributing. And I mean, from there, I mean, I’m the community manager, but I started with small contributions.

that ended up supporting the community in terms of documentation and some other aspects of the project I was excited about and that I could contribute to. So I really think the contributions are dependent on what you’re interested in. And even if there’s some difference in opinion and perspective or background, all of that can make a huge difference for the community and anything from documentation to code or discussion and collaboration will count as valid contribution and effort. So I’d say to anyone that’s wanting to join the Gemara community and start contributing, I think you should just find an area that truly interests you and makes you excited and get involved.

Sally Cooper (14:02)
Oh, that’s great. Well, thanks so much. And before we wrap, we’re going to do rapid, rapid fire. So I hope you’re ready because this is the fun part. No overthinking, no explanations, just the first instinct, okay, that comes to you. And I’m going to bounce. Yes, exactly. I’m going to bounce back and forth and ask you both some questions. Ready?

Jenn Power (14:17)
I’ll close my eyes then.

Sally Cooper (14:25)
Okay, Hannah, you’re up first. Star Wars or Star Trek?

Hannah Braswell (14:29)
Star Wars.

Sally Cooper (14:30)
Nice, I love it.
And Jenn, same question, Star Wars or Star Trek?

Jenn Power (14:335)
Star Wars.

Sally Cooper (14:36)
Okay, we’re all friends here.
Okay, back to Hannah, coffee or tea?

Hannah Braswell (14:42)
Definitely coffee.

Sally Cooper (14:43)
Yay, cheers. That’s solid.
Jenn, morning person or night owl?

Jenn Power (14:49)
Night Owl.

Sally Cooper (14:50)
Ohh that tracks. Hannah, beach vacation or mountains?

Hannah Braswell (14:56)
Hmm beach vacation.

Sally Cooper (14:58)
Nice choice. Jenn, books or movies?

Jenn Power (15:02)
Movies.

Sally Cooper (15:03)
Nice. All right, last round. Hannah, favorite open source mascot?

Hannah Braswell (15:08)
Oh…Zarf. I think that looks like an axolotl. I used to be obsessed with axolotls. And I mean, ever since I saw that, I was like, that’s the mascot.

Sally Cooper (15:18)
I love Zarf too. Cool. Okay. That’s a really strong pick.
Jenn, I’m going to give you the same question. Favorite open source mascot?

Jenn Power (15:26)
actually love the OpenSSF goose. I think it’s so cute.

Sally Cooper (15:30)
Teehee, Honk, he’s the best. Okay, let’s bring it home, Hannah, sweet or savory.

Hannah Braswell (15:38)
Savory.

Sally Cooper (15:39)
interesting. Okay, and Jenn? Spicy or mild?

Jenn Power (15:46)
mild. I can’t handle any spice. I’m a baby.

Sally Cooper (15:51)
love it. That’s amazing. Well, thank you both so much for playing along. And as we wind things down, do you have any other calls to action for our audience if someone’s listening, and they want to learn more or get involved? What is like the best next step for them?

Jenn Power (16:05)
I would say read the white paper. We are looking for feedback on it and that is really a way to understand the philosophy and the architectural goals of Gemara. And if you’re looking to just like, hey I want to learn GRC, that’s a good first step. So I think that’s what I would say.

Sally Cooper (16:28)
Fantastic. Hannah, Jenn, thank you so much for your time today and for the work you’re doing for the open source security community. We appreciate you both. And to everyone listening, happy open sourcing and that’s a wrap.

Feb 09

Love0

What’s in the SOSS? Podcast #54 – S3E6 AIxCC Part 4 – Cyber Reasoning Systems: The Real-World Journey After AIxCC

By Jeff Diecks Podcast

Summary

In this final episode of our AI Cyber Challenge (AIxCC) series, CRob and Jeff Diecks wrap-up the journey from DARPA’s groundbreaking two-year competition to the exciting collaborative phase happening now. Discover how winning teams are taking their AI-powered vulnerability detection systems into the real world, finding actual bugs in projects like the Linux Kernel and CUPS. Learn about the innovative OSS-CRS project that aims to create a standard infrastructure for mixing and matching the best components from different systems, and hear valuable lessons about how to responsibly introduce AI-generated security findings to open source maintainers. The competition may be over, but the real work—and collaboration—is just beginning.

This episode is part 4 of a four-part series on AIxCC:

Listen on Apple Podcasts Listen on Spotify Listen on Overcast Listen on Pocket Casts

Conversation Highlights

00:00 – Welcome and Introduction to AICC
01:37 – OpenSSF’s AI Security Mission: Two Lenses
03:54 – Competition Highlights: What the Teams Discovered
07:43 – Real-World Impact: From Research to Production
10:44 – Lessons Learned: Working with Open Source Maintainers
13:13 – OSS-CRS: Building a Standard Infrastructure
14:29 – Breaking Down Walls: Post-Competition Collaboration
15:39 – How to Get Involved

Episode Links

Transcript

CRob (00:09.408)
Welcome, welcome, welcome to What’s in the SOSS, the OpenSSF’s podcast where I get to talk to the most amazing people in the planet that are either involved or on the outskirts of open source software and open source security. Today, we have a treat. We get to talk to one of my dear friends and teammates, Jeff, and we’re gonna dive into a topic that I really don’t know a lot about today.

So Jeff, why you introduce yourself to the audience and kind of describe what you do for the foundation.

Jeff Diecks (00:44.686)
Yeah, thanks, CRob. And hello, I’m Jeff Diecks. I’m a technical project manager with OpenSSF. And I’ve been involved in open source for 20 plus years now. Goodness. And I am OpenSSF’s lead on the AI cyber challenge program that we work on. And CRob is sort of telling you the truth. He’s been on the three episodes prior to this where he’s learned plenty about AICC, but we’re here to talk a little bit more about this and wrap up the series today.

CRob (01:17.582)
Yeah, these words you use, AI, that isn’t something I hear a lot about. Wink. Could maybe you recap for us, like what is the OpenSSF doing around AI security? And then just maybe give a brief recap about the AI CC.

Jeff Diecks (01:37.028)
Yeah, for sure. So OpenSSF in the world of AI, we have our AI ML Security Working Group that looks at security and AI from kind of two lenses. The first is AI for security, which is what we’ll be talking about here today, projects that help you AI to help improve the security of projects, and security for AI, which is securing all this new world of AI things and all the lessons we’ve learned about securing software. AI is software too and it needs securing. We have a whole suite of projects and work that focuses on that too. Specific to AIxCC, again, it’s the AI Cyber Challenge. It was a two-year competition run by DARPA and ARPA-H. If you’re just hearing this episode first, I encourage you to go back to the first episode in this series with Andrew Carney from DARPA and ARPA-H for an overview of the program. And then we got into some good conversations with a couple of the team leads from some of the winning teams. But the purpose of the competition was to use AI and develop new systems to both find and fix vulnerabilities in open source software that are important to our critical infrastructure. An interesting part about the competition… written into the rules for any of the competitors accepting prize money, they were obligated to release their software as open source.

CRob (03:07.214)
Nice. That’s awesome. Well, yeah, again, I say that a bit tug-and-cheek, but there has been quite a lot of activity, whether it’s in the working group or specific to the follow-ons to the AICC competition, which is what we’re here about today to kind of put a bow on these conversations and help encourage the community to engage and go forward. So we talked about, we talked to a couple of the teams, we talked to Andrew, kind of gave us an overview of the program.

From your perspective and your engagement with the community, we have a new cyber reasoning special interest group within the foundation. So what have the teams been up to subsequently since the August close of the competition?

Jeff Diecks (03:54.414)
Yeah, there’s really two parts of this and I’ve had the great honor of meeting with and speaking with a lot of the teams and learning about what they’ve been doing. But we first started with just conversations about, you know, their experiences with the competition and what they learned, similar to the couple of episodes that we did. And what was really interesting is, you know, every single team, there’s something of value that came out of their system. They excelled in a, you know, at least a specific area.

They were all finding bugs in real world software. Just a couple of highlights from some of the other teams. Team Theori, which was among the three winners, the third place winner, they had a unique approach. Their system, unlike the others, did not use fuzzing. It used pure LLM AI. so just an interesting variation and you potential there for that system to be super flexible because it doesn’t come with some of the requirements of projects already being set up with fuzzing. So interesting to see what becomes of their system there. And then in a couple of other cases, it was really interesting. Teams that have systems that are extremely capable, but for one reason or another,

CRob (05:08.142)
Yeah.

Jeff Diecks (05:18.096)
There was maybe a specific part of their system that just didn’t work well with the scoring mechanisms of the competition itself. So we had a team that was one of the best at generating patches for the issues that it found that was most effective. But as these things go, there was kind of a late change in the architecture of their system during the competition. the part of their system that was supposed to submit all these patches that got generated into the competition for scoring didn’t function correctly and didn’t submit everything. So they didn’t get credit for all their great work. But we’ll talk in a little bit about well, but it’s still a capable system. And now we can use it not for a competition, but for real stuff that’s potentially even more valuable. There was another that it was great at generating proof of vulnerabilities.

CRob (06:13.87)
Mm-hmm.

Jeff Diecks (06:15.328)
And, but they had made some assumptions based on the competition infrastructure and the system just didn’t perform well within the confines of the competition. But what was interesting about the architecture of their system was they would generate a potential result that they thought might have been a finding from part of their system. And then they would submit that potential result out to several other LLMs and have them feedback.

CRob (06:24.718)
Mm-hmm.

Jeff Diecks (06:45.176)
a verdict on whether they thought it might be effective or not. We kind of made the joke. It was like doing the poll of getting eight out of nine dentists to agree and decide on the submission. So those are some highlights from the competition. But you asked about what the team’s been up to in recent months. So that’s been really interesting. DARPA has kind of extended the incentives from their program and they’re offering

CRob (06:48.846)
Hmm.

Jeff Diecks (07:13.552)
incentives and rewards for the teams now taking these systems and using them in the real world against real open source projects and demonstrating that they’re effective there. And if they can demonstrate that, they earn additional reward money, which is encouraging the adoption and the transition of this research into real world usage. So we’ve had some interesting findings there and a few examples there.

CRob (07:27.374)
Awesome.

Jeff Diecks (07:43.248)
We’ve got Team 42 that we’ve been working with. They’ve focused a lot on their system seeming to be very effective in working with the Linux kernel and specifically some of the out-of-tree subsystems. They’ve found and reported several bugs and had some of them accepted, accepted patches. And actually later this week we’ve scheduled and we’re doing a consultation for that team with a kernel maintainer.

to give them feedback and help their research move forward and any guidance they can give on how to make their system more effective. So we’re looking forward to that conversation.

CRob (08:23.36)
Excellent.

Today, so we have a mixture of projects that are being donated. They’re all open source, but some of them are being donated like here, for example. Where would we go from here? What is the group’s thoughts about these, broadly the cyber reasoning systems and kind of what other interests or ideas are floating around to keep the momentum?

Jeff Diecks (08:52.91)
Yeah, there’s a few things. So one, OpenSSF is involved with the teams and we’ve formed, as you mentioned, the special interest group, the cyber reasoning system special interest group as the kind of continued home from the competition for all the teams to continue collaborating together and working there. some interesting developments so far, we’ve hosted having the teams present to one another.

of their work with real open source projects and examples of bugs they’ve found in the process they’ve been following and how they’ve been getting received from open source projects with what they’ve been submitting. So for example, Team Fuzzing Brain, who has donated their CRS systems for OpenSSF to host and support.

They’ve been working against a bunch of projects, but specifically they shared some examples of their work with the CUPS project where they found some bugs, they reported them, they’ve had some accepted patches, and they’ve gotten great feedback from the CUPS maintainers who are very appreciative of their work, both finding bugs and submitting patches, but also helping to generate and expand the fuzzing.

CRob (09:56.408)
God.

Jeff Diecks (10:17.028)
know, harness coverage of the project itself, which the systems are pretty capable of. So we’ve been learning a lot about the reporting process because it’s one thing to have these capable systems, but you know, it’s the world where like you and I and everybody else are a bit, you know, skeptics of just, you know, pure AI things, right? So we’re, working our way through and kind of learning from one another about

CRob (10:19.82)
very nice.

Jeff Diecks (10:44.078)
What’s the best way to keep humans in the mix and how are projects receiving these things? What are some lessons learned? So for example, we had a conversation in a SIG meeting where we’re talking about the patch submission process and some of the projects were kind of reacting. It was perhaps a bit too aggressive to just go ahead and introduce yourself by submitting a patch to the system.

CRob (10:46.51)
Mm-hmm.

Jeff Diecks (11:12.784)
right into the pull request queue of a project. And the group suggested maybe for the next go-round, it’s maybe a more polite way to introduce yourself by opening an issue, reporting how it was found, what it was found, all the supporting information, and then attaching a patch to be considered versus just, hey, here’s a PR. By the way, it came from AI.

So some interesting.

CRob (11:43.022)
Right. And we’ve heard a lot of feedback from upstream about their disinterest in that approach.

Jeff Diecks (11:50.772)
Yeah, well, and was a big focus of the scoring of the competition itself. That was among the feedback that we gave consistently for a couple of years of make sure you’re incentivizing the development of systems that don’t make life more difficult for maintainers, but hopefully make things easier and think about how these things will be received, not just technical capabilities.

But you mentioned, you know, donated projects and, you know, the one that I think is of real interest and, you know, for folks to follow along. So Team Atlanta led the way development of a project that we’re calling OSS-CRS, and bundled in with it, they intend to have something called CRS Benchmark.

And what these are for is it intends to be a standard infrastructure for building and running and evaluating all these CRSs and being able to kind of mix and match and use different parts of different ones for, you know, kind of a combined solution.

So, you know, if we think of a future where we’ve got, you know, a system like we talked about that’s most effective at generating patches, but we’ve got a different one that’s best at finding vulnerabilities.

CRob (12:58.755)
Wow.

Jeff Diecks (13:13.488)
And the hope is that through this standard interface, folks can leverage and kind of fine tune things to get the best performance and the best results out of a combination of systems rather than just relying upon a single one. So you can just think of it the way it’s intended to run. If you just imagine yourself at a command line prompt and you issue a OSS dash bug find dash CRS build.

then give it a configuration and a project, a compatible project, and that’ll build a system to run against. And then you can issue, thing, OSS bug find CRS run config project and the name of harness, and it’ll go off and do its thing. So again, you’re specifying which configurations you’re wanting to use, which subsystems you want it to.

CRob (14:01.666)
Mm-hmm.

Jeff Diecks (14:14.01)
pull from. So they’ve got an interesting roadmap. You know, we’re talking with them and, you know, hoping to bring our community and perspective to help support that project and its development, you know, and adoption into the real world.

CRob (14:29.176)
And I remember us talking around DEF CON last year. And I think the competition and the prize money are great. That was very exciting. But I’m most excited about this kind of phase we’re in now, where we’re seeing the teams with that ethical wall down between them from the competition. Now they’re actually able to talk and collaborate and share ideas. I’m really excited to see the community come together, helping support these students on these ideas.

Jeff Diecks (14:58.916)
Yeah, and that’s been the interesting part and part of why this it’s taken us a bit to get this whole podcast series out through the course of the competition for competition integrity reasons. You know, we were advising the competition organizers, but we weren’t interacting with the teams themselves. So we had to go through a whole process after the finals to introduce ourselves to all of the individual teams and let them know that we’re here and about and you know, the things we offer to help.

support them in the further development of the system. that’s been an interesting few months of lots of great conversations and seeing these teams come together within our working group and special interest group.

CRob (15:39.842)
So you’ve inspired me. I sure would like to know more on how to get involved. How can I do that?

Jeff Diecks (15:41.488)
Ha

Well, if your Monday afternoons at 1 p.m. Eastern are free, we have two different meetings that basically are in that time slot on alternating weeks. Our full AI ML security working group meets again on Mondays at 1 p.m. Eastern time on a biweekly basis. And on the alternating weeks, the cyber reasoning system special interest group meets in that time slot. You can find them.

CRob (16:09.326)
Peace.

Jeff Diecks (16:11.844)
You know, both of those meeting series on our community calendar at opensf.org slash calendar.

CRob (16:18.958)
Well, I want to thank you for helping shepherd and guide the folks in the competitions. We’re seeing some great results come out of this. And I’m really excited to see what our community and these amazing students kind of come up with on how to further the use of AI to help improve security on things. Yeah. And with that, we’ll say this is a wrap.

Jeff Diecks (16:42.308)
Sounds good. Thanks, CRob, and we’ll see you in the meetings.

CRob (16:48.911)
I for one welcome our new robot overlords and I wish everybody a happy open sourcing. Have a great day.

Feb 09

Love0

What’s in the SOSS? Podcast #53 – S3E5 AIxCC Part 3 – Buttercup’s Hybrid Approach: Trail of Bits’ Journey to Second Place in AIxCC

By OpenSSF Podcast

Summary

In the third episode of our AI Cyber Challenge (AIxCC) series, CRob sits down with Michael Brown, Principal Security Engineer at Trail of Bits, to discuss their runner-up cybersecurity reasoning system, Buttercup. Michael shares how their team took a hybrid approach – combining large language models with conventional software analysis tools like fuzzers – to create a system that exceeded even their own expectations. Learn how Trail of Bits made Buttercup fully open source and accessible to run on a laptop, their commitment to ongoing maintenance with prize winnings, and why they believe AI works best when applied to small, focused problems rather than trying to solve everything at once.

This episode is part 3 of a four-part series on AIxCC:

Listen on Apple Podcasts Listen on Spotify Listen on Overcast Listen on Pocket Casts

Conversation Highlights

00:04 – Introduction & Welcome
00:12 – About Trail of Bits & Open Source Commitment
03:16 – Buttercup: Second Place in AIxCC
04:20 – The Hybrid Approach Strategy
06:45 – From Skeptic to Believer
09:28 – Surprises & Vindication During Competition
11:36 – Multi-Agent Patching Success
14:46 – Post-Competition Plans
15:26 – Making Buttercup Run on a Laptop
18:22 – The Giant Check & DEF CON
18:59 – How to Access Buttercup on GitHub
21:37 – Enterprise Deployment & Community Support
22:23 – Closing Remarks

Episode Links

Transcript

CRob (00:04.328)
And next up, we’re talking to Michael Brown from Trail of Bits. Michael, welcome to What’s in the SOSS.

Michael Brown (ToB) (00:10.688)
Hey, thanks for having me. I appreciate being here.

CRob (00:12.7)
We love having you. So maybe could you describe a little bit about your organization you’re coming from, Trail of Bits, and maybe share a little insight into what your open source origin story is.

Michael Brown (ToB) (00:23.756)
Yeah, sure. So Trail of Bits is a small business. We’re a security R &D firm. We’ve been in existence since about 2012. I’ve personally been with the company about four years plus. I work there within our research and engineering department. I’m a principal security engineer, and I also lead up our AIML security research team. So Trail of Bits, we do quite a bit of government research. We also work for commercial clients.

And one of the common threads in all of the work that we do, not just government, not just commercial, is that we try to make it as public as much as we possibly can allow. So for example, sometimes, you know, we work on sensitive research programs for the government and they don’t let us make it public. Sometimes our commercial clients don’t want to publicize the results of every security audit, but to the maximum extent that our clients allow us to, we make our tools, we make our findings, we make them open source. And we’re really big believers in

that the work that we do should be a rising tide that raises all ships when it comes to the security posture for the critical infrastructure that we all depend upon, whether we’re working on hobbies at home and whether we’re building things for large organizations, all that stuff.

CRob (01:37.32)
love it. And how did you get into open source?

Michael Brown (ToB) (01:42.146)
Honestly, I’ve just kind of always have been there. So realistically, you know, the open source community is where a lot of the research tools that I started out my research career. That’s where you found them. So I started off a bit in academia. I got my undergrad in computer science and then went and did something completely different for eight years. And then when I kind of

Uh, you know, for context, I joined the military. flew helicopters for like eight years and basically nothing in computing. But as I was starting to get out, um, the army, I, know, I was getting married, about to have kids. I kind of decided I wanted to be, you know, around the house a little bit more often. um, you know, I started getting a master’s degree at Georgia Tech. They’re offering it online. And then after I did that, I went to go, um, do a PhD there and also work for their, um, uh, their applied research arm, Georgia Tech Research Institute.

So lot of the work that I was doing was, you know, cutting edge work on software analysis, compilers and AI ML. And a lot of the stuff that I, you know, built the tools that I did my research on, they came from the open source community. They were tools that were open sourced as part of the publication process for academic work. They were made publicly and available open source by companies like Trail of Bits before I came to work with them as the result of government research projects.

So, honestly, I guess I don’t really have much of an origin story for when I got there. I kind of just landed there when I started my career in security research and just stayed.

CRob (03:16.814)
Everybody has a different journey that gets us here. And interestingly enough, you mentioned our friends at Georgia Tech, which was a peer competitor of yours in the AICC competition, which you all on Trail of Bits team, I believe your project was called Buttercup. And you came in second place. You had some amazing results with your work. So maybe could you tell us a little bit about the…

Michael Brown (ToB) (03:33.741)
Yeah, that’s correct.

CRob (03:43.15)
What you did is part of the AI CC competition and kind of how your team approached this.

Michael Brown (ToB) (03:51.022)
Yeah. So, um, you know, at the risk of sounding a bit like a hipster, um, I’ve been working at the intersection of software security, compiler, software analysis, AI ML for, you know, basically, um, almost my entire, uh, career as a research scientist. So, you know, dating back to the earliest program I worked on, uh, for, DARPA was back in 2019. And, um, so this was, this was before the large language model was a predominant form of the technology or kind of became synonymous with AI. So.

CRob (04:04.719)
Mm.

Michael Brown (ToB) (04:20.792)
For a long time, I’ve been working and trying to understand how we can apply techniques from AI ML modeling to security problems and doing the problem formulation, make sure that we’re applying that in an intelligent way where we’re going to get good solid results that actually generalize and scale. So as the large language model came out and we started recognizing that certain problems within the security domain are good for large language models, but a lot of them aren’t.

When the AI cyber challenge came around, we always approached this and this, I was the lead designer, my co-designer, Ian Smith. And I, you know, when we sat down and make the, the original concept for what became Buttercup, we always took an approach where we were going to use the best problem solving technique for the sub problem at hand. So when we approached this giant elephant of a problem, we did what you do and you have an elephant and you’ve got to eat it, eat it one bite at a time.

So each bite we took a look at it and said, okay, you we have like these five or six things that we have to do really, really well to win this competition. What’s the best way to solve each of these five or six things? And then the rest of it became an engineering challenge to chain them together. Our approach is very much a hybrid approach. This was a similar approach taken by the first place winners at Georgia Tech, which by the way, if you’ve got to be beat by anybody being beat by your alma mater, it takes a little bit of this thing out of it. So, you know, we came in first and second place. It’s funny, I actually have another Georgia Tech PhD alumni.

CRob (05:33.832)
You

Michael Brown (ToB) (05:42.926)
on my team who worked on Buttercup. So Georgia Tech is very well represented in the AI cyber challenge. So yeah, we’ve always had a hybrid approach. The winning team had a hybrid approach. So we used AI where it was useful. We used conventional software analysis techniques where they were useful. And we put together something that ultimately performed really, really well and exceeded even my expectations.

CRob (05:45.458)
That’s awesome.

CRob (06:07.56)
I can say I mentioned in previous talks, I was initially skeptical about the value that could have been derived from this type of work. But the results that you and the other competitors delivered were absolutely stunning. You have converted me into a believer now that I think AI absolutely has a very positive role can play both in the research, but also kind of the vulnerability and operations management space. What

Looking at your buttercup. What is unique about your approach with the cyber reasoning system with the buttercup?

Michael Brown (ToB) (06:45.39)
Yeah, so it’s funny you say that we converted you. I kind of had to convert myself along the way. There was a time in this competition where I thought, you know, this whole thing was going to kind of be reliant on AI too much and was going to fall on its face. And then, you know, at that point, I’d be able to say like, see, I told you, you can’t use LLMs for everything. But then it turns out, you know, as we got through there, we use LLMs for two critical areas and they worked much better than I thought they would. I thought they would work pretty well, but they ended up working to a much better degree than I thought they actually would. you know, what makes Buttercup unique?

CRob (06:49.852)
Yeah.

CRob (07:00.678)
You

Michael Brown (ToB) (07:15.69)
is that, like I said, we take a hybrid approach. We use AIML for very good problems that are well-suited for AIML. And what I mean by that is when we employ large language models, we use them on small subproblems for which we have a lot of context. We have tools that we can install for the large language model to use to ensure that it creates valid outputs and outputs that can carry on to the next stage with a high degree of confidence that they’re correct.

CRob (07:30.076)
Mm-hmm.

CRob (07:43.912)
Mm-hmm.

Michael Brown (ToB) (07:45.934)
And then in the places where we have to create or sorry, in one of the places where we have to use conventional software analysis tools, those areas are very amenable to the conventional analysis. So, you what I mean by this? good example is, for example, we needed to produce a proof of vulnerability. We have to have a crashing test case to show that when we claim a vulnerability exists in a system, we can prove through reproduction that it actually exists. Large language models aren’t great.

at finding these crashing test cases just by asking it to look at the code and say, hey, what’s going to crash this? They don’t do very well at that. They also don’t do well at generating an input that will even get you to a particular point in a program. But fuzzers do a great job of this. So we use the fuzzer to do this. But one of the things about fuzzers is they kind of take a long time. They’re also more generally aimed at finding bugs, not necessarily vulnerabilities.

CRob (08:36.808)
Mm-hmm.

Michael Brown (ToB) (08:42.702)
So we use an AIML, or Large Language Model based accelerator, or a C generator, to help us generate inputs that were going to guide the fuzzer to either saturate the fuzzing harnesses that existed for these programs more quickly. They would help us find and shake loose more crashing inputs that correspond to vulnerabilities as opposed to bugs. And those things really, really helped us deal with some of the short analysis and short.

processing windows that we encountered in the AI cyber challenge. So was really a matter of using conventional tools, but making them work better with AI or using AI for problems like generating software patches for which there really aren’t great conventional software analysis tools to do that.

CRob (09:28.018)
So as you were going through the competition, which went through multiple rounds, are there anything that surprised you or that you learned that, again, you said your opinion changed on using AI? What were maybe some of the moments that generated that?

Michael Brown (ToB) (09:45.226)
Yeah, so there I mean there were a couple of them. I’ll start with one where I can pat myself on the back and I’ll finish with one where I was kind of surprised. So first, we had a couple of moments that were really kind of vindicating as we went through this. Our opinion going into this was that large language models, couldn’t just throw the whole problem at it and expect it to be successful. So going into this, there was a lot of things that we did that we did

CRob (09:49.405)
hehe

Michael Brown (ToB) (10:14.774)
two years ago when we first started out that, you know, that’s like two years ago, it’s like five lifetimes when it comes to the development of AI systems now. So there were some things that we did that didn’t exist before that became industry standard by the time we finished the competition. So things like putting your LLM queries or your LLM prompts in a workflow that includes like validation with tools or the ability to use tools.

CRob (10:29.298)
Mm-hmm.

Michael Brown (ToB) (10:43.062)
That was something that was not mainstream when we first started out, but that was something that we kind of built custom into Buttercup when it came particularly to patching. And then also using a multi-agent approach. know, a lot of the, you know, I don’t know, a lot of the hype around AI is that, know, you just ask it anything and it gives you the answer. You know, we’re asking a lot of AI when we say, here’s a program, tell me what vulnerabilities exist, prove they exist, and then fix them for me.

And also don’t make a mistake anywhere along the way. It’s way too much to ask. we found particularly with patching, were, back then, multi-agent systems or even agentic systems or multi-agentic systems were unheard of. were still just, we were still using chat GPT 3.5, still very much like chatbot interactions, web browser interactions.

CRob (11:16.564)
Yeah.

Michael Brown (ToB) (11:36.438)
integration into tools was certainly less widespread. So we had seen some very early work on archive about solving complex problems with multiple agents. So breaking the problem down for it. And we used this and our patcher ended up being incredibly good. was our most important and our biggest success on the project. Really want to shout out Ricardo Charon, who’s our leader, the lead developer for our patching agent.

CRob (11:47.976)
Mm-hmm.

Michael Brown (ToB) (12:06.414)
or for a patching system within for both the semi finals and finals in the ICC. He did an incredible job and we really built something that I, like I said, I regard as our biggest success. So, know, sure enough, and as we go through this two year competition, now all of a sudden, you know, multi agentic systems, multi agentic tool enabled systems, they’re all the rage. This is how we’re solving these challenging problems. And also a lot of this problem breakdown stuff that has made its way baked into the models now, the newer thinking and reasoning models from

Anthropic and open AI respectively. They you you can give it these large complicated problems and will do it will first try to break it down before trying to solve it. So you know we were building all that stuff into our system before. It came about so that that’s an area where you know, like I said, we learned along the way that we had the right approach from the beginning and it’s really easy to go back and say that that’s what we learned that we were right. So on the other side of this I will have to say, you know, I’ll reiterate I was really surprised at how well.

CRob (12:53.639)
Mm-hmm.

Michael Brown (ToB) (13:04.11)
language models were able to do some of the tasks we asked it to do. Part of it’s how we approach the problem. We didn’t ask too much of it. And I think that’s part of the reason why the large language models were successful. an area that I thought where it was going to be much more challenging was patching. But it turned out to be an area where, a certain degree, this is kind of an easier version of the problem in general because open source software, which are the targets of the AI cyber challenge, they’re ingested into the training.

CRob (13:08.924)
Mm.

Michael Brown (ToB) (13:31.404)
data for all of these large language models. the models do have some a priori familiarity with the targets. So when we give it a chunk of vulnerable code from a given program, it’s not the first time it’s seen the code. But still, they did an amazing job actually generating useful patches. The patch rate that I expected personally to see was much lower than the actual patch rate that we had, both in the semifinals and in the finals. So even in that first year window,

CRob (13:33.64)
Mm.

Michael Brown (ToB) (13:58.63)
I was really kind of blown away with how well the models were doing at code generating, code generation tasks, particularly small focused code generation tasks. So I think, think large language models are kind of getting a bad rap right now when it comes to like, you know, trying to vibe code entire applications. They’re like, gosh, this, this code is slop. It’s terrible. It’s full of bugs and stuff. Well, well, you did also ask you to build the whole thing. You know, if I asked a junior developer to build a whole thing, they probably also put together some.

CRob (14:07.366)
Yeah.

CRob (14:17.233)
Yeah.

CRob (14:26.258)
Yeah.

Michael Brown (ToB) (14:26.71)
and gross stuff. But when I ask a junior developer to give me a bug fix, much like the large language model, when I ask it for a more constrained version of problem, they tend to do a better job because there’s just fewer moving parts. So yeah, those are are two things I took away. One that, you know, like I said, I get to pat myself on the back for another that was actually surprising.

CRob (14:46.012)
That’s awesome. That’s amazing. So now that the competition is over and what does the team plan to do next beyond this competition?

Michael Brown (ToB) (14:57.098)
Yeah, so I mean, look, we spent a lot of our time over the last two years. A lot of I wouldn’t quite say blood. I don’t think anyone would bled over this, but we certainly had some tears. We certainly had a lot of anxiety. you know, we put a lot of we put a lot of ourselves into Buttercup. And so, you we want people to use it. So to that end, Buttercup is fully available and fully open source. know, DARPA made it a contingent, a contingency of participating in the competition that

CRob (15:09.917)
Mm-hmm.

Michael Brown (ToB) (15:26.892)
you had to make the code that you submitted to the semi finals and the finals open source. So we did that along with all of our other competitors, but we actually took it one step further. So the code that we submitted to the finals is great. It’s awesome, but it runs that scale. It used $40,000 of a $130,000, I think, total budget. And it ran across like an Azure subscription that had multiple nodes.

countless replicated containers. This is not something that everyone can use. We want everyone to use it. So actually in the month after we submitted our final version of the CRS, but before DEF CON occurred, where we figured out that we won, we spent a month making a version of Buttercup that’s decoupled from DARPA’s competition infrastructure. So it runs entirely standalone on its own, but more importantly, we scaled it down so it’ll run on a laptop.

CRob (16:18.696)
Mm-hmm.

Michael Brown (ToB) (16:25.154)
We left all of the hooks. We left all of the infrastructure to scale it back up if you want. So the idea now is that if you go to trail of bits.com slash buttercup, you can learn about the tool. have links to our GitHub repositories where it’s available and you can go, you can go download buttercup on your laptop and run it right now. And if you’ve got an API key, that’ll let you spend a hundred dollars. We can run a demo to show you that, we can find and patch a vulnerability and live.

CRob (16:51.496)
That’s easy.

Michael Brown (ToB) (16:53.164)
Yeah, so anyone can do this right now. So if you’re an organization that wants to use Buttercup, you can also use the hooks that we left back in to scale it up to the size of your organization, the budget that you have, and you run it at scale on your own software targets. So even users beyond the open source community, we want this to be used on closed source code too. So yeah, our goal is for, you you asked what we’re gonna do with it afterward. We made it open source, we want people to use it.

And even on top of that, we don’t want it to bit rot. So we actually are going to retain a pretty significant portion of our winnings of our $3 million prize. We’re going to retain a pretty significant portion of that. And we’re going to use it for ongoing maintenance. So we’re maintaining it. We’ve had people submit PRs that we’ve accepted. They’re tiny, know, it’s only been out for like a month, but you know, and then we’ve also made quite a few updates to the public version of Buttercup afterwards. So it’s actively maintained.

There’s money from the company’s putting its money where its mouth is. We’re actively maintaining it. The people who built it are part of the people who are maintaining it. We are taking contributions from the community. We hope they help us maintain it as well. And yeah, we’ve made it so anyone can use it. I think we’ve taken it about as far as we can possibly go in terms of reducing the barriers to adoption to the absolute minimum level for people to use Buttercup and leverage AI to help them find and patch vulnerabilities at scale.

CRob (18:16.716)
I love that approach. Thank you for doing that. How did you fit the giant check through the teller window?

Michael Brown (ToB) (18:22.574)
Fortunately, that check was a novelty and we did not actually have a larger problem than the AICC itself to solve afterward, which was getting paid. So yeah, we did have the comically large check, you know, taped up in our booth at the AICC village at DEF CON and it certainly attracted quite a few photographs from passersby.

CRob (18:26.716)
Ha ha ha!

CRob (18:31.964)
Yeah.

CRob (18:37.864)
Mm-hmm.

Michael Brown (ToB) (18:47.736)
I don’t know. think if you get on like social media or whatever and you look up AICC, if you see anything, there’s probably lots of pictures of me throwing a big smile up and you two thumbs up underneath the check that random people took.

CRob (18:59.464)
So you mentioned that Buttercup is all open source now. So if someone was interested in checking it out or possibly even contributing, where would they go do that?

Michael Brown (ToB) (19:07.564)
Yeah, so we have a GitHub organization. We’re Trail of Bits. You can find Buttercup there. You can also find our public archives of the old versions of Buttercup. So if you’re interested in what, like, the code that actually won the competitions, you can see what got us from the semifinals to the finals. You can see what won us second place in the finals. And you can also download and use the version that’s actively maintained that’ll run on your laptop. And all three of them are there. Their repository name is just Buttercup.

We are not the only people who love the princess bride. So there are other repositories named butter. Yeah. There are other, there are other repositories named butter cup on GitHub. So you might have to sift it a little bit, but yeah, basically it github.com slash trailer bits slash butter cup, I think is like 85 % of the URL there. don’t have it memorized, but, but yeah, you can find it publicly available and, along with a lot of other tools that, trailer bits has made over the years. So we encourage you to check some of those out as well. A lot of those are still actively maintained.

CRob (19:39.036)
That’s what it was.

Michael Brown (ToB) (20:03.72)
have a lot of community support. believe it or not at last, I counted like something like 1250 stars. buttercup is only like our fifth most popular tool that, that trail of bits has created. So, you know, we, we were quite, we were quite notable for creating some binary lifting tools that are up there. we have also some other tools that we’ve created recently for, parser security, analysis like that, like graftage.

And then also some kind of more conventional security tools like algo VPN, like still rank above buttercup. So as awesome as buttercup is, it’s like, it’s like only the fifth coolest tool that we’ve made as voted on by the community. So check out the other stuff while you’re there too. believe it or not, buttercup isn’t, isn’t, isn’t our most popular, offering.

CRob (20:51.56)
Pretty awesome statement to be able to say. That’s only our fifth most important tool.

Michael Brown (ToB) (20:53.966)
Yeah.

Michael Brown (ToB) (20:58.444)
I don’t know, you know, like personally, I’m kind of hoping that maybe we move up a few notches after people get time to like go find it and, you know, and start it. But, you know, we, we’ve, we’ve made some other really significant and really awesome contributions to the community, even outside of the AI cyber challenge. So I want to really, really stress all of that stuff is open source. We, you know, we, we aren’t just doing this because we have to, we actually care about the open source community. We want to secure the software infrastructure. We want people to use the tool and secure the software for, you know, before they, before they, you know,

Get it out there so that you we we we tackle this like kind of untackable problem of securing this massive ecosystem of code.

CRob (21:37.606)
Michael, thank you to Trey Labitz and your whole team for all the work you do, including the competition runner-up Buttercup, which did an amazing job by itself. Thank you for all your work, and thank you for joining us today.

Michael Brown (ToB) (21:52.802)
Yeah, thanks for having me. You one last thing to shout out there. If you’re an organization, you’re looking to employ Buttercup within your organization. Don’t be bashful about reaching out to us and asking about use cases for deploying within your organization. We know we’re happy to help out there. That’s probably an area that we focus on a little bit less in terms of getting this out the door for average folks to use or for individuals to use. So we’re definitely interested in helping to make sure Buttercup gets used.

Like I said, reach out to us, talk to us if you’re interested in Buttercup, we want to hear.

CRob (22:23.44)
Love it. All right. Have a great day.

Michael Brown (ToB) (22:25.678)
All right, thanks a lot.