Join us at Open Source SecurityCon Europe on March 23, 2026

Category

Podcast

Feb 03
Love0

What’s in the SOSS? Podcast #50 – S3E2 Demystifying the CFP Process with KubeCon North America Keynote Speakers

By Podcast

Summary

Ever wondered what it takes to get your talk accepted at a major open source tech conference – or even land a keynote slot? Join What’s in the SOSS new co-host Sally Cooper, as she sits down with Stacey Potter and Adolfo “Puerco” GarcĂ­a Veytia, fresh off their viral KubeCon keynote “Supply Chain Reaction.” In this episode, they pull back the curtain on the CFP review process, share what makes a strong proposal stand out, and offer honest advice about overcoming imposter syndrome. Whether you’re a first-time speaker or a seasoned presenter, you’ll learn practical tips for crafting compelling abstracts, avoiding common pitfalls, and why your unique voice matters more than you think.

Conversation Highlights

00:00 – Introduction and Guest Welcome
01:40 – Meet the Keynote Speakers
05:27 – Why CFPs Matter for Open Source Communities
08:29 – Inside the Review Process: What Reviewers Look For
14:29 – Crafting a Strong Abstract: Dos and Don’ts
21:05 – From Regular Talk to Keynote: What Changed
25:24 – Conquering Imposter Syndrome
29:11 – Rapid Fire CFP Tips
30:45 – Upcoming Speaking Opportunities
33:08 – Closing Thoughts

Transcript

Music & Soundbyte 00:00
Puerco: Stop trying to blend or to mimic what you think the industry or your community wants from you. Represent – always show up who you are, where you came from – that is super valuable and that’s why people will always want to have you as part of their program.

Sally Cooper (00:20)
Hello, hello, and welcome back to What’s in the SOSS, an OpenSSF podcast. I’m Sally and I’ll be your host today. And we have a very, very special episode with two amazing guests and they are returning guests, which is my favorite, Stacey and Puerco. Welcome back by popular demand. Thank you for joining us for a second time on the podcast.

And since we last talked, you both delivered one of the most talked about keynote at KubeCon. Wow. So today’s episode, we’re going to talk to you about CFPs. And this is really an episode for anyone who has ever hesitated to submit a CFP, wondered how to get their talk reviewed through the CFP process. Asked themselves, am I ready to speak? Or dreamed about what it might take to keynote a major event.

We’re gonna focus on practical advice, what works, what doesn’t, and how to show up confidently. And I’m just so excited to talk to you both. So for anyone who’s listening for the first time, Stacey, Puerco, can you tell us a little bit about yourselves? and about the keynote. Stacey

Stacey (01:48)
Hey everyone, I’m Stacey Potter. I am the Community Manager here at OpenSSF. And my job, I mean, in a nutshell is basically to make security less scary and more accessible for everyone at open source, right? I’ve spent the last six or seven years in open source community building across mainly CNCF projects, Flux, Flagr, OpenFeature, Captain to name a few.

And now focusing on open source security here at OpenSSF. Basically helping people connect, learn, and just do cool things together. And yeah, and I delivered a keynote at KubeCon North America that was honestly, it’s still surreal to talk about. It was called Supply Chain Reaction, a cautionary tale in case security, and it was theatrical. It was…slightly ridiculous. And it was basically a story of a DevOps engineer who I played the DevOps engineer, even though I’m not a DevOps engineer, frantically troubleshooting a compromised deployment. And Puerto literally kaboomed onto the stage as a Luchador superhero to save the day. had him in costume and we had drama.

And then we taught people a little bit about supply chain security through like B-movie antics and theatrics. But it turns out people really responded to making security fun and approachable instead of terrifying.

Adolfo GarcĂ­a Veytia (@puerco) (03:23)
Yeah. Well, hi, and thanks everybody for listening. My name is Adolfo GarcĂ­a-Veytia. I am a software engineer working out of Mexico City. I’ve been working on open source security for, I don’t know, the past eight years or so, mainly on Kubernetes, and I maintain a couple of the technical initiatives here in the OpenSSF.

I am now part of the Governing Board as starting of this year, which is a great honor to have been voted into that position. But my real passion is really helping build tools that secure open source while being unobtrusive to developers and also raising awareness in the open source community about why security is important.

Because sometimes you will see that especially executives, CISOs, and they are compelled by legal frameworks or other requirements to make their products or projects secure. And in open source, we’re always so resource constrained that security tends to be not the first thing on people’s minds. But the good news is that here in the OpenSSF and other groups, we’re working to make that easy and transparent for the real person as much as possible.

Sally Cooper (04:57)
Wow, thank you both so much. Okay, so getting back to call for proposals, CFPs. From my perspective, they can seem really intimidating, but they’re also one of the most important ways for new voices to enter community. So I just have a couple questions. Basically, like, why are they important? So not just about like going to a conference, but why is it important to get

Why would a CFP be important to an open source community and not just a conference? Stacy, maybe you could kick that off.

Stacey (05:32)
Sure, I think this is a really important question. I think CFPs aren’t just about filling conference slots. They’re really about who gets to shape the narrative in our communities and within these conferences. So when we hear the same voices over and over and they show up repeatedly, right, you get the same perspectives, the same solutions, the same energy, which, you know, is also great. You know, we love our regular speakers, they’re brilliant, but

communities always need new and fresh perspectives, right? We need the people who just solved a weird edge case that nobody’s talking about. We need like a maintainer from a smaller project who has insights that maybe big projects haven’t considered, or, you know, we need people from different backgrounds, different use cases and different parts of the world as well. CFPs are honestly one of the most democratic ways we have to surface new leaders, right?

Sometimes someone doesn’t need to be well-connected or have a huge social media following. They just need a good idea and the courage to submit a talk about it, right? And that’s really powerful. And I think when someone gives their first talk and does well, they often become a mentor, a maintainer, a leader in that community, right? CFPs are literally how we build the next generation of contributors and speakers. So every talk is a potential origin story for someone’s open source journey.

Sally Cooper (07:08)
Puerco, what are your thoughts on that?

Sally Cooper (07:11)
And the question again is call for proposals can feel really intimidating, but they’re also one of the most important ways for new voices to enter a community.

Adolfo GarcĂ­a Veytia (@puerco) (07:20)
Yeah. So, I would say that intimidating is a very big word, especially for new people. maybe, Sometimes it’s difficult to ramp up the courage and I don’t want to mislead people into thinking it’s going to be easy. The first ones that you do, you will get up there, sweat, stutter, and basically your emotions will control your delivery and your body, so be prepared for that.

But it’s going to be fine. The next times you’ll do it, it will get better. And most importantly, people will not be judging you. In fact, it’s sometimes even more refreshing to see new voices getting up on stage.

Sally Cooper (08:13)
That’s really helpful. Thank you. I love it. The authenticity that you bring really helps and helps demystify the CFP process. But now let’s pull back the curtain on the review process. How does that work? And Stacey, have you been on a review panel before? Maybe you could talk about like, when you’re reviewing a CFP, what are you actually looking for?

Stacey (08:39)
Yeah, I’ve been on program committees. I’ve been on a program chair or co-chair on different programs and things like that. yeah, it’s a totally different experience, but I think it gives you lot of insight on how to prepare a talk once you’ve reviewed 75, 80 per session, right? It’s sometimes these calls are really big. I know KubeCon has really huge calls, right? But I would say, you know what we’re actually looking for:

So first, is this topic relevant and useful to our audience? Like, will people learn something they can actually apply? And second, like, can this person deliver on what they’re promising? And honestly, we’re looking we’re not looking for perfection, right? We’re looking for clarity and genuine expertise or experience like with that topic.

I would say be clear, be specific with your value proposition in the first two sentences of a CFP. When the program committee can read your abstract and immediately think, “oh that’s exactly what our attendees need,” right? That’s like gold, right? Also, when somebody shows that they understand the audience, that they’re they’re submitting to, right? Are you speaking to beginners or experienced practitioners and being explicit about that?

Adolfo GarcĂ­a Veytia (@puerco) (10:16)
Yeah, I think it’s important for applicants to understand who is going to be reviewing your papers. There are many kinds of conferences and I would… So ours, even though, of course, there’s commercial behind it because you have to sustain the event, like everybody involved in… Especially in the Linux Foundation conferences, I feel…

we put a lot of effort into making the conferences really community events. And I would like to distinguish the difference, like really make a clear cut between what is academic conferences, like purely trade show conferences and these community events. And especially in academia, there’s this hierarchical view of peers.

assessing what you’re doing. In pure trade show conferences, it’s mostly pay to play, I would say. And when you get down to community, especially if you ever applied to present or submit papers to the other kinds of conferences, you will be expecting completely different things. It’s easy to forget that people looking at your work, at your proposals, at your ideas is very, very close and very, very similar to you.

So don’t expect to be talking to some higher being that understands things much better than you. First of all, it’s not one person. It’s all of us reading your CFPs. keeping that in mind, what you need to keep like consider when submitting is what makes my proposal unique. I think that’s a key question. And we can talk more about that in the later topics, but I feel, to me, when I understood that it was sometimes even my friends reviewing my proposal made it so much easier.

Stacey (12:20)
Yeah, I think that’s a really, really good point Peurco makes is knowing that whatever conference you’re submitting for typically, and I say this like if it’s a Linux Foundation event, right? Because those are the ones that I’ve been most involved with. The program committee members are from within the community. They are, they submit an application to say, hey, yes, I would love to review talks. This is like me volunteering my time to help out this conference. Maybe they’re not able to make the conference.

Maybe they are, maybe they’re also submitting a talk. But usually the panel of reviewers is like five, six, up to 10 people, I would say, depending on the size of the conference. So you’re getting a wide range of perspectives reading through your submissions. And I think that’s really important. When I’m trying to select the program committee, I think it’s really important to diversify as well, right? So have voices from all over – different backgrounds, different expertise, different genders, just as much variance as you can have within the program committee panel, I think also makes a difference with the CFP reviews themselves, right?

But that’s kind of how it’s set up, is you pick these five to 10 people to review all of these CFPs, they have usually, it’s like a week or something like that to review everything, and then they rate it on a scale. And then that’s kind of how the program chairs then arrange the schedule is based off of all that feedback. You can make notes in each of the talks that you’re reviewing, you know, put those in there and then, and that’s basically how they’re all chosen. They’re ranked and they have notes, right, within that system.

Sally Cooper (14:08)
Wow, this is really educational. Thank you so much. For folks that are staring at a CFP right now, because there’s some coming up, and I think we’re going to get into that. Let’s get practical. What makes a strong abstract? How technical is too technical? How much storytelling belongs in a CFP? And what are some red flags that you might see in submissions?

Adolfo GarcĂ­a Veytia (@puerco) (14:34)
So, the first big no-no in community events is don’t pitch your product. Even if you trying to disguise it as a community event, the reviewers will … You have to keep in mind that reviewers have a lot of work in front of them. I am sure people, there are all sorts of reviewers, but usually as a reviewer, you see that folks put a lot of effort into crafting their proposals.

If you pitch your product, which is against the rules in most conferences, in the community conferences, the reviewer will instantly mark your proposal down. We can sniff it right away. You have to understand that for us, the more invalid proposals we can get out of the way as soon as possible, that will happen. If it is a product pitch, just don’t.

And then the next one is you have to be clear and concise in the first paragraph or sentence even. So when a reviewer reads your proposal, make sure that the first paragraph gives you an idea of, so this is going to be, I’ll talk about this and it’s gonna like…inspect the problem from this side or whatever, but give me that idea. And then you can develop the idea a little bit more on the next couple of paragraphs, but make sure that the idea of the talk is delivered right away. I have more, but I don’t know, Stacey, if you want to.

Stacey (16:20)
Yeah, no, I think that’s really good advice. would say whatever conference that you’re submitting, being on so many different program committees, I’ve seen the same talk submitted to every conference that has an Open CFP, regardless of the talk being specific to that conference or not. So think that’s key number one is make sure that what you’re submitting fits within the conference itself.

I think not doing a product pitch is key – especially within an open source community, open CFP, right? Those are only for open source, for non-product pitches. I think Puerco makes a really good point with that. But, you know, like, is this conference that I’m submitting this talk to higher level? Is it super technical and adjusting for those differences, right? A lot of times you’ll find in the CFPs that there is room to submit a beginner level, an intermediate level, an advanced level, but typically the conference description and the categories and things like this, you want to be very specific when you’re writing your CFP. You could sometimes you reuse the same CFP you’ve submitted to another conference, but you want to tailor it to each specific conference that you are submitting for.

Don’t just submit the same talk to five different conferences because they are unique, they are specific and you want to make sure that if you want your talk accepted, these are the little changes that make a big difference on really getting down to the brass tacks of what that conference is about and what they’re really looking for. So I always have to, when I’m writing something and when I’m looking at a conference to write it for, I have the CFP page up, I have the about page up for that conference and I’m making sure that it fits within what they’re asking me for, really.

Adolfo GarcĂ­a Veytia (@puerco) (18:20)
Yeah. And I just remember another one. And this is mostly, this happens most in the bigger ones, like the Cubicums and so on. Don’t try to slop your way into the conference. if you, I mean, it’s like, I’d rather see a proposal with bad English-ing or typos than something that was generated with AI. And I’ll tell you why.

It’s not because like, pure hates of AI or whatever. no. The problem with running your proposal into an LLM is that most of the time, so you have to keep in mind, especially in the big conferences, you will be submitting a proposal about the subject that probably then other people will be trying to talk about the same thing. And what will get you picked is your capability of expressing like…getting into the problem from a unique way, your personality, all of those things.

When you run the proposal through the LLM, it just erases them. All sorts of personal, like the uniqueness that you can give it will just be removed. And then it’ll be just like looking at the hollow doll of some of the person and you will not stand out.

Stacey (19:38)
Yeah, I agree completely – and…is it a terrible thing to have AI help you with some of the editing? No, not at all. But write your proposal first. Write it from your heart. Write it from your point of view. Write it from your angle. But do not create it in AI, in the chatbots. Create it from yourself first, and then ask for editing help. That’s fine.

I think a lot of us do that and a lot of people out there are using it for that extra pair of eyes. Do I sound crazy here? Does this make any sense? I don’t know how to word this one particular sentence. That’s fine. But yeah, don’t start that way.

Adolfo GarcĂ­a Veytia (@puerco) (20:19)
Exactly. mean, and just to make it super clear, it’s not that, especially people whose first language is not English like me. I of course use help of some of those things to like at least don’t like introduce many types or whatnot, but just as Stacey said, don’t create it there.

Sally Cooper (20:41)
This is great advice. Thank you both so much. Okay. How about getting accepted for a keynote? Like your KubeCon keynote really stood out. It was technical. It was really funny. memorable, engaging. How does someone prepare a keynote that differs from a regular talk?

Stacey (21:03)
Well, I want to start off by saying that we didn’t know, we weren’t submitting our talk for a keynote, right? We didn’t even know that that was like in the realm of possibility that could happen for KubeCon North America. We just submitted a talk that we thought would be fun, would be good, would give like, you know, some real world kind of vibes and that we wanted to have fun and we wanted to, you know, create a fun yet educational talk.

We had literally no idea that we could possibly have that talk accepted as a keynote. I didn’t know that. And this was my first real big talk. So it was a complete shock to me. I don’t know if you have other thoughts about that, but…

Adolfo GarcĂ­a Veytia (@puerco) (21:50)
Yeah, it sort of messes your plans because you had the talk planned for say 35 minutes and then you have 15 and you already had like 10 times more jokes that could fit into the 35 minutes. So, well…and then there’s also, course, like all of those things that we talked about, like getting nervous. Well, they not only come back, but they multiply in a huge way. I mean, you’ve been there. I don’t know. You get over it.

Stacey (22:28)
I would also say that once we found out that our talk was accepted first, were like, yay, our talk got accepted. And then I think it was like a few days later, they were like, no, no, your talk is now a keynote. So we freaked out, right? We had our little moment of panic. But then we just worked on it. And we worked on it, and we worked on it, and we worked on it, right? So not waiting till the last minute, I would say, to prep your talk.

But we…I think my main goal with this talk, and I have to give so much credit to Puerco because he’s such a good storyteller and he does it in such a humorous, but really technical and sound way. And we worked on this script. We wrote out an entire script because we only had 15 minutes. We went from a 25 minute talk to a 15 minute talk.

And so…pacing was really important, storytelling was really important, but also being funny was like something that I really wanted us to have, which Puerco was really good at too. And I think all of these things trying to squash it down into this 15 minutes was really tough, but I think that’s important to remember about keynotes versus talks is I think keynotes are more like, what is this experience of the talk about? Versus like, let’s get down to really technical details, right? You can do a technical talk that’s 25, 35, 45 minutes, but it’s a keynote. People aren’t going to remember anything from a keynote if you’re digging too, getting too deep in the weeds, right? So that was my focus. And I don’t know, Puerco, if you have anything else to add to that.

Adolfo GarcĂ­a Veytia (@puerco) (24:10)
Yeah, the other is that the audience is so much bigger that your responsibility just grows, especially to deliver, right? So as Stacey said, we actually wrote the script, rehearsed online, in person before the conference. And the experience also in the conference is very different because you have to show up early, you have to do a rehearsal in the prior days before your actual talk. And that’s said – nothing like it didn’t go perfect.
Like we still fumbled here and there and like messed up some of the details and the pacing and whatnot. it’s, I don’t know, at least in our case, it was about having fun and trying to get some of that fun into the attendees.

Sally Cooper (25:01)
Yeah, you really did. It was so fun. I think that’s what stood out.

Okay, one of the biggest barriers to submitting a CFP isn’t skill, it’s confidence. So what would you say to someone who feels like, I’m not expert enough. I don’t know if I have permission to do this. What you know, how do they deal? How do you personally deal with imposter syndrome? And why is it important to make sure that those new and diverse voices do submit at CFP?

Adolfo GarcĂ­a Veytia (@puerco) (25:27)
Oh, I’m an expert. So the first thing to remember, kids, is that Impostor Syndrome will never go away. In fact, you don’t want it to ever go away. Because Impostor Syndrome tells you something very, very important. And that is you are being critical of yourself, of your work, of your ideas. And if you ever stop doing that,

It means one, you don’t really understand the problem or the vastness of the problem that you’re trying to speak about and to talk about in your talk. And the other is you will stop looking for new and innovative ideas. So no matter where you get to, that imposter syndrome will ever be with you.

Stacey (26:20)
I agree. I don’t think it ever goes away. I feel like, you know, I was an imposter at the keynote. Absolutely was, right? Like, I didn’t know what the heck I was doing. I didn’t know what the heck I was saying half the time. I mean, I tried to memorize my lines and do the right thing and come off as this expert. I never, ever feel like an expert about anything, right? Unless I’m talking, I guess, about my cats or my kid or something.

Adolfo GarcĂ­a Veytia (@puerco) (26:47)
Yeah, exactly.

Stacey (26:49)
But yeah, think that’s, yeah, you’re pushing yourself to grow and that’s a good thing, right? So if you feel like an imposter, you know, that’s okay. And we all feel like that.

Adolfo GarcĂ­a Veytia (@puerco) (27:04)
Yeah. And the other, yeah, the other very important thing is think about what you are proposing to, to, to talk about in your talk. it’s supposed to be like new cutting edge stuff, like it’s something interesting, something unique. so it’s okay to feel about that because it’s, it’s a problem that you’re still researching that you’re trying to understand, that – especially think about – think about it this way.
If you propose any subject for your talk, anybody that goes there is more or less assuming that they want to know and learn more about it. if you feel confident enough to speak about it, like people will respond by willingness to attend your talk. That means you are already one little bit of a level above because you’ve done that research, you’ve done that in-depth dive into the subject. So it’s fine.

It’s fine to feel it. I realized that it’s a natural thing.

Stacey (28:05)
And most of the people in the audience are there to support you, to cheer you on, and are not gonna harp on you or say, oh gosh, you messed up this thing or that thing. They’re really there to give you kudos and really support you and be willing to hear and listen to what you have to say.

Sally Cooper (28:25)
Love that. Okay, let’s close the advice portion with a quick round of CFP tips rapid fire style. I’m going to go back and forth so each person can answer. Stacey will start with you. One thing every CFP should do.

Stacey (28:43)
I mean, get to the point as quickly as you possibly can. That would be my thing, right?

Sally Cooper (29:48)
Love it. Puerco, one thing people should stop doing in CFPs.

Adolfo GarcĂ­a Veytia (@puerco) (28:55)
Stop trying to blend or to mimic what you think the industry or your community wants from you. Represent. Always show off who you are, who you came from. That is super valuable and that’s why people will always want to have you as part of a program.

Sally Cooper (29:13)
Stacy, one piece of advice you wish you’d received earlier.

Stacey (29:18)
gosh, would say rejection is normal and not personal. I wish someone had told me that earlier, but that is one big, experience. Speakers get rejected all the time, right? It’s not about your worth. It’s about program balance, timing, and fit. So keep submitting.

Sally Cooper (29:39)
Okay, Puerco and Stacey, both got famous after this Puerco selfie or autograph?

Adolfo GarcĂ­a Veytia (@puerco) (29:44)
Selfie with a crazy face, at least get your tongue out or something.

Sally Cooper (29:50)
Stacey. KubeCon or KoobCon?

Stacey (29:54)
Oh gosh, I feel like this is like JIFF or GIF. And I’m in the GIF camp, by the way. I say KubeCon, even though I know it’s “Coo”-bernetes, I still say CubeCon, so.

Adolfo GarcĂ­a Veytia (@puerco) (30:07)
CubeCon, please.

Sally Cooper (30:09)
Okay, before we wrap up, Stacey, as the OpenSSF Community Manager, can you share some upcoming CFPs and speaking opportunities people should keep an eye on?

Stacey (30:19)
Yeah, so Open Source Summit North America is a pretty large event. I think it’s taking place in Minneapolis in May this year. There’s multiple tracks and there’s lots of opportunities for different types of talks. The CFP is currently open right now, but it does close February 9th. So go and check out the Linux Foundation Open Source Summit North America for that one.

We also have OpenSSF Community Days, which are co-located events at Open Source Summit North America, typically. And these are our events that we hold kind of around the world, but honestly, they’re perfect for first-time speakers as well. They’re smaller, they’re more intimate, and the community is super supportive. Our CFP for Community Day North America is February 15th. So go ahead and…search for that online. You can find them, and we’ll put the links in the description of this podcast so you can find that.

And then be on the lookout for key conferences later on in the year as well. KubeCon North America will be coming up later. Open Source Summit Europe is coming up later in the year. So be on the lookout for those. There’s also within the security space, I know there’s a lot of B-sides conferences and KCDs, which are Kubernetes community days and DevOps days.

If you’re in our OpenSSF Slack, we have a #cfp-nnounce channel that we try and promote and try and put out as many CFPs as we can to let people know that if you’re in our community and you want to submit talks regarding some of our projects or working groups or just OpenSSF in general, that CFP Announce channel is really a great place to keep checking.

Sally Cooper (32:13)
Amazing. Thank you both so much, not just for the insights, but for really making the CFP process feel more approachable and human. If you’re listening to this and you’ve been on the fence about submitting a CFP, let this be your sign. We really need your voice and thank you both so much.

Stacey (33:32)
Thank you.

Adolfo GarcĂ­a Veytia (@puerco) (33:33)
Thank you.

Jan 27
Love0

What’s in the SOSS? Podcast #49 – S3E1 Why Marketing Matters in Open Source: Introducing Co-Host Sally Cooper

By Podcast

Summary

In this special episode, the What’s in the SOSS podcast welcomes Sally Cooper as an official co-host. Sally, who leads OpenSSF’s marketing efforts, shares her journey from hands-on technical roles in training and documentation to becoming a bridge between complex technology and everyday understanding. The conversation explores why marketing matters in open source, how personal branding connects to community building, and the importance of personas in serving diverse stakeholders. Sally also reveals OpenSSF’s 2026 marketing themes and explains how newcomers can get involved in the community, whether through Slack, working groups, or contributing content

Conversation Highlights

00:09 – Welcoming Sally Cooper as Co-Host
01:28 – From Technical Training to Marketing Leadership
03:54 – Bridging Technology and Understanding
06:19 – Why Marketing Makes Open Source Uncomfortable
08:11 – Personal Branding and Career Growth
10:42 – Understanding Community Personas
12:33 – Getting Started with OpenSSF
14:44 – OpenSSF’s 2026 Marketing Themes
16:18 – Rapid Fire Round
17:09 – How to Get Involved

Transcript

CRob (00:09.502)
Welcome, welcome, welcome to What’s in the SOSS, the OpenSSF podcast where we talk to people, projects, and we talk about the ideas that are shaping our upstream open source ecosystem. And today we have a real treat. It’s a very special episode where we’re welcoming a new friend. And this is somebody that you probably know if you’ve been involved in our community for any period of time.

This young lady gets to help us with our messaging and how we present ourselves to the outside world, how we get our messaging out to all those interested OpenSoft community contributors around the globe. And today she’s officially joining Yesenia and I as a co-host of What’s in the SOSS. So I am proud and pleased to welcome Sally Cooper.

Yesenia (01:02.916)
Woo!

CRob (01:07.488)
Sally has been helping lead our marketing wing of efforts for the last several years. So before we jump into kind of what you do within that marketing function, Sally, we would like to hear a little bit about your open source origin story and how you got into technology.

Sally Cooper (01:28.549)
wow. Well, thank you so much, Yesenia and CRob. I’m super excited to be here. And yeah, I started my career a very long time ago. I actually started in tech with hands-on technical roles, working in training, documentation and support, and really helping people understand systems and tools and workflows.

Yesenia (01:52.21)
Yeah, I want to welcome Sally. great to have just another voice on this podcast, putting the hard work that our open source ecosystem is out there and getting more of these other voices. But you were talking about that you started in tech early and for me, that’s new for me. I would love for you to dive into these like technical roles. I think understanding your background in the technical and how you’ve gotten into marketing and working with open-assess that’s just going to relate to folks and understand that.

You don’t always have to be technical or work in a technical field to support your security. So I’d love to understand your background and how you’ve connected your technical background into the transitions you’ve had in your career.

Sally Cooper (02:35.611)
that’s such a good question. Yeah. I think you really nailed it there because you don’t need to always be technical and sometimes you don’t even, you can be technical and you end up in something like marketing for me. So, when I say started in tech, mean, this was like really entry level, hands on, learn it from the ground up. I worked in finance in my first job out of college. I was working at a data processing center and it was really operational.

accuracy, lots of responsibility, really not a lot of glamour. So the thing that kind of was a turning point was that we went through a major systems upgrade and we moved from a legacy system to entirely new software. So suddenly people who had been doing their jobs a certain way for years really were expected to work differently and often overnight. And I became one of the people who could help bridge the gap.

because I understood the technology and how to explain complex systems in an easy to understand manner. And I ended up being in training. So I became a software trainer and trained the whole organization on how to use the software to do their jobs.

Yesenia (03:52.776)
That’s very useful.

Sally Cooper (03:54.649)
Yeah, thanks. It’s funny because we all have to get started somewhere, right? And that’s how it worked out for me. After that, I worked at a startup in B2B e-commerce and continued on with educational software training, writing technical guides, books, some of the first e-learning programs. So I’m definitely dating myself here. But looking back, yeah, looking back, the title marketer wasn’t something that I thought of.

CRob (04:17.772)
Yeah

Sally Cooper (04:24.131)
But I was doing a lot of work in marketing without knowing it, just helping people understand concept topics. So yeah, that’s how I got here. Thanks for asking.

Yesenia (04:37.906)
Yeah, we all date ourself very easily. mean, we’re in tech. It already ages us the minute we walk in. But I think that’s a great understanding and background, right? I think that’s one of the most important skills when it comes to this technical is like, can you bring this high level technical aspect into something that everyday folks can understand and then drive them in? I’m curious from there, now you’re doing marketing. How did you get involved with that?

Sally Cooper (05:06.713)
Yeah, great question. So around the time when my career sort of took off with the technical education, there was something happening in the background. So early 2000s, this was the dawn of YouTube, smartphones were starting to emerge, companies were beginning to realize that technology wasn’t just about features, it was about an experience. And so I find this a very full circle moment because before smartphone, I had an iPod.

It was a pink metallic iPod and I got really obsessed with podcasts. So podcasts were new. It wasn’t just about the music for me. It was really listening to, you know, a conversation that was educational. And I could do that while raising a family, doing, like going for a walk, getting exercise, making dinner. You could have headphones on and just bring yourself into a whole other world.

So yeah, so that’s when I really started like it I also loved the campaign like looking at the billboards and seeing the silhouettes with I You know the iPod and the headphone all of that. So it’s kind of full circle

CRob (06:13.484)
Yeah.

Yesenia (06:19.934)
And it’s really lovely, especially when you see those nice like billboards and like, how much thought has someone taken into that? And like, when you think of like open source, like it’s people’s hobby projects, there’s just like no profit. And I feel like marketing in a sense, I’ve learned it from my own personal knowledge, professional growth, as you could say, there, I realized I was doing marketing without realizing I was doing marketing.

But marketing can just make some people uncomfortable, especially in the open source space. Like, what do you think about that?

Sally Cooper (06:53.463)
Yeah, that’s really valid. Open source is really personal. A lot of projects start off as a hobby, a passion, a side project built on nights and weekends. The word marketing can feel a little uncomfortable. It like, it doesn’t really belong there. I’ve definitely heard that feedback from developers. In open source, we’re not selling software. So it’s a completely new concept for me. I did have some marketing jobs after the educational jobs and

CRob (07:04.014)
Right.

Sally Cooper (07:23.479)
So I’m learning still, I’m learning from all of you and from our community that we’re sharing ideas, tools, practices, and that the currency is really people’s time, attention, and trust. So without marketing, great projects stay invisible, maintainers get burnt out, and users can struggle in silence, and the people who can contribute never even find the door.

CRob (07:50.142)
And this is extremely interesting to me because I observe Yesenia and kind of for the trajectory of her career and so much of your online persona is you do a lot of work of kind of branding yourself and providing advocacy and outlets to help empower other people.

Yesenia (07:58.589)
Yeah.

CRob (08:11.522)
It seems like a really big part of what you do outside of your day job and outside of your foundation work. So from your perspective, Yesi, how do you see these worlds connecting?

Yesenia (08:17.359)
Absolutely.

Yesenia (08:23.39)
I will recently I think it’s an interesting area. I heard this quote from a co worker. I would love to call her but I don’t have her. But it was like, your branding should be getting you the next job, right? Your next step your next opportunity. And as I started in my career, I was really thinking about like,

I kept getting seen and told like I wasn’t technical, but if you looked at my background, it’s in my education. It’s like, how am I not technical? Right. So I really started thinking about like where branding is like where people start meeting you. So your resume is a form of branding, your LinkedIn page is a form of branding. And I really saw it as like sharing a story about yourself, your impact, your value. I really letting them know what they’re getting into before they even reach out to you. So.

It just naturally happened as a way for me to like leave a toxic work environment and get into the next space. And as I realized I was doing it, like I said earlier, I didn’t realize I was doing marketing until somebody was like, you’re marketing. And I’m like, cool.

CRob (09:30.102)
I think what you do is very effective.

Yesenia (09:32.338)
Thank you.

Sally Cooper (09:33.345)
Yeah, I agree. Yesenia, you were an inspiration to me when I first started at OpenSSF because you were so good at branding. You had the cybersecurity big sister. I saw that somewhere. It’s like, yeah. And then you started tagging me on LinkedIn and you just made me feel like I was welcome. And I know that you do that to the community. You make people feel like there’s someone who is technical, but also human who leads with authenticity. So I was super impressed and I always learn so much from you.

Yesenia (09:37.448)
No.

Yesenia (09:45.371)
and

Yesenia (10:02.462)
What you guys gonna make me cry? No emotion. No, there’s no crying about the bars. No need baseball. I just aged myself there. But yeah, I think it’s really about creating those personas. And this is just something that you can do for yourself, that you do for your community, that you do for your projects. It was just something that I realized we just needed to connect people and get them moving. And personas has been talked a lot today.

CRob (10:05.006)
There’s no crying in open source.

Yesenia (10:31.39)
in this conversation. Sally, I love your expert opinion on this. Why do you think they’re so important when it comes to open source marketing?

Sally Cooper (10:42.189)
Yeah, well, CRob and I ran a project along with the OpenSSF staff where about a year ago we polled our community and we asked them a few questions to try to identify who they were, what their job titles were, what was important to them, how they learned about OpenSSF and how we could serve them better. And we came up with a list of personas.

I will link the personas in this transcript, hopefully I can figure that out. But we have software developer maintainers, open source professionals, the OSPOs, security engineers, executives and C-suite. And there’s a whole bunch of titles there. And then we came up with a new one that we hadn’t thought about before, which is funny because now that we’re talking a lot about marketing, there’s a product marketer.

CRob (11:11.662)
you

Yesenia (11:13.146)
Ooh.

CRob (11:36.91)
Mm-hmm.

Sally Cooper (11:36.985)
who is very much someone who is interested in open source software and open source security software. They’re typically a member or looking to become a member of the OpenSSF and they wanna help elevate the people that they work with, the projects that they’re working on, all the great work that their companies are doing in open source. really, Personas help us move from here’s a project to here’s how you ship secure code or

Here’s how we can help you manage risk or here’s how we can help you meet policy requirements. Marketing has really become a service and that’s where personas fit into the mix.

CRob (12:17.794)
Very nice and thinking about this from like, you know, we’re three kind of insiders for the foundation. If someone’s brand new to the OpenSSF and kind of wants to learn more, what does that journey look like for them, Sally?

Sally Cooper (12:33.429)
Yeah, that’s such a good question. So first of all, we’re all really nice and welcoming and you’re all welcome here. So if you have an idea, marketing can help bring that to light. If you are just new to OpenSSF, you can join many of our, actually all of our working groups. We have an open source community. One that would be really beneficial is the bare working group, belonging, empowerment, allyship, and representation and they meet frequently and they record their meetings on YouTube. So if you’re unsure, you can watch a few and learn a little bit more what it would be like to be in a working group at OpenSSF. Strongly encourage you also to join our Slack channel. We will link that and to follow us on social media. You can sign up for our newsletter. We try to meet people where they’re at.

So when we were talking about the personas, we learned that people are on different platforms. Some people would prefer to watch a video or read a blog. And so we try to cater to that, but we’re also always looking for feedback. So join the Slack, make yourself known. Again, if you have an idea, we can help you bring that to light. So we’d love to hear from you.

Yesenia (13:53.181)
And, know, no personal bias, but the bear group does do some awesome work. You know, there’s also, says the co-lead. We’ve also have a few blog posts that was released last year that Sally and her team has helped kind of release that go into how to get started into open source that I know the community as a whole has been sharing with new members as they come into a Slack channel. They’re like, I’m new, how do I get started? So it’s great resources there.

So we’re kicking into 2026, even though my mind keeps thinking it’s 2016. I had to figure out what’s going on there, but you know, one day we’ll go back there. Sally, as an insider, I’d to know what is marketing working on this year for openness, the staff’s mission and the growth of the communities?

CRob (14:30.101)
You

Sally Cooper (14:44.078)
Thank

Yeah, yeah, great question. So OpenSSF exists to make it easier to sustainably secure the development, maintenance, release, and consumption of the world’s open source software. We do that through collaboration, best practices that are shared, and solutions. And so our themes are showing up in 2026 quarterly to help people in our community meet these needs. For Q1, which we’re in now,

We’re focused on AI ML security. Q2, we’re going to talk about CVE, vulnerability transparency.

CRob (15:25.432)
heard of that.

Sally Cooper (15:27.289)
Q3, policy and CRA alignment. Q4 is going to be all about that base. So Baseline and security best practices.

Yesenia (15:41.01)
Very big fancy buzzwords there. So if anyone’s playing bingo as they listen, you got a few.

CRob (15:48.014)
Well, that has been an interesting kind of overview of what’s been going on. But more importantly, let’s move on to the rapid fire part of the show. have a series of short questions. So just kind of give us the first thing that comes off the top of your head. And I want that visceral reaction. Slack or async docs?

Yesenia (15:58.879)
Thank you for watching.

Sally Cooper (16:18.092)
Async docs.

Yesenia (16:21.15)
Favorite open source mascot.

Sally Cooper (16:24.947)
The Base. Honk as The Base.

CRob (16:27.79)
Nice. Love that one. What do you prefer? Podcasts or audiobooks?

Yesenia (16:27.934)
Go, baby.

Sally Cooper (16:33.273)
podcast.

CRob (16:35.662)
Star Trek or Star Wars?

Sally Cooper (16:38.489)
Star Wars.

CRob (16:40.43)
And finally, what’s your food preference? you like it mild or do you like it hot?

Sally Cooper (16:48.939)
medium.

CRob (16:50.188)
Medium? Well, thanks for playing along. So, Sally, if somebody’s interested in getting involved, whether it’s contributing to a project or potentially considering, you know, joining as a member on some level, how do they learn more and do that?

Yesenia (16:52.658)
That’s your question.

Sally Cooper (16:55.033)
Great question.

Sally Cooper (17:09.995)
Amazing. So go to openssf.org. From there, you can find everything you need. We referenced a blog. You can go check out our blog, find out how to contribute a blog. Everyone can join our Slack, join a working group, follow us on social media, subscribe to our newsletter. And we would love to see you at our events. Those are open to all. And if you are a member, please get involved, submit a blog.

Join us on the podcast. We would love to have you. We have a key study program. We also do quarterly tech talks. If you can dream it, we can build it. And the best place to plug in is our marketing advisory council. It meets the third Thursday of every month at 12 p.m. Eastern time. You can also reach out to us at marketing at openssf.org.

CRob (18:02.392)
Fantastic. And I may state how thrilled I am to be adding you as kind of a voice of our community and kind of joining us as a co-host, Sally.

Sally Cooper (18:13.133)
Woohoo!

Yesenia (18:13.374)
Yeah, I’m very excited for a new voice, help offload some of this work and the stories that you’re going to bring the guests we’re going to have on and as you had shared earlier, our marketing for 2026.

Sally Cooper (18:27.982)
Well, thank you so much both for having me. It’s been a pleasure.

CRob (18:31.662)
Excellent. With that, we’ll call it a wrap. I want to wish everybody a great day and happy open sourcing.

Yesenia (18:35.718)
You’re welcome.

Dec 30
Love0

What’s in the SOSS? Podcast #48 – S2E25 2025 Year End Wrap Up: Celebrating 5 Years of Open Source Security Impact!

By Podcast

Summary

Join co-hosts CRob and Yesenia for a special season finale celebrating OpenSSF’s fifth anniversary and recapping an incredible year of innovation in open source security! From launching three free educational courses on the EU Cyber Resilience Act, AI/ML security, and security for software development managers, to the groundbreaking DARPA AI Cyber Challenge where competitors achieved over 90% accuracy in autonomous vulnerability discovery, 2025 has been transformative. We reflect on standout interviews with new OpenSSF leaders Steve Fernandez and Stacey, deep dives into game-changing projects like the Open Source Project Security Baseline and AI model signing, and the vibrant community conversations around SBOM, supply chain security, and developer education. With nearly 12,000 total podcast downloads and exciting Season 3 plans including AI Cyber Challenge competitor interviews, CFP writing workshops, and expanded global community initiatives in Africa, we’re just getting started. Tune in for behind-the-scenes insights, friendly competition stats on our most popular episodes, and a sneak peek at what’s coming in 2026!

Conversation Highlights

00:00 – Celebrating OpenSSF’s Fifth Anniversary
02:52 – Educational Growth and New Initiatives
05:51 – Community Voices and Leadership Changes
08:45 – The Role of Community Manager
11:44 – Open Source Project Security Baseline
14:47 – AI and Machine Learning in Open Source
17:47 – Software Bill of Materials (SBOM) Discussions
20:34 – Podcast Highlights and Listener Engagement
22:26 – Looking Ahead to Season Three

Episode Links

Transcript

CRob (00:05.428)
Welcome, welcome, welcome to What’s in the SOSS. Today I’m joined with my co-host, Yesi, and we got a really great recap for everybody. We’re gonna be talking about the whole last year’s season of What’s in the SOSS, some of the amazing people that she and I got to interview. Yesi, I’m excited to actually get to talk with you today.

Yesenia (00:13.58)
Hello.

Yesenia (00:30.318)
I know, I got co-host and I never got to co-host with you and here we go. But today’s exciting because it’s not just celebrating everyone’s impact and everything awesome that’s been done in the open source community, but this year’s actually OpenSSF’s fifth year anniversary. That was amazing. I just found out. I was like, whoa, good episode.

CRob (00:47.44)
Wait!

CRob (00:53.646)
Yeah, some of us have been around a whole five years, so it’s not quite a surprise, but hey. That’s right. So I mean, kind of looking back over the last year, we had so many amazing things that both our community did and then like we’ve highlighted through the podcast. You know, let’s you know, we had a whole section where we worked with our.

Yesenia (00:58.798)
But at least we’ve made it longer than COVID. That’s fine.

CRob (01:18.704)
Linux Foundation education team on a whole cybersecurity skills framework to try to help coach new people into the profession and try to help identify skills that employers would want to hire. And I know this has been talked about a little bit in the bear working group, right?

Yesenia (01:36.598)
Yes, it’s something that we’re also using to consider as we bring in more contributors that are newer to this space. This is like a really good framework and a functional structure of how we can bring in these folks and help them scale up as well as helping these open source contributors.

CRob (01:53.368)
Right, and as we’re upskilling, you know, the crew and the back was really busy. We issued three whole new courses this year. All three exactly.

Yesenia (02:02.318)
Free courses and across the different very important spaces because who isn’t talking about CRA and AI? right there. Like it’s right there for you. got an hour long video on each. You got a nice little badge at the end. And for our software development managers, we can also talk about security. So those are, you know, three new courses if you’re checking out on how to expand your education.

You have the LFD 125, which is your security for software development managers. Two on my bucket list because they impact my work, which is understanding the EU Cyber Resilience Act. That’s LFEL 1001. wonder, my binary math is a little rustic, but curious what that converts to. And then our secure AI ML driven development. This one, I know a few people in the…

The BEAR working group that I’ve taken it and good feedback and BEARRRR!. But not even these new courses, but just the group in general. We have new LFS, new OpenSSF members joining us.

CRob (03:14.96)
That was pretty cool. And I think you actually got the opportunity to interview Stacey when she started, right? She’s our new Community Manager.

Yesenia (03:23.456)
Yeah, if you haven’t worked with the open-ended stuff, you haven’t met Stacey’s great community manager, really there. I wanted to say a word, but we’re on live, so I can’t. But she’s really driving. It’s good episode too. She got on our podcast, shared a little bit of her background. And I know she works closely with the Bear community, helping drive a lot of the operations. But we also had a new general manager. You got to interview.

CRob (03:51.384)
Right, yeah. Yeah, my new boss Steve, Steve Fernandez joined us around the first quarter and he brings with us a real kind of business and corporation focused background. So he’s really helped kind of mature a lot of the stuff we do around here and enhanced the scope of the services that we offer the community.

Yesenia (04:13.006)
And there was one more. I don’t know, I can’t put my finger on it. There was one more new member. Hmm.

CRob (04:16.75)
Hmm

Well, we did have a new co-host this year. Hello?

Yesenia (04:22.306)
That’s right, it’s me! Yes! Sound more now!

CRob (04:28.747)
Very exciting. Yeah. And overall, the podcast kind of focused on current topics, new and interesting projects like our security baseline. We had a couple talks around CRA and I know that we are, we’ll kind of save this a little bit as a teaser for next year, but we did several talks talking about AI.

Yesenia (04:49.902)
And there we did also talk on the AIxCC, which, you know, they’re going ahead and pushing security into the future with their autonomous vulnerability discovery. I know working in my past that that autonomous vulnerability discovery is such a complex, huge issue that I’m excited somebody’s driving deeper into that and working with OpenSSF.

CRob (05:12.752)
And I think I mentioned to you in some of the podcasts, I came into the whole AIxCC competition incredibly skeptical that I was unsure of the value that AI tools would bring into this space. But after we got the results, I was just floored. The fact that like the top team had over a 90% accuracy rate in finding and writing a fix for vulnerabilities.

Yesenia (05:39.078)
Wow.

CRob (05:41.836)
The second place team was only in the high 80% success ratio…only. Yeah, like there’s some amazing stuff and that really kind of convinced me that this is there’s some value in this space. And I think there’s, I’m really looking forward to some of the collaboration with around the cyber reasoning systems and a lot of the new things we’re doing in the AI space right now.

Yesenia (05:59.663)
Do know if they’re continuing it for next year?

CRob (06:06.116)
The competition isn’t continuing, but we will be continuing to work with DARPA and ARPA-H and the different competitors. We’ve already lined up. You’ll see some podcasts coming out in the early next year where we’re talking to the different competition teams. And several of those groups already are working to donate their software to the OpenSSF to help continue to grow a community and continue the development and refinement of these systems. There’s going to be some amazing stuff out of the AIML Working Group next year.

Yesenia (06:24.087)
Nice.

Yesenia (06:34.68)
Yeah, because I can just imagine with the percentage of the intrigue, just the research and the technical architecture of how they designed this to be able to produce such results. I know it’s going to be a huge impact into our open source and the security overall. But it’s for one year, you know, we had educational growth, governance, maturity, policy collaboration, our supply chain security. That’s one of my favorite words. I got earrings for it. It’s sharper.

CRob (06:51.631)
Yeah.

Yesenia (07:04.864)
And then you got AI, know, the preflow of that that’s come in. It’s really hit the open source in a real way. And I’m excited and I love that the podcast is capturing how we’re evolving in these spaces with the voices from our community.

CRob (07:19.172)
Mm-hmm. So let’s talk about those community voices a little bit. You mentioned that I had the opportunity to kind of talk with Steve, our new General Manager, and it was interesting that, know, Steve spent some time in his podcast, which was the title was Enterprise to Open Source, kind of talking about Steve’s journey. And he really kind of focused in on how his decades ofbeing a consumer of open source really is forming his current role as a steward of open source right now.

Yesenia (07:56.931)
Yeah, after listening to that, it was, it was, it’s understanding of why he got the position considering his background in the space, like, and just since he started the changes that’s happened in open source and the growth of what is, you know, Steve’s vision from where he bridges enterprises’ risks mindset with that of open source. Like that is something we definitely need to consider when it comes to it, because one of their major consumers is our enterprises.

And I know he’s played a big role in the Baseline and maturing that foundation of it. From listening to the episode, I know he talks about like those decades of consuming and then stepping into this and really calling security a hidden greatness, which is the work that you only notice when it’s missing or you get impacted, right? And this is for even the everyday person is like, you won’t realize that you need that security privacy until youknow, credit cards are stolen, right? So, but for him really coming in and turning those enterprise pain points into what is OpenSSF roadmap this year, and the greater is really helping organizations ship safer software.

CRob (09:09.88)
I agree. Now let’s talk about showing up fully. This was the interview you did with Stacey, our new Community Manager. What highlights would you like to share out of that conversation?

Yesenia (09:19.874)
This one, I love this, this is one of my favorites. Stacey came in and she’s had this background of becoming a community manager for open source communities. And she really kicked the ground running and was pushing that train. Like she’s behind that train moving it. But her real focus was around belonging, that authenticity, the inclusion and connecting BEAR with DevRel. And even though they’re two different working groups under us.

We have a very similar mission, just a different scope. So being able to come in as a community member and really ground how much community work is underpinned and all the technical work. I’ve seen her show up fully to the calls to not just Bear, but the other working groups and just making sure that she drives that community first mindset. And she connects with the maintainers, the members, the newcomers, and just making sure everyone’s being heard and felt. So,absolutely love that and you know there’s so much more into that 

CRob (10:24.24)
She also does a pretty amazing keynote.

CRob (10:29.968)
You’ve got to watch the video. It’s amazing.

Yesenia (10:45.43)
and I didn’t get to see the keynote but I you gotta watch it I’ve heard so many with her and Puerco

CRob (10:53.368)
And that’s, I think another interesting thing kind of pivoting around the community manager role. We have so many things going on across all the technical initiatives and working groups. It’s hard to kind of keep track of all of it. And that’s why having this role of that community manager is so important to be that connective tissue between our folks in the community that are contributing with staff, with the Board and the TAC. So it’s really important to have that, that role to help keep us balanced and focused.

Yesenia (11:22.306)
Yes. And let’s not forget, the podcast wouldn’t be the podcast without Stacey sitting here, listening to us, editing, publishing it. Big kudos if you ever see Stacey and you do her podcast. Please let her know she’s working really hard behind the scenes. She’s listening to us right now. So tons of kudos to her.

CRob (11:39.352)
Absolutely. Well, thinking about, thinking about what came up next is, the Open Source Project Security Baseline was a big effort for us, both in our community and within the whole broader LF. We did a, yeah, yeah. And we did a great podcast with two of the maintainers, Eddie Knight and Ben Cotton. And the title was a Deep Dive into the Open Source Project Security Baseline. And, you know, I thought that was.

Yesenia (11:53.932)
You helped push a lot of that.

CRob (12:08.752)
Pretty amazing little chat because both Eddie and Ben approached this project from the perspective of an upstream maintainer. We want to do whatever we can to remove work and burden from upstream and allow them to focus on creating amazing software and not necessarily have them have to worry about a compliance checklist, so to speak.

Yesenia (12:33.998)
And what I know with the Baseline, it ties together several projects.

CRob (12:39.596)
Yeah, have the Baseline itself is the catalog, which is the brains of the whole operation. And that details a list of requirements that should be done in the course of software development, publication and consumption. And then we have the orbit working group, which actually is the kind of the home for the baseline. And the ORBIT working group has a series of software projects.

That help try to automate or enable a lot of these different techniques. So we have things like around managing, making policy-based decisions in your CI pipeline, like a minder or a Gemara. We have a security insight spec that’s all part of the Orbit Working Group. And that’s a way for people to express how they are achieving some of these requirements. So like, for example, if you’re a project and you make, you issue SBOMS.,

You can make a security insights file to tell people how to find your SBOM. So they don’t have to come continually emailing you asking you for more information.

Yesenia (13:47.119)
And I heard a very quotable famous quote come out of this podcast, which was, oh, we got to put this on a t-shirt. “Give maintainers a way to show their security work, not just promise it.” Because that’s a huge thing. You’re working on these projects day and night in the stereotypical basement. And no one really cares unless they’re impacted, not in that sense. But it’s nice that we could show.

Have a way for maintainers to show their security work, give themselves a kudos and acknowledgement for the hard work putting it together.

CRob (14:19.279)
Right.

CRob (14:23.21)
And that’s where I’m very excited. And this kind of ties in with Steve’s vision and strategy is that projects like Baseline or SLSA, these are things that help downstream, meet your boardroom expectations. But all of these things are created and curated by the community. So again, we try to wherever possible focus in on the maintainer experience and making things easier. And I just love thatkind of dual purpose that we’re trying to help both up and downstream at the same time.

Yesenia (14:56.824)
Yeah. And then this year we also going back into those educational pieces, like some other episodes we talked to it was, you know, David Wheeler’s new AI ML Development Software. We have the Cybersecurity Skills Framework that we talked about earlier. And from there, we had that conversation with Sarah. think you interviewed Sarah on the AI competition model signing. What was your takeaway from that?

CRob (15:25.168)
Yeah, that was really great. So that was right as, so we have an AI ML Working Group is one of our technical initiatives and they’ve been around for about three years. And it was a little bit of a slow start where they did a lot of talking and evaluating and kind of setting up liaison relationships with the, there’s a whole cast of characters that are involved in AI security in the upstream ecosystem. And when I talked with Sarah, it was right after they’d had two publications.

The first was an AI Model Signing Project where they were leveraging a Sigstore and In-toto to help consumers understand, here is a signed model or a signed artifact. On this day, theycreated this artifact and it’s been untampered with since. So again, it’s trying to help provide more information into the pipeline so people can make risk-based decisions.

Yesenia (16:02.837)
interesting.

CRob (16:21.774)
And then right after that, they also released a white paper and of talking about how to integrate DevSecOps practices into machine learning and LLM development. And that’s been a really important artifact where it’s helped us realize, recognize that there are a lot of people involved in creating air quotes here, AI stuff, whether it’s an application, you’re training a model, you’re trying to go to market with something. There’s a lot of personas that are involved and onmost of them aren’t classically trained software engineers or cybersecurity practitioners. So the white paper kind of highlights these other people that participate in this creation process and talks about some techniques that are both old, you know, from AppSec, what we’ve done for 25, 30 years that have worked well, that could be applicable in the AI space. But then they also talk about some new ideas because these technologies are a little different and it does requiresome new ways of thinking, of being able to interrogate the different gizmos, whether it’s GPUs or eGenTech. So each technique requires some a little bit different tools to help protect them.

Yesenia (17:33.487)
Yeah, I’m glad you brought up the white paper because I was about to be like, I read the white paper. It was actually a good piece of knowledgeable guidance and information on how to Model Sign that I’m bringing into my own industry. It’s a good read, you know, and then we have other reads like the CRA compliance that we had a conversation with Alpha-Omega and the Erlang group. Those are also two good episodes to watch or to listen to.

Yesenia (18:03.522)
When it comes to the CRA. But, you we’ve talked about Baseline, we talked about GUAC, we’ve talked about SLSA, but the other card on, you know, the other bingo card for 2025 is SBOM. What episodes do we have on that?

CRob (18:14.746)
That’s right.

What episodes did we have on software bill of materials? 

CRob (18:51.608)
Right, we did do several things around SBOM. We had the opportunity to talk with Kate Stewart, who’s been a leader within the software build material space almost since the beginning. She represents SPDX, which is one of the two tools that most people use to create software builds materials, with the other one being Cyclone TX that our friends over at OWASP care take. And that was really interesting kind of talking about Kate’s perspective of the evolution of these things.

And then more recently, I had the opportunity to talk with the chief security officer of Canonical, my former coworker, Stephanie Domas. And we talked about a bunch of different things. And SBOM was kind of wrapped up in that conversation and talking about just challenges within the current regulated space that both commercial entities like an.

Yesenia (19:26.445)
Ooh.

CRob (19:41.546)
Canonical will face, but also upstream open source maintainers as well. So really engaging conversations around supply chain and software bill materials. The GUAC conversation was also really good and kind of important. That’s a very useful tool to help you get wisdom out of your SBOMS. Wisdom.

Yesenia (20:00.601)
Wisdom. Word of the day. It’s awesome. Considering it’s OpenSSF’s fifth year, just this year’s reflection on podcasts, we’ve really covered on multiple areas of the community, has been working on. And just my favorite thing about this whole thing is the little competition that’s going on against these podcast episodes where our guests have come in and asked, what’s my number? What’s my view? So as of today’s recording, we have the Mike Lieberman’s talk on GUAC SLSA and securing open source at 611. GitHub’s Mike Hanley, transforming department of NO . at 406.

Yesenia (20:55.886)
Eric Brewer and the future of open source software at 370, Vincent Danen and the Art of Vulnerability Management at 328. I’m so glad my dislexia, is not switching these numbers. And lastly, we have Sonatype’s Brian Fox and the Perplexing Phenomenon of Downloading Known Vulnerabilities at 327. So if you want to help these folks out,

Yesenia (21:25.644)
Give it a listen and let’s see if we can change the top episodes by the end of the year.

CRob (21:31.024)
It’s kind of a curious peek behind the scenes where guests will come in and do their podcast. And they’re very interested. It’s not vanity, but people like to hear that their work is valued. And so there is very healthy competition and some little bragging rights that Mr. Lieberman will kind of say, well, I have the most downloaded open as a podcast. So it’s just kind of fun, like a friendly little healthy competition. And again, and focusing in on some of these key areas of supply chain security, application development, software build materials and such.

Yesenia (22:06.19)
Yeah, it’s crazy to see that we’ve, across all the episodes, been about 11,800 total downloads and just 6,000 in 2025. So big thank you to our listeners, our supporters for that. I think it’s the first year of this podcast or second.

CRob (22:24.526)
Second, the second. And that actually kind of gives us our segue towards the end here. We’re talking about a lot of things that happened during 2025. And we are about to publish our annual report where you can kind of dive in and double click on some of these details. We’ll provide a link as this podcast is published that you can look at the report that will link into things like our five-year anniversary or our work with DARPA on AICC or all these amazing things around the baseline. So that’s, I’m really excited to kind of share that annual report with everybody that touches on a lot of the topics that Yacenya and I have talked through and many others. And that kind of moves us on. We’re going on to bigger and better things. 2026 is going to be season three.

CRob (23:17.88)
And I think we’ve got some really interesting topics kind of queued up.

Yesenia (23:21.464)
Are we gonna share? Are we gonna share? Are we gonna be nice to our listeners?

CRob (23:24.944)
I think everyone’s on the nice list. We can share that with them. Yeah, you’re going to see us starting off the year with kind of a full court press around AIxCC and AI and ML security topics. We have a bunch of work queued up with some of the cyber reasoning system competitors. We’ll talk with some of the competition organizers, again, talking more with our community experts around these very important topics and maybe unveiling some new projects that our AI team has in the hopper. That’s going to be very exciting. We’re going to have some very special guests from around the world of open source and public policy and research. And we’re going to have some very recognizable names that may have been in the show or a part of our community’s orbit we would love to reengage with and talk more with.

CRob (24:21.358)
So thinking about, you’re going to see multiple series of episodes around the AIxCC competition in particular. We’re going to be focusing in on industry and research stars. So we’re going to try to find some well-known voices out in the research community, joining some of our maintainers and kind of talking about some big picture conversations in the ecosystem. And then you’ll see many more things around our education efforts.

Would you like to talk about some of the stuff I know that Bear’s preparing to do?

Yesenia (24:51.822)
For Bear, we have very exciting things for next year. Not associated with that sports team. We have the next mentorship for the summer that we’re going to be producing. We’re working towards those details. We’re working with a group out in Africa for having an open source, what is it called? Open Source Security and Software Africa Group for the primary focus on doing speaking engagements, holding meetups and conferences in Africa. Cause there’s a huge community group there that have nowhere really to go. with global restrictions and visas, they’re very limited. So helping them kind of grow that out, share out some tips and tricks that we’ll be sharing on social just to drive more awareness into these projects and to these teams. And of course our community office hours, which have also had a lovely set of community members that have come in and shared their journeys, education pieces and blogs that have been recently produced like, Sal and Ejiro have produced about newcomers into the open source. We’re working on getting part three released, but you can find part one and two out in OpenSSF’s blog main page.

CRob (26:39.576)
Excellent. And I’m also excited that we’re going to be doing some special education segments on the podcast around how to write a good call for papers abstract. And then how to build your first conference talk, which is something that again, a lot of these newcomers haven’t had experience with that. Some of us that have been around the block a little bit can help share some of the wisdom we’ve earned over the last couple of Right.

Yesenia (26:46.83)
Yes.

Yesenia (27:02.358)
My try on era.

CRob (27:07.512)
And with that, I want to thank you for coming on board and being our co-host. You’ve really brought in a nice set of energy and a fresh perspective when you’re talking with our community members. And I wanted to remind everybody, as we are preparing for season three, if you have ideas or suggestions for topics, please email marketing at openssf.org. We would love to hear your episode pitches, your CFP stories, if you want to do some demos or have case studies.

Yesenia (27:12.119)
Absolutely.

CRob (27:35.822)
Or you just have just general projects that help the broader OpenSSF mission of improving the security of open source software for everybody forward. So thank you again, Yesenia. It’s been a pleasure. I’m looking forward to another exciting year of talking with you again. All right. Happy open sourcing, everybody.

Yesenia (27:50.776)
Thanks, CRob. To the next episode.

Nov 19
Love0

What’s in the SOSS? Podcast #45 – S2E22 SBOM Chaos and Software Sovereignty: The Hidden Challenges Facing Open Source with Stephanie Domas (Canonical)

By Podcast

Summary

Stephanie Domas, Canonical’s Chief Security Officer, returns to What’s in the SOSS to discuss critical open source challenges. She addresses the issues of third-party security patch versioning, the rise of software sovereignty, and how custom patches break SBOMs. Domas also explains why geographic code restrictions contradict open source principles and what the EU’s Cyber Resilience Act (CRA) means for enterprises. She highlights Canonical’s work integrating memory-safe components like sudo-rs into the next Ubuntu LTS. This episode challenges assumptions about supply chain security, software trust, and the future of collaborative development in a regulated world.

Conversation Highlights

00:00 – Welcome
01:49 – Memory safety revolution
02:00 – Black Hat reflections
03:48 – The SBOM versioning crisis
06:23 – Semantic versioning falls apart
10:06 – Software sovereignty exposed
12:33 – Trust through transparency
14:02 – The insider threat parallel
17:04 – EU CRA impact
18:50 – The manufacturer gray area
21:08 – The one-maintainer problem
22:51 – Will regulations kill open source adoption?
24:43 – Call to action

Transcript

CRob (00:07.109)
Welcome welcome welcome to what’s in the sauce where we talked to the amazing people that make up the upstream open source ecosystem these are developers maintainers researchers all manner of contributors that help make open source great today I have an Amazing friend of the show and actually this is a repeat performance. She has been with us once before

We have Stephanie Domas. She’s the chief security officer for Canonical. It’s a little company you might have heard about. So welcome for joining and thank you for joining us today, Stephanie.

Stephanie Domas (00:45.223)
Thank you for having me again for a second time.

CRob (00:48.121)
I know. It’s been a while since we chatted. So how are things going in the amazing world of the Linux distros?

Stephanie Domas (00:58.341)
Yeah, so just for people who aren’t as familiar with canonical because we do have a bit of a brand thing, we are the makers of Ubuntu Linux. So connect the dots for everyone. So the world of the distros is always a fun place. There has been a lot of recent excitement around NPM hacks, supply chain hacks. And so the world of archives, running archives, running a distro, there’s never a dull moment.

CRob (01:08.422)
Yay.

Stephanie Domas (01:28.475)
So on our distro front, right, we’ve been excited to try and on the security front, focusing on, we’re taking a fresh eye to how to introduce things like memory, save sort of core components into our operating systems. So a sudo RS, so a Rust implementation of sudo is now a part.

CRob (01:40.231)
Ooh.

Stephanie Domas (01:49.085)
Ubuntu will be a part of our next LTS, which is something we’re really excited. We’re looking for more opportunities to replace some of these fundamental components with memory safe versions of them. So that’s also exciting.

CRob (01:52.604)
Nice.

CRob (02:00.167)
That’s amazing. My hat is off to all of you for taking the leadership role and doing that. That’s great. I had the great opportunity to participate in a panel with you recently at Hacker Summer Camp. So kind of, I know it’s been a little bit of time, reflecting back, what was your favorite experience from the Black Hat Def Con, B-Sides, Diana week?

Stephanie Domas (02:24.609)
Yeah, and so I’m always one of those people who one of my favorite things is just the ability to reconnect with so many people in the industry. That’s the one time of year where actually despite the fact that you and I physically live near each other, that actually tends to be the one time a year we see each other in person. so extending that to all of just the great people I’ve known in the industry and getting to see them. The panel you spoke about was another real highlight for me. So the panel for those who didn’t get the privilege of attending wasn’t asked me anything on open source. And it was great because there was such a variety of interests right there. People who are interested in what it’s like to be a maintainer. What does it mean to use open source in enterprise? What does it mean to try and think of monetization models around open source? And so the diversity of conversation, which is really exciting to find people who are in the space really unfamiliar with open source or trying to figure out how do I reason about it and to be able to have those conversations with them was really exciting for me.

CRob (03:24.515)
I agree, I thought that was a great experience and I would love to see us do that again sometime. Excellent. Well, let’s move on to business. Today you have some things you’re interested in talking about. So let’s talk about one of my favorite topics, a software bill of materials and also talking about sovereignty within software.

Stephanie Domas (03:29.725)
I’ll put in a good word.

CRob (03:48.867)
From your perspective, how are you seeing these ideas and the tooling around SBOM, how are you seeing that adopted or used within the ecosystem today?

Stephanie Domas (04:00.421)
Yeah, so I will say, I’ll preface it with SBOMs continue to show a lot of really great promise. I do think there is a lot of value to them. I think they’re really starting to deliver on some of those benefits from trying to implement them at scale, right across our entire distro, across the archives, right? There are still some implementation challenges. So one of the things that’s been on my mind a lot recently is around versioning.

CRob (04:28.273)
Ooh.

Stephanie Domas (04:29.863)
So I feel like every week I see some new vendor or I get approached by some vendor whose business model is to create patches for you on the open source that you use to help maintain your security.

It’s a really interesting business proposition. get why these companies are popping up. But when I start to think of S-bombs and some of the difficulty we’re having around implementing them, this idea of version control in open source, when we have companies coming out of the woodwork to create patches, just creates this swirl of chaos.

To add some more, to add a specific example around there, you think about semantic versioning, right? When we release the version of something, it’s 5.0.0. When we release a patch, when the original manufacturer, the upstream releases a patch, it’s 5.0.1, right? Then 5.0.2. And so S-bombs come into play because the S-bombs goal is to understand the version of the thing that is inside of your piece of software. And then it uses that to kind of look up and say, hey, what are any known security issues in this version. But as I continue to see and have even talked to some of these businesses whose whole value proposition is, we will actually maintain and develop patches for you for the things that you are using, you’re breaking this whole idea of semantic versioning that now you are potentially carrying a tremendous number of patches for security specifically that aren’t representing that versioning number. And so then the question becomes like, what do you do with that? If I’m using internally 5.0.0 and I’ve used one of these companies to develop a security patch for me, what does that mean to my version number? What do I put in my S-bomb when I have created my own security patches that I have chosen not to upstream? What does that mean? And so…

Stephanie Domas (06:23.931)
I see this as a really unique problem to open source, right? Where there’s, there’s a lot of upsides to open source, like the tremendous amount to be specific, but the code base being open and allowing people to develop their own patches now creates this slurry of confusion of, don’t think version numbers will be the thing we can rely on in S bonds. And I don’t know what the answer is, right? I don’t actually, I don’t have a good approach to this when we’re starting to see, you know, customers using these companies create their own patches, they use some internal vulnerability scanner, and then they come back to us and say, hey, there’s these problems, these reports I’m getting from my scanner that’s scanning my S-bombs, and we’re sitting there confused of like, but because you’re carrying these other patches, I mean, it’s causing so much confusion where inside of the company, they don’t even realize. And so they get these scanning reports, so they come to us. They’re like, well, I don’t know how to answer this question because you’ve done something else internally, which is great. You like that, but your risk position, you’ve hired one of these companies to give you patches. So it’s one of these interesting things that’s been on my mind recently of, don’t know what the answer is, but I think SBOM’s inversion control and open source is going to be a really big struggle in the coming years as we see more of these offshoots of, of patches and security maintenance coming from not upstream and with no intention to upstream them.

CRob (07:50.415)
And I think, you we talk, get to deal with a lot of global public policy concerns and thinking about like the CRA where manufacturers are, have a lot of pressure to produce these patches. They’re encouraged to share them back upstream, but they’re not required. So I imagine the situation that you’re describing where these vendors are kind of setting up this little cottage practice.

You’re only going to see more of that. again, it’s going to cause a lot of confusion for the end users and a lot of, I’m sure, frustrated calls to your support line saying, I’m using package one, two, three, well, the actual, canonical version, pun intended. Ours is XYZ. Again, I bet that would cause a lot of customer downstream confusion.

Stephanie Domas (08:39.579)
Yeah, the EU CRA is one of the forcing functions that I think will bring a lot of this to light is where you start to have these obligations around patching and add a testing to this to your customers. But then again, you start to get this.

untenable spaghetti muster. How is that virgining happening when that manufacturer has potentially again use one of these businesses to create patches internally that they did not upstream. And so what does that mean for the S-bomb of the users when you give that to the users? How are they supposed to make informed risk decisions about that when the version number in that S-bomb is either follows upstream, in which case it misrepresents the security posture or they’ve come up with their own versioning system that’s unique to the internal patches that they’re carrying. And so again, the poor users are left without being able to make those informed decisions. So yes, the EU CRA, one of my favorite topics, is it’s gonna be a forcing function for that. And I think it’s gonna…

Again, there’s no answers. Like I think we’re going to be forced to try and make some of these decisions about like, what, what does that mean? Right? How do you, how do users reason about their SBOMs when the version numbers in them may not make sense anymore.

CRob (09:53.423)
It’s good to know that you’ve jumped into one of the hardest problems in software engineering, versioning. Do you want to tackle the naming issue as well while we’re at it?

Stephanie Domas (10:03.205)
I’m here to solve all, actually I’m just here to call out the problems.

CRob (10:06.247)
So somewhat adjacent, you mentioned that you wanted to talk about kind of this trend in the ecosystem around sovereignty. The EU thinks about it as digital sovereignty, but we’ve seen this concept in a lot of different places. So what are some of your observations around this, air quotes, new movement?

Stephanie Domas (10:32.121)
Yeah, so sovereignty is a really interesting one. For those who haven’t been too exposed to it, we’re seeing a big push in things like data sovereignty, which is focused on where is my data, right? Is my data contained in some, name some geographical border?

The big one that has been coming up for me a lot recently is software sovereignty. So there are some countries and some regulations that are taking the approach that the way to drive trust in a piece of software is by where it was developed, where the developers that wrote that piece of code come from. And on a surface, I…

I can see why they’re thinking that, right? I don’t necessarily agree with that. I can see why they may be thinking, hey, if this software was developed by people with the same ideals as me, maybe I can assume that it is for the same purpose that I agree with, right? I can see that argument, but I don’t love that argument. So the thought process to me is, is this actually the most effective path to true security, right? What they’re trying to achieve through this focus on software sovereignty is in fact more trust in the software. And being economical, right? That our solutions are open source, right? When this has come up for me for ambitions to sell to different federal governments, right? And so I sit in these conversations where contractors and consultants are trying to explain to me like, well, here’s what you have to do to sell to this name federal government here.

And you sit there thinking like, none of this makes any sense for open source. It’s kind of the antithesis of it, right? Attestation that the code was only worked on by people in certain countries. And I keep sitting there thinking, this is a wild way to derive trust in software. And so I sit there asking myself, why is this happening, right? On one side of the fence, there’s the, yes, I derived trust in software by who developed it.

Stephanie Domas (12:33.157)
The other side of that and the side that I think open source solves is, it not a more robust defense, right? To not think about national borders, but instead globally accessible and audible code bases. And again, they’re in these meetings. So CMMC was a reaching one for those in the U S but they have them in all over the world sitting there thinking like there’s absolutely no way for me to achieve anything beyond level one in CMMC because I can’t and have no interest in attesting to build pipelines that are only from certain citizens. And again, it’s one of these where it’s it’s confusing to me. So it’s been on my mind sitting there in these meetings about this and support sovereignty, right? Support sovereignty has actually come up as well for us. We have experience not just governments, but customers now who are seeking support to only be performed by people in name geographic border. And so we’re having to try and develop strategies again around support sovereignty. And again, I go back to if the codebase is globally accessible and auditable, I just feel like that’s a better way to solve this problem of deriving trust in what is done in the software, even from a support perspective, if you want to derive trust in what changes they’re proposing, what code they’re proposing to you. The fact that you can see it all should be how you derive trust, not where that person is physically sitting in the world. So that’s me on my snowboard. Sovereignty for the moment.

CRob (14:02.611)
And I, I like in this movement to back when I was an in-house practitioner in a InfoSec team, that the business always was dismissive of the insider threat vector. The fact that they were so focused on external actors that these, that somebody’s going to come in and hack the organization. But you look at reports like the Verizon data breach report that comes out every year and consistently every year, the most damaging cybersecurity breaches are people outside of your organizational trust boundary. And that’s, know, they cause the most damage. They are the most frequent, whether it’s ignorance or malice, you know, the, the, is the root cause of the problem, but it’s still someone inside of your organization. And to extrapolate that from a nation, you’re going to have a much broader spectrum of people, even outside of the, your small enterprise. So I always thought I was kind of scratched my head why people were so dismissive of this because there’s evidence that shows that the insiders are actually the ones that can cause that have consistently caused the most damage and then when you blow that up.

Stephanie Domas (15:11.773)
And I can tell you, yeah, as someone who spent time as an adjunct professor at The Ohio State University teaching software development.

There’s a lot of not great coders out there, right? So even if their ideals align, even if like there is nothing associated with code coming from a domestic location, it ensures quality, right? Again, even if you’re not worried about malice, there’s a lot of bad developers out there. The ability to have a portable code base should be your number one focus. And so yeah.

CRob (15:46.119)
And it’s the whole, have the thousand eyes, transparency perspective. But what I’ve always loved about open source is the fact that it’s meritocracy based. That, you know, the best ideas get submitted. have, we are able to pull ideas from everywhere that, know, anyone’s idea is potentially equally as valid as everybody else’s. So I love that aspect and that makes the software stronger because it helps ideally.

Coach those developers like you alluded to that might not be as skilled or as aware of security techniques and ideally helps them improve themselves, but also helps ideally not let those bad commits get into the code.

Stephanie Domas (16:25.777)
Yeah, and I’ll be the first to admit that not all open source is quality, right? It’s a variety of things. But the point is that you have the ability to determine that yourself before you choose to use the code. So depending on your risk posture, you get to make that decision.

CRob (16:43.43)
Mm-hmm.

Absolutely. So again, kind of staying on this topic, from your perspective and like the developers and then the customers you get to work with, how has this new uptick of global cyber legislation impacted your mission and impacted your customers?

Stephanie Domas (17:04.698)
Yeah, so I’ll actually I’m going to circle back to our favorite legislation, which is the EU CRA. So this has been a really interesting one because it’s.

CRob (17:10.01)
Yay!

Stephanie Domas (17:15.549)
It’s forced a lot more light, I think from the customers we’re working into the open source they’re using, how they’re consuming it and putting a lot more intentionality into if I continue to consume it this way, what does that mean for my, my liability associated with this piece of code? So in a way it’s been really good that it’s forcing customers to think more about that. As you mentioned earlier about patching and thinking about how am I getting this patch? How am I incorporating this patch? Part of the EU CRA is requiring for security patching for the defined lifecycle of the device. And again, that’s actually driving some really interesting conversations about, okay, yeah, that makes sense for the piece of software I wrote, but I had these pieces that I was consuming in the open source space and what does that mean for these? And so I do think it’s driving a good conversation on what does that mean? I worry that there is still some confusion in the legislation and until we get the practice guidance, we won’t have the answers to around that enterprise consumption of open source. And what does that mean from my risk perspective, right? If you consume so, and I’ll also caveat. like there is a carve out in the EU CRA for open source stewards. Canonical is defied, we make open source, but we are not actually an open source too, right? Because we have monetization models around support and because we want to support our customers, we are actually considered a manufacturer. So we’re taking a manufacturer role. So a manufacturer role over open source is also kind of a big gray area in the practice guidance and the legislation of like, well, what does that mean? Because again, despite the fact that we’re taking manufacturer on Ubuntu, the amount of upstream dependencies of us is astronomical. And so what does that mean? Right? What does that mean for us taking liability on the Debian kernel? I don’t know.

It’s bit confusing at the moment of what that means because we are in fact in manufacturer consuming open source. then to our customers who we have business relationships with, right? We are signing CRA support clauses, right? We are giving them, we are taking manufactured. They don’t have to take manufacturer on the open source they consume from us. But again, it’s a bit unclear of what entirely that means. We are actively involved in things like the Etsy working group that is defined some of these practice standards, particular and operating systems and Ubuntu is one of our flagship products. So we’re involved in that, that working group to try and define like, what exactly does this mean for operating systems? But despite being best known for Ubuntu, we actually have a very large software portfolio. And there’s a lot of software in our portfolio that doesn’t have a vertical practice guidance being made. And so this like general horizontal one that would define the requirements for those products not really see much activity on that yet. So there’s also big unknown around. don’t know what those expectations are for our products right now, which again, have tremendous upstream dependencies. And in some case, we are a fraction of that code base, right? What canonical produces in that code base is a fraction of the overall code base. But when people consume it through, what does that mean? Because we took the manufacturer role on it.

CRob (20:33.415)
Yeah. Well, and you touched on it a little bit earlier that all open source is amazing. Not all of it is great software. It’s a big spectrum. You look at a student project or someone running an experiment that you stumble across on GitHub is a very different experience than like operating with Kubernetes or the Linux kernel or anything from like the Apache foundation. like when you have these large mature teams, they have a lot of different capabilities as opposed to random project you find on the internet that somebody might not have intended for it to be commercialized.

Stephanie Domas (21:08.416)
Yeah, and I was reading an article recently on, I think it was an NPM archive that they had done their statistical analysis on, but they had done all this interesting analysis on the number of essentially the number of different handles who had committed to a piece of software and their overall conclusion right was that.

CRob (21:23.911)
Mm-hmm.

Stephanie Domas (21:26.301)
The majority of open source was one maintainer on this archive, right? That they showed of the, you know, N number of most popular downloads over 50 % of them. And I can’t remember the number. It might’ve been something closer to 80 % was one person. And so again, it’s not open source has a lot of amazing things. Not all of it is well written, not all of it is well maintained. so again, it’s sort of forcing regulations like the EU CRA are forcing people to take a much better look at their supply chain to understand, you know, where am I consuming that from? that something that is well maintained? Because I can’t just take the old version, say it’s stable and not worry about it from there. Can’t do that anymore under the CRA. I have to do security patching on this thing and now I’m potentially responsible for it and what does that mean? I do worry and I hope that we won’t see a recoil of enterprises using open source because of that fear, right? Because there are large libraries out there where there aren’t people willing to step up and take the maintainer role on them. And that makes enterprises afraid to consume those products under the EU CRA. And so that’s a fear that I do have. And again, it’s because it’s a bit of a gray area about how the liability, if that open source repo decides to take steward status, what does that mean for the enterprise consuming that? If it’s something small, it’s a tiny library, maybe the risk isn’t large, but it may be a really meaningful part of the application.

And if there’s no, if you’re not consuming it through somebody willing to take manufacturing role in your chain, will you still be willing to consume that piece of open source? So I do worry about that. I hope that as the practice guidance comes out, that is clarified so there’s better understanding of what that means and we won’t see that recoiling. But I have tried to talk to some of the legislators that are working on that and actually they haven’t been able to answer that question for me yet either. Is there going to be clarification because this is an area of concern. So hopefully between now and more practice guidance, we get more clarification so we don’t see that recoiling.

CRob (23:34.915)
And I know that at least from the horizontal and vertical standard standpoint, we are very close to starting to get public drafts being able to be delivered sometime fourth quarter in 2025. And ideally that’ll allow us to see what the legislators are thinking and where they are planning on landing on some of these issues. And hopefully we get some clarifications. And ideally we get the chance to provide feedback and say, hey, great work.

But have you considered X, Y, and Z to help make this more precise and more clear?

Stephanie Domas (24:11.387)
Absolutely, and my understanding that, hopefully I’m remembering the numbers correctly, I think there’s 15 vertical standards being written in Etsy right now. And so while that’s an astronomical amount of practice guidance, so like tremendous amount of work being done, that still won’t cover all the products. And so there’s still going to be a tremendous number of products that are sitting outside of those 15 vertical standards. And so again, the question will be like, but what does that mean for all the other products that are not in these 15?

CRob (24:43.431)
So as we wind down, what thoughts do you have? What do you want our audience to take away? What actions would you like them to take based off this conversation?

Stephanie Domas (24:55.825)
Yeah, depending on your role in the space. mean, I’m going to go ahead and plug open SSF, right? So the issues that we talked about, like what does SBOM mean as open source starts to become divergent, right? All of these things, I think will only be solved in organizations like the open SSF, right? Who has this bringing all the collective parties together to think about what it means. Same with some of these gray areas I mentioned about in the EU CRA and what does that mean? We need those types of organizations. if anyone listened to this and was like, well, maybe I have ideas on how to solve because all Stephanie did was complain about the problem, but I want to be a part of the solution. Look at joining the organizations like OpenSSF. That’s where the solutions will come from. Right. I can see my side of the problem, but I can’t, I can’t, even if I had ideas, I can’t independently solve it. Organizations like OpenSSF can do.

CRob (25:44.902)
Right.

It’s very much a community set of issues and I think collectively we’re stronger in having a better solution together. My friend, I need to arrange an ice cream social event in the near future before we’re covered in snow. But thank you for your time Stephanie and thank you to you and the team over at Canonical for all your hard work. with that, we’re going to call this a wrap. Thank you everybody.

Stephanie Domas (25:56.805)
Agreed.

CRob (26:17.019)
Happy open sourcing out there.

Nov 04
Love0

What’s in the SOSS? Podcast #44 – S2E21 A Deep Dive into the Open Source Project Security (OSPS) Baseline

By Podcast

Summary

In this episode of “What’s in the SOSS,” CRob, Ben Cotton, and Eddie Knight discuss the Open Source Project Security Baseline. This baseline provides a common language and control catalog for software security, enabling maintainers to demonstrate their project’s security posture and fostering confidence in open source projects. They explore its integration with other OpenSSF projects, real-world applications like the GUAC case study, and its value to maintainers and stakeholders. The role of documentation in security is emphasized, ensuring secure software deployment. The effectiveness of the baseline is validated through real-world applications and refined by community feedback, with future improvements focusing on better tooling and compliance mapping.

Conversation Highlights

00:00 Welcome & Introductions
02:40 Understanding the Open Source Project Security Baseline
05:54 The Importance of Defining a Security Baseline
08:49 Integrating Baseline with Other OpenSSF Projects
11:42 Real-World Applications: The Glock Case Study
14:21 Value for Maintainers and Other Stakeholders
17:29 The Role of Documentation in Security
20:37 Future Directions for the Baseline and Orbit
23:26 Community Engagement and Feedback

Transcript

CRob (00:11.23)
Welcome, welcome, welcome to What’s in the SOSS, where we talk to upstream maintainers, security experts, and just generally interesting luminaries throughout the upstream open source ecosystem. My name’s CRob. I’m the security architect for the OpenSSF, and I’m also your host for What’s in the SOSS. Today, we have two amazing friends of the show, amazing community members and developers and contributors across the open source ecosystem. So I want to welcome Eddie and Ben. Do you want to take a moment just to introduce yourselves and kind of explain what your deal is?

Eddie (01:02)
Yeah, my deal is I am in Amsterdam with you at 9 AM with a completely different energy level than you have right now. I am loving this. This is this is awesome. Eddie Knight from Sonatype. I do a lot of work across the Linux Foundation related to security compliance.

Ben (01:20)
I’m Ben Cotton. I’m the open source community lead at Kusari. I’m the leader of the OSPS Baseline SIG and a member of the Orbit Working Group.

CRob (01:29)
Awesome. Great talks today. We’re going to be diving into the OSPS Baseline, the catalog, and ORBIT, GUAC, and a whole bunch of different topics. So let’s set the stage here, gentlemen. The Baseline. Folks have been hearing about this off and on over the last few months, but maybe we could explain this in plain English, like what is the Open Source Project Security Baseline and talk about the catalog?

Eddie (01:57)
All right, I’ll let Ben give the official answer since he’s the project lead for it. Baseline’s a control catalog that helps maintainers and consumers of software have a clear definition of good for their projects and their project security. Ben, you want to give a more real answer?

Ben (02:16)
Yeah, I mean, it’s what it says on the tin, right? It’s a baseline for open source projects to follow in terms of their security hygiene. And it’s not really about the software they’re producing. It’s about the practices that are used to build that software. So the analogy that I recently came up with as we were going back and forth on this is it’s like health department regulations that say, “You have to wash your hands. You can’t pick up the food off the floor and then give it to the customer.” It doesn’t say that the quality of your recipe has to taste good. But you have to use secure practices. So we’ve developed a catalog of controls at three different tiers, the idea being that new projects, small projects, projects that are more trivial in nature just have like a sort of a bare minimum of like, yeah, everyone’s got to do this. Everyone needs to wash their hands before they start cooking food.

CRob (03:14)
I appreciate that.

Ben (03:15)
And that is important for SOSS, right?

CRob (03:18)
Right.

Ben (03:18)
As you go up the levels, know, go up to level three, like that’s really big projects that are, you know, lots of contributors, typically well resourced, at least relative to other open source projects and really important to the world of technology as we know it. And so those have to have the tightest security requirements because they’re rich targets for attackers.

With the baseline, the motivation is like, this is not a list of all the cool things you could do. It’s do this. One of the requirements we have is there is no should – there is only must. Because we don’t want to be having maintainers spending a lot of time chasing down all these things that they could do. We understand that open source maintainers, which we are, are often volunteers doing stuff in their spare time without necessarily any real security training. And so we need to give them straightforward things that are meaningful to enhancing their project security posture.

CRob (04:27)
This is, think, the first time ever on the planet anyone has ever referred to security as cool. Thank you, sir. I appreciate that as a longtime securityologist. So let me ask you, gentlemen both, why do you think it’s so important to define this baseline? And why is that important for open source projects?

Ben (04:47)
So I think the most important thing that’s been coming up in conversations I’ve had with people here in Amsterdam and other places is like, It gives us a common language. Go be more secure is not helpful. It doesn’t tell anyone anything.

And with baseline, especially with these different levels, you can say, our project is secure. We meet baseline level one. We meet baseline level two. Now there’s a common language. We all can know what that means because there is an explicitly defined catalog that says what these levels are. And then conversely, If I’m a vendor or manufacturer of some product and I use an open source project and I want them to be more secure because I have regulatory obligations, I can go to them and say, I really need you to be at baseline level two. We can help you with these specific tasks. And now we’re talking the same language. We have this common understanding of what this means as opposed to you’re not secure enough or you need to be more secure.

CRob (05:52)
Love that.

All right, so from your perspectives, and I think you might have touched on this a little bit, Ben, but how do we think that the baseline makes it easier for maintainers and developers who are already so busy with just their general feature requests and bug burn down?

Eddie (06:09)
So we started this journey a long time before we ever started saying baseline, right. My very first interaction with CNCF before I ever did any commits on OpenSSF, I was just kind of like, maybe attending a call here and there. We were doing this, it was like a security bug bash. And maybe we had called it the slam by this point. We wanted to solve for folks who were doing really cool stuff and everybody in the conversation knew that their stuff was being built well and properly and everybody’s washing their hands and stuff like that, right? But we didn’t have a way to demonstrate that outward and say like, hey, this project is running a clean kitchen. You should trust this more than just a random, you know street food vendor, whatever the open source equivalent of that is.

We want to boost confidence in the projects in our ecosystem. And, back then we had the CLO monitor because it was just for CNCF. And there was this set of rules of like, these are the things that we expect CNCF projects to do. And when we could go to a project and say, and I would pull out my phone on the KubeCon floor and be like, click through, type in your project name, pull it up, see like, this is where you’re scoring right now, right? And the scoring part brings all of its own baggage. But the point is like, there’s this list, right? And they’re like, that’s all you need? That’s it? That’s all you needed me to do, right? And so we had projects that were able to increase their own like personal maintainer confidence in their project. Like, oh man, I’m actually doing a really good job here.

All I needed to do was like shift this, rename this section header in a file so it could be detected. And now people see that I’m actually doing this stuff. And so you’re dramatically boosting our own like confidence in our work, but then you’re also boosting the public confidence in it. And this source is just having a list, right? Now that list for CNCF is not, it did not prove to be scalable and compatible with the broader ecosystem. It’s like, well, we don’t do this, we don’t do that.

So having baseline is a way of saying, let’s get that list, let’s get those wins that we experienced within the CNCF and make that possible for everybody by making it this like, not just CNCF, but agnostic baseline of controls that are good for projects.

CRob (08:53)
And those of us that have come from enterprise computing, the term baseline is very common practice as you’re deploying systems and applications. There’s a checklist of things you have to do before that system is able to go live. I love it. So thinking about the catalog, I realized that we have a lot of other assets within the OpenSSF, a lot of great projects and guidance. Could you maybe touch on some of the other projects within the OpenSSF that Baseline works with / integrates to?

Ben (09:21)
Yes. Yeah. So there is this whole working group called ORBIT in which Baseline sits. And it’s really about generating some of the tooling. So we use Gemara to sort of the scaffolding, I guess, for the control catalog. And it’s a software project that provides a framework to build control catalogs on. We do that. We’re working on tooling to automate some of the baseline evidence collection to make it easier for maintainers to you know, quickly evaluate where they are and what tasks they need to do to achieve their desired level. There’s a very smart man who has done a lot of mapping. This CRob guy has done a lot of work to map baseline to existing things like the best practices badge, as well as other external frameworks like the CRA.

CRob (10:29)
I’ve heard of that.

Ben (10:30)
Right?! Various NIST guidance, you know, really kind of make it so that, you know, baseline gives you not just, you know, confidence in your security posture, but then also gives you pointers to, you know, these more regulatory kind of control catalogs, where if you have a vendor coming to you and saying, hey, we need you to be secure, you can say, well, here, here’s what we meet. Here’s the list of things now you know. You know, so we really try, you know, we want to make sure that baseline is a part of an ecosystem and not just this really good idea that we have off in the corner that is sort of academic.

CRob (11:14)
That’s that’s excellent. That actually helps me pivot to my next set of questions. Let’s move out of the ethereal world of theory and talk about some real world applications of this. We just recently released a case study where we worked with another OpenSSF project named GUAC. And I just loved reading this case study. Could you maybe walk us through what the project was trying to prove and how baseline helped the GUAC project?

Eddie (11:44)
Yeah, that one was actually remarkably easy because all I had to do was yell at Ben and then it was suddenly done. [Crob laughing]

Ben (11:53)
So, you know, with that case study, we had the advantage of I’m a contributor to GUAC and then also as the baseline SIG Lead, like there’s some good overlap there. You know, so really what we were looking at is, you know, sort of a two pronged approach. One, you GUAC, the graph for understanding artifact composition, is a software supply chain security project. It would be really bad if it were, say, compromised in a supply chain incident, right? So, when you’re a security project, you have to have your own house in order. And so, you know, from the beginning, the project has really been done with that in mind. But we want to see, like, you know, validate our assumptions. Like, are we actually doing these things that, you know, are sort of the best practices

CRob (12:41)
Make sense.

Ben (12:43)
And then also, like, you know, from the baseline perspective, we want to get that real world, like here’s an actual project using it. What are the things that are unclear? What are the things that makes that don’t make sense? What are the things that are really easy? And so, you know, with that, we were, was able to use the Linux foundations, LFX insights, now has some automated

evidence collection. And so that, you know, was able to mark up a lot of boxes off right away. Some things are like, well, that’s just how GitHub works. So check, check, check. And so in the space of an hour or so, I was able to do…

CRob (13:35)
An hour?!

Ben (13:26)
An hour. I got level one and level two almost done. There were like four or five things where I was like, I’m not sure if we as a project would consider this sufficient or in a couple cases like we don’t actually document our dependency selection process. There is one, we don’t just add things William Nilliam, but you know we just need to write that down because as new contributors come on they need to know too and so like you know it was the amount of work that actually needed to be done to check the boxes off was really low. Which was very you know good news for me on both sides because I was gonna be the one doing the work.

And I’m the one trying to tell people like, you should use baseline to evaluate your project security. And so we really would love to have more projects do that sort of initial run and give us that feedback and help us. We spent a lot of time as a SIG with probably two dozen people at least have been involved over the last two years.

Coming up with these ideas, debating, you know, what needs to be included, what needs to be excluded. Eddie and I recently spent several hours coming up with definitions of words that were very precise so that we could be very clear and unambiguous. Like when we say project, this is what we mean in this context. and, we’ve tried very hard to keep this as a not a security wish list, but like a practical set of things for real world project maintainers. But, even with dozens of people involved, that’s only dozens of experiences. We want this to be something that’s useful to all eleventy billion open source projects out there.

So we need some more like real world do this, come back and tell us, hey, this doesn’t make sense. “This really is not a fit.” “My project can never do this because” – that kind of information.

CRob (15:36)
That’s awesome. From your perspective, as a maintainer, not that you’ve gone through this for GUAC, as a maintainer, how does that add value to you? What are you hoping to leverage out of that experience beyond the project itself, but as a GUAC maintainer, what are you hoping to gain from going through this baseline process?

Ben (15:58)
Well, I think the first thing is that it just gives confidence that like, yep, we’re doing the right things. We are doing what we can to reasonably protect ourselves from security incidents, particularly supply chain compromise, because GUAC isn’t running a service or anything.

And then, you know, being able to build on that. And then, you know, if, you know, we get emails like, Daniel Stenberg gets from, you know, car manufacturers and stuff like that, you know, we can, you know, just be like, yep, here’s our, our baseline level go have fun – (Daniel, if you’re listening, I would love for the cURL project to try out the baseline) and then you can just be like Yep, here’s my statement that we meet this baseline level as of this date. Have fun. If you want more, send me a check.

CRob (16:59)
So Eddie, we’ve talked about the maintainer a lot. But let’s talk about some other personas that are involved with the baseline and might care. From like an OSPO or an executive or security engineer perspective, what do you see the value of a project going through and having achieved some level of baseline.

Eddie (17:20)
Oh yeah. I mean, any metric badge check mark, right? It’s always helpful because going off of the number of GitHub stars only gets you so far.

CRob (17:35)
Right.

Eddie (17:36)
Especially now, we see that there’s actually an inverted relationship between stars and security for Hugging Face projects.

CRob (17:46)
Huh, really?

Eddie (17:47)
Yeah. Like there’s like somebody, well damn, now I’m gonna have to like find the research and actually show it to you to back my claim up. But yeah, was a little while ago somebody posted something where they found that it used to be more stars is more eyes. More eyes is more critiques. More critiques is more security, right? But for like ML projects, these kinds of things that you find on Hugging Face are the folks who are doing something fishy are pretty good at spoofing stars.

CRob (18:27)
Gaming the system, huh? I don’t like that. That makes me sad. And angry.

Eddie (18:33)
Yeah. And it’s like the more fishy that their thing is, the better their skill set is at spoofing stars. So it’s just kind of a weird thing. So when we have something like the best practices badge, Like, CNCF loves that, like the TOC loves that. Within TAG security and compliance, we obviously also love, it was not meaning to be a contrast statement. You like shook your head, you’re like, what, do you guys disagree? No, we don’t disagree. But there is also this desire to have something that is a little bit more fleshed out, right, which is why we were like, real big on CLO monitor and things like that. So the more fidelity that the badge has the more interesting it is. But I mean anything anything that can help accelerate that selection process is really helpful for the like The OSPO type of personality that you’re talking about.

CRob (19:37)
It’s been interesting kind of working with these projects and then being like a downstream consumer it there are many tasks within application development and app sec that are very difficult to measure. And some things are, I can verify what your settings are in your source forge. I can validate if you’re using multi-factor authentication or not. But there’s like just some tasks that are very difficult. And I’m excited that it’s not a solved problem yet, but the team has a lot of great ideas. And I think things like using security insights and other tools, to help provide that additional evidence showing that yes, here’s our policy. And a lot of the baseline encompasses some tasks that developers don’t always love, which is things like documentation.

Eddie (20:36)
Yeah, we have a lot of documentation checks. That is the number one question that we get, which is a fair question set. But one of the most common question sets is just like, what does documentation have to do with security?

CRob (20:49)
So Eddie, what does documentation have to do with security?

Eddie (20:53)
This is one of those situations where I actually struggle to answer at first. I have an answer. But the first 10 seconds is me going, why is this even? Isn’t it obvious? This is obvious, right? And then I look around the room and it’s like, it’s not obvious. OK. So there’s a couple different types of documentation that we need. So we need the things that you would put in a SECURITY.MD.

Just where do I go if I find a bug, if I find a security vulnerability? Who should I contact? Where should I post this information? What should I expect back from me? Those types of things. But then there’s also stuff if I’m using the project. If I need to run GUAC, Is GUAC secure by default? Is everything locked down when I turn it on? So it might be a little bit harder to turn on and deploy into my cloud infrastructure or whatever, but I don’t need to worry about it. Or is it the opposite? Is it insecure by default? Because almost all projects are insecure by default. The goal is to get more users. So you make it easy to deploy. And that means that when you turn this on, it’s going to have root access to something, it’s gonna have some network traffic that would not be recommended for prod, things like that. And so if we don’t have clear, easily accessible documentation with like a link that people know how to get to that kind of thing, like if this isn’t created and it’s not in a place that people know about it, then you’re actually deploying software that can be secure, but in practicality for users, there’s a high likeliness that they’re going to deploy it in securely. So you might have done your job, but people aren’t gonna be using it in the secure fashion because you haven’t documented it well enough or made it available or clear to them. And those are just like the two that come straight to mind. Like there’s a few different documentation needs that we have.

Ben (23:00)
And some of that, the documentation controls too are around like project policy in terms of, and I mentioned the dependency selection process. you can’t rely on, well, everyone knows this because one, people forget, two, if it’s not written down, everyone knows it, but everyone might know a slightly different thing. And then, you three, hopefully you’re bringing new contributors into your community. They need a place to learn about these things. And so, you know, having some of those things like, you know, we look for dependencies, you know, we prefer that they are actively maintained that they have, you know, maybe an OpenSSF Scorecard score above a certain threshold or like maybe there’s an advanced algorithm you use to mix a bunch of things together and then figure out, you know, maybe, you know, if it’s, you know, a project within an ecosystem, you don’t pull in just random things off of package repository, you have an internal repository that you mirror things into to protect from things like that right but if that’s not written down if that’s not you know clearly documented for the people who need it it’s not going to get followed.

CRob (24:15)
So let’s get out our crystal balls and look into the future. You know what do you guys see for orbit the catalog and just this general let’s work in general?

Eddie ()
What do we see for the future? So we’ve right now we’ve stabilized the control catalog, I would like to, I would like to make it a Gemara complete control catalog, right? So it lists out the capabilities, the threats and the controls, right? Because we’ve written a threat informed control catalog, but we haven’t captured and documented, what threats are we mitigating with this? So I think that’d be pretty cool. How close are we to doing that? I don’t know.

The other thing is just getting, more people to actually demonstrate compliance with the controls? think most projects, especially in the LF, are gonna be predominantly compliant already. Like you’ve already done all this stuff. We just want to be able to tell everybody that you’ve done it.

CRob (25:16)
Get credit for your homework.

Eddie (25:18)
Yeah, we wanna give you credit for this, right. And so that’s gonna be a big lift is going through and doing that hour of work with GUAC. Doing that hour of work with all of these different projects kind of adds up. So that’s gonna be something that I hope happens very soon. Within CNCF, we did it in the security slam fashion, right? So OpenTelemetry, Meshery, OSCAL-COMPASS, and Flux actually, were all part of that in the spring. And that went pretty well. Where the breakdown happened was on multi-repo projects like OpenTelemetry. I think it was 70 repos.

Yeah, like a lot of repos. think Kubernetes is double that, right? Yeah. So when you have so many different repos and we need to go in and say, here’s where the documentation is for this repo. Here’s where the policy is for this repo, right? It gets a little bit bumpy. And I think there’s still some room for improvement on how we’re capturing and doing those evaluations. say, I think I have a backlog. I know. There’s improvement on that.

But as more people are going about that and giving feedback, like Ben comes and says, this is where something took 20 minutes, but I expected it to take five. Then we can actually make those improvements and improve our backlog, refine our backlog a little bit.

Ben (26:51)
Yeah, and I would, know, to Eddie’s point and you mentioned earlier, CRob, but we do not have fully complete tooling to measure all the measurable things yet. And so that’s an area that the Orbit Working Group is working on as a group. We’ve also had some sort proto discussions about having a catalog of examples. What does a dependency selection policy look like? What does this documentation thing look like?

In baseline itself on my backlog includes like just going out real world example, you know, from Fedora, from curl, from Kubernetes, from wherever, like here are some things that look like what we would suggest you have. And then, you know, ideally, I think we’d also want to have a project that is just templates for each of these things that are templatizable. Like you don’t have, you know, so code of conduct licenses, those are pretty well established.

A lot of this other stuff like what what is sort of like the platonic ideal of a security MD file? What is you know the best dependency selection policy that people can just you know do a choose your own adventure? I want this this this put it together. This is what makes sense for my project. Here you go. It’s no it’s of no use to anyone to have everybody writing this from scratch over and over again, especially if they’ve not seen an example of it before.

CRob (28:21)
So as we wind down here. are the calls to action do you have for the community or whether it’s developers in the OpenSSF or just kind of unaffiliated maintainers? What would you like folks to take away from this?

Ben (28:37)
I would love them to look at the open source project security baseline, baseline.openssf.org and evaluate your project against it and give us feedback. What worked? What didn’t? What do you think? Why isn’t this there? We want this real world feedback on the control catalog so we can make sure it is actually fit for the purpose we’ve designed it for. So for me personally, that’s the biggest takeaway I want from people listening to this.

Eddie (29:09)
Complain loudly. That’s what I want. We are trying to create an accelerator. We’re trying to improve the ecosystem. We’re trying to improve the lives of maintainers. And any single place where this is slowing down a maintainer, that is outside of intent. That is a design flaw of some kind. If this is slowing you down, if this is confusing, if you’re getting pushback from some end user who now thinks that you’re doing worse than before you started, before baseline existed, right? We heard that feedback from somebody. It’s like, hey, LFX Insights turned on their scanner, and now I have a user who thinks that our project’s doing a bad job with security. And it’s like, oh, well, that didn’t meet expectations.

CRob (30:00)
That was an unintended consequence.

Eddie (30:02)
Yeah. And it was that perception was inaccurate. The tests were accurate but imprecise, right? They nailed exactly what the tests were trying to do. They were very, very, very much there, but not, they weren’t aiming in the right direction, right? And so we refined like, okay, let’s zone that in, move it closer to the bullseye on what we’re trying to achieve. And I think we’re getting a lot better at that. But that’s because somebody came and ruffled our feathers and was like, hey, you’re not doing what you said you’re trying to do. we thought we were.

CRob (30:43)
Right.

Ben (30:45)
Yeah. And I just want to point out that the baseline is itself an open source project with open public meetings, pull requests welcome. We truly do want feedback and contribution from people who have tried things out or don’t understand. I shared very early on on my social media accounts and a guy I know came back and was like, we could never meet this. And it turns out the wording was just awful. We did not make this clear at all. And yeah, we fixed that. it’s like, OK, we went back and forth a few times. All right, this is our intent. We have now captured it well. And I think the wording is a lot better on that because people were confused and asked questions.

CRob (31:30)
Well, and to your patches welcome comment, we’ve had decent engagement with open source maintainers. I would love to see us have more downstream GRC security people giving us feedback from your perspectives. What other compliance regimes or laws would you like to see? And did we get our compliance mapping right? Is it spot on? Does it speak to the needs you need to have to defend yourselves against auditors and regulators?

Well, Eddie and Ben, two amazing community members, friends of the show here. Thank you for your time. Thank you for all you do across your fleet of open source projects that you contribute to and maintain. And with that, we’re going to call it a day. I want to wish everybody a wonderful day here from sunny Amsterdam. And happy open sourcing. Bye, everybody.

Eddie (32:23)
Thanks CRob.

Ben (32:24)
Thanks, CRob.

Oct 16
Love0

What’s in the SOSS? Podcast #42 – S2E19 New Education Course: Secure AI/ML-Driven Software Development (LFEL1012) with David A. Wheeler

By Podcast

Summary

In this episode of “What’s In The SOSS,” Yesenia interviews David A. Wheeler, the Director of Open Source Supply Chain Security at the Linux Foundation. They discuss the importance of secure software development, particularly in the context of AI and machine learning. David shares insights from his extensive experience in the field, emphasizing the need for both education and tools to ensure security. The conversation also touches on common misconceptions about AI, the relevance of digital badges for developers, and the structure of a new course aimed at teaching secure AI practices. David highlights the evolving nature of software development and the necessity for continuous learning in this rapidly changing landscape.

Conversation Highlights

00:00 Introduction to Open Source and Security
02:31 The Journey to Secure AI and ML Development
08:28 Understanding AI’s Impact on Software Development
12:14 Myths and Misconceptions about AI in Security
18:24 Connecting AI Security to Open Source and Closed Source
20:29 The Importance of Digital Badges for Developers
24:31 Course Structure and Learning Outcomes
28:18 Final Thoughts on AI and Software Security

Transcript

Yesenia (00:01)
Hello and welcome to What’s in the SOSS, OpenSSF podcast where we talk to interesting people throughout the open source ecosystem. They share their journey, expertise and wisdom. So yes, I need said one of your hosts and today we have the extraordinary experience of having David Wheeler on a welcome David. For those that may not know you, can you share a little bit about your row at the Linux Foundation OpenSSF?

David A. Wheeler (00:39)
Sure, my official title is actually probably not very illuminating. It says it’s the direct, I’m the director of open source supply chain security. But what that really means is that my job is to try to help other folks improve the security of open source software at large, all the way from it’s in someone’s head, they’re thinking about how to do it, developing it, putting it in repos, getting it packaged up, getting it distributed, receiving it just all the way through. We want to make sure that people get secure software and the software they actually intended to get.

Yesenia (01:16)
It’s always important, right? You don’t want to open up a Hershey bar that has no peanuts in the peanuts, right? So that was my analogy for the supply chain security in MySpace. Because I’m a little sensitive to peanuts. I was like, you know, you don’t want that.

David A. Wheeler (01:22)
You

David A. Wheeler (01:31)
You don’t want that. And although the food analogy is often pulled up, I think it’s still a good analogy. If you’re allergic to peanuts, you don’t want the peanuts. And unfortunately, it’s not just, hey, whether or not it’s got peanuts or not, but there was a scare involving Tylenol a while back. And to be fair, the manufacturer didn’t do anything wrong, but the bottles were tampered with by a third party.

Yesenia (01:40)
Mm-mm.

David A. Wheeler (01:57)
And so we don’t want tampered products. We want to make sure that when you request an open source program, it’s actually the one that was intended and not something else.

Yesenia (02:07)
So you have a very important job. Don’t play yourself there. We want to make sure the product you get is the one you get, right? So if you don’t know David, go ahead and message him on Slack, connect with him. Great gentleman in the open source space. And you’ve had a long time advocating for secure software development in the open source space. How did your journey lead to creating a course specifically on secure AI and ML driven development?

David A. Wheeler (02:36)
As with many journeys, it’s a complicated journey with lots of whens and ways. As you know, I’ve been interested in how do you develop secure software for a long time, decades now, frankly. And I have been collecting up over the years what are the common kinds of mistakes and more importantly, what are the systemic simple solutions you can make that would prevent that problem and eliminating it entirely ideally.

Um, and over the years it’s turned out that in fact, for a vast number, for the vast majority of problems that people have, there are well-known solutions, but they’re not well known by the developers. So a lot of this is really an education story of trying to make it so that software developers know how to do things. Now it’s a fair, you know, some would say, some would say, well, what about tools? Tools are valuable. Absolutely.

If to the extent that we can, we want to make it so that tools automatically do the secure thing. And that’s the right thing to do, but that’ll never be perfect. And people can always override tools. And so it’s not a matter of education or tools. I think that’s a false dichotomy. It’s you need tools and you need education. You need education or you can’t use the tools well as much as we can. We want to automate things so that they will handle things automatically, but you need both. You need both.

Now, to answer your specific question, I’ve actually been involved in and out with AI to some extent for literally decades as well. People have been interested in AI for years, me too. I did a lot more with symbolic based AI back in the day, wrote a lot of Lisp code. But since that time, really machine learning, although it’s not new, has really come into its own.

And all of a sudden it became quite apparent to me, and it’s not just me, to many people that software development is changing. And this is not a matter of what will happen someday in the future. This is the current reality for software development. And I’m going to give a quick shout out to some researchers in Stanford. I’ll have to go find the link. So who basically did some, I think some important studies related to this.

David A. Wheeler (04:59)
When you’re developing software from scratch and trying to create a tiny program, the AI tools are hard to beat because basically they’re just creating, know, they’re just reusing a template, but that’s a misleading measure, okay? That doesn’t really tell you what normal software development is like. However, let’s assume that you’re taking existing programming and improving it, and you’re using a language for which there’s a lot of examples for training. Okay, we’re talking Python and Java and, you know, various widely used languages, okay?

David A. Wheeler (05:28)
If you do those, turns out the AI tools can generate a lot of code. Some of it’s right. So that means you have to do more rework, frankly, when you use these tools. Once you take the rework into account, they’re coming up with a 20 % improvement in productivity. That is astounding. And I will argue that is the end of the argument. Seriously, there are definitely, there are companies where they have a single customer and the customer pays them to write some software. If the customer says never use AI, fine, the customer’s willing to pay 20 % improvement, I will charge that extra to them. But out in most commercial open source settings, you can’t throw, you can’t ignore a 20 % improvement. And that’s current tech, that’s not future tech. mean, the reality is that we haven’t seen improvements like this since the switch from hut, from assembly to high level languages, the use of, you know, the use of structure programming, I think was another case we got that kind. And you can make a very good case that open source software was that was a third case where you got that digital, productivity. Now you could also argue that’s a little unfair because open source didn’t improve your ability to write software. makes you didn’t have to write the software.

David A. Wheeler (06:53)
But that’s okay. That’s still an improvement, right? So I think that counts. But for the most part, we’ve had a lot of technologies that claim to improve productivity. I’ve worked many over the years. I’ve been very interested in how do you improve productivity? Most of them turned out not to be true. I don’t think that’s true for AI. It’s quite clear for multiple studies. mean, not all studies agree with this, by the way, but I think there’s enough studies that there’s a productivity improvement.

David A. Wheeler (07:21)
It does depend on how you employ these, that, but you know, and they’ll get better. But the big problem now is everyone is on the list. This is a case where everyone, even if you’re a professional and you’ve been doing software development for years, everybody’s new at this part of the game. These tools are new. And the problem here is that the good news is that they can help you. The bad news is they can harm you. They can do.

David A. Wheeler (07:50)
They can produce terribly insecure software. They can also end up being the security vulnerability themselves. And so we’re trying to get ahead of the game and looking around what’s the latest information, what can we learn? And it turns out there’s a lot that we can learn that we actually think is gonna stay on the test of time. And so that’s what this course is, those basics they’re gonna apply no matter what tool you use.

David A. Wheeler (08:17)
How do you make it say you’re using these tools, but you’re not immediately becoming a security vulnerability? How is it so that you’re less likely to produce vulnerable code? And that turns out to be harder. We can talk about why that is, but that’s what this course is in a nutshell.

Yesenia (08:33)
Yeah, I know I had a sneak preview at the slide deck and I was just like, this is fantastic. Definitely needed it. And I wanted to take a moment and give a kudos to the researchers because the engine, the industry wouldn’t be what it is today without the researchers. Like they’re the ones that are firsthand, like try and failing and then somebody picks it up and builds it and it is open source or industry. then boom, it becomes like this whole new field. So I know AI has been around for a minute.

David A. Wheeler (09:01)
Yeah, let me add that. I agree with you. Let me actually separate different researchers because we’re building on the first of course, the researchers who created these original AI and ML systems more generally, obviously a lot of the LLM based research. You’ve got the research specifically in developing tools for developing, for improving software development. And then you’ve got the researchers who are trying to figure out the security impacts of this. And those folks,

Yesenia (09:30)
Those are my favorite. Those are my favorite.

David A. Wheeler (09:31)
I’m Well, we need all of these folks. But the thing is, what concerns me is that remarkably, even though we’ve got this is really new tech, we do have some really interesting research results and publications about their security impacts. The problem is, most of these researchers are really good about, you know, doing the analysis, creating controls, doing the research, publishing a paper, but for most people, publishing a paper has no impact. People are not going to go out and read every paper on a topic. That’s, know, they have work to do basically. So if you’re a researcher makes these are very valuable, but what we’ve tried to do is take the research and boil it down to as a practitioner, what do you need to know? And we do cite the research because

David A. Wheeler (10:29)
You know, if you’re, if you’re interested or you say, Hey, I’m not sure I believe that. Well, great. Curiosity is fantastic. Go read the studies. there’s always limitations on studies. We don’t have infinite time and infinite money. but I think the research is actually pretty consistent. at least with Ted Hayes technology, we, can’t guess what the great grand future holds.

David A. Wheeler (10:55)
But I’m going to guess that at least for the next couple of years, we’re going to see a lot of LLMs, LLM use, they’re going to build on other tools. And so there’s things that we know just based on that, that we can say, well, given the direction of current technology, what’s okay, what’s to be concerned about? And most importantly, what can we do practically to make this work in our favor? So we get that 20 % because we’re going to want it.

Yesenia (11:24)
Yeah, at this point in time, we’re seedling within the AI ML piece. What you said is really, really important. It’s just like, so much more to this. There’s so much more that’s growing. And I want to take it back to something you had mentioned. You’re talking about the good that is coming from the AI ML. And there is the bad, of course. And for the course that you’re coming out, what is one misconception about AI in the software development security that you hope that this course will shatter? What myth are you busting?

David A. Wheeler (11:53)
What myth am I busting? I guess I’m going to cheat because I’m going to respond with two. It’s by the fact that I actually can count. guess, okay, I’m going to turn it into one, which is, guess, basically either over or under indexing on the value of A. Basically expecting too much or expecting too little. Okay, basically trying to figure out what the real expectations you should have are and not go outside that. So there’s my one. So let me talk about over and under. We’ve got people.

Yesenia (12:30)
Well, I’m going to give you another one because in software everything starts with zero. So I’ll give you another one.

David A. Wheeler (13:47)
Okay, all right, so let me talk about the under. There are some who have excessive expectations. We’ve got the, you know, I think vibe coding in particular is a symptom of this, okay? Now, there are some people who use the word vibe coding as another word for using AI. I think that’s not what the original creator of the term meant. And I actually think that’s not helpful because it’s a whole lot like talking about automated carriages.

Um, very soon we’re only going to be talking about carriages. Okay. Everybody’s going to be using automation AI except the very few who don’t. Okay. So, so there’s no point in having a special term for the normal case. Um, so what, what I mean by vibe coding is what the original creator of the term meant, which is, Hey, AI system creates some code. I’m never going to review it. I’m never going to look at it. I’m not going to do anything. I will just blindly accept it. This is a terrible idea. If it matters what the quality of the code is. Now there are cases where frankly the quality of the code is irrelevant. I’ve seen some awesome examples where you’ve got little kids, know, eight, nine year olds running around and telling a computer what to do and they get a program that seems to kind of do that. And that is great. I mean, if you want to do vibe coding with that, that’s fantastic. But if the code actually does something that matters, with current tech, this is a terrible idea. They’re not.

They can sometimes get it correct, but even humans struggle making perfect code every time and they’re not that good. The other case though is, man, we can’t ever use this stuff. I mean, again, if you’ve got a customer who’s paying you extra to never do it, that’s wonderful. Do what the customer asks and is willing to pay for it. For most of us, that’s not a reasonable industry position. What we’re going to need to do instead is learn together as an industry how to use this well. The good news is that although we will all be learning together, there’s some things already known now. So let’s run to the researchers, find out what they’ve learned, go to the practitioners, basically find what has been learned so far, start with that. And then we can build on and improve and go and other things. You don’t expect too much, don’t expect too little.

Yesenia (15:28)
Yeah, the five coding is an interesting one, because sometimes it spews out like correct code. But as somebody who’s written code and reviewed code and like done all this with the supply chain, I’m like. It’s like that extra work you gotta kind of add to it to make sure that you’re validating your testing it and it hasn’t just accidentally thrown in some security vulnerability in in that work. And I think that was. Go ahead.

David A. Wheeler (15:51)
What I can interrupt you, one of the studies that we cited, they basically created a whole bunch of functions that could be written either insecurely or securely as a test. Did this whole bunch of times. And they found that 45 % of the time using today’s current tooling, they chose the insecure approach. And there’s reason for this. ML systems are finally based on their training sets.

They’ve been trained on lots of insecure programs. What did you expect to get? You know, so this is actually going to be a challenge because when you’re trying to fight what the ML systems are training on, that is harder than going with the flow. That doesn’t mean it can’t be done, but it does require extra effort.

Yesenia (16:41)
We’re going extra left at that point. All right, so you your one and I gave you, know, one more because we started at zero. Any other misconception that is being bumped at the course.

David A. Wheeler (16:57)
Um, I guess the, uh, I guess the misconception sort is nothing can be done. And, uh, of course the whole course is a, uh, a, a, a stated disagreement with examples, uh, because in fact, there are things we can do right now. Now I would actually concede if somebody said, Hey, we don’t know everything. Well, sure. Uh, you know, I think all of us are in a life journey and we’re all learning things as we go. Uh, but that doesn’t mean that we have to, um, you know, just accept that nothing can be done. That’s a fatalistic approach that I think serves no one. There are things we can do. There are things that are known, though maybe not by you, but that’s okay. That’s what a course is for. We’ve worked to boil down, try to identify what is known, and with relatively little time, you’ll be far more prepared than you would be otherwise.

Yesenia (17:49)
It is a good course and I the course is aimed for developers, software engineers, open source contributors. So how does it connect to real world open source work like those that are working on closed source versus that open source software?

David A. Wheeler (18:04)
Well, okay, I should first quickly note that I work for the Open Source Security Foundation, Open Source is in the name, so we’re very interested in improving the security of open source software. That is our fundamental focus. That said, sometimes the materials that we create are not really unique to open source software. Where it can be applied by closed source software, we try to make that clear. Sometimes we don’t make it clear as we should, but we’re working on that.

Um, and frankly, in many cases, I think there’s also worth noting that, um, if you’re developing closed source software, the vast majority of the components you’re using are open source software. I mean, the average is 70 to 90 % of the software in a closed source system software system is actually open source software components. Uh, simple because it doesn’t make sense to rebuild everything from scratch today. That’s not a, an M E economically viable option for most folks. So.

in this particular case for the AI, it is applicable equally to open source and closed source. It applies to everybody. and this is actually true also for our LFD 121 course on how to develop secure software. And when you think about it, it makes sense. The attackers don’t care what your license is. just, you know, they just don’t care. They’re going to try to do bad things to you regardless of, of the licensing.

And so while certainly a number of things that we develop like, you know, the best practices badge are very focused on open source software, you know, other things like baseline, other things like, for example, this course on LFT 121, the general how to develop secure software course, they’re absolutely for open source and closed source. Because again, the attackers don’t care.

Yesenia (19:53)
Yeah, they just they just don’t they’re they’re actually just trying to go around all this like they’re trying to make sure they learn it so that they know what to do. Unfortunately, that’s the case. And this course you said it offers a digital badge. Why is this important for developers and employers?

David A. Wheeler (20:11)
Well, I think the short answer is that anybody can say, yeah, I learned something. But it’s, think, for, I guess I should start with the employers because that’s the easier one to answer. Employers like to see that people know things and having a digital badge is a super easy way for an employer to, to make sure that, yeah, they actually learn at least the basics of, you know, that topic. you know, certainly it’s the same for you know, university degrees and other things. You when you’re an employer, you want the, it’s very, important that people who are working for you actually know something that’s critically important. And while a degree or digital badge doesn’t guarantee it, it at least gives that additional evidence. For people, mean, obviously if you want, are trying to get employed by someone, it’s always nice to be able to prove that. But I think it’s also a way to both show you know something to others and frankly encourage others to learn this stuff. We have a situation now where way too many people don’t know how to do some what to me are pretty basic stuff. You know I’ll point back to the LFD 121 course which is how to develop secure software. Most colleges, most universities that’s not a required part of the degree. I think it should be.

David A. Wheeler (21:35)
But since it isn’t, it’s really, really helpful for everybody to know, wait, this person coming in, do they not, they’ve got this digital badge. That gives me much higher confidence going in as somebody I’m working with and that sort of thing, as well as just encouraging others to say, hey, look, I carried enough to take the time to learn this, you can too. And both LFD 121 and this new AI course are free, so that in there online so you can take it at your pace. Those roadblocks do not exist. We’re trying to get the word out because this is important.

Yesenia (22:16)
Yeah, I love that these courses are more accessible and how you touched on the students, like students applying for universities that might be more highly competitive. They’re like, hey, look, I’m taking this extra path to learn and take these courses. Here’s kind of like the proof. And it’s like Pokemon. It’s good to collect them all, know, between the badges and the certifications and the degrees.

I definitely that’s the security professional’s journeys. Collect them all at this point with the credibility and benefits.

David A. Wheeler (22:46)
Well, indeed, course, the real goal, of course, is to learn, not the badges. But I think that badges are frankly, you know, you collecting the gold star, there is nothing more human and nothing more okay than saying, Hey, I, you know, I got to I got a gold star for if you’re doing something that’s good. Yes, revel in that. Enjoy it. It’s fun.

David A. Wheeler (23:11)
And I don’t think that these are impossible courses by any means. And unlike some other things which are, know, I’m not against playing games, playing games is fun, but this is a little thing that’s both can be interesting and is going to be important long-term to not only yourself, but every who uses the code you make. Cause that’s gonna be all of us are the users of the software that all developers as a community make.

Yesenia (23:42)
Yeah, there’s a wide range impact from this, not just like even if you don’t create software, just understanding and learning about this, you’re a user to understanding that basic understanding of it. So I want to transition a little bit to the course because I know we’re spending the whole time about it. Let’s say I’m a friendly person. I signed up for this free LFEL 1012. Can you walk me through the course structure? Like what am I expected to take away from the course in that time period?

David A. Wheeler (24:09)
Okay, yeah, so let’s see here. So I think what I should do is kind of first walk through the outline. Basically, I mean, the first two parts unsurprisingly are introduction and some context. And then we jump immediately into key AI concepts for secure development. We do not assume that someone taking this course is already an expert in AI. I mean, if you are, that’s great. It doesn’t take, we’re not spending a lot of time on it, but we wanna make sure that you understand the basics, the key terms that matter for software development. And then we drill right into the security risks of using AI assistance. I want to make it clear, we’re not saying you can’t use just because something has a risk, everything has risks, okay? But understanding what the potential issues are is important because then you can start addressing them. And then we go through what I would call kind of the meat of the course, best practices for secure assistant use.

You know, how do you reduce the risk that the assistant itself doesn’t become subverted, it starts working against you, things like that. Writing more secure code with AI, if you just say, write some code, a lot of it’s gonna be insecure. There are ways to deal with that, but it’s not so simple or straightforward. For example, it’s pretty common to tell AI systems that, hey, I’m an expert in this topic and suddenly it gets better. That trick doesn’t work.

No, you may laugh, but honestly, that trick works in a lot of situations, but it doesn’t work here. And we’ve actually researched showing it doesn’t work. So there are things that work, but it’s more it’s, it’s more than that. And finally, reviewing code changes in a world with AI. Now, of course, involves reviewing proposed changes from others. And in some cases, trying to deal with the potential DDoS attacks as people start creating far more code than anybody can reasonably review. Okay. We’re have to deal with this. and frankly, biggest problem, frankly, the folks who are vibe coding, you know, they, they run some program. It tells them 10 things. I’ll just dump all 10 things at them. And no, that’s a terrible idea. you know, and the, the curl folks, for example, have had an interesting point where.

They complained bitterly about some inputs from AI systems, which were absolute garbage, wasted their time. And they’ve praised other AI submissions because somebody took the time to make sure that they were actually helpful and correct and so on. And then that’s fantastic. you know, basically you need to push back on the junk and then find and then welcome the good stuff. And then, of course, a little conclusion wrap up kind of thing.

Yesenia (27:01)
I love it. it was a good outline. was not seeing it. Is this like videos accomplished with it or is this just like a module click through?

David A. Wheeler (27:10)
Well, basically we, we group them into little chapters. forgot what their official term is. It’s chapters, section modules. I don’t remember what the right term is. I guess I should, but basically after you go that, then there’s a couple of quiz questions and then a little videos. Basically the idea is that we want people to get it quickly, but you know, if it’s just watch a video for an hour, people fall asleep, don’t remember anything. That’s the goal is to learn, not just, you know, sleep through a video.

David A. Wheeler (27:39)
So little snippets, little quiz questions, and at the end there’s a little final exam. And if you get your answers right, you get your badge. So it’s not that terribly hard. We estimate, it varies, but we estimate about an hour for people. So it’s not a massive time commitment. Do it on lunch break or something. I think this is going to be, as I said, I think this is going to be time well spent.

David A. Wheeler (28:07)
This is the world that we are all moving to, or frankly, have already arrived at.

Yesenia (28:12)
Yeah, I’m already here. think I said it’s a seedling. It’s about to grow into that big tree. Any last minute thoughts, takeaways that you want to share about the course, your experience, open source, supply chain security, all of the above.

David A. Wheeler (28:27)
My goodness. I’ll think of 20 things after we’re done with this, of course. well, no, problem is I’ll think about them later in French. believe it’s called the wisdom of the stairs. It’s as you leave the party, you come up with the point you should have made. so I guess I’ll just say that, you know, if you develop software, whether you’re not, whether you realize or not, it’s highly likely that the work that you do will influence many, many.

Yesenia (28:31)
You only get zero and one.

David A. Wheeler (28:54)
About many, many people, many more than you probably realize. So I think it’s important for all software developers to learn how to develop software, secure software in general, because whether or not you know how to do it, the attackers know how to attack it and they will attack it. So it’s important to know that in general, since we are moving and essentially have already arrived at the world of AI and software development, it’s important to learn the basics and yes.

Do keep learning. Well, all of us are going to keep learning throughout our lives. As long as we’re in this field, that’s not a bad thing. I think it’s an awesome thing. I wake up happy that I get to learn new stuff. But that means you actually have to go and learn the new stuff. And the underlying technology remark is, it’s actually remarkably stable in many things. This is a case though, where, yes, a lot of things change in the detail, but the fundamentals don’t. But this is something where, yeah, actually there is something fundamental that is changing. One time we didn’t use AI often to help us develop software. Now we do. So how do we do that wisely? And there’s a long list of specifics. The course goes through it. I’ll give a specific example so it’s not just this highfalutin, high level stuff. So for example,

Pretty much all these systems are based very much on LLMs, which is great. LLMs have some amazing stuff, but they also have some weaknesses. One is in particular, they are incredibly gullible. If they are told something, they will believe it. And if you tell them to read a document that gives them instructions on some tech, and the document includes malicious instructions, that’s what it’s going to do because it heard the malicious instructions.

David A. Wheeler (30:48)
Now that doesn’t mean you can’t use these technologies. I think that’s a road too far for most folks. But it does mean that there’s new risks that we never had to deal with before. And so there’s new techniques that we’re going to need to apply to do it well. And I don’t think they’re unreasonable. They’re just, you know, we now have a new situation and we’re have to make some changes because of new situation.

Yesenia (31:11)
Yeah, it’s like you mentioned earlier, like you can ask it to be an expert in something and then it’s like, oh, I’m an expert. That’s what I laughing. I was like, yeah, I use that a lot. I’m like, the prompt is you’re an expert in sales. You’re an expert in brand. You’re an expert in this. And it’s like, OK, once it gets in.

David A. Wheeler (31:25)
But the thing is, that really does work in some fields, remarkably. And of course, we can only speculate sometimes why LLMs do better in some areas than others. But I think in some areas, it’s quite easy to distinguish the work of experts from non-experts. And the work of experts is manifestly and obviously different. And at least so far, LLMs struggle to do that.

David A. Wheeler (31:54)
This differentiation in this particular domain. And we can speculate why but basically, the research says that doesn’t work. So don’t do that. Do there are other techniques that have far more success, do those instead. And I would say, hey, I’m sure we’ll learn more things, there’ll be more research, use those as we learn them. But that doesn’t mean that we get to excuse ourselves from ignoring the research we have now, even though we don’t know it.

David A. Wheeler (32:23)
We don’t know everything. We won’t know everything next year either. Find out what you need to know now and be prepared to learn more Seagull.

Yesenia (32:32)
It’s a journey. Always learning every year, every month, every day. It’s great. We’re going to transition into our rapid fire. All right, so I’m going to ask the question. You got to answer quiz and there’s no edit on this point. All right, favorite programming language to teach security with.

David A. Wheeler (32:56)
I don’t have a favorite language. It’s like asking what my children, know, which of my children are my favorite. I like lots of programming languages. That said, I often use Java, Python, C to teach different things more because they’re decent exemplars of those kinds of languages. But so there’s your answer.

Yesenia (33:19)
Those are good range because you have your memory based one, which is the see your Python, which more scripts in the Java, which is more object oriented. So you got a good diverse group.

David A. Wheeler (33:28)
Static type, right, you’ve got your static typing, you’ve got your scripting, you’ve got your lower level. But indeed, I love lots of different programming languages. I know over 100, I’m not exaggerating, I counted, there’s a list on my website. But that’s less impressive than you might think because after you’ve learned a couple, the others tend to not, they often are similar too. Yes, Haskell and Lisp are really different.

David A. Wheeler (33:55)
But most Burmese languages are not as different as you might think, especially after you’ve learned a few. So I hope can help.

Yesenia (34:01)
Yeah, the newer ones too are very similar in nature. Next question, Dungeon and Dragon or Lord of the Rings?

David A. Wheeler (34:11)
I love both. What are we doing? What are you doing to me? Yeah, so I play Dungeons and Dragons. I have watched, I’ve read the books and watched the movies many times. So yes.

Yesenia (34:24)
Yes, yes, yes. First open source project you ever contributed to.

David A. Wheeler (34:30)
Wow, that is too long ago. I don’t remember. Seriously, but it was before the term open source software was created because that was created much later. So it was called free software then. So I honestly don’t remember. I sure it was some small contribution to something somewhere like many folks do, but I’m sorry. It’s lost in the midst of times back in the eighties. Maybe. Yeah. The eighties somewhere, probably mid eighties.

Yesenia (35:00)
You’re going to go to sleep now. Like,

David A. Wheeler (35:01)
So, yeah, yeah, I’m sure somebody will do the research and tell me. thank you.

Yesenia (35:09)
There wasn’t GitHub, so you can’t go back to commits.

David A. Wheeler (35:11)
That’s right. That’s right. No, was long before get long before GitHub and so on. Yep. Carry on.

Yesenia (35:18)
When you’re writing code, coffee or tea?

David A. Wheeler (35:22)
Neither! Coke Zero is my preferred caffeine of choice.

Yesenia (35:26.769)
And this is not sponsored.

David A. Wheeler (35:28.984)
It is not sponsored. However, I have a whole lot of empty cans next to me.

Yesenia (35:35)
AI tool you find the most useful right now.

David A. Wheeler (35:39)
Ooh, that one’s hard. I actually use about seven or eight depending on what they’re good for. For actual code right now, I’m tending to use Claude Code. Claude Code’s really one of the best ones out there for code. And of course, five minutes later, it may change. GitHub’s not bad either. There’s some challenges I’ve had with them. They had some bugs earlier, which I suspect they fixed by now.

But in fact, I think this is an interesting thing. We’ve got a race going on between different big competitors, and this is in many ways good for all of us. The way you get good at anything is by competing with others. So I think that we’re seeing a lot of improvements because you’ve got competing. And it’s okay if the answer changes over time. That’s an awesome thing.

Yesenia (36:36)
That is awesome. That’s technology. And the last one, this is for chaos. GIF or GIF?

David A. Wheeler (36:42)
It’s GIF. Graphics. Graphics has a guck in it. And yes, I’m aware that the original perpetrator doesn’t pronounce it that way, but it’s still GIF. I did see a cartoon caption which said GIF or GIF. And of course I can hear it just reading it.

Yesenia (36:53)
There you have it.

Yesenia (37:05)
My notes is literally spelled the same.

David A. Wheeler (37:08)
Hahaha!

Yesenia (37:11)
All right, well there you have it folks, another rapid fire. David, thank you so much for your time today, for your impact contribution to open source in the past couple decades. Really appreciate your time and all the contributors that were part of this course. Check it out on the Linux Foundation website. then David, do you want to close it out with anything on how they can access the course?

David A. Wheeler (37:38)
Yeah, so basically the course is ecure AI/ML-Driven Software Development its numbers LFEL 1012 And I’m sure we’ll put a link in the show. No, I’m not gonna try to read out the URL But we’ll put a link in there to to get to it But please please take that course. We’ve got some other courses

Software development, you’re always learning and this is an easy way to get the information you most need.

Yesenia (38:14)
Thank you so much for your time today and those listening. We’ll catch you on the next episode.

David A. Wheeler (38:19)
Thank you.

Oct 07
Love0

What’s in the SOSS? Podcast #41 – S2E18 The Remediation Revolution: How AI Agents Are Transforming Open Source Security with John Amaral of Root.io

By Podcast

Summary

In this episode of What’s in the SOSS, CRob sits down with John Amaral from Root.io to explore the evolving landscape of open source security and vulnerability management. They discuss how AI and LLM technologies are revolutionizing the way we approach security challenges, from the shift away from traditional “scan and triage” methodologies to an emerging “fix first” approach powered by agentic systems. John shares insights on the democratization of coding through AI tools, the unique security challenges of containerized environments versus traditional VMs, and how modern developers can leverage AI as a “pair programmer” and security analyst. The conversation covers the transition from “shift left” to “shift out” security practices and offers practical advice for open source maintainers looking to enhance their security posture using AI tools.

Conversation Highlights

00:25 – Welcome and introductions
01:05 – John’s open source journey and Root.io’s SIM Toolkit project
02:24 – How application development has evolved over 20 years
05:44 – The shift from engineering rigor to accessible coding with AI
08:29 – Balancing AI acceleration with security responsibilities
10:08 – Traditional vs. containerized vulnerability management approaches
13:18 – Leveraging AI and ML for modern vulnerability management
16:58 – The coming “remediation revolution” and fix-first approach
18:24 – Why “shift left” security isn’t working for developers
19:35 – Using AI as a cybernetic programming and analysis partner
20:02 – Call to action: Start using AI tools for security today
22:00 – Closing thoughts and wrap-up

Transcript

Intro Music & Promotional clip (00:00)

CRob (00:25)
Welcome, welcome, welcome to What’s in the SOSS, the OpenSSF’s podcast where I talk to upstream maintainers, industry professionals, educators, academics, and researchers all about the amazing world of upstream open source security and software supply chain security.

Today, we have a real treat. We have John from Root.io with us here, and we’re going to be talking a little bit about some of the new air quotes, “cutting edge” things going on in the space of containers and AI security. But before we jump into it, John, could maybe you share a little bit with the audience, like how you got into open source and what you’re doing upstream?

John (01:05)
First of all, great to be here. Thank you so much for taking the time at Black Hat to have a conversation. I really appreciate it. Open source, really great topic. I love it. Been doing stuff with open source for quite some time. How do I get into it? I’m a builder. I make things. I make software been writing software. Folks can’t see me, but you know, I’m gray and have no hair and all that sort of We’ve been doing this a while. And I think that it’s been a great journey and a pleasure in my life to work with software in a way that democratizes it, gets it out there. I’ve taken a special interest in security for a long time, 20 years of working in cybersecurity. It’s a problem that’s been near and dear to me since the first day I ever had my like first floppy disk, corrupted. I’ve been on a mission to fix that. And my open source journey has been diverse. My company, Root.io, we are the maintainers of an open source project called Slim SIM (or SUM) Toolkit, which is a pretty popular open source project that is about security and containers. And it’s been our goal, myself personally, and as in my latest company to really try to help make open source secure for the masses.

CRob (02:24)
Excellent. That is an excellent kind of vision and direction to take things. So from your perspective, I feel we’re very similar age and kind of came up maybe in semi-related paths. But from your perspective, how have you seen application development kind of transmogrify over the last 20 or so years? What has gotten better? What might’ve gotten a little worse?

John (02:51)
20 years, big time frame talking about modern open source software. I remember when Linux first came out. And I was playing with it. I actually ported it to a single board computer as one of my jobs as an engineer back in the day, which was super fun. Of course, we’ve seen what happened by making software available to folks. It’s become the foundation of everything.

Andreessen said software will eat the world while the teeth were open source. They really made software available and now 95 or more percent of everything we touch and do is open source software. I’ll add that in the grand scheme of things, it’s been tremendously secure, especially projects like Linux. We’re really splitting hairs, but security problems are real. as we’ve seen, proliferation of open source and proliferation of repos with things like GitHub and all that. Then today, proliferation of tooling and the ability to build software and then to build software with AI is just simply exponentiating the rate at which we can do things. Good people who build software for the right reasons can do things. Bad people who do things for the bad reasons can do things. And it’s an arms race.

And I think it’s really both benefiting software development, society, software builders with these tremendously powerful tools to do things that they want. A person in my career arc, today I feel like I have the power to write code at a rate that’s probably better than I ever have. I’ve always been hands on the keyboard, but I feel rejuvenated. I’ve become a business person in my life and built companies.

And I didn’t always have the time or maybe even the moment to do coding at the level I’d like. And today I’m banging out projects like I was 25 or even better. But at the same time that we’re getting all this leverage universally, we also noticed that there’s an impending kind of security risk where, yeah, we can find vulnerabilities and generate them faster than ever. And LLMs aren’t quite good yet at secure coding. I think they will be. But also attackers are using it for exploits and really as soon as a disclosed vulnerability comes out or even minutes later, they’re writing exploits that can target those. I love the fact that the pace and the leverage is high and I think the world’s going to do great things with it, the world of open source folks like us. At the same time, we’ve got to be more diligent and even better at defending.

CRob (05:44)
Right. I heard an interesting statement yesterday where folks were talking about software engineering as a discipline that’s maybe 40 to 60 years old. And engineering was kind of the core noun there. Where these people, these engineers were trained, they had a certain rigor. They might not have always enjoyed security, but they were engineers and there was a certain kind of elegance to the code and that was people much like artists where they took a lot of pride in their work and how the code you could understand what the code is. Today and especially in the last several years with the influx of AI tools especially that it’s a blessing and a curse that anybody can be a developer. Not just people that don’t have time that used to do it and now they get to of scratch that itch. But now anyone can write code and they may not necessarily have that same rigor and discipline that comes from like most of them engineering trades.

John (06:42)
I’m going to guess. I think it’s not walking out too far on limb that you probably coded in systems at some point in your life where you had a very small amount of memory to work with. You knew every line of code in the system. Like literally it was written. There might have been a shim operating system or something small, but I wrote embedded systems early in my career and we knew everything. We knew every line of code and the elegance and the and the efficiency of it and the speed of it. And we were very close to the CPU, very close to the hardware. It was slow building things because you had to handcraft everything, but it was very curated and very beautiful, so to speak. I find beauty in those things. You’re exactly right. I think I started to see this happen around the time when JVM started happening, Java Virtual Machines, where you didn’t have to worry about Java garbage collection. You didn’t have to worry about memory management.

And then progressively, levels of abstraction have changed right to to make coding faster and easier and I give it more you know more power and that’s great and we’ve built a lot more systems bigger systems open source helps. But now literally anyone who can speak cogently and describe what they want and get a system and. And I look at the code my LLM’s produce. I know what good code looks like. Our team is really good at engineering right?

Hmm, how did it think to do it that way? Then go back and we tell it what we want and you can massage it with some words. It’s really dangerous and if you don’t know how to look for security problems, that’s even more dangerous. Exactly, the level of abstraction is so high that people aren’t really curating code the way they might need to to build secure production grade systems.

CRob (08:29)
Especially if you are creating software with the intention of somebody else using it, probably in a business, then you’re not really thinking about all the extra steps you need to take to help protect yourself in your downstream.

John (08:44)
Yeah, yeah. think it’s an evolution, right? And where I think of it like these AI systems we’re working with are maybe second graders. When it comes to professional code authoring, they can produce a lot of good stuff, right? It’s really up to the user to discern what’s usable.

And we can get to prototypes very quickly, which I think is greatly powerful, which lets us iterate and develop. In my company, we use AI coding techniques for everything, but nothing gets into production, into customer hands that isn’t highly vetted and highly reviewed. So, the creation part goes much faster. The review part is still a human.

CRob (09:33)
Well, that’s good. Human on the loop is important.

John (09:35)
It is.

CRob (09:36)
So let’s change the topic slightly. Let’s talk a little bit more about vulnerability management. From your perspective, thinking about traditional brick and mortar organizations, how have you seen, what key differences do you see from someone that is more data center, server, VM focused versus the new generation of cloud native where we have containers and cloud?

What are some of the differences you see in managing your security profile and your vulnerabilities there?

John (10:08)
Yeah, so I’ll start out by a general statement about vulnerability management. In general, the way I observe current methodologies today are pretty traditional.

It’s scan, it’s inventory – What do I have for software? Let’s just focus on software. What do I have? Do I know what it is or not? Do I have a full inventory of it? Then you scan it and you get a laundry list of vulnerabilities, some false positives, false negatives that you’re able to find. And then I’ve got this long list and the typical pattern there is now triage, which are more important than others and which can I explain away. And then there’s a cycle of remediation, hopefully, a lot of times not, that you’re cycling work back to the engineering organization or to whoever is in charge of doing the remediation. And this is a very big loop, mostly starting with and ending with still long lists of vulnerabilities that need to be addressed and risk managed, right? It doesn’t really matter if you’re doing VMs or traditional software or containerized software. That’s the status quo, I would say, for the average company doing vulnerability maintenance. And vulnerability management, the remediation part of that ends up being some fractional work, meaning you just don’t have time to get to it all mostly, and it becomes a big tax on the development team to fix it. Because in software, it’s very difficult for DevSec teams to fix it when it’s actually a coding problem in the end.

In traditional VM world, I’d say that the potential impact and the velocity at which those move compared to containerized environments, where you have

Kubernetes and other kinds of orchestration systems that can literally proliferate containers everywhere in a place where infrastructure as code is the norm. I just say that the risk surface in these containerized environments is much more vast and oftentimes less understood. Whereas traditional VMs still follow a pattern of pretty prescriptive way of deployment. So I think in the end, the more prolific you can be with deploying code, the more likely you’ll have this massive risk surface and containers are so portable and easy to produce that they’re everywhere. You can pull them down from Docker Hub and these things are full of vulnerabilities and they’re sitting on people’s desks.

They’re sitting in staging areas or sitting in production. So proliferation is vast. And I think that in conjunction with really high vulnerability reporting rates, really high code production rates, vast consumption of open source, and then exploits at AI speed, we’re seeing this kind of almost explosive moment in risk from vulnerability management.

CRob (13:18)
So there’s been, over the last several, like machine intelligence, which has now transformed into artificial intelligence. It’s been around for several decades, but it seems like most recently, the last four years, two years, it has been exponentially accelerating. We have this whole spectrum of things, AI, ML, LLM, GenAI, now we have Agentic and MCP servers.

So kind of looking at all these different technologies, what recommendations do you have for organizations that are looking to try to manage their vulnerabilities and potentially leveraging some of this new intelligence, these new capabilities?

John (13:58)
Yeah, it’s amazing at the rate of change of these kinds of things.

CRob (14:02)
It’s crazy.

John (14:03)
I think there’s a massively accelerating, kind of exponentially accelerating feedback loop because once you have LLMs that can do work, they can help you evolve the systems that they manifest faster and faster and faster. It’s a flywheel effect. And that is where we’re going to get all this leverage in LLMs. At Root, we build an agentic platform that does vulnerability patching at scale. We’re trying to achieve sort of an open source scale level of that.

And I only said that because I believe that rapidly, not just us, but from an industry perspective, we’re evolving to have the capabilities through agentic systems based on modern LLMs to be able to really understand and modify code at scale. There’s a lot of investment going in by all the major players, whether it’s Google or Anthropic or OpenAI to make these LLM systems really good at understanding and generating code. At the heart of most vulnerabilities today, it’s a coding problem. You have vulnerable code.

And so, we’ve been able to exploit the coding capabilities to turn it into an expert security engineer and maintainer of any software system. And so I think what we’re on the verge of is this, I’ll call it remediation revolution. I mentioned that the status quo is typically inventory, scan, list, triage, do your best. That’s a scan for us kind of, you know, I’ll call it, it’s a mode where mostly you’re just trying to get a comprehensive list of the vulnerabilities you have. It’s going to get flipped on its head with this kind of technique where it’s going to be just fix everything first. And there’ll be outliers. There’ll be things that are kind of technically impossible to fix for a while. For instance, it could be a disclosure, but you really don’t know how it works. You don’t have CWEs. You don’t have all the things yet. So you can’t really know yet.

That gap will close very quickly once you know what code base it’s in and you understand it maybe through a POC or something like that. But I think we’re gonna enter into the remediation revolution of vulnerability management where at least for third party open source code, most of it will be fixed – a priority.

Now, zero days will start to happen faster, there’ll be all the things and there’ll be a long tail on this and certainly probably things we can’t even imagine yet. But generally, I think vulnerability management as we know it will enter into this phase of fix first. And I think that’s really exciting because in the end it creates a lot of work for teams to manage those lists, to deal with the re-engineering cycle. It’s basically latent rework that you have to do. You don’t really know what’s coming. And I think that can go away, which is exciting because it frees up security practitioners and engineers to focus on, I’d say more meaningful problems, less toil problems. And that’s good for software.

CRob (17:08)
It’s good for the security engineers.

John (17:09)
Correct.

CRob (17:10)
It’s good for the developers.

John (17:11)
It’s really good for developers. I think generally the shift left revolution in software really didn’t work the way people thought. Shifting that work left, it has two major frictions. One is it’s shifting new work to the engineering teams who are already maximally busy.

CRob (17:29)
Correct.

John (17:29)
I didn’t have time to do a lot of other things when I was an engineer. And the second is software engineers aren’t security engineers. They really don’t like the work and maybe aren’t good at the work. And so what we really want is to not have that work land on their plate. I think we’re entering into an age where, and this is a general statement for software, where software as a service and the idea of shift left is really going to be replaced with I call shift out, which is if you can have an agentic system do the work for you, especially if it’s work that is toilsome and difficult, low value, or even just security maintenance, right? Like lot of this work is hard. It’s hard. That patching things is hard, especially for the engineer who doesn’t know the code. If you can make that work go away and make it secure and agents can do that for you, I think there’s higher value work for engineers to be doing.

CRob (18:24)
Well, and especially with the trend with open source, kind of where people are assembling composing apps instead of creating them whole cloth. It’s a very rare engineer indeed that’s going to understand every piece of code that’s in there.

John (18:37)
And they don’t. I don’t think it’s feasible. don’t know one except the folks who write node for instance, Node works internally. They don’t know. And if there’s a vulnerability down there, some of that stuff’s really esoteric. You have to know how that code works to fix it. As I said, luckily, agent existing LLM systems with agents kind of powering them or using or exploiting them are really good at understanding big code bases. They have like almost a perfect memory for how the code fits together. Humans don’t, and it takes a long time to learn this code.

CRob (19:11)
Yeah, absolutely. And I’ve been using leveraging AI in my practice is there are certain specific tasks AI does very well. It’s great at analyzing large pools of data and providing you lists and kind of pointers and hints. Not so great making it up by its own, but generally it’s the expert system. It’s nice to have a buddy there to assist you.

John (19:35)
It’s a pair programmer for me, and it’s a pair of data analysts for you, and that’s how you use it. I think that’s a perfect. We effectively have become cybernetic organisms. Our organic capabilities augmented with this really powerful tool. I think it’s going to keep getting more and more excellent at the tasks that we need offloaded.

CRob (19:54)
That’s great. As we’re wrapping up here, do you have any closing thoughts or a call to action for the audience?

John (20:02)
Call to action for the audience – I think it’s again, passion play for me, vulnerability management, security of open source. A couple of things. same. Again, same cloth – I think again, we’re entering an age where think security, vulnerability management can be disrupted. I think anyone who’s struggling with kind of high effort work and that never ending list helps on the way techniques you can do with open source projects and that can get you started. Just for instance, researching vulnerabilities. If you’re not using LLMs for that, you should start tomorrow. It is an amazing buddy for digging in and understanding how things work and what these exploits are and what these risks are. There are tooling like mine and others out there that you can use to really take a lot of effort away from vulnerability management. I’d say that for any open source maintainers out there, I think you can start using these programming tools as pair programmers and security analysts for you. And they’re pretty good. And if you just learn some prompting techniques, you can probably secure your code at a level that you hadn’t before. It’s pretty good at figuring out where your security weaknesses are and telling you what to do about them. I think just these things can probably enhance security open source tremendously.

CRob (24:40)
That would be amazing to help kind of offload some of that burden from our maintainers and let them work on that excellent…

John (21:46)
Threat modeling, for instance, they’re actually pretty good at it. Yeah. Which is amazing. So start using the tools and make them your friend. And even if you don’t want to use them as a pair of programmer, certainly use them as a adjunct SecOps engineer.

CRob (22:00)
Well, excellent. John from Root.io. I really appreciate you coming in here, sharing your vision and your wisdom with the audience. Thanks for showing up.

John (22:10)
Pleasure was mine. Thank you so much for having me.

CRob (22:12)
And thank you everybody. That is a wrap. Happy open sourcing everybody. We’ll talk to you soon.