craig

• 00:57 – Guest introductions
• 01:56 – Dave and Jay offer insight into why CoSAI was necessary
• 05:16 – Jay and Mihai explain the complementary nature of OpenSSF’s AI/ML Security Working Group and CoSAI
• 07:21 – Mihai digs into the importance of proving model provenance
• 08:50 – Dave shares his thoughts on future CoSAI/OpenSSF collaborations
• 11:13 – Jay, Dave and Mihai answer Omkhar’s rapid-fire questions
• 14:12 – The guests offer their advice to those entering the field today and their call to action for listeners

• David LaBianca on LinkedIn
• Mihai Maruseac on LinkedIn
• Jay White on LinkedIn
• CoSAI homepage
• OpenSSF AI/ML Security Working Group homepage
• Get involved with OpenSSF

Jay White soundbite (00:01)
We are always talking about building these tentacles that spread out from the AI/ML security working group and the OpenSSF. And how can we spread out across the other open source communities that are out there trying to tackle the same problem but from different angles? This is the right moment, the right time and we’re the right people to tackle it. 

Omkhar Arasaratnam (00:18)
Hi everyone, and welcome to What’s in the SOSS? I’m your host Omkhar Arasaratnam. I’m also the general manager of the OpenSSF. And today we’ve got a fun episode for y ‘all. We have not one, not two, but three friends on to talk about CoSAI, OpenSSF AI and ML, how they can be complementary, what they do together, how they will be focusing on different areas and what we have ahead in the exciting world of security and AI/ML. So to begin things, I’d like to turn to my friend David LaBianca. David, can you let the audience know what we do?

Dave LaBianca (00:57)
Yep, hey, so I’m David LaBianca. I’m at Google and I’m a security engineering director there and I do nowadays a lot of work in the secure AI space.

Omkhar Arasaratnam (01:06)
Thanks so much, David. Moving along to my friend Jay White. Jay, can you tell the audience what you do?

Jay White (01:16)
I’m Jay White. I work at Microsoft. I’m a security principal program manager. I cover the gamut across open source security strategy, supply chain security strategy, and AI security strategy.

Omkhar Arasaratnam (01:23)
Thank you, Jay. And last but not least, my good friend Mihai. Mihai, can you tell us a little bit about yourself and what you do?

Mihai Maruseac (01:30)
Hello, I am at Google and I’m working on the secure AI framework, mostly on model signing and supply chain integrity for models. And collate with the GI, I collate the OpenSSF AI working group.

Omkhar Arasaratnam (01:43)
Amazing. Thank you so much and welcome to one and all. It is a pleasure to have you here. So to kick things off, who’d like to tell the audience a little bit about CoSAI, the goals, why did we need another forum?

Dave LaBianca (01:56)
I can definitely jump in on that one, Omkhar. I think it’s a great question. What we saw since, you know, ChatGPT becoming a big moment was a lot of new questions, queries, inbounds to a whole bunch of the founders of CoSAI surrounding, hey, how are you doing this securely? How do I do this securely? What are the lessons learned? How do I avoid the mistakes that you guys bumped into to get to your secure point today?

And as we all saw this groundswell of questions and need and desire for better insight, we saw a couple things really happening. One is we had an opportunity to really work towards democratizing the access to the information required, the intelligence required, the best practices required to secure AI. And then everybody can execute to their heart’s content at whatever level they’re able to, but it’s not about not knowing how to do it. So we knew that that was our goal. 

And then why another forum? It was because there’s amazing work going on in very precise domains. OpenSSF is an example, but also great work in Frontier Model Forum, in OWASP, in Cloud Security Alliance, on different aspects of what you do around AI security. And the gap we saw was, well, where’s the glue? Where’s the meta program that tells you how you use all these elements together? How do you address this if you’re an ENG director or a CTO looking at the risk to your company from the security elements of AI?

How do you approach tying together classical systems and AI systems when you’re thinking about secure software development, supply chain security? How do you build claims out of these things? So the intent here was, well, how do we make the ecosystem better by filling that gap, that meta gap, and then really working hand in hand with all of the different forums that are going to go deep in particular areas? And then wherever possible, you’ll figure out how we fill any other gaps we identify as we go along.

Omkhar Arasaratnam (04:00)
That’s great. Jay, Mihai, anything to add there?

Jay White (04:02)
Nothing to add, just a bit of a caveat. When David and I spoke way back early on in this year, I was extremely excited about it because as he said, what’s the glue that brings it all together? And you know, up to that point, Mihai and I had already started the AI/ML Security Working Group under the OpenSSF. We’re sitting here thinking about security from the standpoint of well, what’s happening now with these open large language models? How are we creating security apparatus around these models? How is that tying into the broader supply chain security apparatus? And what ways can we think about how to do that kind of stuff? 

And then of course, when I met David, I said, man, this is phenomenal. We are always talking about building these tentacles, right? The tentacles that spread out from the AI/ML security workgroup in the OpenSSF. How can we spread out across the other open source communities that are out there trying to tackle the same problem but from different angles? So, this the right moment, at the right time and we’re the right people to tackle it.

Omkhar Arasaratnam (05:01)
That’s a great summary, Jay. It takes a village for sure. Now we have two of the OpenSSF AI work group leads on the podcast today. So, I mean, how does this relate to the work that we’re doing there, guys? Sounds very complementary, but could you add more color?

Jay White (05:17)
The way that we think about this is well, let’s start with the data. Let’s start with the models and see how we can build some sort of guidance or guideline or spec around how we sign models and how we think about model transparency. And then of course, bringing on a SIG, the model signing SIG, which actually built code. We have an API, a working API that right now we’re taking the next steps towards trying to market. I’ll let Mihai talk about that a little bit further. As a look forward into this conversation, I sit in both CoSAI and AI/ML Security Working Group. So, when we get to that level of discussion, the tie-in is amazing. But Mihai, please talk about the technical stuff that we got going on.

Mihai Maruseac (06:03)
We have two main technical approaches that we have tackled so far in the working group and they are very related. So one is model signing and the other one is trying to get moving forward for some way of registering the supply chain for a model. So the provenance, the SLSA provenance are similar. And I’m saying that they are both related because in the end, in both of these, we need to identify a model by its hash, so we to compute the digest of the model. 

As we work for the model signing, we discover that just simple hashing it as a Blob on disk is going to be very bad because it’s going to take a lot of time. The model is large. So we are investigating different approaches to make hashing efficient. And they can be reused both in model signing and in provenances and any other statement that we can do about AI supply chain. They would all be based on the same hashing scheme for models.

Omkhar Arasaratnam (07:02)
And Mihai, maybe to drill into that a little bit for the folks listening in the audience. So, we can use various hashing techniques to prove provenance, but what does that solve? Why is it that we want provenance of our models? What does that allow us to better reason over?

Mihai Maruseac (07:21)
Yeah, so there are two main categories that we can solve with the hashing of a model. One is making sure that the model has not been tampered with between training and using it in production. We have seen cases where model hubs got compromised and so on. So we can detect all of these compromises before we load the model. The other scenario is trying to determine a path from the model that gets used into an application to the model that got trained or to the data sets that have been used for training. 

When you train a model or when you have a training pipeline, you don’t just take the model from the first training job and put it directly into the application. In general, there are multiple steps, fine-tuning the model, combining multiple models, or you might do quantization. You might transform a model from one format to another.

For example, right now, a lot of the people are moving from pickle formats to safe tensile formats. So each of these steps should be recorded. In case there is some compromise in the environment, you will be able to detect it via the provenance.

Omkhar Arasaratnam (08:27)
Got it. David, I know that your focus has been more on the leadership of CoSAI, but certainly, it’s not the first time you and I have spoken about OpenSSF. I’m curious as you look across the work at OpenSSF, if there’s other opportunities where we may be able to collaborate in CoSAI and how you see that collaboration evolving in the future.

Dave LaBianca (08:50)
I think it’s a great question. We have three in our real work streams. One of them is fundamentally around AI software supply chain, secure software development frameworks. The other two are preparing the defender and AI security governance. But the beginning, the inception of this conversation around that CoSAI wanted to do something in this AI secure supply chain space was conversations with Mihai and others at Google, and Jay, and realizing that there were actually lots of opportunities here. 

You know, one of them that was really straightforward from everybody was, hey, nobody’s really just looking for a provenance statement when you’re a CTO, CIO, director or the like. They want to claim there’s something they’re trying to prove to the outside world or at least state to the outside world. How you compose all of those elements together, especially when it’s not just a model, it’s your entire application set that you’re binding this to. 

It’s the way you did fine-tuning or the way you’re using large token prompts, pulling it all together and being able to make that claim. There needs to be guidance and best practices and things that you can start from so that you don’t have to figure this out all out yourself. So that was one key area. 

Another area was there’s really truly amazing efforts going on in OpenSSF in this element of the AI space on provenance. One of the things that we feel that a group like CoSAI can really help with is collaborating on what are those additional not technical bits of how you prove or create the provenance statement, but what are the other things that over time a practitioner would want to see in a provenance statement or be able to be attested to with a providence statement so that the provenance statement actually ties more closely to a claim in the future? You know, things like, hey, should we have to state what geography the training data was allowed for as part of a statement as you go forward? Things like that. 

So bringing that AI expertise, that ecosystem expertise around things that people want to do with these solutions. And then working with and collaborating with OpenSSF on what does that mean? How do you actually use that? Do you use that in a provenance statement? We see that that’s the type of amazing opportunity, especially because we have really wonderful overlap. Having Jay and Mihai as part of this and all of the other board members that are doing OpenSSF, we see this really great opportunity for collaboration. There’s always that possibility that teams bump heads on things, but like the idea that we’re all working towards the same mission, the same goal, it should be good challenges as we get to the weird edges of this in the future.

Omkhar Arasaratnam (11:13)
And that’s certainly one of the biggest benefits of open source that we can all collaborate. We all bring our points and perspectives to a particular problem area. So at this part of the show, we go through three rapid-fire questions. This is the first time we’ve had three guests on. So I’m going to iterate through each of y’all. Feel free to elaborate. As with any of these answers, I’m going to give you a couple choices. But a valid answer is no, Omkhar, actually, it’s choice number X and here’s why. Fair warning: the first question I’m gonna make judgment and the first question is spicy or mild food? Jay White, let’s begin with you.

Jay White (11:53)
You know what? Somewhere in the middle. I like some things spicy. I like some things mild. I like my salsa spicy, but I’m not a fan of spicy wings.

Omkhar Arasaratnam (12:02)
I mean that was a politically correct statement. Let’s see if Mihai has something a little more deterministic.

Mihai Maruseac (12:08)
I am kind of similar to Jay, except on the other side. I like the spicy wings, but the mild salsa.

Omkhar Arasaratnam (12:15)
Oh man, you guys, you guys, you guys need to run for office. You’re just trying to please the whole crowd. Dave from your days on Wall Street, I know you can be much more black and white with, are you a spicy guy or a mild guy? 

Dave LaBianca (12:29)
Always spicy. 

Omkhar Arasaratnam (12:30)
Always spicy. See, that’s why Dave and I are going to hang out and have dinner next week. All right. Moving into another controversial topic: text editors, Vim, VS Code, or Emacs? Let’s start with Mihai.

Mihai Maruseac (12:43)
I use Vim everywhere for no matter what I’m writing.

Dave LaBianca (12:45)
I mean, it’s the only right answer. I mean, there’s only one right answer in that list and Mihai just said it. So, I mean, like that’s easy.

Omkhar Arasaratnam (12:51)
Absolutely. How about you, Jay? What are you going to weigh in with? 

Jay White (12:54)
It’s Vim. It’s the only answer.

Omkhar Arasaratnam (12:59)
The last controversial question, and we’ll start with Mr. LaBianca for this. Tabs or spaces?

Dave LaBianca (13:09)
That we even have to have this argument is more of the fun of it. It’s got to be spaces. It’s got to be spaces. Otherwise somebody’s controlling your story with tabs. And like, I don’t want that. I want the flexibility. I want to know that I’m using three or I’m using four. It’s spaces.

Omkhar Arasaratnam (13:23)
There’s a statement about control embedded in there somewhere. Mihai, how about you?

Mihai Maruseac (13:28)
I prefer to configure a formatter and then just use what the formatter says.

Omkhar Arasaratnam (13:34)
Ahh, make the computer make things consistent. I like that. Jay?

Jay White (13:38)
I’m spaces. I’m with David. I had to use a typewriter early on in life.

Omkhar Arasaratnam (13:42)
Hahaha. I got it.

Dave LaBianca (13:47)
I have those same scars, Jay.

Jay White (13:48)
Hahaha. Yeah!

Omkhar Arasaratnam (13:52)
So the last part of our podcast is is a bit of a reflection and a call to action. So I’m going to give each of you two questions. The first question is going to be what advice do you have for somebody entering our field today? And the second question will be a call to action for our listeners. So Mihai, I’m going to start with you, then we’ll go to Jay and wrap up with Mr. Labianca. Mihai, what advice do you have for somebody entering our field today?

Mihai Maruseac (16:52.204)
So I think right now the field is evolving very very fast, but that shouldn’t be treated as a blocker or a panic reason. There is a firehouse of papers on archive, a firehouse of new model formats and so on, but most of them have the same basis and once you understand the basis it will be easier to understand the rest.

Omkhar Arasaratnam (14:35)
Thanks, Mihai. What’s your call to action for our listeners?

Mihai Maruseac (14:39)
I think the principle call to action would be to get involved into any of the forums that we are talking about AI and security. It doesn’t matter which one, start with one and then from there expand as time allows to all of the other ones.

Omkhar Arasaratnam (14:52)
Jay, what advice do you have for somebody entering our field today?

Jay White (14:56)
Fundamentals, fundamentals, fundamentals. I can’t stress it enough. Do not start running. Start crawling. Take a look at you know, what was old because what was old is what is new again, and the perspective of what is old is lost on today’s engineer for automation and what’s new. So, your perspective might be very welcomed, especially if you concentrate on the fundamentals. So, anyone coming in today, become a master of the fundamentals, easiest path in and that way your conversation will start there and then everyone else who’s a bit more advanced will plus you up immediately because respect to be given to those fundamentals.

Omkhar Arasaratnam (15:39)
Completely agree. Fundamentals are fundamentals for a reason. And what is your call to action for our listeners?

Jay White (15:46)
So, my call to action is going to be a little different. I’m going to tackle this making a statement for everyone but targeting underrepresented communities because I also co-chair the DE&I working group inside of OpenSSF as well. I feel like this is an excellent opportunity not just for me to tackle it from this standpoint in terms of the AI/ML security working group but also for CoSAI as well. Look, just walk into the walk into the room. I don’t care whether you sit as a fly on the wall for a couple of minutes or whether you open your mouth to speak. Get in the room, be seen in the room. If you don’t know anything say, hey guys, I’m here, I’m interested, I don’t know much, but I want to learn. And be open, be ready to drink from the firehose, be ready to roll up your sleeves and get started. And that’s for the people in the underrepresented community.

And for everyone I would say, I would generally say the same thing. These rooms are free. The game is free. This is free game we’re giving you. Come in and get this free game and hold us accountable. And the merging of information security in general, cybersecurity down into this great AI engine that’s spinning back up again. The merging of these worlds are colliding at such a rapid pace, it’s an excellent time to get in and not know anything. Because guess what? Nine times out of ten, the people in there with the most to talk about don’t know anything either. They’re just talking and talking and talking until they say something right. So, get in and be ready and open to receive.

Omkhar Arasaratnam (17:28)
Those are extremely welcoming words and that outreach is welcome and amazing. Wrapping things up, Mr. LaBianica, what advice do you have for somebody entering our field today?

Jay White (17:41)
So I think honestly, it’s gotta be leaning into where Jay was going. For me, foundational security is the way you don’t speed run the last 40 years of vulnerabilities in your product, right? And whether you like the academic approach of reading and you want to go read Ross Anderson’s Security Engineering, rest in peace, whether you want to find it yourself, but there is so much knowledge out there that’s hidden in silos. 

And this doesn’t just go for people who starting their career. Twenty years in, if you’ve never looked at what information warfare sites the house have found or what signals intelligence have found, like if you haven’t looked across the lines and seen all the other ways these systems go wrong, you’re still working from a disadvantage. So it’s that foundational element and learn the history of it. You don’t have to worry about some of that stuff anymore, but knowing how you got here and why it’s happening now is so critical to the story.

And then especially with AI, please, please don’t forget that, yes, there’s an ooh shiny of the AI and it’s got new security problems, but making sure you’ve prevented access to your host, that you’ve got really strong authorization between systems, that you know why you’re using it and what your data is. Like these things are fundamental, and then you can worry about the more serious newer threats in these domains. But if you don’t do that stuff, really, really hard to catch up.

Omkhar Arasaratnam (18:56)
Words of wisdom to completely agree. And your call to action for our listeners, Dave?

Dave LaBianca (19:02)
I’m gonna tie it together, both Jay and Mihai, because I think both stories are super important. CoSAI is based on this idea of we really wanna democratize security intelligence and knowledge around securing AI. You can’t do that when it’s a single voice in the room, when it’s just Google or just tech companies in the US or just take your pick on what that just is. So my call to action is one, our work streams are starting, please lean in or anybody’s workstreams. It doesn’t have to be CoSAI’s workstreams. Lean in, bring your voice to the table because we need the different viewpoints in the room. We need the 10th person in the room going, wait, but I need a cost-effective solution that works on this low-end device in this region. Nobody can fix that if we don’t make sure that folks are in the room. 

Yes, you have to hold us accountable to make sure we make that space as Jay was saying, but we then also need those voices in the room. And regardless of where you come from, we need that contribution and that community building because there’s no way you can ever pick up anything whether it’s from Microsoft or Anthropic or IBM, Intel or Google and then say that’s gonna work for me. You need those diverse inputs, especially on the security side, right? They’ll let you go, OK well my company thinks about it or my entity thinks about it this way and I need to then figure out how I Find solutions that build to it So I think, you know, get involved, bring your voice to the table and help us all see the things that we’re missing that we’re kind of blind to because of either, you know, we work at a big tech company or whatever.

Omkhar Arasaratnam (20:29)
Thanks so much, Dave. As we close out, I’d love a quick, how should folks that are interested get involved with CoSAI as well as the AI/ML workgroup at the OpenSSF? Maybe we’ll start with Mihai. Mihai, if somebody’s interested in getting involved at the OpenSSF AI ML workgroup, where do they go to? How do they start?

Mihai Maruseac (20:49)
So join a meeting and we can go from there. The meeting is on the OpenSSF calendar every other Monday. 

Omkhar Arasaratnam (20:55)
For anyone that’s looking for where to check that out go to openssf.org, and it’s all up there on the home page. Dave if folks want to get involved with CoSAI, per your generous invitation, where can they go to learn more and how can they get plugged into those work groups that are spinning up?

Dave LaBianca (21:11)
So first things first, go to one word coalitionforsecureai.org. That gets you your starting points around how to find us and how to see where we go. Look at our archives of our email lists. Look at our GitHub repo that shows what we’ve currently published around our governance and our controlling rules. And then in September, look out for the calls to action from our work streams around participation and the rest. And then it’ll be exactly as Mihai said. Join us. Come troll on a GitHub repo, whatever you want to do, but find a way to get engaged because we’d love to have you there.

Omkhar Arasaratnam (21:40)
Amazing. Jay, anything to add in terms of new folks looking to join either project as you span both?

Jay White (21:45)
I’m accessible all over the place. You can find me on LinkedIn. But find me, find Mihai, find David, and contact us directly. We are more than happy to usher you in and bring you in, especially once the workstreams spin up in the Coalition for Secure AI. But Mihai and I were there every other Monday at 10 A.M. Pacific Standard Time. 

Omkhar Arasaratnam (22:08)
All right, well, thanks so much, guys. Really appreciate you joining, and I look forward to y’all helping to secure and democratize secure AI for everyone.

David LaBianca (22:16)
Hey, Omkhar, thank you for having us.

Omkhar Arasaratnam (22:18)
It’s a pleasure.

Announcer (22:19)
Thank you for listening to What’s in the SOSS? An OpenSSF podcast. Be sure to subscribe to our series of conversations on Spotify, Apple, Amazon or wherever you get your podcasts. And to keep up to date on the Open Source Security Foundation community, join us online at OpenSSF.org/get involved. We’ll talk to you next time on What’s in the SOSS?

In this episode, Omkhar chats with Mike Hanley, Chief Security Officer and SVP of Engineering at GitHub. Prior to GitHub, Mike was the Vice President of Security at Duo Security, where he built and led the security research, development, and operations functions.

After Duo’s acquisition by Cisco for $2.35 billion in 2018, Mike led the transformation of Cisco’s cloud security framework and later served as CISO for the company. Mike also spent several years at CERT/CC as a Senior Member of the Technical Staff and security researcher focused on applied R&D programs for the US Department of Defense and the Intelligence Community.

When he’s not talking about security at GitHub, Mike can be found enjoying Ann Arbor, MI with his wife and nine kids.

• 01:21  Mike shares insight into transporting a family of 11
• 02:02  Mike’s day-to-day at GitHub
• 03:53  Advice on communicating supply chain risk
• 08:19  Transforming the “Department of No” into the “Department of Yes And…”
• 12:44  AI’s potential impact on secure open software and, specifically, on software supply chains
• 18:02  Mike answers Omkhar’s rapid-fire questions
• 19:26  Advice Mike would give to aspiring security or software professionals
• 20:38  Mike’s call to action for listeners

• Mike Hanley on LinkedIn
• DARPA AI Cyber Challenge
• Get involved with the OpenSSF

Mike Hanley soundbite (00:01)

Core to everything that we do is this thesis that good security really starts with the developer. Internally, for our engineers, that means the developers who are Hubbers. But it’s also all of our customers and all of the developers and all of the open source maintainers and communities that are using GitHub. 

Omkhar Arasaratnam (00:18)
Welcome to What’s in the SOSS? I am this week’s host, Omkhar Arasaratnam. I’m also the general manager of the OpenSSF. Joining us this week is Mike Hanley. Mike is the CSO and SVP of engineering at GitHub. Prior to joining GitHub, Mike was the vice president at Duo Security where he built and led the security research, development and operations function. After Duo’s acquisition by Cisco, for many billions of dollars in 2018, Mike led the transformation of Cisco’s cloud security framework and later served as CISO for the company. And when he’s not talking about security at GitHub, Mike can be found enjoying Ann Arbor, Michigan with his wife and eight kids. Hold on. Do we need a pull request, Mike?

Mike Hanley (01:02)
I think we do. I think we need to update that to nine kids.

Omkhar Arasaratnam (01:04)
Well, congratulations to the Hanley family on a ninth. 

Mike Hanley (01:08)
Thank you, Omkhar I appreciate it.

Omkhar Arasaratnam (01:11)
I’ve got to ask you a question that every parent has probably asked you. How do you transport your family around? I mean, logistically, I’m curious.

Mike Hanley (01:21)
We have one of those amazing vans that looks like we’re running a summer camp. We have a Ford Transit, and it’s all-wheel drive, because I’m in Ann Arbor, Michigan, so we’re a four-seasons kind of area, so we need to make sure we can get around in the snow. But it’s great. It’s just got the side door, so when we get to school, everybody just sort of files out in a line, and we throw the door shut, and we’re off to the races. So Ford Transit, 12 seats. I got space for one more. (Laughter) So we love that thing. It’s great for getting around with the whole fam.

Omkhar Arasaratnam (01:48)
That’s amazing, a feat of engineering. Speaking of engineering, you’re the CSO and SVP of engineering at GitHub. Can you walk me through your role overseeing both of these teams and what it means for you, what it means for how you think about secure software development at GitHub?

Mike Hanley (02:02)
Yeah, so I initially joined GitHub a little over three years ago to be the first chief security officer. So bringing all the security programs together and then sort of writing the next chapter of that story. And about a year after I got to GitHub, so a little over two years ago, I said yes to the opportunity to also take on running all of engineering at GitHub. And having those two hats may seem a little bit unique on the outside, but I think it’s actually very natural for us at GitHub because core to everything that we do is this thesis that good security really starts with the developer and internally for our engineers, that means the developers who are Hubbers at GitHub building everything related to GitHub. 

But it’s also all of our customers and all the developers and all the open source maintainers and communities that are using GitHub. We want security to start with them as well and start with in a way that works for them as well. So secure in their accounts. So things like the 2FA work that we’ve done, but also that it’s easy for them to get security value into the projects and communities that they are a part of.

Omkhar Arasaratnam (03:05)

You’re the home for the world’s developers, and GitHub has this unique role in helping to secure the software supply chain, which is an area that can be really difficult for leaders to understand in depth and breadth. How are you advising organizations that are on their journey to better understand their supply chain? What would you advise security leaders or developers tackling this issue? 

I mean, as, I consider myself a software engineer who’s been doing security for a long time. Not every security leader has a background like you or I, where it comes from a place of software engineering. I’m curious how you articulate the complexity to those that may be more oriented towards risk or infrastructure, things of that nature.

Mike Hanley (03:53)
Yeah, I think the biggest danger, Omkhar, in terms of organizations or teams understanding their supply chain risk is they think too small or they have a too narrow of a lens. It’s like, well, my supply chain is secure because I know or have an attestation as to what totally composes the software that I’m actually building, packaging and publishing. And while that’s good, and certainly it’s something that organizations should do — and if they’re doing it great pat on the back to them because frankly, they’re already a leg up over a lot of other places that are doing nothing — but it’s important not to stop there. 

It’s…we’ve seen this actually with if you look at a lot of the mainstream incidents that have kind of hit the news in the last few years, of course you’re very familiar with all these, but they’ve involved things like attacking build systems. Some of them have been back doors in software

Some of them have been insider threats, and you sort of have this range of potential attack vectors on the security of your software. So I think you need to consider things like, of course, having an inventory of what’s there and doing, you know, sort of the bread and butter vulnerability management that you would, and dependency management that you, that you would or should be doing as a best practice. 

But it’s also considering, you know, how do I trust or how do I assess the organizations that are actually producing the software that I depend on? Are they secure in their persons? How do you think about the accounts and communities that are actually contributing to that work? Do I understand all the third-party integrations and tools that I’m using, of which many organizations I would suggest don’t have a full inventory of many of those? When we look at third party integrations on GitHub, there are a vast sort of sea of those that customers and developers and organizations use. 

Having an understanding of like, what’s plugged in where? What do they have access to? The people who develop and operate that integration or the service I’m integrating with, what’s their security look like? And I think it’s really just understanding this very broad kind of network of effects that happens goes well beyond just, like, the idea that somebody could potentially commit a backdoor, which is obviously an important thing to assess or that there might be a vulnerability and a dependency that you have. These are actually important things to assess, but your threat model needs to be much, much broader than that. 

And I think for every organization that’s really worried about their code getting backdoored, good, they should be thinking about that. But they also need to make sure they actually go look and say like, what third-party applications of our developers authorized sort of full and unfettered access to our build systems on? If you haven’t done that, you need to make sure that you’re looking at some of those things. And this has really actually informed a lot of the decisions that we’ve tried to make at GitHub in the course of the last few years. 

I mean, the one that I think is what I hope will be one of the most impactful things that’s happened in supply chain security in the last several years was actually driving two-factor authentication adoption. And you’ll remember the sort of package hijacking attacks that happened on the NPM ecosystem in late 2021. And that was a really interesting set of learnings for us because the adversary incentives are so clear. If I can just get one piece of code to run a Monero miner on this package that’s in the top 500 on NPM, I’m going to get at least 10,000 downloads an hour out of that. 

And so the incentive is very high to go attack those accounts, but the vast majority of maintainers and developers that we found at the time actually weren’t using 2FA. And it’s sort of interesting to look and say, well, in the IT world, if you started a corporate job tomorrow, you would get a managed laptop, an account, and something that was enrolled in 2FA, because that’s now a best practice and a standard in IT. Yet we don’t have, or we haven’t quite caught up with those types of standards and best practices in the broader sort of developer ecosystem. 

So I think the 2FA work that we did — while it was hard to get many, many millions of developers enrolled in 2FA — that’s the kind of thing that just raises the bar for everybody and it substantially increases the attack cost of a supply chain attack because you’re kind of crowding out account takeover and sort of these other low hanging fruit that would most commonly be exploited by a bad guy. 

Omkhar Arasaratnam (07:38)
I think that’s a great point and thanks for all the great work that you all have done in order to increase some of the security basics like 2FA more broadly. I want to jump back to your role at GitHub. And as I mentioned before, as somebody that personally identifies as a software engineer who’s been doing security for a really long time, you know, software engineers the world over have oft complained about the security department coming in and saying, you can’t do this, blah, blah, blah. And then their velocity slows down and they can’t do the cool thing because security person said no. Both teams roll up to you. How do you balance that? How do you, how do you see that impacting your day to day? Is that a, is that a good thing? Bad thing?

Mike Hanley (08:19)
What you described is what I often call the Department of No. And really, I think a modern security team, especially in a modern software-driven organization, of which the vast majority of companies are at this point, you have to be the Department of Yes And. Which is, yes, we want to do the new thing. We assume that our engineers, our finance people, our marketing people have the right goals for the company. And we want to figure out how to make that particular endeavor successful and contemplate the risk management needs that we have as a company. 

We want to make sure the good thing happens while making sure that the bad thing does not happen. I think for us internally, having both teams under one roof while there’s traditionally this separation that you mentioned, that separation comes with challenges because it actually, in many cases, encourages the security team to sort of sit in the ivory tower and come down and say, well, Omkhar, you can’t do that because reasons. And usually that engagement happens

at the 11th hour, right? I mean, it’s very rare that you get sort of negative security feedback early in a model like that.

And I find that by having the teams together, I mean, literally all the security leaders are sitting in the same calls as the engineering leaders because they all are part of the same team. And we have this notion internally that really everybody in the company is on the security team. And obviously, that means something specific for engineering because they’re doing development work that impacts the security of the company, the security of our stakeholders. They’re building things to make sure that we can realize good security outcomes, especially for open source maintainers and our paying customers. So they kind of have a threefold mission there where they’re helping with security.

But it’s also true for the finance people who report a gift card scam from our CEO or the HR and recruiting folks who are looking out for fake potential employees who are trying to scam their way into the company. So that idea that everybody’s on the security team is really a cultural approach for us to make sure that everybody’s a part of that broader network. And so this is basically the antithesis of this idea that humans are the weakest link. In fact, we view them as the strongest part of our security program and actually an enhancement of all the tooling that we have by getting everybody engaged. 

But specific to engineering, I think it’s great because it actually makes sure the incentives are tightly aligned, right? Like I’m responsible for making sure that the website is running and secure all the time. Like those are the two things that are effectively my set of responsibilities to the business and they are not in any way intention. In fact, if you look at the vast majority of our investment profile, it is actually going toward those two things. Now we’re actually doing a lot of net new features and we’re building new things all the time and we’ve got you know experimentation on bets that we want to place in the future, but the overwhelming amount of our work goes to to those two priorities that I mentioned a minute ago.

You know we’re really geared toward like making sure that security is a good experience for everybody not not a bad one, which doesn’t mean we don’t say no from time to time. But I think you minimize the number of no’s by having security as deeply engaged in what’s happening at the edge as possible, because the security team can’t actually be everywhere all at once. Our security team is not equivalent to the number of engineers that we have in the company. I don’t know. I’ve yet to meet anybody who can say that. And I don’t think I will in this lifetime. Certainly not with the jobs reports that we see that suggest there’s a vast shortage of security talent and expertise. And certainly you and I see this when we’re thinking about how to help open source maintainers. It’s just not out there. It doesn’t exist en masse. And it doesn’t exist in sort of broadly available sense. 

But if you’re really leaning into the idea that we want to help the engineers and you recognize that security team is not going to be a part of great security work all the time, you actually want to make sure the engineers are the superheroes from a security standpoint or the finance folks are the superheroes from a security standpoint and you shift that mindset toward one that’s actually trying to drive that outside the security team. This actually works really, really well for us, and I think creates a really tight coupling between those two teams. And it also allows us to, I think, focus on like higher order concepts. 

So for example, the idea not just that we do security, but that we do it well, and we do it in a way that’s actually a great experience. So we talked a little bit about 2FA, spending the time with the design team and the product teams to figure out what is doing this well and in such a way that it’s not alienating, that it’s actually a good experience, that people actually adopt it at scale without feeling like it’s being foisted on them in a counterproductive way. 

Because when you end up in that scenario, people find a way to get around your security controls, whether they’re your internal developers or stakeholders that you’re affecting with an initiative like that. So really having everybody together under one roof driving those common goals, I think has actually been like very, very healthy for both the company’s internal security objectives, but also for our ability to affect broad change in the open source ecosystem as well with things like that 2FA initiative that we did.

Omkhar Arasaratnam (12:44)
I think you’re spot on. Humans are ingenious. Humans will want to do the thing that provides them the least friction. Let’s make the security, the secure thing, the thing that provides least friction. Your story about everyone’s accountability being security. It’s not just the security team, it’s finance, it’s engineering, et cetera, reminds me and sounds a little corny, but 20 years ago when I was just getting into security, I was at this conference and everybody had this lanyard that said, I’m a firewall. And, you know, 20 years later, it seems kind of corny, but the idea was security was everyone’s accountability and it didn’t need to be backstopped by SecOps. If somebody at the frontline could stop that upstream, the benefit was even larger. 

Now, if we switch gears a little bit and think about everyone’s favorite topic, AI. How do you think AI will impact open source security, and what ways do you see it helping us to secure the software supply chain?

Mike Hanley (13:42)
My view on this is very bullish and it is summed up as this. I think AI will be two things. One is it will redefine the idea of shifting left in software development, which is, you know, we’ve been talking about this for more than 10 years at this point. The idea obviously that you move security value as far left as possible. What that’s meant to date has been, you’re generally getting feedback sort of at CI/CD time, right? Like after you’ve written the code, submitted it for review and it’s going through testing, that’s typically when you’re getting feedback. So that’s left obviously of boom in most cases where boom is your thing has been deployed to production, you find a vulnerability or you get breached. 

AI is basically saying when you are in the editor, bringing your idea to code through the keyboard or other input device, at that moment, you don’t just have security expertise, you actually have an AI pair programmer who has a vast range of knowledge and expertise in a variety of topics, accessibility, security, scalability, the particular language or framework that you’re using, right there with you, giving you more secure suggestions, helping you check your work, and literally just paired with you the entire time. 

So I think that is the realization of what we really want with shift left, because you can’t actually go any further left than when you’re bringing your idea to code in the editor. And I think that’s the highest value place to add security value and it’s also the best experience for developers because you are in the flow doing the work in that moment, not, hey, I committed my changes up, I’m waiting for tests to run while I go get lunch and then I come back or maybe even it’s the next day depending on how slow your build systems are. 

The next day I get feedback and I gotta go, what was I thinking at that moment when I did this? So that shift left and adding security value right in the idea in real time is huge and that is uniquely available to us because of the advances in AI, certainly with GitHub Copilot, but there’s others out there as well that are trying to do similar things. So that’s one. 

The second is AI gives us an opportunity not just to help address vulnerabilities in new and existing code, right? So this idea that as I’m writing it, I can get higher quality, more secure code in real-time, but also I’m adding value to things like my CI/CD. So it’s, I’m getting better suggestions from automatically getting suggested fixes from some of the tooling that I have now, instead of just giving, hey, there’s a problem here. It’s there’s a problem here and here’s how you should fix it. So there’s tons of value there, but, but it also enables the idea to retrospectively fix things. 

And I think this is one of the like grand challenges that we’ve had frankly in, in security generally, but especially in open source security Omkhar as you know, like a lot of the building blocks that we have for everything that we do and experience are maintained by a small number of people who don’t necessarily have robust access to security resources. And in many cases, fundamental building blocks of the internet are unmaintained at this point. Major projects that people depend on are unmaintained. And that is a huge challenge for the whole ecosystem. 

But the idea that, particularly through AI and agents, that we might actually be able to at scale

ingest process and refactor or fix bugs at scale in open source projects to me is super exciting, super compelling. It’s a way to supercharge, not just your internal developers who are writing first party code, but actually help supercharge open source developers or open source communities and really empower them to go do this work. They’re like, they may not be incentivized to do, they may not have the resources to do, they may not have the expertise as part of their community of maintainers and their project to go do. That to me is super compelling. 

And you know, it’s interesting when you say like, well, we’re dependent on these things, if only we could figure out what to do about it. And often, we talk about, well, we can deploy people to go help. And yeah, that’s good. That helps for the short term. But that doesn’t scale. And it doesn’t necessarily help with these massive excavations that it takes to go back into projects that have 10, 20, or more years of history behind them. So I’m excited that AI can actually help us get the leverage that goes beyond what we can do with human scale, especially where we have tactically deployed things or sort of volunteer firefighter resources that can go help with a small number of things. AI is going to give us the lift, I think, to go scan and support and help fix and really renovate, maybe is a nice way to put it, some of the projects that we depend on for the next 10, 20, and 30 years.

Omkhar Arasaratnam (18:02)
That’s a great point, Mike. And it’s also why we look forward to programs like the AI Cyber Challenge from DARPA and seeing the work that’ll come out of that. Switching gears a bit, we’re going to go into rapid-fire mode. So I’m going to ask you a series of questions, give you a couple of possible answers. And of course, it’s up to you whether to choose one of them or say, hey, Omkhar, you got it wrong. Actually, this is how I feel. So are you ready to go?

Mike Hanley (18:30)
I’m ready, go.

Omkhar Arasaratnam (18:32)
Spicy vs. mild food?

Mike Hanley (18:35)
It’s basically always spicy for me.

Omkhar Arasaratnam (18:37)
Always spicy. I knew we got along good for a reason. Now this is a controversial one. We talked about bringing AI to the IDE. So let’s talk about text editors, Vim, VS code or Emacs?

Mike Hanley (18:48)
My tastes have changed over time. I would say I’m more of a Vim guy now, but at times I’ve actually just used Nano, which I don’t know that that’s a popular answer. (Laughter) I’m not proud of that, to be clear, but at times that was the preferred editor in a prior life.

Omkhar Arasaratnam (19:03)
Why does Nano bring such shame? (Laughter) All right, tabs or spaces?

Mike Hanley (19:09)
I’m generally a tabs, tabs kind of guy. Keep it, keep it simple.

Omkhar Arasaratnam (19:12)
Awesome. Alright, closing things out, Mike, you’ve had a wonderful illustrious career so far. What advice do you have somebody for somebody entering our field today? Be it from a software engineering or security perspective?

Mike Hanley (19:26)
I think for either one, find an opportunity to meet people and get involved and do something, have something tangible. And the great thing about open source is this is actually one of the very best ways to build your resume because your work is easily visible. You can show people what you’ve done. You see a lot of resumes, a lot of experiences, a lot of schools.

But what can really differentiate, I think, is when you can say, like, here’s a thing that I did, that I built, and I learned something unique from that. And you don’t always necessarily get that from just going through things like coursework. When you’ve really had to, like, duct tape something together and deal with the messy reality of getting things to work the way you want them to work outside of a sandbox, I think there’s a lot of value and sort of real-world knowledge that comes from that that is impossible to…there’s no compression algorithm for experience is a way I’ve heard this put previously.

And just hacking on the weekends with some of those projects or finding some people who you want to work on was for the sole sake of just learning how to do something new is incredibly valuable and it’s a great way to stand out.

Omkhar Arasaratnam (20:25)
That’s great advice. A mentor of mine once said, well, you should never test in prod learning and prod is a, is a useful lesson.?last question for you. What’s your call to action for our listeners?

Mike Hanley (20:38)
I think it’s similar. It’s getting involved in some meaningful way. But I think going back to some of the things that we talked about earlier, Omkar, just to give a slightly different answer to that, would be ask some hard questions in your organization about your understanding of open source security, and particularly your supply chain. I do think when I think about 20-plus years in the security space, it is one of those things that stands out to me as unique in that many organizations in security still don’t do the basics. I mean, it is 2024, and we still have to tell people to turn on two -factor authentication. So that is what it is, I would say. 

But it’s also true that not every organization does a great job with inventory management and with configuration management, or even just sort of mapping out their broader ecosystem and infrastructure. And I think just going back and saying, like, do we really understand our supply chain? Do we really understand our dependencies? Do we really have, like, provenance of our assets, our inventory, our vulnerability management? 

So I think again, in the context of open source security, like really just going back and saying, like, do we really know how we use this stuff and challenge maybe what the assumptions have been in the previous weeks and months and years? I guarantee you that whatever you’re looking at is far too narrow. And I think that can be an important conversation. It can really help your organization up-level its supply chain because again you might find healthy initiatives out there that are low-hanging fruit that you can take advantage of, so that’d be maybe a call to action is to have that conversation in your next security team meeting.

Omkhar Arasaratnam (22:02)
Great advice from Mike Hanley, the CSO and SVP of engineering at GitHub. Mike, thanks so much for joining us on What’s in the SOSS? and we look forward to having you on again soon, perhaps.

Mike Hanley (22:13)
Thank you Omkhar, great to be here.

Announcer (22:15)
Thank you for listening to What’s in the SOSS? An OpenSSF podcast. Be sure to subscribe to our series of conversations on Spotify, Apple, Amazon or wherever you get your podcasts. And to keep up to date on the Open Source Security Foundation community, join us online at OpenSSF.org/get involved. We’ll talk to you next time on What’s in the SOSS?

In this episode, Omkhar Arasaratnam visits with Aeva Black, who currently serves as the Section Chief for Open Source Security at CISA, and is an open source hacker and international public speaker with 25 years of experience building open source software projects at large technology companies.

She previously led open source security strategy within the Microsoft Azure Office of the CTO, and served on the OpenSSF Technical Advisory Committee, the OpenStack Technical Committee, and the Kubernetes Code of Conduct Committee. In her spare time, Aeva enjoys riding motorcycles up and down the west coast.

• 01:37- Aeva describes a day in the life at CISA
• 02:38 – Details on the use of open source in the public sector
• 04:27 – Why open source needs corporate investment to maintain security
• 06:20 – Aeva shares what their second year at CISA looks like
• 07:58 – Aeva answers Omkhar’s rapid-fire questions
• 09:28 – Advice for people entering the world of security
• 10:16 – Certs are nice to have, but they aren’t everything
• 10:42 – Aeva’s call to action for listeners

• Aeva Black on LinkedIn
• CISA Open Source Security Roadmap
• Get involved with the OpenSSF community

Aeva Black soundbite (00:01)
The burden of securing open source — its ongoing maintenance, its testing, quality assurance, getting signing —  to make open source continue to be deserving of the trust we’ve all placed in it that can’t rest solely on unfunded volunteers. Companies have to participate, shoulder up and help.

Omkhar Arasaratnam (00:19)
Welcome to What’s in the SOSS? I’m your host Omkhar Arasaratnam. I am the general manager of the OpenSSF. And today we have my good friend Aeva Black joining us. Hi Aeva!

Aeva Black (00:32)
Hi, Omkhar, thanks so much for having me on today.

Omkhar Arasaratnam (00:34)
It’s a pleasure. Now, to start things off, why don’t you tell our listeners a little bit about your title and what you do?

Aeva Black (00:43)
Sure. So my official title is Section Chief for Open Source Security. Sounds kind of anime. I like it. I’m also a technical advisor here at CISA, the US Cybersecurity and Infrastructure Security Agency. We’re so enthusiastic about security, put it in our name twice. What I do day to day is just kind of work on solving open source security problems that I have been working on before, but now on this end of the fence.

Omkhar Arasaratnam (01:10)
Well, as I think I’ve told you in the past, my son is a huge anime fan. I literally had to bring a check bag back with me from Tokyo with all the various paraphernalia. But aside from indulging in my excitement about hearing CISA titles associated with anime, can you tell us a little bit more about the day-to-day? I mean, Section Chief sounds like a pretty cool role and you have been involved in the community for a while. What’s your day to day look like Aeva?

Aeva Black (01:37)
Honestly, you know, my previous careers, I often wrote code these days. Day-to-day looks more like answering emails, hopping on meetings, whether they’re internal meetings or interagency meetings, meetings with the open source communityies, but it’s a lot of talking and writing and speaking in public.

Omkhar Arasaratnam
When you announced new role at CISA and that you’d be joining CISA —  I think it was late last summer, I think about August, if memory serves — I was incredibly excited because I’ve seen CISA over the years take a stronger, better, more supportive approach when it comes to open source software. And I was really excited to see somebody like you that has had such a long history of open source support and advocacy join CISA. Can you talk to me about what it looks like on the inside, as everybody’s sitting back with their developer keyboard, clickety clacking, doing git commits all day? Has, has the government evolved into pure open source? How’s that going?

Aeva Black (02:38)
You and our listeners might be surprised to realize just how much both federal and state governments have always used open source. Our, our friend Deb Bryant — she’s been around, used to be at Red Hat, helped out in the open source initiative — used to actually run the open source programs office for the state of Oregon more than 10 years ago. So I think what it looks like today in CISA is pretty much what it has looked like. There’s more clearance, there’s more coverage, should we say, for folks who want to contribute to open source as part of their day job.

We’ve seen that get written down in sort of a guidance way, both in the DoD CIO’s memo a couple years ago, the DHS CIO memo for all DHS agencies includes groups like Coast Guard to use more open source, to contribute to open source, to be good participants in the community. So we’re seeing certainly more support for that. But again, folks across government have always used open sourcing. My first moment of realizing that, probably 2008, I saw some folks from the US Navy give a talk on using MySQL in a cluster running in their ships for battlefield awareness. It was the best database they could find at the time for what they needed. So it’s really nothing new.

Omkhar Arasaratnam (04:00)
Thanks for letting us know. I hadn’t realized that. And it’s very encouraging to hear that not only are we seeing broad adoption of open source within private sector, but also within the public sector. Now, security is a really important mission with a near infinite problem space, especially when it comes to open source security. You’ve been doing this for a while, where should we start? Because it seems like we could start just about anywhere and still have a life’s work ahead of us.

Aeva Black (04:27)
Yeah. Like you said, I’ve been doing this a while since the late 90s, and really as part of my job since the early 2000s. What hasn’t changed: the breadth and the diversity of open source communities is our strength, it is global participation in these communities. And so for today, in light of some of the recent threats against open source, and the pretty big compromises or vulnerabilities in open source that have affected products, we still need to recognize that open source is maintained mostly by volunteers in a participatory community-driven approach.

And yes, of course, companies have a big role to play too. But money isn’t always the solution, but research and common sense have shown that it usually is part of the solution. The burden of securing open source, its ongoing maintenance, its testing, quality assurance, getting signing all of those sorts of activities to make open source continue to be deserving of the trust we’ve all placed in it, that can’t rest solely on unfunded volunteers. Companies have to participate, shoulder up and help. that the transparency in open source, the promise that anyone can modify and study the source code, that transparency has to also be sort of dialed up for the amount of code that’s out there today. There’s so much more code than there used to be in open source, and the ratio of number of humans reviewing code to amount of code published has changed. That increases the risk a bit.

Omkhar Arasaratnam (05:57)
That’s some great advice as to where to start. Now we can slowly see over the horizon, the holiday season fast approaching. I know I’m, you’ve certainly had some great accomplishments. We’ve had some great shared work that we’ve done together. As you look to your second year in your role, what are your priorities? What’s in front of you and what would you like us to focus on?

Aeva Black (06:20)
Yeah, for myself and my team here at CISA, I’ll share that I knew things would be different in the public sector. It’s my first time in a public sector role. Hiring in any role is never as fast as we want it to be. We find a great candidate and the machinery of the organization, private sector or public sector, it’s always slower than we wish. So one of my priorities is continuing to grow my team and to bring more knowledge about open source and from the open source community into roles in the public sector, not just in my team, right, but across the agency and supporting other teams that don’t yet have as much knowledge about open source. Right? So a lot of internal awareness and training in terms of outward work. There’s been a lot that I find really encouraging with groups like FreeBSD’s attestation to the NIST secure software development framework.

A year ago, I had thought that there was no way to make the SSDF work for open source. And I was proven wrong and I’m delighted by that. And now I’m seeing a number of additional foundations and projects working towards a similar goal with their community and their funders working together to raise the bar on how and what secure assurances can be made about the process by which community-stewarded open source is developed. It’s not interesting who’s writing it, but how is it written? How is it tested? How is it assured? I’m really encouraged to see more of that and look forward to partnering with folks, including the OpenSSF towards more of that.

Omkhar Arasaratnam (07:58)
And we look forward to working with you, Aeva. So now is the time in the podcast in which we move to the rapid-fire section. I’m going to prompt you with a couple of different answers. There’s always a possibility that I’ve missed something, and you give me what you’d prefer your answer to be. Now I feel like I have some insight to the first question because we’ve eaten together several times,

Aeva Black (08:23)
That we have!

Omkhar Arasaratnam (8:24)
But spicy versus mild food, Aeva?

Aeva Black (08:27)
It depends if it’s Indian food, spicy, if it’s Mexican medium to mild.

Omkhar Arasaratnam (08:32)
And if it’s sushi, mild.

Aeva Black (08:34)
I mean, jalapenos on sushi can be really good.

Omkhar Arasaratnam (08:37)
Hmm. Yes. Yes, I agree. I take that back. Fair enough. Or a nice spicy salmon roll, perhaps.

Aeva Black (08:45)
True. Yeah.

Omkhar Arasaratnam (08:47)
Alright. Text editor of choice: Vi,  VS code, Emacs?

Aeva Black (08:52)
Easy, easy. Vim. I’ve always used Vim. I have my system set up, put me in Emacs, I usually have to shell, like use a different shell to kill it because I get stuck.

Omkhar Arasaratnam (09:00)
(Laughter) Well, I mean, Emacs is an operating system on its own, to be fair. (Laughter)

Aeva Black (09:04)
Yeah, just not one that I’m comfortable in.

Omkhar Arasaratnam (09:06)
I am also a Vim person, so shared, shared joy there. Tabs are spaces?

Aeva Black (09:13)
Spaces.

Omkhar Arasaratnam (09:14)
I knew it. Awesome. All right, Aeva, we’re wrapping up now. So in closing out, I have two final questions. The first one, what advice do you have for somebody entering our field today?

Aeva Black (09:28)
I wish I had an entire podcast on just this one, but really find your hyper-focus. For a lot of us, we can get stuck on things. Figuring out how to get stuck on the things that were good for my career helped me out early on. And building a T-shaped set of knowledge, so go deep first. Once you’ve gone as far as you want to go, then do it again on a different topic, and that builds breadth over time. Certs are nice to have to get past resume filters, but your network is everything. Maintain relationships across jobs. That’s the second big piece of advice I’d give.

Omkhar Arasaratnam (10:05)
I’ll let you in on a secret. I think the last cert that I got was as a Red Hat certified engineer in 2002. Do you want to share with the audience what last cert you got, if any?

Aeva Black (10:16)
It’s the if any part. Yeah. (Laughter) I considered a couple of certs back in the old MySQL days, early career. I never bothered with the Linux certs or the networking certs because I’ve just logged into a system and show that I knew my stuff.

Omkhar Arasaratnam (10:35)
Absolutely agree. Last question, Aeva. What’s your call to action for our listeners?

Aeva Black (10:42)
Well, for the listeners that are or work at a company, be a responsible consumer of open source. And that means participating in the project so you have insight. It means vetting the code and staging it appropriately locally. If you’re not a large corporation, but a member of a community, then my advice is make sure you’re building your community with stable governance and documented norms so that companies can understand how to work with you and that you behave as a group of a community in a predictable way. Predictable release cycles, predictable vulnerability management, all of those sorts of activities as an open source developer help to grow the project. And leave breadcrumbs, leave gaps for new contributors to fill and make sure you’re passing down the ladder to the next generation of contributors.

Omkhar Arasaratnam (11:38)
Excellent advice as always. Aeva Black, thank you so much for joining us on What’s in the SOSS?

Aeva Black (11:43)
Thanks so much for having me, Omkhar. See you around.

Announcer (11:46)
Thank you for listening to What’s in the SOSS? An OpenSSF podcast. Be sure to subscribe to our series of conversations on Spotify, Apple, Amazon, or wherever you get your podcasts. And to keep up to date on the open source security foundation community, join us online at openssf.org slash get involved. We’ll talk to you next time on What’s in the SOSS?

Andrew Pollock is a Senior Software Engineer at Google, currently working on https://osv.dev. With a background as an Enterprise Security Engineer, he has extensive experience in large-scale Linux Systems Administration and GCP Security. Andrew is passionate about the human factors in security, focusing on scalable solutions, great user experiences and self-service opportunities. He has primarily worked in Linux/Unix environments as a Site Reliability Engineer or Security Engineer, with a strong interest in process improvement and automation.

• 00:52 – Andrew shares his background as a “mid-90s data nerd”
• 02:31 – Managing vulnerabilities in the open source ecosystem
• 03:57 – How to navigate inconsistent metadata
• 06:26 – The challenge of source attribution
• 07:54 – The rapid-fire round
• 09:15 – Andrew’s advice to open source developers
• 10:22 – Andrew’s call to action to developers

• Andrew Pollock on LinkedIn
• Getting to know the Open Source Vulnerability format
• OSV.dev
• Get involved in the OpenSSF community

Andrew Pollock soundbite (00:01)
The beautiful thing about open source is it’s open to all, it’s accessible. You can play around with things, you can break it, you can fix it. It’s fairly approachable. Most of the larger projects have vibrant communities around them that are fairly welcoming and inclusive. 

CRob (00:18)
Hello everybody, I’m CRob. I do security stuff on the internet and I also am a community leader within the OpenSSF, the Open Source Security Foundation. One of the really cool things I get to do as part of the OpenSSF is to host What’s in the SOSS?  where I talk to amazing people from across the open source ecosystem. And today I have my friend Andrew Pollack from Google. G’day, sir.

Andrew Pollock (00:41)
G’day, CRob, how you doing?

CRob (00:43)
I’m doing wonderful. Thank you for asking. Andrew, could you maybe give the audience a little bit of your backstory, your origin? How did you get into open source and what are you doing today?

Andrew Pollock (00:52)
Yeah, so I’m a computer nerd of the mid-90s. I grew up in the MS-DOS era and out of high school, I started studying an information technology degree at uni and I was also working as a mainframe operator at the Brisbane City Council where I also encountered Unix and found Linux in its early existence to be just way cooler than DOS. My Linux distro of choice was Debian and I quite enjoyed using Debian and aspired to become part of that project, and so I joined Debian in the early noughties. I became a Debian developer and started participating in the Debian project proper. And that was sort of my ground level sort of involvement with open source. And I’m still a Debian project member today, but nowhere near as active as I would like to be.

CRob (01:47)
And what are you doing today within the open source ecosystem?

Andrew Pollock (01:50)
Today I work on OSV, which is a couple of different things, right? So the part that intersects with the OpenSSF is the OSV schema. And the part that I more actively work on is osv.dev, which is an open source implementation of a database using the OSV schema to aggregate and enrich those records.

CRob (02:16)
Very nice. Well, I know amongst other things, you’re a big data guy. So let’s talk about vulnerability data and why it’s important. Could you maybe explain some of the key kind of data points that we need to effectively manage vulnerabilities in this wacky ecosystem?

Andrew Pollock (02:31)
Yeah, you know, this is all about the data. So I really enjoy looking at the data in aggregate form and zooming out and looking at challenges with using that in bulk. So OSV provides vulnerability information about software packages at the source code level predominantly and it also provides Linux package of vulnerability information as well. 

And the thing that we need to be able to convey that information accurately is a good package name and good version information because we want to address the use case of vulnerability scanning, first and foremost, and then vulnerability remediation for things that are detected. So step one, to be able to identify what’s vulnerable, you need to know, you need to have a package name that’s meaningful within the ecosystem that you’re sort of wanting to operate on. And then you need to, at a minimum, know what version you need to be beyond to not be impacted by the vulnerability.

CRob (03:43)
I bet you see, let’s call it mildly, some inconsistencies across our amazing ecosystem. Could you maybe speak to some of the challenges that you see within vulnerability metadata today?

Andrew Pollock (03:57)
OSVs got multiple different data sources today, and a lot of them come from language ecosystems, which have some sort of curation within themselves so they’re fairly internally consistent with their packages because they know their own sort of backyard. Where I started looking more broadly was the problem space of C and C++ software, which doesn’t sort of have a centralized repository of vulnerability information. So we had to look at the CVE space for that, and we started pulling CVEs from the NVD to try and figure out which ones of them related to software that was not being covered by OSV-native vulnerability information.

And that’s where — me as a data nerd — just lost my mind because things were very, very inconsistent. So the challenges in that space are around naming. As I said, knowing what the package is to be able to identify it and, to a lesser degree, versioning as well because there’s no consistent versioning scheme when you go from one random open source project to another. And not all open source projects even follow any particular release practice or versioning scheme. So you don’t know anything about a string that is being called a version.

CRob (05:31)
Can you maybe just give us some insight to how you might try to handle some of those problems?

Andrew Pollock (05:35)
I would obviously advocate for projects as part of their maturation process to adopt release management practices that would include things like a formal versioning scheme. SemVer is the one that immediately springs to mind. It’s a standardized format. You can make some pretty clear assumptions about how that one walks and talks. Calva is another one that I know of. So that’s really, really helpful for reasoning about a project.

CRob (06:08)
So can you maybe talk to — you and the others in the OSP project — what you’re attempting to solve within the open source ecosystem? You know, how do you find these authoritative sources? Like how do you know what the right source is to try to make some of these attributions?

Andrew Pollock (06:26)
For the more formalized languages, we don’t have to find the source. The source sort of defines itself, right? So for your Pythons and your Gos and your Rusts and those sorts of things, they have a curated data source and they’ve chosen to supply that data natively in the OSD format and that’s great, right? So where we’re having to do that synthesis ourselves is where things get trickier. And so the challenge that we have is firstly determining what is the authoritative source for a particular piece of software. So that’s sort of 90% of the time that’s attributing it back to a GitHub repo. And then figuring out what the versions are in that repo. 

So we’ve got a real mix where there’s somebody in the business of providing accurate authoritative data, and then one where we’re having to do some inferences about it ourselves. And for the ones where we’re doing the inferences ourselves, we just have to look at the existing vulnerability data that’s available to us, so the CPE records and other data sources like the NVD CPE dictionary and look for hints as to what the what the repository might be. And that’s either metadata in the CPE dictionary or metadata on the CPE itself. So reference URLs typically.

CRob (07:54)
Well, I think it’s amazing work that you all are doing for the community. I think it’s very helpful for researchers and downstream and also community members. So I really appreciate it. Let’s move on to the rapid-fire part of our talk today. First off, spicy or  mild food.

Andrew Pollock (08:12)
I’m on the milder end of things. I value my taste buds and my sense of taste, so I don’t like to destroy them by too spicy stuff.

CRob (08:20)
Very nice. What’s your favorite whiskey?

Andrew Pollock (08:23)
Ooh, well funnily enough I just talked about taste bud destruction, but Laphroaig.

CRob (08:29)
Laphroaig. Very nice. Yummy. So being a developer from your background. Vi or Emacs?

Andrew Pollock (08:37)
Oh, Vi all the way.

CRob (08:39)
Yes! Another fan. And finally, tabs or spaces, sir?

Andrew Pollock (08:44)
Uhh, spaces.

CRob (08:46)
Spaces. All right. Any any rationale why?

Andrew Pollock (08:49)
I think I like my consistency, right? And so if you have tabs, you’re at the whims of what the editor’s tab spacing is, whereas if they’re spaces, they’re spaces. It’s unequivocal.

CRob (09:02)
Very nice. Well, and as we wrap up, what advice would you give somebody that’s kind of coming into this space? They want to become a new open source developer or contribute to a project, or they even want to get into cybersecurity. What advice do you have for those folks?

Andrew Pollock (09:15)
Well, my career background is I’m very self-taught, and I’d like to think that that’s still a feasible career path for people today. And the beautiful thing about open source is it’s open to all, it’s accessible. You can tinker with things as a hobbyist, you can tinker with things in your own time, you can play around with things, you can break it, you can fix it. It’s fairly approachable in a sort of incremental way. And most of the larger projects have vibrant communities around them that are fairly welcoming and inclusive. 

And so stay curious, experiment, learn by breaking things and putting them back together, I think that’s a great way of learning. I’m an experiential learner myself, so I think that’s a great way to learn. I think open source is a fabulous sort of ground level way to get involved.

CRob (10:08)
That’s awesome. I think that’s excellent advice for somebody looking to get into this crazy space we all live in. Closing out, do you have any call to action anything you want to try to inspire our listeners to get into or help out with?

Andrew Pollock (10:22)
Yeah, given that I’m spending a lot of time looking at vulnerabilities in the aggregate, my sort of call to action to developers is to think about the bigger picture when you take a dependency in code that you’re writing because that’s normally how known vulnerabilities come into your code that you’re working on. So the OpenSSF has some really awesome best practices guides around evaluating a dependency or a project before you might want to take it on as a dependency. 

And we’ve got some other teams in the Google open source security team that they’re doing a lot of dependency analysis type insights. So it’s very easy to just take a dependency on board to solve a problem. But if you don’t sort of look at the entire graph of dependencies that are behind that sometimes you’re actually sort of taking on quite a liability. So that would be my sort of piece of advice to sort of developing in the open source space is sometimes it might be better off just re implementing some functionality yourself rather than grabbing something that solves the need.

CRob (11:32)
Yeah, that’s unknown that you don’t know who how made it and what’s inside it.

Andrew Pollock (11:37)
Could be an iceberg.

CRob (11:39)
Exactly. Well, Andrew, I really thank you for your time, everything you do for the community, your time today. Thank you all and have a great day.

Announcer (11:47)
Thank you for listening to What’s in the SOSS? An OpenSSF podcast. Be sure to subscribe to our series of conversations on Spotify, Apple, Amazon or wherever you get your podcasts. And to keep up to date on the Open Source Security Foundation community, join us online at OpenSSF.org/getinvolved. We’ll talk to you next time on What’s in the SOSS?

Bec Rumbul is the Executive Director and CEO of the Rust Foundation, a global non-profit stewarding the Rust language, supporting maintainers, and ensuring that Rust is safe, secure, and sustainable for the future. She holds a PhD in Politics and Governance, and has worked as a consultant and researcher with governments, parliaments and development agencies all over the world, advocating for openness and transparency, and developing tools to improve digital participation.

• 02:57 Bec shares her day-to-day activities with the Rust Foundation
• 04:53 Bec on her sometimes tricky responsibilities during her time at the U.N.
• 06:35 How Bec communicates the importance of memory safety and Rust with stakeholders
• 09:47 Surprises related to organizations that are adopting Rust
• 11:50 Impediments to Rust adoption
• 13:44 Bec answers Omkhar’s rapid-fire questions
• 15:49 Advice Bec would give a non-technical person entering a technical field
• 17:09 Bec’s call to action for listeners

• Bec Rumbul on LinkedIn
• Rust Foundation homepage
• Get involved with OpenSSF

Soundbite (00:01)
Omkhar: VI, VS Code or Emacs — favorite text editor?

Bec: That’s a trap. In the Rust community we would never ,ever deign to tell anyone what their preference should be. We welcome all preferences in the Rust community.

Omkhar: Oh, well answered!

Bec: (laughter)

Omkhar: Alright!

Omkhar Arasaratnam (00:19)
Welcome to What’s in the SOSS? I’m your host Omkhar Arasaratnam, and with me today we have my good friend Rebecca Rumbul. 

Bec Rumbul (00:26)
Thank you very much for having me.

Omkhar Arasaratnam (00:28)
So we’ve known each other for a while, but for our audience, why don’t you introduce yourself, your title, and what is it that you do?

Bec Rumbul (00:36)
Ah, okay, so yeah, as you say, I’m Bec Rumbul, Executive Director and CEO of the Rust Foundation. And what do I do there? Well, I try and keep the wheels on the bus. I try and keep people happy. I try and support all of our wonderful maintainers. And I ask a lot of people for money so that we can keep on doing that.

Omkhar Arasaratnam (00:57)
So I’ve inferred you’re not cutting a lot of code, but as an example, what was your day today? What did you end up doing to give our audience an idea of what the day in the life of a CEO might be?

Bec Rumbul (01:10)
Sure, yep, no, I do not do any coding at all and I don’t recommend anyone ever ask me to try. That’s the surest way to get things to fall apart, I suspect. The great thing about the job is there are very few regular days actually. I get to talk to this wonderful wide spectrum of really just interesting and intelligent individuals. I get to speak to people inside Big Tech who are using Rust, who are thinking of adopting Rust, people that are maintainers and people that have been kind of building the language for years and years and years and are very personally invested in it, finding out from them how we can best support them and how we can make sure that they are able to write wickedly good code securely, for instance. 

I obviously have staff, which is great because they are the people that I’m people that do the real work, not me. So I spend a lot of time coordinating with them, trying to figure out what our priorities as a foundation should be. And I have a wonderful board as well, made up of the community and our corporate sponsors who, you know, help to provide strategic direction, oversight, potentials for funding, that kind of thing.

As I said, I’m always on the lookout to fundraise so that we can keep providing this wonderful language to everyone that wants to use it.

Omkhar Arasaratnam (02:28)
You’re also our associate member rep for the OpenSSF. So thank you for those contributions as well. Now, in talking before this and prepping for the podcast, we were discussing your history leading up to your current position. You’ve been at the Rust foundation for just over, what is it, like two-and-a-half years now? What led you to this path? It certainly wasn’t a long history of computer science. It’s quite an interesting past though.

Bec Rumbul (02:57)
I kind of Forrest Gumped my way into this. I’ve never been one of those people with a very straight, very focused career ladder. Before this I did a lot of consulting for the UN, a lot of digital democracy work, a lot of research, looking at how to kind of empower citizens of countries all over the world to hold their politicians to account, to make better laws, to enable parliaments themselves to support that with the politicians that sit inside them. 

So I did years of that and I really enjoyed obviously doing the digital aspect and finding new open source tools to help people with that but also working on the democracy side, you know, the kind of consensus-driven, decision-making side, figuring out how that can be done well.

And that was one of the aspects of the job at Rust that really kind of called to me. Yes, okay, technically I’m in the CEO’s seat, but actually, I have very little power. The power is really in the hands of the community, it’s in the hands of the board. There’s an awful lot of people that are involved in helping us to make the best possible decision, not just the one that’s most kind of expedient for me at the time because I’m the boss.

With this amazing new language that was just emerging when I came into the role, and I had this opportunity to nurture this thing that I don’t even think many people realized how important it was going to grow to be. So yeah, playing mum to the Rust programming language as well has been fascinating and a real privilege.

Omkhar Arasaratnam (04:35)
What a very interesting past and what impactful work you’ve done in the past. I’d love to delve into that a little more. Is there one particular aspect of the advocacy work you’ve done previously that you’re really proud of, maybe a little embarrassed by? Let’s give the audience something that some insight to Bec’s world prior to Rust.

Bec Rumbul (04:53)
I did some very random stuff and I did some stuff that was required politically but maybe wasn’t really embedded in the hearts and minds of people that I was working with. So I’m not going to name specific countries, but I have worked with some parliaments in some very authoritarian regimes, shall we say? So, yeah, I’m not going to name countries because I don’t want to upset anyone. But yeah, there were some times where I definitely had some, you know, towing the line, not really wanting to give people the kind of power that genuinely enables people to make democratic decisions. 

But that said, I did, you know, I did some amazing work with parliaments in Kenya, in South Africa, in Ghana where they were really invested in digitizing and really invested in trying to empower the local citizenry to help make laws better. 

Omkhar Arasaratnam (05:46)
You know, it’s an election year. I’m not going into what that means for us over here in the States. I’m going to leave that aside. Thank you for sharing that though. And you have done some amazing work and it’s really interesting to see how that work has now led you to where you are now. So speaking of where you are now, Rust. I hear it’s going to fix all of our memory safety problems, right?

Bec Rumbul (06:07)
Yeah, of course. We can take the next question now. (Laughs)

Omkhar Arasaratnam (06:09)
Done. Next. I mean, it’s interesting. I think what would be interesting for us to learn is from your perspective, as someone who admittedly isn’t a technologist and somebody that is focused on improving things, making the world a better place, how do you frame the rationale for using Rust and how do you touch on things like memory safety when it comes to the discussions you’re having with your stakeholders?

Bec Rumbul (06:35)
It’s so important to be able to pitch what Rust can do and the kind of memory safety feature at the right level so that people can genuinely understand. It’s really easy to bamboozle people really quickly when you start getting really techie. And whilst I don’t write code, obviously I’ve worked in the area long enough so that I understand how these things work. But I’m hyper-conscious that certainly when you’re dealing with big world people, not just techie, techie people, these things can get very very complex and, you know, you can see people’s eyes glaze over very quickly. 

So the way I try and explain memory safety is to kind of tell people about you know some of the big hacks that they’ve heard of and how actually, you know using, different kinds of code that operate in different ways might have prevented some of those things. Not every single one, every big vulnerability is slightly different. But memory safety is this one feature that means that actually it’s really, really difficult to just have like really low-level errors or really sort of small mistakes that are just human errors, they’re not computer errors most of the time, they’re human. So any kind of safety net like a memory-safe language means that that’s just not possible. 

Obviously, there are many other potential vulnerabilities out there that memory safety won’t fix. But it was really important to have organizations like Google for instance releasing their research on using Rust where they’ve come out publicly and said actually using breast means that 70% of their vulnerabilities are gone because Rust is memory safe, it’s just automatically clearing those. So in terms of an economic view, not just a security view, that’s a hell of a lot of people that are doing forward-facing code now, not trying to fix something and digging through code, trying to fix something that already exists. 

So that’s kind of the way I tend to approach it. It’s not perfect. It’s still an imperfect pitch, but I think because governments are now getting involved in security is suddenly after, you know, so very many years in the wilderness being seen as a bolt-on now it’s being given the attention it needs more people are actually, I think getting up to date on just the theory of memory safety if not the actual ability to code it.

Omkhar Arasaratnam (09:06)
The the angle that you took in terms of expressing the economic benefit as well as that safety net in my mind, I contrast this. We had Christoph Kern from Google on the podcast a few episodes ago, and you may imagine that Christoph had a very computer science point of view. So I love the fact that both of you have brought these, I’ll say, from  different perspectives, on the same topic. And it’s very interesting to hear that. Switching gears, where are you seeing interest in adoption of Rust that was surprising? Like who was trying to adopt Rust now that you were just like, huh, I didn’t think about that?

Bec Rumbul (09:47)
I think I’m most surprised and encouraged by how quickly the safety-critical industry has noticed and started to prepare the ground for Rust adoption. Because obviously, you know, safety-critical, it is the most important sort of sector for having really secure, really high quality, high performing software. The fact that that sector has been kind of the first off the blocks in looking at Rust and figuring out how it can be used, I think, was really interesting. 

Obviously, again, you know things like speed and performance of Rust appeals to that sector as well, but these are serious people building serious stuff, right? So it’s encouraging that that is a sector that’s looking really hard at this. That said, I love seeing Rust popping up in different places. Obviously, it’s kind of great in terms of Wasm. Rust embedded is growing and growing at the moment. But I love it when someone sort of pops up and says, my company is using this but I’m not allowed to say anything about it publicly. Damn, please talk about it publicly because that gives other people confidence as well. There’s loads of people I’d say that are, you know, loads of CTOs at the moment that are kind of rust curious.

And yeah, we’re having kind of quiet conversations, but they don’t, you know, they just want to dip their toe in at the moment and they’re looking around for other organizations to kind of see what they’re doing. But very few are willing to kind of stick their head up and say, actually, no, we’ve done this and this was good and this was bad and this is what you should think about. 

Omkhar Arasaratnam (11:32)
What do you see as the next major challenges for Rust? I mean, it seems like everybody’s all in and even those that are just Rust curious for right now are certainly dipping a toe in the water. Other than, you know, the who’s going to go first mentality? What are the other impediments we have to adopting Rust today?

Bec Rumbul (11:50)
One, Rust notoriously has a steep learning curve. I actually think that that is being flattened. Where we were two, two-and-a-half years ago in terms of teaching Rust is very different to where we are now. And there’s lots and lots of good quality training stuff out there. And large tech organizations are better set up now to migrate whole teams across to Rust. So I think there is still a bit of a hangover from that, but I don’t think it’s as much of a problem as it was before. 

I think one of, and what I’ve seen in some conversations, is that because Rust is so new and young, an awful lot of people in positions of responsibility don’t know it. They learned C++ when they were doing their comp sci degree or in the early days or Python, and even though we’re getting an awful lot of people at grassroots level, I do think there’s a reticence among people who are quite a bit higher up and who have to make these huge financial decisions about whether they’re going to invest that heavily in this because, obviously, they just don’t have that kind of personal firsthand experience of it. So I think there’s a little bit of that. 

The tech mini-slowdown last year didn’t help anyone, I don’t think. Certainly, if you’re kind of looking at doing iIf the whole sector is feeling a bit kind of sluggish, it’s probably not the time to invest. That said, I do think the momentum is there and I think we’ll be having a very different conversation in a couple of years’ time.

Omkhar Arasaratnam (13:33)
That makes a lot of sense. Alright, Ms. Rumbul, we are going to move into the rapid-fire round. Are you ready? 

Bec Rumbul (13:42)
As I’ll ever be.

Omkhar Arasaratnam (13:44)
Some of these, some of these questions may lean a bit technical. I will give you a choice of a set of answers. There’s always the, no, Omkhar, you didn’t get that right. But I think moreover, some of these questions lean very tech heavy. I would like your point of view as to how your community reasons over some of these questions, should you be privy to when they come up. 

Bec Rumbul (14:07)
Okay.

Omkhar Arasaratnam (14:08)
Now, the first one is not techy. Spicy or mild food? And I think I know the answer to this.

Bec Rumbul (14:13)
Spicy food. You only live once.

Omkhar Arasaratnam (14:15)
Yes, that’s why we’re friends. That’s why we’re friends. Now here, here come the techie ones. Vi, VS code or Emacs favorite text editor?

Bec Rumbul (14:27)
That’s a trap.

Omkhar Arasaratnam (14:29)
Hahaha!

Bec Rumbul (14:31)
The answer is, in the Rust community, we would never ever deign to tell anyone what their preference should be. We welcome all preferences in the Rust community.

Omkhar Arasaratnam (14:41)
Oh, well answered! All right. Let’s see how adeptly you dodge the next question. Tabs or spaces?

Bec Rumbul (14:50)
Oh, I don’t care. And I know that’s not the right answer. I’m supposed to choose a hill to die on here, but life’s too short. (Laughs)

Omkhar Arasaratnam (15:01)
Life is too short, let’s drink wine?

Bec Rumbul (15:03)
Life’s too short, lets drink wine. And you know, the previous answer also applies. We would never, ever suggest a preference to people. It’s whatever they’re comfortable with. 

Omkhar Arasaratnam (15:11)
You know, I recently took some time off personal vacation and I actually went through the Rustlings course. So I will let you know my thoughts once I complete it. Thus far, this old C programmer had to learn some new tricks, but it was very insightful. Closing things out. What advice do you have for somebody entering our field today? And normally the person I’m asking this, they’re normally somebody that spent multiple decades in security or multiple decades in software engineering. I’d like you to answer this question from the perspective of somebody that’s thinking about entering the field as a leader in a code-hosting nonprofit. 

Bec Rumbul (15:49)
I think there is room for everyone in open source. That’s the most amazing thing about this kind of community. And there are a lot of different skills that are really needed here. I think I might have been hired because the board was interested in bringing in some skills that don’t, there weren’t too many of those skills in the community because people who have mad coding skills don’t spend all of their time looking at spreadsheets and writing board agendas, right? And chasing people and trying to charm them out with their money. 

So I think, you know, my advice is it’s fine, it can be intimidating coming in and speaking with all these people that are so very much smarter than you. But you have things that they don’t. And the whole of open source is desperately in need of a whole kind of range of skills, from project management to administration bureaucracy, to event management, to moderation and community management, all of these things. It’s not just about the code. If it was just about the code, open source wouldn’t work. It’s all about creating that code together in a community of people that are just kind of pulling in the right direction with the same values.

Omkhar Arasaratnam (17:00)
What great advice. Last question for you, Bec. What’s your call to action for our listeners? What would you have them do after listening to this podcast?

Bec Rumbul (17:09)
It’s kind of building on my last point actually, you know, we’re always in need of people to help us do great stuff and to give us opinions that come from different places to where we are. My kind of call to action is get involved — even if you’re not a security professional or, you know, someone that’s going to bang out lines and lines and lines of code for fun of an evening — if you’re really interested in helping a community grow and developing amazing software and securing our shared online world, your skills helping to manage the community or do administrative things or management things are just invaluable. So turn up, join a community, have fun.

Omkhar Arasaratnam (17:48)
Bec Rubmul, thank you so much for joining us on What’s in the SOSS? And all the best in leading the Rust community to newer and greater heights. Thank you for all you do.

Bec Rumbul (17:58)
Thank you.

Announcer (17:59)
Thank you for listening to What’s in the SOSS? An OpenSSF podcast. Be sure to subscribe to our series of conversations on Spotify, Apple, Amazon or wherever you get your podcasts. And to keep up to date on the Open Source Security Foundation community, join us online at openssf.org/getinvolved. We’ll talk to you next time on What’s in the SOSS?

Brian Fox is Co-founder and Chief Technology Officer at Sonatype, bringing over 28 years of hands-on experience driving software development for organizations of all sizes, from startups to large enterprises.

A recognized figure in the Apache Maven ecosystem and a longstanding member of the Apache Software Foundation, Brian has played a crucial role in creating popular plugins like the maven-dependency-plugin and maven-enforcer-plugin. His leadership includes overseeing Maven Central, the world’s largest repository of open-source Java components, which recently surpassed a trillion downloads annually.

As a Governing Board member for the Open Source Security Foundation, Brian actively contributes to advancing cybersecurity. Working with other industry leaders, he helped create The Open Source Consumption Manifesto, urging organizations to elevate their awareness of the Open Source Software (OSS) components they use.

• 00:57 Brian shares his background
• 03:56 The confusing trend of downloading assets on Maven Central with known vulnerabilities
• 08:16 How this trend continues in other repos
• 11:08 Brian and CRob discuss Log4Shell
• 16:54 Brian answers CRob’s rapid-fire questions
• 18:46 Brian’s advice for up-and-coming security professionals
• 19:50 Brian’s call to action

• Brian Fox on LinkedIn
• Sonatype’s 2023 State of the Software Supply Chain Report
• Get involved with OpenSSF

Brian Fox soundbite (00:01)
The customer is not gonna accept as an excuse, like, why did you get hacked? Well, my direct dependencies were fine, it’s those indirect ones that were bad. Like, nobody cares, right? Like if you buy a pie from a bakery and you get sick, it’s not acceptable for them to go, well it was my sugar provider that did a bad job, here. 

CRob (00:18)
Hello everybody, welcome to What’s in the SOSS? I’m CRob. I do security stuff on the internet. I’m also a community leader within the Open Source Security Foundation, and I’m one of the hosts of this amazing podcast. Today I’ve got my friend Brian Fox. He’s the CTO and co-founder of Sonatype and he is an amazing open source contributor and he’s a great collaborator with us and a lot of industry efforts. So I want to say, Brian, welcome.

Brian Fox (00:49)
Yeah, thanks for having me.

CRob (00:50)
Could you maybe share a little bit about your origin story? How you got into open source and kind of the stuff you’re into today?

Brian Fox (00:57)
Oh, sure, my origin story. That’s a long one, but short version is I dabbled in open source before it was I think called open source in college actually, some PERL CGI open source stuff way, way back in the day. But I would say my first real introduction into like durable open source, let’s say — I had I dabbled with a JSP library at SourceForge. And I don’t really count that, but you can see they’re all connected in a straight line. I got involved with Apache Maven super early

in the 2.0 alpha days. I wrote some plugins. I was basically doing development management during the day, had some folks working for me, and we were trying to transform our builds from Ant to Maven and move from CVS, I guess it was, to Subversion. 

Yeah, I’m dating myself a little bit. And we were running into some issues, and so because I didn’t have much opportunity to code during the day, I was coding at night. So I would go home and start working on fixing bugs and writing plugins that the folks working for me during the day the next day would then use to move forward. And so I wrote some pretty popular plugins, the Dependency plugin and the Enforcer plugin that later came from, the Code Haas Project — Code Haas was sort of an adjacent open SourceForge, if you will, that eventually got pulled into Apache Maven proper. And I came along with it as a committer, was later a project management committee member, and for a while after we started Sonotype was even the chair of that project. 

And, and for, for historical reasons that, you know, we could have a whole podcast on, you know, the Maven central repository has always been a separate project from the Apache Maven software base. And basically, a handful of us, and then later Sonatype, have basically been the stewards of that all along. So, I know where all the bodies are buried and have lots of war stories from, from open source Java and Maven Central in general, from, for the last 20-plus years. I suppose.

CRob (03:03)
So I’m a little bit of a data nerd and I know you share that kind of passion for data and numbers. Your organization, Sonatype, puts out an annual report, well many of annual reports, but the one I’m most interested in is the State of Open Source Supply Chain Report. That’s one, I’ve referenced that many, many years. It’s pretty amazing. And I want to kind of talk about some of your findings from the most recent one. Maybe we can kind of, that’ll get us into this topic. But you know, when I was looking at the 2023 report, I noticed that you had many statements, one of which was you noted that 96% of all vulnerable downloads from Maven Central had known fixes available. What’s up with that? Why are folks still downloading known vulnerable packages when there’s a fix available?

Brian Fox (03:56)
Yeah, yeah, this has been my soapbox, as you know, for a couple of years. 2022 was the first year we published that stat. Sadly, it was unchanged in 2023. It did not get better. What does this stat mean exactly? At a point in time, when a thing that is in Maven Central, and I think off the top of my head, I’m not certain. I think around 12% of the things in central have a known vulnerability. But they’re skewed, obviously, towards more popular versions of it. So when those things are downloaded, looking at the point in time that it was downloaded, was there already a fix for that vulnerability available? So in other words, it’s not a case of this vulnerability is out there and people are using it because it’s not fixed.

After Log4Shell, which I’m sure we’ll get into a bit more, you know, we saw a lot of people talking about like open source should do a better job of fixing their stuff. And it’s like, well, when you sit where I am and you look at that and you go, well, wait a minute, when these things are being consumed from the repository, 96% of the time, the fix already is available. So what does that tell you? It tells you that the consuming organizations are not choosing to update for the fix. It shouldn’t be a 0% stat, right? There’s always going to be vulnerabilities that in certain contexts just don’t apply, and so you can continue to use the thing, or you have other mitigating controls.

But 96% is crazy. And the fact that it doesn’t change. And so why does that happen? There’s lots of reasons. And I’ve been recently in blog posts and articles exploring more the psychological side of things. I think there’s, humans are wired to procrastinate and hope that the future will be better. And I think there’s an element of that, that people look at it and go, well, you know, we’ve lived with these things so long, you know, I’m afraid to do an update. Dr. Wheeler and I have kind of toyed with the idea that API changes introduce a bunch of these problems, or at least the fear and the history of API breakages have caused people to be afraid to make an update. 

And so there’s lots of little reasons that cause organizations to just sit there and not update and therefore continue to consume these known vulnerable things. And so I’ll ask the obvious question because when I point this stat out, everybody turns around and asks the same question, which is, Brian, why do you make vulnerable versions available for download in Maven Central at all? Right? And so, you know, part of that is precedence in history that Maven Central is known for its stability, that we don’t allow authors to just up and decide to unpublish something. For those of you that remember left-pad and NPM years ago, it was a little bit of a tit-for-tat, and a maintainer removed a package and it broke like the internet, right? And so for reasons like that, we don’t allow things to just up and disappear. 

Most of the time, like I said, not everything is universally applicable. So I use an analogy. For some people, peanut butter can kill them. They are deathly allergic to peanut butter. I can go into the store right now, I buy peanuts, buy peanut butter, buy things with peanut butter in them. Why? Because I happen to, fortunately, not be allergic to peanut butter. I do not have that vulnerability, or at least that vulnerability does not apply to me. So like that analogy, why should we take every single component down just because it has a vulnerability and make it impossible to reproduce old things, break things that don’t need to be broken? 

And I draw a distinct line between a vulnerability and a known malicious package. A known malicious package is like saying we’re going to continue to sell salmonella-tainted  food because some people luckily don’t get affected by it. Like, no, that’s very, very different, right? So salmonella components, yeah, they’re coming out immediately. Peanut butter, they should be labeled, they should be understood so that people that have vulnerabilities can know about it, right? And so that’s why we don’t just disappear components merely because there’s a vulnerability. 

CRob (08:04)
You have a lot of observability into Maven Central. Do you feel that you know the pattern you’re seeing does that probably exist in other repositories as well? Do you think it’s similar? 

Brian Fox (08:16)
I know it does. Yeah, I know it does. How do I wanna phrase this? There are similar problems. They don’t have the same root in the origin of it. So, for example, Maven has a strong namespace in the group ID, right? So typically that would be the reverse DNS of your company, just like Java package standards. That’s where we, we didn’t invent it. We just followed the Java standard. And when, when somebody comes to Maven Central, we validate that they control that domain and that namespace, or alternatively they can use their project URL at GitHub as an example, right? And, and we do the validation and that’s so somebody can’t show up to the repository and pretend to be Eclipse and start publicizing things. You can’t pretend to be Apache and start publishing things. 

That’s an important piece of this. The other piece of it is that Maven prefers stability over auto-updates. So it prefers staying on a version number. So it’s sort of an anti-pattern in Maven land to say, just get me the latest version of this thing all the time. So that choice, that design choice, has unintentionally created the issue that I talked about, that people aren’t upgrading, right? That’s an unintended side effect of that. But now let’s look at what’s happening on ecosystems like NPM, Python, Ruby, where the tool has made the choice that we are always gonna check and fetch the latest version unless I’m told not to, right? 

That’s why you have the concept of a lock file to lock it down because if you don’t lock it, it’s going to fetch it. And also, unfortunately, those ecosystems also don’t have a strong namespace enforcement. So what happened starting around 2017, the attackers figured this out and they start publishing components into the repository that have confusingly similar names, basically a typosquat of a name. And since there’s no namespace enforcement, it’s very hard for consumers to tell the difference between am I getting the legit component or not?

And you couple that with the fact that the tool is going to auto-update these versions all the time, which means if somebody were to compromise an actual legitimate JavaScript project and put a fake component up there — and this has happened many times — the consumers will fetch it almost immediately. So you’ve created this situation where it’s easy for the attackers to put stuff in the repo and everybody will consume it almost immediately so you have a ready willing audience, which means you’re seeing on those ecosystems this massive stat we talk about, 350,000 known malicious component attacks in the last couple of years and it’s been doubling year over year. 

That’s only happening in those ecosystems. So we don’t have this problem in Maven because there’s not the auto-update and there is the namespace. But we’re having the massive attacks on these other ones because they do, right? So it’s sort of like these design decisions lead to different unintended consequences, I guess I would say.

CRob (11:08)
So let’s let’s talk about a specific example. Our friend Log4Shell that ruined many, many IT folks holidays in December a year or two back. I have an old stat: there were almost 250,000 known downloads of the malicious packages of Log4j which was about 29% of all the downloads at that time.  How is it that people are still grabbing that known malicious package? Why is that with such a widely known vulnerability?

Brian Fox (11:45)
(Laughs)  Yeah, I wish I knew, honestly. So I’m looking at the latest stats and anybody listening to the podcast, you can see them yourself. We have a resource center that’s been up since 2021. It’s at sonatype.com/log4j. So you can see those stats yourself. As we sit here right now, we’re looking at 405 million downloads since it was announced and worse the last seven-day average is 35% of those are of those non-vulnerable versions. And so it’s actually gone backwards in the last year. We were down at one point to, I think it was in October when we published the report, it was in the twenties and it took multiple years to get there, which was really disappointing, but it slid backwards for reasons that I have not been able to explain. 

And so, I also use this one sort of as the perfect bellwether for the reasons that you outlined, right? That it’s fairly easy to exploit. It’s fairly publicized. Everybody knows about it. And it was sort of one of these situations very early on that every vendor was getting asked by their customers, did you remediate this? And they would have to explain why. 

And so it created this situation that it was easier to just upgrade than it was to explain why you weren’t upgrading. So like I said before, I wouldn’t expect the uptake of every vulnerability to be 100% updates. In Log4Shell,  I think that is closer to true, and so we should expect to see the uptake here to be you know closer to, you know, 80, 90, 95% I would expect. I’ve done some research recently to look into the origin of this because you know people will often try to justify like, it’s just a few people with broken builds that are running all the time, right? Because it does seem incredulous. 

I’ve looked into the vulnerable downloads within the last month and they’re coming from millions of IPs worldwide. You can’t pin it down to, this is like a provisioning script that’s running on EC2 or something like this, right? Or some serverless configuration that might be happening in a Lambda, you know, weird things like that. No, it’s all over the map. 

And when I’ve looked at like the cohort of dependencies when I’ve taken the IP numbers that are fetching these vulnerable versions and I look at the cohort of dependencies to see like, is it one application everybody’s built building or something weird. Is it one place that’s causing this as a transit of dependency? I’ve not been able to put my finger on it. It is all over the map, which just tells me it’s kind of what I suspected in the beginning. It’s just average usage and folks for whatever reason just freaking haven’t upgraded. (Laughs) 

So it’s really disappointing because if it were any of the other things I mentioned, we could go attack that at its source and have a big impact. Instead, it’s messaging like this to get people to pay attention. And this is, it kind of intersects the SBOM area. If you can’t produce an SBOM, kind of implies you don’t know what’s inside your software. If you don’t know what’s inside your software, why should we expect that you’re doing a good job of updating it?

CRob (15:493)
So thinking about solutioning this a bit, do you have any advice for what developers or downstream consumers can do as they’re ingesting open source components to avoid this continual downloading of known vulnerable packages?

Brian Fox (15:15)
Yeah, I mean, tooling has existed for a long time to help you understand your entire transitive hull of your application. You know, Sonatype’s been providing this for, what year is it now? 14 years. (Laughs)So this is not a new thing. And you know, in the world, the regulated world, we’re seeing legislation all over the place from the US government, from European Union, PCI standards, everybody’s pushing towards, you need to be able to produce an SBOM. And this is part of the reason why, that it’s no longer acceptable to just blindly ignore what’s inside your software. And we saw for years organizations were only focused on the direct dependencies, the things their developers use, and ignored the transitive dependencies. 

But there’s more transitive dependencies than direct ones. So your odds are of pulling in something bad are even higher further down. It’s quite literally the iceberg. 70% of it is coming from these open source transitive dependencies. You need to have visibility in that because your customer is not going to accept as an excuse, like, why did you get hacked? Well, my direct dependencies were fine. It’s those indirect ones that were bad. Nobody cares. If you buy a pie from a bakery and you get sick, it’s not acceptable for them to go, well, it was my sugar provider that did a bad job here. Like I’m sorry like you know you still created this thing and sold it to me you’re still responsible. 

And I think that ultimately that’s what it’s gonna take that some form of liability reform to the point where organizations are actually responsible for the outcomes is needed to change this behavior. 

CRob (16:54)
Well, I think this is gonna be the first of an intermediary step in an ongoing conversation. And hopefully we can help raise th at awareness and get folks focused in on this unique problem. Let’s move on to the rapid fire section of the interview here. Got a couple questions and we’re gonna see what your thoughts are. First question, Brian, spicy or mild food?

Brian Fox (17:20)
Depends on the day, but let’s go with spicy today.

CRob (17:23)
(Sound effect “Oh, that’s spicy) Nice. Being a developerologist as your background, what’s your preference? Vi or Emacs?

Brian Fox (17:34)
Oh, Vi.

CRob (17:35)
Huzzah! What’s your favorite type of whiskey?

Brian Fox (17:41)
Ooh, ooh. I don’t know. I like variety. How’s that? I like to explore new ones. I have one that they make here — actually, it’s in Vermont, just over the border — they make it from maple syrup. And that one is really unique. So I’m gonna go with that. How’s that?

CRob (17:58)
All right, that sounds like a delight. And our final,  most controversial question, tabs or spaces?

Brian Fox (18:06)
(Sighs)  Spaces.

CRob (18:09)
Spaces, very good. There are no wrong answers, but that’s interesting how that causes a lot of controversy.

Brian Fox (18:17)
Yeah, you’re not going to ask about, you know, Alman versus K&R? I mean, that that always is related to tabs and spaces.

CRob (18:24)
(Laughter) Maybe in season two.

Brian Fox (18:28)
Yeah!

CRob (18:30)
Well, thanks for playing along. (Sound effect: That’s so sweet!) As we close out, what advice do you have for  someone that’s thinking about entering this field, whether it’s being an open source developer or getting into the field of cybersecurity? What advice do you have for newcomers?

Brian Fox (18:46)
I think my advice is always the same. Pick something that you’re interested in, whether it’s robotics or build software, whatever, and go find an open source project and try to get involved. It’s a great way to learn a whole bunch of skills. I mean, to be able to learn how to write better code, but also how to navigate, you know, the inevitable politics when you have more than, you know, two humans involved. The communication skills, the politics, the collaboration. 

I think you can learn a lot from open source, especially if you’re, you know, let’s say in high school or college and you don’t necessarily have the ability to get a job here. You can go get some real-world experience through open source projects and there’s open source for basically everything. So you know if you’re into rockets or planes whatever go find something it’s out there. It’s even easier today than it was, you know, 20 years ago, right? And and that would be my advice. 

CRob (19:44)
And finally, do you have a call to action for our listeners to help kind of inspire them?

Brian Fox (19:50)
If your organization can’t immediately assess — if I were to tell you about a new vulnerability right now that you never heard of, if you can’t immediately assess — if you’re even using that component anywhere in your organization or further, you can’t quickly produce the list of applications that are using that, then you’re basically powerless to respond to this next problem, right? That situation I just described is what most of the world went through in, what, December 9th, 2021, when Log4Shell dropped on them. 

We have studies, it was in the 2022 report, and I think we probably repeated it in 23, that showed organizations that had tooling in place were mediated their portfolio of thousands of applications within days versus other organizations that spent six months doing triage. So if you can’t immediately have a system that can tell you, are you using this component anywhere, any version of it, and you can’t get to the point of saying, we are using this precise version in these applications, then you need to solve that problem immediately. If only to prepare for the inevitable next thing than to prepare for the legislations that are going to require you to have the SBOMs.

And I would add, if you have solved that, then you need to be looking at these intentionally malicious problems because those require different solutions. They’re not going to show up in SBOMS. They’re trickier because they’re attacking the developer as soon as they download them. So if you think you have your inventory under control, you need to ask yourselves, how would I know if a developer downloaded one of these typosquatted components that may have dropped a backdoor or might have had custom code that simply exfiltrated data directly in the open source? Your traditional scanners are not going to pick this up. So I guess that’s two things depending on where you sit on that spectrum.

CRob (21:39)
Well, excellent. I appreciate you have some amazing advice and those are some interesting things to think about and act on. As always, I really appreciate your partnership and your ongoing contributions to help make the whole ecosystem better, Brian. You’re an amazing partner.

Brian Fox (21:53)
Yeah. Likewise. Thanks, CRob, for inviting me.

CRob (21:55)
Cheers!

Announcer (21:56)
Thank you for listening to What’s in the SOSS? An OpenSSF podcast. Be sure to subscribe to our series of conversations on Spotify, Apple, Amazon or wherever you get your podcasts. And to keep up to date on the Open Source Security Foundation community, join us online at OpenSSF.org/get involved. We’ll talk to you next time on What’s in the SOSS?

Arun Gupta is vice president and general manager of Open Ecosystem Initiatives at Intel Corporation and the OpenSSF Governing Board Chair. Arun has been an open source strategist, advocate, and practitioner for nearly two decades. He has taken companies such as Apple, Amazon, and Sun Microsystems through systemic changes to embrace open source principles, contribute, and collaborate effectively.

On July 9-10, the OpenSSF will attend the 2024 OSPOs for Good symposium hosted by the UN. Arun and What’s in the SOSS? co-host Omkhar Arasaratnam will lead a session called “Engaging the Open Source Community.”

Following the symposium on July 11, attendees are invited to come to a secondary event, What’s Next for Open Source? It will feature a collection of curated workshops to discover how to build and gather the skills you need to move forward with open source. Omkhar is coordinating the security track and presenting opening remarks. Arun will offer closing remarks. 

• 02:13 – Arun’s general outlook on security and life
• 03:39 – Arun shares his personal background and illustrious career history
• 09:04 – Comparing the OpenSSF and the Cloud Native Computing Foundation (CNCF)
• 13:30 – Arun details his work with the United Nations
• 16:42 – Areas that a lot of security professionals are getting wrong
• 18:20 – Arun answers Omkhar’s rapid-fire questions
• 19:08 – Advice Arun would give to aspiring security professionals
• 20:40 – Arun’s call to action for listeners

• OSPOs for Good 2024
• What’s Next for Open Source event
• Arun Gupta’s LinkedIn profile
• CNCF homepage
• United Nations Sustainable Development Goals
• Get involved with OpenSSF

Announcer (00:01)
A quick programming note: On July 9th and 10th, the OpenSSF will attend the 2024 OSPOs for Good symposium hosted by the UN. What’s in the SOSS? co-host Omkhar Arasaratnam and today’s guest Arun Gupta of Intel will lead a session called “Engaging the Open Source Community.”

Following the symposium on July 11th, attendees are invited to attend a secondary event, What’s Next for Open Source? It will feature a collection of curated workshops to discover how to build and gather the skills you need to move forward with Open Source. Omkhar is coordinating the security track and presenting opening remarks. Arun will offer closing remarks. 

For more information, check out the links in this podcast’s episode description.

Arun Gupta soundbite (00:45)
Sometimes security folks focus too much on the technology. Take a step back. Have that empathy for the customer. Does the customer understand that language? Are you talking in their language? Is the intent of your dialog landing the impact on the customer who’s listening to that discussion? 

Omkhar Arasaratnam (01:03)
Welcome to What’s in the SOSS? I am your host and the general manager of the OpenSSF Omkhar Arsaratnam. And with us today, we have Arun Gupta. Arun, tell us what you do.

Arun Gupta (01:18)
Omkhar, thank you for having me here. I’m really happy and excited to be here. I am the Vice President and General Manager for Open Ecosystem Team at Intel. And that’s my day job. As part of that, I coordinate open source strategy across the entire company, whether it’s software, hardware, all through the stack, why we contribute, how we contribute, how do we bring alignment across different business units.

So that’s quite an exciting venture actually. In addition, because of Intel’s legacy, it allows me to do a lot of chop wood, carry water kind of work in the community. So I’m really fortunate and very grateful to be the chair of the OpenSSF governing board, in addition to the chair of the Cloud Native Computing Foundation governing board as well.

Omkhar Arasaratnam (02:07)
Holy cow, where do you find time to sleep? And you run as well, you’re a runner, is that right?

Arun Gupta (02:13)
I like to run. I think the running is what gives me… I was listening to a podcast this morning and it talks about self-compassion. And I think that’s something that I’m really big on. So it’s very important to be compassionate to yourself. Make sure you’re taking care of yourself so that you can do all of these other things. 

I must say I’m really blessed and fortunate in that sense that people like the way I think, like the way I operate, like the way I treat them, going back to Maya Angelou’s quote, really. So, and I think that’s what has helped me get into the leadership position. There are a lot of wonderful people in this world, but you know, making sure you are listening to people, engaging with people, taking care of them, being empathetic to them. Those are some of the traits that you really need to be in this leadership position. But it really, gives me a satisfactory feeling at the end of the day, being the governing chair of the two of the largest Linux Foundation foundations essentially, and drive them forward.

Omkhar Arasaratnam (03:15)
Your contributions to the community have been numerous and this certainly isn’t your first first day in open source. And I think your numerous contributions to the community is part of the reason why you’ve been elected to such prestigious positions within these two foundations. Can you talk to us a little bit about your history in open source? How long have you been doing it? How’d you get your start?

Arun Gupta (03:39)
Yeah, I grew up in India. I moved to the United States. I moved to the United States back in ‘98. And I was very fortunate to literally go to sun.com/jobs and apply for a job. And I was one of the original JDK team members. And gosh, over two decades ago, we started changing the culture at Sun Microsystems at that point of time. It was a very close source company, Solaris, Netscape application server, all that. 

And then that’s when we started changing the culture at Sun. How do we take this closed source application server and make it an open source application server? And we realized it’s not just about putting the source code over the firewall, but it’s really bringing that people process, culture change, essentially, all of that kind of coming together, essentially. And that sort of…so over 20 years ago is when that bug got into me and I found it very exciting. It’s like, wow, you know, this is core competency of the company and you’re putting that out in the open, but yet that allows you to collaborate with your partner and be able to compete with them as well. That was quite exciting. So back in 2003, 2004 timeframe is when I started getting into that movement and it was still new at that point of time. 

But then, over the last 20 years, that’s the only way I’ve lived and operated exclusively. From Sun, I went to Red Hat, where you will see on their walls of their offices, “First they laugh at you, then they fight with you, and then you win.” And that kind of mantra kind of gets into your blood because that’s the open source philosophy, right? Then I was at CouchBase, then I was at Amazon, part of the open source strategy team, where I was on loan to multiple service teams crafting their open source strategy. 

I remember launching Amazon EKS, Amazon’s managed Kubernetes service back in 2017, and educating the service team that, hey, how do you participate in the open source community? What does it mean? There’s a concept of social dynamics, social engineering that you need to understand. You can’t just submit a pull request and expected to be accepted. So I think that’s the norm that I taught. And then I was on loan to multiple service teams. 

After Amazon, I spent a couple of years at Apple and I was fortunate enough to craft their first open source program office. So I built their first open source program office, went all the way up to the multiple executives, building that case, why Apple should contribute to open source, and a lot of fun over there.

But over the last couple of years, Greg Lavender, our CTO reached out and he says, “Arun, we would like to build open ecosystem culture back at Intel.” And so I’m very fortunate enough here. After a very long time, I feel very happy and excited that all through my management chain to Pat Gelsinger, I don’t have to explain what open ecosystem is. They are the ones that are really pushing the boundary and the entire company is built on…we believe walled gardens prohibit innovation. We believe open ecosystem creates an equitable playground for multiple players to collaborate and also increase the total addressable market so that you can do more fun things on top of that. So I think in that sense, very fortunate to be working at Intel and very fortunate and blessed to be working in this open source movement for the last couple of decades.

Omkhar Arasaratnam (07:07)
What an inspirational story. And it’s, I will second that Intel is definitely one of the examples of an organization that really gets open source. As an old kernel guy, it always used to make me smile to see that the new bits for whatever the new processor was would hit the kernel well ahead of the Silicon being released to the street. And there was a big focus on upstream as well as maintaining the ecosystem that we all enjoy. So thank you Intel and thank you for the work that you do there Arun. 

Arun Gupta (07:41)
It keeps it sustainable. The reason we contribute is because, as you said, Intel has been the largest corporate contributor to Linux kernel for 15 plus years. We contribute there because our customers, when they buy a laptop from Fry’s or Best Buy or an online retail store, they expect when they download Ubuntu or whatever operating system of their choice, it would work out of the box and be able to leverage the latest processors.

And that’s the reason, honestly, we contribute to 300 plus open source projects, whether it’s Linux kernel, PyTorch, TensorFlow, Kubernetes, OpenJDK, and a wide range of projects, because it’s a customer obsession that truly gets us there. And that’s what makes open source sustainable as well.

Omkhar Arasaratnam (08:24)
See, I know you’ve been doing this a long time because you mentioned Fry’s and they’ve been defunct for three years.

Arun Gupta (08:30)
(Laughter) I still love that place. It’s funny because in our neighborhood, Fry’s have been converted into a pickleball court now here.

Omkhar Arasaratnam (08:38)
(Laughter) No kidding. We’ll have to play pickleball the next time I see you. 

Arun Gupta (08:42)
That’s right, yeah!

Omkhar Arasaratnam (08:43)
Switching gears slightly, let’s talk a little bit about the work that you do within Linux Foundation as the board chair for both CNCF as well as the OpenSSF. These are big tasks. I’d love to understand what similarities you see between the security community at the OpenSSF and the cloud native community at CNCF.

Arun Gupta (09:04)
A lot of commonality. They are both, as at Intel we call as, BHAG. Big Hairy Audacious Goals. Both these foundations have those BHAGs essentially. I mean, if you think about CNCF is about how do we make cloud native computing ubiquitous, no matter where you are? And similarly, OSSF, Open Source Security Foundation, talks about how do we secure open source software for the greater public good? 

But there is definitely a lot of similarity between the two foundations. They’re both Linux foundation sub-foundations. They both have a governing board. There are 28 members in CNCF and 23 in OpenSSF per my count this morning. They both have a technical body like CNCF has the technical oversight committee, and OpenSSF has technical advisory council. So both have that element. Now, CNCF also has a technical advisory group, which is about security, where they dig into the details of how do you secure cloud native infrastructure? Security is the most boring thing, right? I mean, it works until it doesn’t work and then everything breaks. So I think that’s a super important element. So you could…

Omkhar Arasaratnam (10:13)
When it’s done well, it’s very boring.

Arun Gupta (10:15)
(Laughter) Right exactly. (Laughter) So I think it’s very important that security is job number one, even in cloud native computing. You can make it ubiquitous, but if it’s not secure, it’s absolutely useless in that sense. So I think that’s the way they think about it. There is a tag security where there is deep focus on how do we make sure that we are making this secure? But so far, that focus has been only on the cloud native computing. And I think that’s exactly where OpenSSF shines up. 

OpenSSF is fulfilling a gap. which is looking at a bigger, broader landscape to identify how do we secure the broader open source software? That’s where tools like OpenSSF Scorecard, Salsa, Sigstore, these are the tools. There is no need for CNCF or any other foundation to create those. That’s where OpenSSF is bringing out these tools that will plug in right there the gaps that CNCF is feeling and any other foundation is feeling.

Within OpenSSF and CNCF, of course, there is a lot of collaboration, but the tools that OpenSSF are creating are available for the broader open source community. So whether you are Apache or Eclipse or in any other community for that sake, those tools are widely available. And let’s be deliberate, let’s be conscious about what kind of interactions can be done to make the cloud native computing more secure so that it fulfills both of our joint agendas and win -win situation. 

And honestly, the way OpenSSF looks at it is as we are creating tools, we can create the tools in silo, but if those tools are not implemented or agreed upon by other communities, again, they’re going to be meaningless. So really making sure that as we are creating this OpenSSF scorecard, how they could be adopted across a wide range of CNCF projects, whatever specifications we come up with, we created secure software development guiding principles. Like, how do you make sure that your software is built using a secure covenant? Now we could come up with a covenant, but really working with CNCF saying that, okay folks, as you are building your project, here are these guiding principles that you should be following. 

So I think in that sense, there’s a very strong cohesion between, the stuff that is being done by OpenSSF and then implemented by CNCF. And again, the idea is if there are gaps identified, there is a clear communication channel, which is more important so that they can give feedback to us. There is of course a public channel, but there is a strong backend channel as well, which enables that high bandwidth communication for the leaders to communicate and share details.

Omkhar Arasaratnam (12:54)
Absolutely, and we have definitely benefited from that back channel and I think the community has definitely benefited from the cohesion, as you put it, that’s been brought together. One of the reasons that many of us get involved with open source and a lot of us are passionate about open source is due to the fact that it’s a public good. I know you’ve done a lot of work with the UN as well and would love to hear your thoughts on the intersection of open source as a public good and what the UN envisages how open source can help the globe.

Arun Gupta (13:30)
Yeah, when the United Nations created these Millennium Development Goals — what they used to call as MDGs at the turn of the millennium, smack at the beginning of the century — those goals were, again, BHAGs, you know, big hairy audacious goals. No poverty, no hunger, no crime, racial, you know, minimize racial injustice, gender equality, beautiful climate policies, you know, policies and all of really wonderful audacious goals. 

And as I’ve been involved with the UN for the last year or so, it’s been really exciting and very humbling experience, because it’s very clear, you have these goals, but the way to solve these goals, of course, is a human element. But a large part of it is a technology element. So last year, I was involved with TED AI, which is a brand new conference, which is again a section of TED, a type of a TED conference that was started in San Francisco last year. 

So last year we worked with TED and the UN to run a hackathon. And the hackathon basically had about 130 participants from all over the country, which basically took a shot at how can we solve this UN sustainable development goals using open source technologies, leveraging AI and cloud native technologies, essentially? vSo that was pretty exciting. 

A couple of months ago, we had KubeCon Paris, and that’s where we had again a very tight collaboration with the United Nations and the Office of Technology within the UN. Really, really good discussion. There were folks from the United Nations who came to the cloud native hacks, which is basically the hackathon that we did at KubeCon, where they talked about the importance and the relevance of the Sustainable Development Goals. These were started at Millennium Development Goals, but 2015 they realized it’s not just about the Millennium, it’s about the sustainability of the humankind. So the name was changed from MDGs to SDGs. 

A very beautiful, a very humbling effort. And I’m very excited to continue that partnership with the UN going forward. Looking forward, we are going to KubeCon Salt Lake City. So we’re going to have a cloud native hacks over there. Highly, highly encourage to bring more and more such places where we can bring that UN hackathon to different events and make an impact to the SDG, essentially making the world a bit more sustainable.

Omkhar Arasaratnam (15:55)
Those are definitely some big, hairy, audacious goals, but also, I think, goals that are good for humankind. And it’s very encouraging to hear this kind of collaboration. I’ve been doing security for a long time. I’ve been doing security for about 20 years. But I always self-identify as a software engineer first that happens to have been doing security for a very long time. 

With that perspective, I personally find there’s a lot of things that security folks just, I guess in their intent of being incredibly security-oriented, that they miss from your perspective. As a software engineer for a very long time. What are security folks getting wrong? 

Arun Gupta (16:42)
Yeah, I think when I think about a conversation, I always think in terms of empathy, that what is my end customer? What do they want? What is the problem that I’m solving for them? That’s super important. Sometimes the security folks focus too much on the technology. Take a step back. Have that empathy for the customer. Does the customer understand that language? Are you talking in their language? Is the intent of your dialogue landing the impact on the customer who’s listening to that discussion? 

The second problem, which is funny enough, is not the technology. Humans are often the weakest link in security. So as security professionals, we sometimes overlook the importance of training and awareness programs for employees. Or we underestimate the potential impact of social engineering attacks. How we could have people just maneuver their way, particularly given how prevalent open source is, how 90 to 9 % of the infrastructure is relying upon open source. For two years, somebody could just social engineer their way into it and then plant something in the software is pretty dynamic. So I think how do we understand the social engineering part of it?

And I guess the last part really is the comms part of it. We need to work very closely with other departments — IT, legal management, developers — making sure the comms are being sent out on a regular basis, the trainings are being done regularly. So focusing on these elements would only make it that much more impactful.

Omkhar Arasaratnam (18:20)
Valuable insight is always Arun. We’re gonna move into the rapid-fire section now.  Okay, spicy or mild food?

Arun Gupta (18:29)
I would say spicy. I’ve always been a spicy person. I like that.

Omkhar Arasaratnam (18:32)
All right, text editors, Vi, VS Code or Emacs?

Arun Gupta (18:36)
Vim, actually.

Omkhar Arasaratnam (18:38)
Vim is the winner! Now this is a highly controversial question. Tabs versus spaces? 

Arun Gupta (18:44)
Oh, yeah, spaces, baby spaces. 

Omkhar Arasaratnam (18:47)
Spaces, all right!

Arun Gupta (18:48)
Yeah, I’m not gonna lose a relationship over it, but spaces it is.

Omkhar Arasaratnam (18:50)
(Laughter) All right, to close us out Arun, for somebody that’s entering our field today, maybe somebody that just graduated from an undergrad in comp sci or somebody that’s making a career change to move into our field, what advice would you have for them?

Arun Gupta (19:08)
Yeah, I was talking to a friend’s son actually, you know, this kid is in high school and up until now he wanted to be a lawyer. And then one day he just comes to the house and he says, I want to be a cyber security professional. And my eyes immediately lit up. I said, “Oh, that’s fantastic! What do you want to do?” And like, I had a very interesting conversation with him. And of course I pointed him to all the training and the certifications and the courses that are offered by OpenSSF. 

My general advice is with ChatGPT with so much of internet resources available, which were not available when I was in college or when I was growing up initially, there is no lack of knowledge. Have that genuine curiosity, dig into it. Don’t be afraid of AI. Embrace it, use tools like ChatGPT to bounce your ideas, build that prompt engineering skill. 

What do you want to really do? Dig into the why of it. Look under the hood, see what’s going on and what could you do? And most importantly, if you find there is a place where you can scratch your itch, do it, contribute. And the more you contribute, the more you collaborate, the more you get your name out there, the more you build the credibility. And, remember, it’s a marathon, it’s not a sprint. So be in it for the long haul.

Omkhar Arasaratnam (20:35)
That’s great advice, Arun. What’s your call to action for our listeners? What would you have them do following this episode?

Arun Gupta (20:40)
I would really encourage them, again, seems to be self-serving, but I would really encourage them to take a look at openssf.org. Look at all the wonderful resources, training, education, certifications that we provide over there. Take a look at that. Come to an event. Go to your local meetup. And the last one, which is very important that I’ve seen particularly people who are graduating out of college don’t have that imposter syndrome. You know, I was there exactly where you are right now 25, 30 years ago, and all it takes is perseverance, grit, and resilience. So just have that in you and roll with it.

Omkhar Arasaratnam (21:22)
Thank you so much, Arun. It’s been a pleasure having you and hope to speak to you again very soon.

Arun (20:21:26)
Thank you so much.

Announcer (22:27)
Once again, for more information about OpenSSF’s activities at the OSPOs for Good symposium and the follow-up event, What’s Next for Open Source, check out the links in the episode description of this podcast. And be sure to catch every episode of What’s in the SOSS by subscribing to the podcast on Spotify, Apple, Amazon or wherever you get your podcasts. And to learn more about the OpenSSF community, visit openssf.org/getinvolved. We’ll talk to you next time on What’s in the SOSS?

The world of software bill of materials (SBOMs) is both complex and fascinating. And few people know the SBOM community better than Adolfo García Veytia — aka Puerco — Staff Software Engineer at Stacklok. Puerco is also a Technical Lead with Kubernetes SIG Release specializing in supply chain improvements to the software that drives the automation behind the release process.

He’s also one of the original authors of OpenVEX, an OpenSSF project working towards a minimal implementation of VEX that can be easily embedded and attested. Puerco is also a contributor to the SPDX project and a maintainer of several SBOM OSS tools. He’s passionate about writing software with friends, helping new contributors and amplifying the Latinx presence in the cloud-native community.

• 01:04 – Puerco shares his background
• 02:21 – What SBOMs are and why they’re so important
• 06:42 – An overview of standards in the SBOM space
• 09:58 – Puerco details his work on VEX projects
• 14:05 – Puerco enters the rapid-fire portion of the interview
• 15:06 – Advice Puerco would offer aspiring open source or security professionals
• 16:12 – Puerco’s call to action for listeners

• Adolfo García Veytia’s LinkedIn page

Adolfo García Veytia soundbite (00:01)
So imagine if you were looking at the video and you see the cook not washing their hands when they cook. That would suck, right? We see a transparent supply chain kind of in the same spirit. The more information you have about your supper, you may be able to do better decisions on whether or not to consume it.

CRob (00:18)
Hello everyone, I’m CRob. I do security stuff on the internet and I’m also a community leader at the OpenSSF. And one of the cool things I get to do as part of the OpenSSF is I get to co-host What’s in the SOSS podcast. With us this week, we have my friend Adolfo, who goes by the handle Puerco out there on the internet. Puerco, bienvenido, al show.

Adolfo García Veytia (00:41)
Gracias, CRob. It’s super exciting to be in a podcast, but also in a podcast with one of my favorite people in the world. So thank you.

CRob (00:50)
I know, right? For those uninitiated in the audience, they might not be aware of the origin story of yourself. Could you maybe just explain a little bit of like what you do with open source and upstream and just kind of maybe some of the projects you’ve worked on?

Adolfo García Veytia (01:04)
Yeah, for sure. I’ve been working on open source projects probably over 10 years or so. I started contributing back in the era of Perl, writing, you know, filing bug reports and documentation for Perl modules. Then did some contributions for PHP. And then when I really started doing more contributions was when I joined the Kubernetes project.

I started up going up the ladder and I became one of the technical leads for Kubernetes Secret release where I specialized on the release process and working on the security features that we have in our releases. And then from then, I started creating lots of different tools that we needed to secure Kubernetes itself, which some of them became projects of their own. And now I’m working on some of the same stuff here in the OpenSSF.

CRob (01:59)
Nice, excellent. So I have a very important topic that you and I get to talk about all the time, but the audience might not be as familiar with. Let’s talk a little bit about software bills of materials, SBOM. Could you maybe describe why SBOMs are important for both developers and also downstream consumers?

Adolfo García Veytia (02:21)
Yeah, for sure. So SBOM, the software bill of materials, to give it a description of what it is in a short sentence, it’s just a list of components that make up your software. That’s the most basic definition of it. Some people may be familiar with SBOM or if not what it is with a term because of some legal requirements and regulation that has come up.

But the fact is that SBOM is kind of the base of the transparent supply chain. So if you’ve seen the news in the past couple of years, there’s a big push to make our software supply chain more secure. And that means can I do good decisions on the risk I’m taking when I’m ingesting third-party software? And to me and to some other SBOM enthusiasts, SBOM is kind of the base of that transparent supply chain. So the way we’re trying to make the supply chain more secure is combating it with information.

So whenever, imagine if you went to a restaurant and before you tried your dinner, imagine you could know exactly the ingredients that went to it, who cooked it. Imagine if you could see a video when they were cooking it. That would give you the ultimate assurance of your plate, right? Of your dinner plate. So imagine if you were looking at the video and you see the cook not washing their hands when they cook. That would suck, right? So we see a transparent supply chain kind of in the same spirit. So the more information you have about your software, you may be able to do better decisions on whether or not to consume it.

And this one is kind of the first layer of that transparent supply chain. As a developer, when you have an SBOM about your project, you kind of have a key to go to a lot of different services that are starting to come up to ask information about your dependencies. So for example, I think the most basic use case is you go to a security scanner and you present an SBOM and say, okay, scanner, give me all of the vulnerabilities that you know are present in this SBOM. And based on the information on your SBOM, the scanner replies back. So that’s kind of the basic use case I see for it.

CRob (04:38)
That’s pretty cool. And I’m correct in remembering there are different types of SBOM. SBOM isn’t just one monolithic thing. You might issue or create an SBOM at different points within the development process, right?

Adolfo García Veytia (04:5:0)
Yeah, exactly. So as the software lifecycle moves forward from idea to software repository or code base to builds and deployment, more information becomes available. And some of the information that goes into your SBOM may not be, it may not be possible to know it at the different stages of the software lifecycle.

So to give an example, if your project requires OpenSSL and it’s dynamically linked, you will not know the effective version until you deploy that binary and it links against your system binaries. So based on that idea that information becomes available as the software moves forward, different kinds of SBOMs have been developed. So there’s the design SBOM, which captures more or less the plan that you want to do.

There’s the source SBOM that looks that a generator looks at your code base, extracts the information that it can and gives it back to you. And then there’s a build SBOM where once you run the build, your compiler or interpreter may do the decision on which version of the dependencies it’s gonna pull down. And even the dependencies of the dependencies, because those may change as your dependencies get new releases. And that gets captured.

And then there’s the, I think the next one is analyzed where basically you take something like an SCA scanner and then point it at your artifact, make some deductions by looking at it from the outside. Gives you that, gives you another kind of SBOM. And finally, there’s the deploy SBOM, which looks at your software once it’s installed and running in a system.

CRob (06:32)
Wow. It’s a lot to keep track of. What types of tools or what are some of the standards that people might bump into in this SBOM space?

Adolfo García Veytia (06:42)
Yeah, so there are two main standards of SBOM. One is SPDX from the Linux Foundation, and the other one is CycloneDX, which is hosted on OWASP, currently undergoing standardization as well. Both standards are more than enough to capture that list of materials. Both standards share more or less the same abstractions when regarding that list of components. And both standards have also grown to capture much more than just a list of dependencies. They can capture things like build provenance and information about machine learning and AI workloads and others.

CRob (07:21 )
And I know that there’s a couple tools that you’re personally involved with. Can you talk about your project, Protobom, and then the bomctl?

Adolfo García Veytia (07:30)
Protobom was born out of a contract or yeah, I think the Radware’s contract from DHS and CISA. They put out like a call for to design a way to exchange information between those formats, and then the company I was working on back at the time, we looked at it and decided that it was a good opportunity to create one abstraction that can handle any SBOM data so that you could basically work with any kind of present or future SBOM formats and without having to care about the implementations. There are a bunch of reasons why. I could happily go, but probably it’s too boring for non-SBOM nerds like me.

CRob (08:19)
No, I think that’s really cool that such a thing exists. Pretty awesome.

Adolfo García Veytia (08:23)
If you think about SBOM tooling as a stack, at the very bottom you have the very strong foundations of both formats, CycloneDX and SPDX, but then Protobom is kind of the next layer in the stack. So that gives you like a universal IO layer to write and read between two formats in your application. That sits on top of Protobom.

And on top of Protobom, we’ve seen a number of tools starting to use it to interact with the formats, but also work with SBOM data. One of those is a bomctl. So full disclosure, I’m not yet part of the bomctl project. I work closely with them because it’s based on Protobom. And the idea of bomctl is that it will be a CLI tool to basically do the most basic operations that most people need to do with SBOMs. Things like updating data in your SBOMs, mixing them, changing formats, those basic operations that most people need to do when they get an SBOM, process an SBOM or share an SBOM are going to be handled by bomctl.

The aim is that bomctl, will provide that great user experience in the CLI, but also the idea is that Protobom will house the required functionality to perform those operations.

CRob (9:43)
So let’s talk about another SBOM-adjacent effort that you and I get to collaborate on together. VEX, the vulnerability exchange. Could you talk about what VEX is and how it kind of plugs into or complements an SBOM?

Adolfo García Veytia (9:58)
Yeah, for sure. So VEX, the way I see VEX helping the overall situation of the secure supply chain is that its main goal is to reduce noise. So part of the work that anybody trying to assess the risk in their supply chain involves triaging vulnerabilities. So when you have a super transparent supply chain as enabled by SBOM and other technologies, you start to get more information.

And with that information, you get things like false positives in scanners. And when your SBOM starts to capture things like your build environment or your build tooling and the dependencies in our container image, you start getting more and more components, which leads to more and more false positives. So to combat this, the idea of VEX came up in the SBOM circles organized by mainly CISA. So VEX, the Vulnerability Exploitability Exchange, is a way for software producers or I hate the word producers, but maybe software authors or maintainers to communicate the impact that vulnerabilities and their components have on their software.

So to give you an idea, if I share a container image and it has some old operating system package that I’m not using, it cannot be triggered in any way, that vulnerability may show up in my user scanners when they scan my container image, but they may not be affected. So VEX is a mean for me as a software publisher to create a small bit of information that gets communicated to my consumers, ideally to their scanners, so that those warnings can be, if not turned off, given more context so that they can make better decisions, and especially to help them triage those vulnerabilities more in a more efficient way.

CRob (12:00)
And when folks are issuing VEX statements, there’s a couple different types, a couple diffrent states that that statement can represent. And what are those?

Adolfo García Veytia (12:10)
Well we think about VEX as a one-shot message that turns off the lights in my security scanner, so to speak. But in reality, VEX is designed to be a communications channel to inform downstream consumers of the whole life cycle of the assessment of vulnerability in my project. So when a new vulnerability gets discovered or reported in one of my dependencies, VEX can give me different statuses that I can communicate to inform them about the evolution of the assessment. So you start by issuing a VEX statement that says that the vulnerability is under investigation, letting them know that you’re aware of it and they’re looking at it. So it’s not getting ignored.

Then the next one, once you have an assessment, you can send another message telling them you’re affected. So if it pops up in their scanner, it’s a true positive, but more importantly, you can send through the VEX channel some extra information about how to mitigate or other information. Or you can also let them know that it’s not affected and then you can inform them why not and that message can potentially turn off some lights or warnings in scanners and alerts. And the last one is fixed, right? So if I got a new release but that new release is showing up as vulnerable, you can send a fixed message.to let them know that this new release is not affected.

CRob (13:35)
It sounds like a really useful kind of emerging tool. I’m looking forward to watching this develop.

Adolfo García Veytia (13:42)
Yeah, I mean, we’re excited on how this is evolving and because it’s a really simple communications channel.

CRob (13:50)
Excellent. Well, let’s move on to the rapid-fire part of our questions. I’m gonna have a couple questions, and generally they’re binary, but if you want to add a little embellishment, please do. First question, spicy or mild food?

Adolfo García Veytia (14:05)
Oh, spicy, of course. I’m Mexican, what else?

CRob (14:12)
(Laughter) Well played, sir. Next question, VI or Emacs?

Adolfo García Veytia (14:18)
VI but not by choice, just by default on my distro.

CRob (14:23)
Do you have an alternative, better alternative?

Adolfo García Veytia (14:25)
Yes, I use JOE.

CRob (14:27)
I haven’t heard
of that one. I’ll have to look that up. Very nice. And our last question, very controversial out there in the ecosystem, tabs or spaces?

Adolfo García Veytia (14:37)
So my thinking is tabs, but I’ve been finding out that spaces plays better with others. So I like tabs because you get control of the indentation visually, but most everywhere it doesn’t work as expected. So I end up defaulting to spaces.

CRob (14:57)
Excellent, excellent. So now as we wind down, do you have any advice to someone? A young developer or someone getting into open source or cybersecurity that any advice for these newcomers to our field?

Adolfo García Veytia (15:06)
Oh yeah. So first of all, the two most important pieces of advice that I can give: one, do not be afraid to take on projects, to ask questions, most importantly, ask your questions. You’ll find that most open source nowadays is very friendly. And the other is show up. Most of those communities are built by people who recognize each other. Even just showing up, showing your face, hearing about the problems, giving simple or complex opinions, everything is super valid. Sometimes just listening to others rant is super valuable. And that’s how you find yourself super quickly on a track to become a maintainer of one of some of the most important projects out there. Yeah.

CRob (15:59)
That’s awesome. Thank you. That’s good advice for newbies. And final question here. Do you have some kind of call to action? Are you looking to inspire our listeners to maybe take up some causes or help out somewhere?

Adolfo García Veytia (16:12)
Yeah, for sure. So I probably could have some calls to action for both projects. So for Protobom, I think we’re looking for folks who maintain SBOM tools. So right now, the strongest implementation is in Go. So if you have an SBOM tool in Go and you want to or you’re planning to start a new SBOM project in Go, come talk to us or look at our repos in github.com slash Protobom.

We hope that you’ll find the project very compelling and helpful for your new project or existing project. So let us know if something is missing or whatever. Also, if you’re familiar with SBOM and in another language, we would like to see more implementations of Protobom in other languages. That’s one.

And for OpenVEX, help us bootstrap the ecosystem. So we’re trying to spread little VEX feeds wherever we can so that when you build artifacts, we can start recognizing those VEX feeds and trying to issue more accurate vulnerability scans. So if you want to help out, let us know and we can help you kick off your your VEX stream.

CRob (17:29)
Excellent. Well, thank you so much for joining us, Adolfo. I really appreciate your collaboration and your leadership in the upstream ecosystem. Thank you for joining What’s in the SOSS? today.

Adolfo García Veytia (17:39)
No, thank you for inviting me, CRob. I’m always happy to chat with you.

CRob (17:43)
Excellent.

Announcer (17:44)

Thank you for listening to What’s in the SOSS? An OpenSSF podcast. Be sure to subscribe to our series of conversations on Spotify, Apple, Amazon or wherever you get your podcasts. And to keep up to date on the Open Source Security Foundation community, join us online at OpenSSF.org/getinvolved. We’ll talk to you next time on What’s in the SOSS?

Christopher Robinson (aka CRob) is the Director of Security Communications at Intel Product Assurance and Security. He also serves as the Open SSF’s Technical Advisory Committee (TAC) Chair. And soon, CRob will step into another role: co-host of What’s in the SOSS? With 25 years of enterprise-class engineering, architectural, operational and leadership experience, Chris has worked at several Fortune 500 companies with experience in the financial, medical, legal, and manufacturing verticals. He also spent six years helping lead the Red Hat Product Security team as their Program Architect.

• 00:57 – CRob’s day-to-day activities and his affiliation with the OpenSSF
• 03:15 – The insight CRob will bring to the podcast as co-host
• 05:46 – What developers writing “post-bang” code should be considering
• 08:40 – Lessons open source could learn from corporate and vice versa
• 12:17 – CRob explores the evolution of open source
• 14:22 – Crob answers Omkhar’s rapid fire questions
• 15:57 – CRob’s advice to people entering the cybersecurity field
• 18:18 – CRob’s call to action for listeners: give back

• CRob’s LinkedIn page
• More content with CRob

CRob soundbite: (00:01)
First and foremost, open source is agile. And that’s something that corporations need to understand. And that it moves at a totally different velocity and isn’t necessarily beholden to month-end change freezes or a year-end close when you’re trying to balance the books. So open source is always moving.

Omkhar Arasaratnam (00:19)
Welcome to What’s in the SOSS? I’m your host Omkhar Arasaratnam and the general manager of the Open Source Security Foundation, the OpenSSF. Joining us today is a good friend of mine. He sometimes goes by Christopher Robinson, he sometimes goes by the security Lorax, but most often he goes by CRob. CRob, welcome to the show!

CRob (00:41)
Hey, thanks for having me Omkhar.

Omkhar Arasaratnam (00:43)
It is a pleasure.

CRob (00:46)
The pleasure’s mine, sir.

Omkhar Arasaratnam (00:48)
So, other than being a security Lorax, can you tell us your title and what you do in your day job as well as the work you do with the OpenSSF?

CRob (00:57)
So I like to say that I really don’t do anything is my claim to fame. I just write and I talk on podcasts like this stuff. But my title is I’m the Director of Security Communications for Intel. So I help make the internet a little less sad about vulnerabilities that impact our portfolio, do crisis communications, I work with our PSIRT and whatnot. And then the other half of my time is spent towards upstream community work like our collaboration at the OpenSSF.

Omkhar Arasaratnam (01:28)
For our listeners that may not have attended a TAC meeting, things of that nature, do you want to talk to what your role is as TAC Chair, what the TAC does and how the TAC works in the OpenSSF to help make open source software a little more secure?

CRob (01:45)
Right. So the Technical Advisory Council, or TAC, is a technical body. We are voted on every year, and some of our seats are appointed by the governing board. But the duty of the TAC is to take a look at the technical initiatives of the foundation. So things like software projects or work on specifications and standards or guides. And we help steer that, making sure things are aligning with the strategic direction of the foundation and they all support the general overall kind of ecosystem uplifting of open source security for everybody.

Omkhar Arasaratnam (02:21)
Amazing, and thank you for all the work that you do. So much of the amazing projects that we have under the OpenSSF thrive under the tutelage of the TAC, and thank you for that. Now as the saying goes this ain’t your first rodeo. You’ve been doing security for a while. 

CRob (02:40)
Are you saying I’m old?

Omkhar Arasaratnam (02:42)
It was implied. (Laughter) And for our audience, the many years of wisdom and experience that CRob brings to the table is one of the reasons that CRob is our new co-host on What’s in the SOSS? As we all believe, he’s going to bring a lot of that experience to the interviews in speaking with some of our guests. CRob, I’m going to put you on the spot, my friend, some thoughts as the newest minted co-host of What’s in the SOSS? What, what do we have in store?

CRob (03:15)
I’m really excited. I’ve got some, a whole list of folks lined up, so we’re going to talk about things like why are people still installing known vulnerable packages? We’re going to talk about large corporations adopting a lot of the projects of the OpenSSF and kind of understanding how an actual person could do some of that. We’re going to talk about coordinated vulnerability disclosure. We’re going to talk about the Linux distros and how kind of their role in helping support security of the ecosystem. So again, I think we have a lot of amazing content, some amazing people that are contributors to the community that we’re going to talk to.

Omkhar Arasaratnam (03:49)
I’m very excited. I can’t wait to hear all the interviews that you have lined up and, and learn more about how people are adopting the work that we’ve been working so hard on. In your career in which you’ve done many things, you spent a lot of time on what I would call the post-bang side. So vulnerability comes out or an exploit occurs in the field. Can you help orient our audience to some of the work that you’ve done in that arena and some of the unique experiences that you’ve had?

CRob (04:22)
Yeah, absolutely. I’ve spent almost seven years of my career working as part of Red Hat product security where under Mark Cox, I helped run the program with him and then eventually partnered with Vincent Danen, who’s now the current leader there. And so I have a lot of experience on both pre- and post-bang. But for my particular skillset is the post-bang piece, trying to help make sure that when that public disclosure goes out, and the world is aware of something going on, trying to make sure that they have adequate information and access to the fixes so that they can treat the problem that’s being disclosed. 

And it’s important that it’s sometimes overlooked as when the responders or developers are in the heat of the moment trying to fix the problem. They’re not thinking about the downstream consequences or what happens when this thing goes public. You don’t want to be on the front page of The Register or The New York Times. So it’s trying to think about things like that and making sure that people can defend themselves.

Omkhar Arasaratnam (05:22)
You have a very particular set of skills, Mr. CRob. What should our developers be thinking of as they’re writing code to make the role of that person that has to deal with the after the fact, the post-bang, a little bit easier? What can they be doing in development so that it’s a little less, the incident is a little less exciting?

CRob (05:46)
I have a couple pieces of advice. First and foremost, all projects should talk about — whether it’s a single maintainer or a team — you need to sit down and think about how, when you get a vulnerability report, how you’re going to handle that and ultimately how you’re going to disclose it. And in the industry, we call that a vulnerability disclosure program or a security policy. So it’s important to have that. And that helps set expectations both for when the finder comes to you — they’re making a demand of something getting fixed. It also helps your downstream understand exactly what you’re gonna do. Are you going to fix something? How quickly are you gonna fix something? So having that documentation upfront and that agreement helps set the expectation so that when things go public, ideally it goes smoothly for your downstream. 

And next, you need to understand kind of what your project depends on. Very few projects are self-contained and they only have code and only use code written within that project, you have dependencies, you have libraries and other components that you’re calling out to. So understanding what you depend on helps you react when one of those components has a vulnerability. And then that’ll ideally help avoid your downstream asking, are you affected? Are you affected? So understanding your dependencies, consider writing an SBOM for your team’s use, your project’s use, but also consider sharing that for your downstreams, because that’s valuable information to them to understand and helps with their vulnerability remediation. 

Next up, I would say that find a big brother or a big sister in the community. There’s a lot, open source has been around for three decades now that since Linus released the kernel, find someone that you look up to, that you respect and ask them advice. Line up people that when you’re in the middle of a crisis, people you can talk to that might have more security expertise than you so that you can ask questions and help get help if you need it. But having that community, big brother, big sister is invaluable to give you that advice and guidance and suggestions on how you might be able to do something. 

And then finally, I would encourage all developers to embrace the mentality of Kaizen, that continuous improvement. So what you do today isn’t necessarily going to make you successful tomorrow. So think about it with every release, with every vulnerability that you fix, think about how you can adjust your practices or tweak your tools so that maybe you can avoid that situation going forward.

Omkhar Arasaratnam (08:08)
That’s some great advice and certainly speaks from those — how many decades of wisdom was it that we were up to — so, sorry, I’ve said too much. With your experience, I’m wondering between the time that you’ve spent on open source and in corporate, what lessons can be learned from either side? Like what should corporate security be thinking of that open source does incredibly well? And what does open source need to think about that maybe corporate security has been doing a little bit better of a job?

CRob (08:40)
That’s a really interesting question. I think, and I can address both points of that. First off, for those of you that aren’t aware, at its core, open source software is about sharing. It’s about openness and transparency and meritocracy, where the best ideas win. So, first and foremost, open source is agile. And that’s something that corporations need to understand, that it moves at a totally different velocity and isn’t necessarily beholden to month-end change freezes or year-end close when you’re trying to balance the books. So open source is always moving. So just being aware of that and that agility not only is in the process, it’s also in their thinking where they are able to ingest new diverse ideas, different perspectives. 

And that’s something that corporations potentially can learn from kind of seeing this attitude and embracing the fact that, you know, somebody, a grizzled veteran of a quarter century in development and security might not have all the answers. And somebody, a new junior person might have just as valid an insight into the problem as everybody else. Next up, where developers are incredibly creative people, but kind of like me, they tend to be a little lazy. So wherever possible, they love automation. And that helps them become more efficient, helps them be that creative source of inspiration and ideas. So open source is all about automating. And that’s something I would really recommend. 

Back when I was in several corporations I was at, we always looked for opportunities to automate a process. And whether this is part of your CI/CD pipeline, part of your deploying servers or configuring Kubernetes clusters, whatever. If there’s like, a repetitive low-value task, make a robot do it for you, write automation to make that go away. And then again, just thinking about open source very much kind of avoids that monoculture and they avoid monolithic thinking. So again, you’re going to have really great diversity of thought and diversity of background. And I would again, encourage corporations to embrace that.

Now flipping this on its head, corporations are generally, hopefully, good at making money and good at directing resources and managing time and priorities. So that’s where open source kind of falls down a little bit. We’re not quite as organized and disciplined as we could be. And where I would, the very first and foremost thing, and you’ll see this, it varies widely by project and community, but documentation and process are key because it not only helps you draw in new members if they can understand the kind of what your processes are and how they can interact with it. 

But it also helps your downstream understand what you’re doing, how you’re doing it and having procedures like tabletop exercises where we went through this effort and the vulnerability disclosures working group at the foundation. We’ve been partnering to help create these resources so that an upstream project can understand the value of doing a tabletop exercise or going through a threat modeling exercise to understand how their software can be broken. 

So these processes, the discipline and rigor — not that I’m saying you would totally want to be inflexible — but open source definitely can adopt some of these things to help improve the quality of life of the maintainers and the downstream.

Omkhar Arasaratnam (11:52)
Make the incidents boring.

CRob (11:54)
Right, exactly. It should be just a non-event. It’s just another patch.

Omkhar Arasaratnam (11:58)
That’s definitely some interesting lessons that can be learned from either side. Now, in your experience in using open source, I’m sure over the years you’ve seen it evolve. How have you seen it from your earliest days engaged with open source to where we are now? Have you seen that evolution?

CRob (12:17)
So when I started off, I ran a corporate engineering and operations team and we were responsible for first web security of a large financial institution. And then I moved over to the distributed side, the distributed server side, where I ran the Unix Linux team. So back then, when I first got into this space, the idea of large companies using open source was generally almost exclusively a cost play where we had feature parity, you know, that the Unix and Linux were generally the same. And it was pure cost savings where you could deploy a server for one quarter the cost of a traditional Unix device. 

And, you know, there wasn’t a lot of innovation, but getting those ideas from Linux and open source into these large financial institutions helped them understand and be able to harness some of the innovation and velocity that happens upstream. That’s where you started to gradually over the last decade or so, I’ve seen companies go from, we’re just trying to save a couple bucks to, hey, there’s some amazing innovations here. I want to integrate these features into my own development practices or my own portfolio so that I can be that, you know, take advantage of some of these cutting edge things. 

And that’s like how things like Kubernetes and containers all came about because that was a way of trying to help improve operational efficiency. So now again, it’s kind of transitioned to where we are now where people are participating upstream and helping set some of that innovation being very engaged upstream to help steer things to have good outcomes downstream. It’s a nice evolution and open source software development. It used to be we were all kind of a Waterfall waterfall style of development model or it was very slow and regimented.

And now we’ve changed to agile methodologies and that’s the de facto way how software is developed today.

Omkhar Arasaratnam (14:14)
It’s certainly amazing to see how adoption has changed over the years, and I can’t wait to see what’s coming next. 

CRob (14:21)
Yeah, me too.

Omkhar Arasaratnam (14:22)
So now, CRob is when we transition to rapid fire. So here’s how rapid fire works. Can I ask you a few questions? 

CRob (14:30)
You may.

Omkhar Arasaratnam (14:31)
And in asking those questions, I’ll provide a set of pre-scripted answers or the right answer may be, no, Omkhar, you got that wrong. Here’s what I think. Are you ready to go?

CRob (14:43)
I am, sir.

Omkhar Arasaratnam (14:44)
Alright, first one. Spicy or mild food?

CRob (14:48)
Spicy, no other answer is valid.

Omkhar Arasaratnam (14:50)
Totally agree. Spicy or nothing. And I recall from a few meals that we’ve shared and a few drinks that we’ve shared, you know how to hold your spice, man. You do all right. All right, this next one’s a bit trickier. Favorite text editor: Vim, VS Code, or Emacs?

CRob (15:08)
VI, colon q, baby!

Omkhar Arasaratnam (15:15)
There you go! Exclamation mark. Quit without saving. Now the last one is incredibly controversial. Tabs or spaces?

CRob (15:21)
(Exasperated sigh) I like tabs, but I understand that spaces have their place and are useful. So again, I’m not a purist. I would lean towards tabs, but you’ll see some occasional spaces. 

Omkhar Arasaratnam (15:33)
That’s an incredibly diplomatic answer. 

CRob (15:36)
I do my best. 

Omkhar Arasaratnam (15:38)
In wrapping things up, CRob, what advice would you have for somebody entering our field today? Be they somebody graduating from college versus maybe somebody who’s made a career change and decided for whatever reason to get in a cyber. I mean, that’s, I don’t know why they’d make that decision, but what advice would you give somebody entering our field?

CRob (15:57)
I have so much advice. It’s hard to choose just one, but I will say this. In my experiences in the last (intentional murmur) century of doing cybersecurity and software development–

Omkhar Arasaratnam (16:08)
What was that, three centuries?

CRob (16:10)
Pretty close. I think that there are so many interesting things going on, so many different and unique aspects of the trade. And I would, it’s very easy to get overwhelmed. It’s like you’re a kid in the candy shop and you want a little of everything. And then when you do that, you get sick and you vomit.

So my advice for new folks is find things that you’re interested in, that you’re passionate about, that excite you, and go find people that have that similar interest. You’ll do better work when it’s focused on things that spark joy for you. And then secondly, once you find your people, so to speak, you find your community, whatever the type of specific nuance you’re interested in, whether it’s GRC or application security or risk management, talk to people.

Find someone that is a mentor or a role model. Reach out to them and engage with them. It’s been my experience that people within both open source and cybersecurity, they like to talk and they like to talk about themselves and share their experiences. And it might be intimidating. You might think of someone as an air quotes rock star that they’re superhuman, but they’re not. They’re just people just like you. And they like to share their experiences. 

And I have personally benefited from having a long trail of mentors in this space where people have been incredibly generous with their time and helped teach me concepts that I was unfamiliar with, or I was able to, you know, in reverse, I helped mentor a lot of people as well. I also work with an organization called ISC2, and I’m a certified cybersecurity person. And part of our code of ethics, part of the idea behind those certifications is improving the trade. So, my contribution to that is helping groom the next generation of folks. 

So,  I feel obliged to do this and many people in my space do. So again, find people that you look up to, talk to them, have them share their experiences and you’ll learn more over coffee or an adult beverage than you ever will from a book or classroom.

Omkhar Arasaratnam (18:08)
Some sage wisdom and for what it’s worth, CRob, I consider you to be one of our rock stars. 

CRob (18:13)
Aww, thank you.

Omkhar Arasaratnam (18:14)
Last question. What’s your call to action for our listeners? 

CRob (18:18)

I talk a lot at cybersecurity conferences and every time it’s generally about open source security and every presentation I have a similar slide where I say, chances are very good, 90 to 98% that you’re using open source software. Why aren’t you contributing back? And contribution isn’t necessarily development or money.

But show up to those communities, give those communities feedback, help them out with some docs, take some notes in a meeting, just participate in the conversation. Be that sounding board as a developer’s testing out new features. If you have the skill set and you are a development person, contribute some patches, fix some bugs, look at their backlog. 

And that is a huge help to a project. Somebody comes in and either provides them feedback or helps them work on their backlog. And that’s the best way that you get a lot of value from this free software. It’s just simple, easy ways to get back.

Omkhar Arasaratnam (19:18)
Patches are welcome.

CRob (19:19)
Patches are always welcome.

Omkhar Arasaratnam (19:21)

CRob, thank you for joining us today on What’s in the SOSS? It’s my pleasure that you’re going to be joining us as co-host. So I’m really looking forward to hearing your episodes as well. And yeah, thanks for all the support and all that you do for the community.

CRob (19:34)
Very welcome. I’m looking forward to it. Thank you.

Announcer (19:37)
Thank you for listening to What’s in the SOSS? An OpenSSF podcast. Be sure to subscribe to our series of conversations on Spotify, Apple, Amazon or wherever you get your podcasts. And to keep up to date on the Open Source Security Foundation community, join us online at OpenSSF.org/getinvolved. We’ll talk to you next time on What’s in the SOSS?

Matt Knight is Head of Security at OpenAI, where he builds IT, privacy and security programs. His teams also collaborate on security research with teams across OpenAI and with the broader security research community. Their goal is to explore the frontier of AI, understand its impacts and maximize its benefits, especially in the cybersecurity domain.

• 00:40 – Matt’s duties at OpenAI
• 01:52 – Matt’s accidental journey into cybersecurity
• 05:18 – The intersection of AI and open source
• 06:45 – Matt’s thoughts on how AI can help security professionals
• 08:53 – Details on the AI Cyber Challenge (AIxCC)
• 10:53 – Matt answers Omkhar’s rapid-fire questions
• 12:29 – Advice Matt would give to aspiring security professionals
• 13:00 – Matt’s call-to-cation for listeners

• Matt Knight’s Linkedin page
• GNU Radio
• AIxCC Challenge
• OpenAI Cybersecurity Grant Program

Matt Knight soundbite (00:01)
AI has the potential to help cybersecurity practitioners where they’re constrained. That’s important because cybersecurity engineers face a lot of constraints. Every security team is constrained by capabilities, is always up against pressure to be faster and is always up against pressure to access greater scale. 

Omkhar Arasaratnam (00:18)
Welcome to What’s in the SOSS? I am your host Omkhar Arasaratnam. I am also the general manager of the OpenSSF. Today we have a good friend of mine Matt Knight Matt. Why don’t you tell us what you do?

Matt Knight (00:31)
Hey, my name is Matt Knight. I’m the head of security at OpenAI.

Omkhar Arasaratnam (00:34)
I feel like you’re burying the lead here. What does the head of security at OpenAI do? I mean, it doesn’t sound like a boring job.

Matt Knight (00:40)
Yeah, it keeps me on my toes. So I joined OpenAI back in 2020 and have been building the security, privacy and IT programs since then. Before OpenAI, I spent most of my career protecting companies and institutions that had comparable high-value research and technology. I also co-founded a company called Agitator that focused on security research. And if you go far enough back, I started my career as an electrical engineer before getting into security.

But these days I spend most of my time focused on security engineering and building the systems for developing and deploying advanced AI. My teams and I also collaborate on security research with teams across OpenAI and with the broader security research community to explore the frontier of this technology, understand its impacts and also maximize its benefits, specifically as far as I’m concerned, on the cybersecurity domain. 

And internally this involves using large language models wherever we can to enable our security program. And yes, even doing some open source work of our own, too. So it’s great to be here and I look forward to a great discussion.

Omkhar Arasaratnam (01:46)
Wonderful. Thanks, Matt. Sounds like you’ve had quite a journey in terms of security. Why don’t we start at the beginning? How’d you get into security?

Matt Knight (01:52)
To put the bottom line up front: accidentally. So I started my career as an electrical engineer, as I mentioned, I studied EE in college and EE is a pretty big field. You can be sort of on one end of the spectrum, you can be doing analog electronics on the other end, you can do digital, coupled with software engineering.

And I was always more on the digital side. So my first job out of college, I was working as an embedded software engineer, writing software for wireless networking stacks. And it was pretty interesting work, but I got to a point in my work where I found that I needed a spectrum analyzer to debug a system I was working on. And if you’ve ever had to buy lab equipment, you know that it’s really expensive. 

But right around that time, there was this open source project called GNU Radio that was getting a lot of buzz. And GNU Radio was really cool because it was this powerful open source signal processing toolkit that enabled basically like using software to implement all these different radio engineering and signal processing tools.

And between GNU Radio and some low-cost commodity hardware, I was able to get my hands on, I was able to basically build my own spectrum analyzer to help me do my work in developing and debugging these wireless systems. So I had this toolkit for monitoring the spectrum and it was pretty useful for that, but I kind of kept playing with it and found that you could use it not only to, you know, capture and analyze signals, you could also use it to replace signals, to generate your own signals. 

And, you know, realized that, you know, when you, if you captured a signal and replayed it, a lot of devices would just accept it and would, would, you know, treat that as valid. And that really freaked me out. Also got me, you know, was sort of my first contact with how, you know, vulnerable much of that ecosystem was. And I kind of couldn’t look away from it. 

So, I started doing security research on my own nights and weekends, wound up having the opportunity to make a career out of it. And wound up doing a lot of work in that, you know, open source or in that, that wireless security space, a lot of which was supported by this really vibrant open source community at the time. And I did that for a while and then made the choice a couple of years later to make a career transition into what I’m doing now.

And I’ve been spending roughly the last decade playing defense. I still have a lot of passion for the wireless space, but these days I’m spending my time protecting companies rather than doing wireless research.

Omkhar Arasaratnam (04:16)
You might have come up with the Flipper Zero before the Flipper Zero.

Matt Knight (04:19)
Flipper Zero is pretty cool. No, I was working with other sort of open source and some proprietary bits of hardware, but really underpinning all of it was GNU Radio. GNU Radio is a really, really powerful open source tool. There’s a ton of great academic and commercial work and research being done on it.

They have a great community around it. I’ve spoken to their conference a couple of times. And if you go far enough back, I open sourced at least one GNU Radio module myself based on research that I’ve done. So quick shout out to that community. It’s still going strong. And I’m always impressed with the great folks who work on that toolchain are coming up with.

Omkhar Arasaratnam (04:57)
I’m happy to hear that. So open source, it’s been with you for a really long time. Let’s talk about your day job now. So you’re working on a lot of cutting-edge stuff. As we think about AI, large language models, generative AI, how much of that world is supported by open source? What’s that look like?

Matt Knight (05:18)
Quite a bit of it is derived from open source. And I’d say that most companies are, to some degree, leveraging open source and also building their own. If we look at many of the frameworks that the AI industry leverages, think things like PyTorch and TensorFlow, they started in various ways within companies but now are open source and are sort of robustly supported by broad communities.

And if you go beyond the frameworks, there are of course, myriad dependencies that companies depend on to do their research and also to run their infrastructure. And of course, much of the world’s AI training infrastructure runs on Linux, which is, you know, of itself, of course, open source. So I’d say that by and large, you know, open sources is pretty important to AI research and innovation. 

But beyond AI, you know, much of the tools that the security industry uses too, you know, have open source connections too. So there’s a network security tool called Zeek that most security, well, I shouldn’t say most, but many security teams use in different ways that’s really powerful. And then, you know, in other domains like code scanning, we’ve got some newer tools like Semgrep and CodeQL that are really powerful. 

Omkhar Arasaratnam (06:25)
So we talked about how open source is a lot of foundational components of what we use in terms of AI today and how open source is a foundational component in a lot of the security tools we use. What if we inverted that? How can we use AI to improve the security of open source? Do you have any thoughts on that?

Matt Knight (06:45)
I do. So my teams and I spend a portion of our time studying and analyzing how advances in AI may impact cybersecurity. And we want those benefits to be, of course, as broadly distributed as possible. And what more deserving beneficiary of that than the open source software ecosystem?

And a thesis that I’ve sort of been refining here is that AI has the potential to help cybersecurity practitioners where they’re constrained. And that’s important because cybersecurity engineers face a lot of constraints. Every security team, to some degree, is constrained by capabilities, is always up against pressure to move faster, and is always up against pressure to access greater scale. 

Do you have the expertise to find and fix the security problems wherever they may lie? Can you find and fix the problems fast enough to mitigate issues before they turn into real problems? Can you fix the problems wherever they happen to exist? There’s always more code to analyze, there’s always more logs to analyze, there’s always more that you can do to get leverage. Can you access them efficiently? 

So we are finding that AI is broadly useful to alleviate some of these pressures. And we as a team look to incorporate language models into our own work wherever we can. Now, of course, it is necessary to be aware that these tools are very imperfect on their own. They have things that they’re really good at and they also have a lot of downsides. So we’re looking for places in which we can implement these tools to benefit our work while also managing the drawbacks and downsides. 

Omkhar Arasaratnam (08:30)
Sounds good. So we’re recording this just after the Open Field Competition for AIxCC closed. It closed on April 30th. Can you share with the audience a little bit about the AIxCC, the AI Cyber Challenge, how OpenAI is involved, obviously, OpenSSF is a supporter as well.

Matt Knight (08:53)
I’m happy to and Omkhar, I think we first met at DEF CON last year in connection with the AI Cyber Challenge and I’m glad to be supporting this initiative along with OpenSSF. So the AI Cyber Challenge really has a great mission at its core, which is to find and fix vulnerabilities in the open source software that powers and underpins the critical infrastructure that we all rely on.

And it’s very timely because we’ve seen this great explosion in AI capabilities that’s largely been driven by language models. And while we’ve seen so much capability growth in language models, I believe that static analysis, that is finding and fixing vulnerabilities in source code, is an area where language models have historically underperformed. I think it’s a rich area for research, but because the capabilities are still emergent, I think success in this challenge is gonna involve a lot more than just like clever prompt engineering to get results. 

But the challenge is great because it engages a robust security research community. It brings a whole bunch of folks who wouldn’t necessarily participate in a program like this into the fold. And it’s also gonna happen and play out publicly. I think the semi-finals and finals are slated to be at DEF CON, which will be a great way to get even more of the community involved. And I’m pretty enthusiastic about what it’s gonna produce. If we look at where conventional static analysis tools fall short, AI and language models have the potential to really bring different capabilities to this domain to help, I think, fill in some areas that could really benefit the sort of static analysis tool ecosystem.

Omkhar Arasaratnam (10:36)
I’m really looking forward to seeing what our competitors come up with as well. We’re going to move into the rapid-fire section. So I’m ready when you’re ready. And the right answer for any of these may be one of the answers I provide or no, Omkhar, I actually, I think it’s something else. So are you ready?

Matt Knight (10:55)
Let’s go, hit me.

Omkhar Arasaratnam (10:57)
Spicy or mild food?

Matt Knight (11:00)
Okay, so I have Irish heritage and I grew up in a family and household where salt was an exotic spice. So my answer may surprise you. I am a spicy food guy all the way.

Omkhar Arasaratnam (11:14)
All right, man. Well, we’ve got…we’re going to be grabbing a meal soon. I hope you’re…I may bring some hot sauce with me. Now, the next couple of questions are very engineering-focused. Text editor of choice: VI, VS Code or Emacs?

Matt Knight (11:32)
I am a VI or Vim person all the way. And, I mean, beyond it just being the first thing that I learned, it’s on everything. You know, it’s on your, it’s on your Linux, you know, laptop, it’s on your, you know, all the servers you’re going to jump onto, but it’s also on a lot of embedded systems. You know, you’ve got the small low profile and embedded versions of it that you see in various places. So it’s a pretty useful editor to fall back on.

Omkhar Arasaratnam (11:59)
I’m also a Vim guy, so full support here. Last but not least, tabs or spaces?

Matt Knight (12:05)
Spaces and specifically two of them.

Omkhar Arasaratnam (12:09)
(Laughter) Excellent Thanks so much for for going through that. Now Matt, as we close out the podcast, what advice do you have for somebody entering our field today — somebody that’s either a new grad? Just completed their undergrad in comp sci or somebody that may be switching careers. What advice do you have for somebody entering today?

Matt Knight (12:29)
Love this, love this question. The world is changing beneath our feet very quickly with whether it’s the emergence of AI or just sort of more generally the pace at which the software ecosystem or the security ecosystem moves. So my advice to anyone who’s getting started is really to stay curious. And if you commit to that and a lifetime of learning, just enjoy the ride.

Omkhar Arasaratnam (12:55)
And the last question for you, what’s your call to action for our listeners?

Matt Knight (13:00)
A couple of things here will be exciting to the listeners here. The first is that we at OpenAI are hiring. So if you’re interested in AI or language models or security, please do give us a look. I also want to just briefly mention our cybersecurity grant program. This is something that all your listeners should feel encouraged to participate in. But we’re giving out a million dollars in cash incentives plus API credits to fuel innovation and research in defense of cybersecurity. 

We love open source as part of that. So if you are working on or want to work on some sort of open source innovation to help benefit the ecosystem, we’d love to take a look and fund it. Just some ideas of things that we’re excited about. So, you know, porting code to memory-safe languages. If you want to look at applying AI to that, that would be awesome. We think that confidential computing for GPUs could be a pretty important layer for protecting AI services going forward. And we’d love to fund some work around that, and other ideas too. We’re always looking for research collaborations with the broader community. So we’d love to hear from you. And just lastly, two of my colleagues were at Black Cat Asia, Paul McMillan and Fotios Chantzis. They actually open sourced some of their work coming out of that. Some automation that we built at OpenAI to help enable our work and help our teams move faster. So if that sounds interesting, I encourage you to go check that out.

Omkhar Arasaratnam (14:25)
Thanks so much, Matt. Really appreciate your time. Thank you for joining us on the podcast.

Matt Knight (14:29)
My pleasure, thanks for having me.

Announcer (14:31)
Thank you for listening to What’s in the SOSS? An OpenSSF podcast. Be sure to subscribe to our series of conversations on Spotify, Apple, Amazon or wherever you get your podcasts. And to keep up to date on the Open Source Security Foundation community, join us online at OpenSSF.org/getinvolved. We’ll talk to you next time on What’s in the SOSS?