Intro Music & Promotional Clip (00:00)
Security was one of those things where we did not compete. It is table stakes for everybody. Our users deserve better than just like, “I’m gonna get this fix out faster than you because I’m not going to tell you about it.” None of us thought that and I believe today that still rings true – we don’t we don’t believe that. That’s part of the open source community that works.
CRob (00:25)
Welcome, welcome, welcome to Big Thoughts Open Sources, where we get the opportunity to talk with people from across the open source ecosystem, talking about supply chain security, software security, AI security, and a lot of other topics. Today, we are joined by a friend of the show and a longtime contributor and collaborator with the OpenSSF, Vincent Danen. You’re the Vice President of product security at Red Hat?
Vincent Danen (00:54)
Hey, CRob. Yes, yes, that is I have the dubious honor of being the VP of product security at Red Hat, as you well know, because we’ve known each other.
CRob (01:04)
A long time. Probably longer than some of the audience members are old.
Vincent Danen (01:09)
well, hopefully you didn’t say alive. I don’t think it’s been that long. We’re not that old.
CRob (01:15)
Hahaha
Well, so today we have a whole exciting series of questions. We’re going to have a great dialogue about broadly open source security. But before we get there, let’s kind of get in the way back machine. I want you to kind of describe to the audience. What is your open source origin story? How did you get into this whole little unique slice of heaven?
Vincent Danen (01:44)
Oh man, we’re going back, the way back machine is right. So I’ve been at, I mean, I’ve been a Red Hat for 17 years, was at Mandriva or Linux Mandrake when I joined them eight years prior to that. So I’ve been doing open source for over 25 years.
CRob (02:01)
Wow!
Vincent Danen (02:02)
Yeah, yeah, I think the first, actually it’s really funny, the first, not the first distro that I used, but the one that I used seriously, I was actually to run my BBS.
And that was like Red Hat Linux 5.
Yeah, it was Red Hat Linux 5, but it didn’t work with some of my SCSI drives. So I switched over to Linux Mandrake, which I think they were trying to be a little tempestuous, impetuous. They were like version six. And so I was using that, which is when I started volunteering, actually doing Package Management for Linux Mandrake while I was doing some Linux consulting on the side, I ran my own business. And they… enjoyed the work that I was doing. So they’re like, hey, you want to do that? Plus I was writing technical articles. If you go back like Tech Republic and a couple other outfits, I was writing a whole bunch of articles on Linux and open source, which was a ton of fun and learned a lot. And, and then I started working for Linux Mandrake doing packaging and documentation. And then one day they were like, Hey, one of these guys wants to be working on the kernel full time and not dealing with some of these vulnerabilities like in wu-ftpd and some of these older things are like, there’s only one per month roughly that you’re gonna have to do. you wanna take that on? And I was like, yeah, sure, whatever, I can do that. And that one a month turned into one a week, turned into one a day, turned into how many hundreds a day are we up to now? Like it’s a lot. So I kind of, fell into the whole security space by accident because somebody suckered me into it. And 25 years later, I’m still doing it. So I must’ve found my…found my niche.
CRob (03:41)
Yeah, well it’s something that I’ve had the opportunity to work alongside you. You’re very passionate about, which is important because it’s not a fun job all the time, right? It can be pretty hard.
Vincent Danen (03:52)
no, no, it’s definitely not a fun job. Almost all the time, at least these days, right? I mean, I think any honest security practitioner will tell you that. But I mean, the open source part of it is a blast, right? Like I, I mean, I have drunk all the Kool-Aid. I am a firm believer in open source, not just open source software, but like the open source ethos, the mythology around open source and the philosophy and the principles and
Man, these are things to live by and I love it.
CRob (04:26)
Yeah. So let’s start to dive into our topic today. Broadly, vulnerability disclosure. And let’s start off with kind of the elephant in the room, so to speak. So a lot of our work today in coordinating vulnerabilities goes through the CVE program, which is a program that’s funded by the US federal government. And there’s a couple of different
You have the CVE program, you have the National Vulnerability Database. But broadly, the CVE has been having some problems with funding, some uncertainty with politics. And in the meantime, the vulnerabilities are rapidly increasing. So from your perspective, and kind of with this 25 year history, what has worked well as part of vulnerability disclosure specific to CVE or not and kind of where are we starting to see it come apart at the seams.
Vincent Danen (05:31)
Great question. I think the actual CV part of it is very central to it, right? Because if I go back to those 25 years, when CV first started, it was basically in response to, we have a vulnerability, it’s called six different things, and nobody knows what they’re talking about, right? So I mean, like, I still remember old send mail vulnerabilities, which would have like a bug force ID and X force ID.
Bob’s ID, and then there was a CV name and the CV was that promise of we will have one name for every vulnerability. So when you’re asking about a buffer overflow or send mail, because Lord knows there was enough of them back then, which one are you talking about?
Right, so if I’m sitting here and I’m a defender and I’m supposed to fix it, I would like to know which patch you’re referring to so I can tell you whether or not that patch was applied, whether or not it’s applicable, all of those sorts of things. So CV was great, it lived up to that promise, every vulnerability got a CVE name, everybody knew what it was called and what it was referred to. I think nowadays the biggest challenge that we have is one, CV isn’t comprehensive.
CRob (06:40)
Yeah
Vincent Danen (06:41)
Right, so if you look at, and it drives me nuts, like GitHub security advisors. There’s a bunch of those without CV names. Valid vulnerabilities, right, but they don’t have a CV name. So when you’re looking at it, it’s like you have customers or people are asking you, have you fixed this GHSA?
And it’s an advisory, I don’t know, what’s the CV name, right? So we don’t necessarily treat everything as it doesn’t exist unless it has a CV, because that’s not how vulnerabilities work, but I find it very frustrating, and I think this is where we’re starting to see some cracks in the seams, is like, GHSAs are valid, and they are, but they don’t have a CV name and nobody cares. It’s like, they’re not out there, like, I don’t know, GitHub if you’re listening, you could have a mechanism as a CNA where you basically assign CVs to all these things, right? Or work with those maintainers to be able to say like, hey, you should have a CV or some kind of policy there. But when we’re looking at all of the noise that came out last year, right? With all the funding kerfuffle and all of this other stuff. And then everyone’s like, I know, I’m going to create a GCV or some other type of CVE or a…
Like we were going back to the bug force IDs and the X force IDs. Even this will probably make you laugh because this is a bit of a blast from the past to like the DWF. Right? Like CV worked and they were like, ah, no, we should have DWF too. And then like that didn’t work. And so I worry about all of these competing ecosystems when we had something that worked really, really well. It was basically a dictionary. It just described a thing and identified the thing.
And that’s it. That’s all we need. We don’t need any of the other stuff. Now I think that maybe CV is going a little bit off the rails and lean more into like NVD land where NVD was like, we’re going to enrich all this stuff. We’re going to give you this extra data. We’re going to give it a CVSS score and all this jazz. That was never what CV was meant for. It was never meant to say, this is the severity. This is a CVSS. Whatever it was like, this is the thing. This is the coded effects. Downstream users, upstream users, you figure it out. right? And that model works well.
CRob (08:56)
That’s been, I think, absolutely agree that CVE is that one identifier to rule them all. That way everybody had kind of that same shared understanding. And in Red Hat, that CVE may have these three effects and in Windows it might have these 12 other effects. But it was nice that everybody had, this is what the problem is. This is the root cause.
Vincent Danen (09:19)
Yeah, and when you, this is my beef with NVD, right? Because they started out as a database for the US government to rate things in the context of the government’s use. Everyone and their dog decided like, this is a cool database, we’re gonna use it too, right? And it was a very much one size fits all, right? And you noted different effects, like the thing I love about open source is it’s one, right? It is ubiquitous, it is everywhere and everybody is using it.
But, that means you’ve got different compilers. You’ve got different versions of interpreted languages, so like your Go’s, your Pythons, your Rubys. You have different operating systems with different security contexts, like Windows operates very differently than Linux does. Mac OS is similar, but there’s different compilers, right? And then everybody compiles these things differently. They have different hardening flags and all of these sorts of things. So unless you look at it from the vendor context of like, this is how it’s used. This is the security context around it. This is how it’s compiled. Those scores and those severities are going to be different. Like Mac OS doesn’t have SELinux, but we do. It’s enabled by default. Shame on you if you turn it off. It works really well. But I mean, we’ll rate those things in that context, right? Or with somewhat like the Fortify source or SSP or some of these other compiler flags. I mean, I’m not a Mac guy other than a user. Like I don’t develop on it.
Do those hardening flags exist in that C compiler? Literally no idea. But that’s going to change something from like remote code execution to a denial of service, right? In a lot of cases. So those have to be accounted for.
CRob (11:02)
Yeah, exactly. One thing, so from your perspective, you mentioned that you went from one CVE a week to one a day to one an hour. It’s interesting from your perspective and kind of your interactions with the upstream and downstream communities, how has this rapid acceleration of more information, more vulnerabilities being shared. How do you think that’s affecting people?
Vincent Danen (11:35)
Um…I think people are giving up on the notion of coordinated vulnerability disclosure
CRob (11:41)
Say it isn’t so.
Vincent Danen (11:42)
Which I actually think is terrible. No, I think it’s terrible. Think. The thing I think to realize when it comes to open source is you have up streams, you have down streams, and you have a wide variety of everything else. You’re never gonna be able to coordinate everything across everybody who’s a piece of open source, right? Like that’s just a fact. But there are, I’ll say congregations of different types of open source that are available from different vendors. And that’s where you have your commercial open source vendors, right? So whether it’s Canonical or SUSE or Red Hat or whatever, right?
They are large distributors who provide significant chunks of what that open source is being used out on the internet, particularly when it comes to critical infrastructure. I don’t know this for fact, but I do believe that like a nuclear reactor isn’t going to be updating from the Linux kernel directly, nor are they going to be building all of their source code, because why won’t they?
CRob (12:38)
Hope not.
Vincent Danen (12:40)
Their job is to make sure that nuclear energy is run safely and securely and…all of those sorts of things, their job is not to build an operating system or to maintain it, right? So they’re going to be going to a commercial enterprise open source vendor, and they’re gonna be going for that. So the notion that we’ve had for probably the last 20 years is, I don’t wanna call them cabals, because that sounds bad…
But like groups of people who dealt with embargoed vulnerabilities representing major open source vendors, that basically account for what I would say is 80, 90 % of the critical infrastructure that’s being used. Red Hat is one of them, but it is by no means unique. We would share information because we had these tenants, like I remember when I was back at Mandriva and I was working with Red Hat and I was working with SUSE and I was working with Debian, security was one of those things where we did not compete. We agreed very much that it is table stakes for everybody. Our users deserve better than just like, I’m gonna get this fixed out faster than you because I’m not going to tell you about it. And this is my competitive advantage. None of us thought that. And I believe today that still rings true. Don’t believe that. That’s part of the open source community that works. Right now, what I’m seeing is people are not disclosing to those channels and they’re just making things available online, right?
I don’t know when this is going to go live, so I don’t know in reference to time, but like last week is copy fail, that came out and that was just public and a lot of enterprise open source vendors were caught flat-footed because it was just disclosed, there was no coordination, right? Now I’m not saying that the Linux kernel is the only place that this is happening, it obviously isn’t, this one just happened to be pretty remarkable in terms of its publicity, right?
And I just think this is going to get worse unless we figure out how to make this better. And we do have those mechanisms, but it’s, I don’t know if it’s an education thing. I don’t know if it’s part of like some of the stewardship in, you know, CRA land, you know, where foundations are involved. But of course not all open source communities have foundations.
So it’s really complicated, but there’s a few places where people have traditionally gone, or they’ll go to a vendor, an open source enterprise vendor and report, like this happens to Red Hat all the time, people report vulnerabilities to us, and we will work with the community and kind of do that middleman coordination to make sure that everybody has a fair chance of understanding the vulnerability, contributing to its fix, and then having a standard date for when these things go public, right?
CRob (15:36)
Yeah.
Vincent Danen (15:37)
And I think we have to work harder at that because if we don’t, we get more copy fail like things where it’s like a zero day that shouldn’t have been.
CRob (15:48)
I have a personal hypothesis about this in that much like development today, we have a lot of non-traditional people kind of dabbling in the vulnerability disclosure space and they don’t have that experience or that understanding of how things work in the ecosystem. And that’s one of the things like I can say that we are working to try to help educate and provide these consistent means because it’s important. The whole idea about coordinated vulnerability disclosure is that everybody involved has the ability to have some type of solution so that when it goes public, the end users have the ability to get treatment and make themselves whole.
Vincent Danen (16:29)
Yep. Oh no, 100 percent. And I think it’s just going to get worse with the, I mean, you’ve seen what it looks like the last week or two. If you look at like a CV details and the number of CVEs coming in week over week now, like I think someone was telling me at least when it came to like Red Hat, just with some of the vulnerabilities that were like, we’ve almost eclipsed what we got all of last year already. Right. So I mean, like this is insane. And…
CRob (16:51)
already. Wow.
Vincent Danen (16:52)
Already. Right. So I mean, like this is insane. And
Like the coordination around all of this, like as much as I like to say AI is useful for a bunch of stuff, like it’s really great for finding things, right? AI doesn’t coordinate disclosure. People have to do that, right? So there’s still a, mean, people are wondering about human loop and value of security researchers, like that’s it, right? Part of it is the relational piece, the coordination, things like that. Like AI is good at finding it. Hopefully it’s good at fixing it, but we have to be able to coordinate and then everybody else has to be able to apply these things and everyone’s personal acceptance of AI is different as well. So do humans want to validate things?
I’m sure that some upstreams will want to do that. So you have to account for that as well. But it’s just the sheer volume is going to be the hard part. And I think people will get frustrated because, I’ve submitted my 3,800 CVs or CV, potential CV reports. No one’s getting back to me. So I’m just going to go dump these on Twitter or X now.
CRob (17:53)
And since you invoked the name of our shared friend, let’s pivot a little bit and talk about how AI is changing the economics of vulnerability discovery and not necessarily focusing on the economics of vulnerability response. So from a PSIRT perspective, which is a product security and incident response team, which is the organization that you represent within Red Hat.
How do you today kind of distinguish between something that AI helped with the research and versus the volume of just AI slop where you’re just getting a huge volume of reports and how can we help maintainers with this kind of situation?
Vincent Danen (18:40)
Yeah, I think that the usage of AI slop is diminishing, right? Like last year, 100%, lots of garbage was floating around. From what I’ve seen, a lot of the reports and things that even we’ve experimented with, it’s actually really good now. So there’s less slop, more true positives, those sorts of things, right? I think it’s going to get to a spot where there’s…
There’s a difference between the quality and the quantity. I think the quality has definitely gotten better. I would say as a recipient of a report, I would be asking for a proof of concept as well. Don’t just generate a bunch of things and then just shoot it over the wall. Years ago, people used to do that with fuzzing, They’d fuzz image magic or something like that, and then give us a tarball of all these fuzzed files and be like, hey, we think there’s some problems here.
Like, thanks, could you please point them out to me? Like, don’t just give me a million fuzzer files of random input and then like some of them crash and some of them don’t. Like you gotta weed through some of this stuff yourself first, right? So the notion of like test it first, like actually be able to reproduce it, make sure it’s valid is really good. Particularly since I think a lot of people are using AI tools. And this is my hypothesis. A lot of people are using AI tools right now to submit bugs to bug bounty programs and get their payouts. Right? In fact, we’ve seen some bug bounty programs close because they’re like, well, I mean, you guys are just going to be using AI for this stuff, not vetting it at all. We’re just going to stop paying these things. And I actually think we’re, I think the days of bug bounty programs are done.
CRob (20:28)
I think it’s definitely limited. would agree with that.
Vincent Danen (20:31)
You know, because all we’re doing then is we’re giving people money and they’re investing in these tokens to do all this work, knowing that they’re gonna get this payoff, right? And the incentive isn’t there. Now I’ve always been a, prior to AI, I’ve been opposed to bug bounty programs, just because if you’re gonna be doing this for open source, this is called giving back to the community, right? Like you shouldn’t get paid for this. You should like, you find, I find a bug in a piece of software, I feel like it’s my duty as the end user of that software, the recipient of that value that it gives me to give that to the maintainer, whether it’s a security bug or regular bug, right?
CRob (21:10)
Mmmhmm. Well, and that’s an interesting…we’ve heard a lot of feedback from the upstream community around the volume of these things. And thinking about it, unlike older tools like static analyzers, which sometimes required very expensive licenses for, and only, that was kind of the dominion of commercial enterprises, with these new large language models.
Some of them you could do open models, local things for free or low cost, but then even the well-known name brand frontier models, you can get in at $20 a month. So the economics for the research, the finding piece is greatly reduced. So, outside of a proof of vulnerability or concept, what else would you recommend for people that are trying to use these tools and are hoping to…build a relationship or at least have a good interaction with upstream.
Vincent Danen (22:07)
Yeah, it’s honestly vetting it, right? Like don’t just find it. Excuse me, find it and vet it. Like you gotta put a little bit of elbow grease in there because if you don’t, at the end of the day, you’re just telling my guys like, you gotta do it. Or you’re telling upstream, you gotta do it, right? And if you’ve got 100 different people submitting basically the same vulnerability, because I mean, we’re seeing that now, right?
CRob (22:29)
Mm-hmm.
Vincent Danen (22:31)
Interestingly, for a very long time, I’ve seen, you know, people will report a vulnerability and a month later, somebody else completely random, completely different reporting the same vulnerability. So this is not a new phenomenon exclusive to AI. Like it happened. It’s just the scale and speed of it is everything that’s speeding up. Right? So if we’re going to be dumping all of this stuff on maintainers or vendors or any other recipient of this, you got to do your due diligence. Like they’re going to have to do some level of human effort to triage these things.
I think it’s our responsibility and job to figure out how do we use these tools to help with that triage to make it faster, right? I was actually having a conversation with someone this morning, the same kind of idea, right? Where they’re like, well, I think that humans need to validate all of these things. And I’m like, I agree to an extent, but if we’re being flooded with this stuff, we literally can’t because then we’re all sitting on backlogs of stuff that we’ll call embargoed, but isn’t actually embargoed because somebody else is finding it and reporting it somewhere else.
And then we get into that whole, just even the coordination of CVE names, nevermind the coordination of responsible disclosure. So we have to get better at, like, how do we do rapid vulnerability assessments or judging the validity of a report with another agent or a series of agents? There was a talk, I actually think it was at the SOSS Fusion event maybe two years ago.
And someone was talking about this idea of AI decisions made by consensus, right? Where you have like three or four different agents that are designed differently, trained differently, looking at the same problem. And if all four of them or three of them or whatever your panel is agree that it is a thing, then you go do that thing.
CRob (24:22)
That’s interesting.
Vincent Danen (24:23)
If you have one of them disagree and it has valid, then it triggers a human review, right? And I think that if we…build systems like that, then we’re able to get through that backlog, do the triage quickly and reserve the things that maybe the models aren’t confident on. Then we get a human being to look at.
CRob (24:42)
Mmmhmm. And that’s been one. The great thing about open source is all the source code is freely publicly available that you can look at it, make changes and submit it back upstream. Bad thing about open source is that all the source code is publicly available and anybody can look at it and make changes,
Vincent Danen (25:00)
Yep.
CRob (25:01)
Which again, you mentioned that the kind of the duplicative reports has always been a problem. And even thinking about like the dawn of fuzzers, how all these tools create a large volume of information. You get a lot of data, but kind of identifying the wheat from the chaff is hard because you might have a critical vulnerability hidden inside of thousands of either knot bugs or very low things that would yield very little outcome for an attacker.
Vincent Danen (25:34)
Yeah, but these are the same problems we’ve always had, right? It’s just a tool that’s different, right? If I look at like a SAS T-scanners, right? I mean, my goodness, there’s so much noise in those things. It is a needle in a haystack trying to find a true positive, a true vulnerability in these things. You’ll find a lot of weaknesses, great defects, like, yes, we should improve the code and whatnot. But most of them are like, just, they don’t mean anything, right?
And so a lot of people get frustrated and I find a lot of these scanning tools run for compliance reasons, not for actual security reasons. Right. And this is actually one of the things that makes me really happy about some of these advances in these AI tools is like we can actually zero in on that signal and get rid of a lot of the noise, especially now, right? We’ll call AI slop noise. And there was some signal buried in there, but there was a lot of slop around it right now that that stuff is basically being fixed, it’s drained off, it’s better now. And I expect three
Three months from now, it’s going to be better. Six months from now, it be even better. A year from now, it’ll probably blow our minds because these things just keep iterating, and they just keep getting better. So we’re going to end up having better tools. I don’t think SAS tools have gotten better over the last decade. They’re basically the same thing. And now we have people talking about using AI to filter through SAS findings. Well, what does that tell you? Now we’re going to use AI to figure out how much of this is signal in here.
Now we’re going to use AI to figure out how much of this is signal.
CRob (27:01)
A tool to fix my tool.
Vincent Danen (27:03)
Totally, totally. But now we don’t even need that other tool because the tool that would have fixed that tool is actually doing a better job. Right. And so I actually think this is great. We just have to ride the wave and wait for it to come down, right? And normalize a little bit.
CRob (27:20)
Right. Well, and one thing that kind of offered a glimmer of hope to kind of improve the signal to noise ratio was one of our favorite topics, software bill of materials.
Vincent Danen (27:35)
Mm.
CRob (27:36)
And it was awesome. Like when it was initially presented, governments jumped on it and put requirements out. seeing with like the EU CRA, SBOMs are part of that requirement. But from upstream,There’s limited utility for an upstream maintainer to, they’re rarely if ever gonna use an SBOM. But as you work your way down the supply chain it becomes increasingly more important. From your perspective as one of those, you’re a downstream from upstream. What metadata from your perspective is required for people like you to be able to make these risk-based decisions based off of the…packages that you’re sharing and supporting for your customers.
Vincent Danen (28:23)
Yeah, I mean, it’s for me, it’s the basic stuff, right? It’s the name of the package, it’s the version, it’s a release. A git commit, right? So that as I’m traversing git history, I know like this version equates to this git commit and I can see, because then I can actually start trolling through the git commits for fixes and stuff to know which versions are affected by things and whatnot. And I think there’s some, I can’t remember who was talking about this at Vulcan a couple of weeks ago, but there’s a really neat, maybe it was the folks from Google, but like some very neat get graphing things where they can actually start to determine, because it git is messy with all of the branches and commits and forks and everything else. It’s like, what’s that window of risk? When was it introduced? When was it remedied? How many branches are in there so we can see if we’re on this branch, was it affected or not, right? That’s the sort of information that was useful for me as a downstream reproducer, right?
There’s a lot of stuff that is useful from like a legal compliance perspective, like licensing and whatever else. But I mean, I think fundamentally, folks who assemble a collection of open source don’t necessarily need an SBOM from upstream, right? Because they’ll be assembling all of these things. if I look at it from my perspective, I produce an SBOM for RHEL, like for the operating system. I don’t produce an SBOM for OpenSSH or an SBOM for OpenSSL, right? That’s a component in my SBOM, right? Because nobody’s looking for you know, where is ssl.c in my compiled code, right? They’re just looking for like, where’s my open ssl library? Where do I have it all over my estate, right? The difference will be for some of these, and different ecosystems are different, but like the ones that come to mind are like npm, where you take all of these different dependencies for a project.
and you smush them all into one file to minimize it. And I don’t really, can’t tell what’s in there or what’s not. SBOMB for that would be really helpful because I might need to fix a vulnerability in one of those dependencies that I’m as a downstream, I’m just pulling the one package that’s inclusive of a bunch of other things. Go is another example, right? All of these things shoved into one binary and like there are tools to be able to determine what it was and whatnot, but I have to go look at the binary itself.
That means I can’t do an analysis if I have like an SBOM aggregator tool where I’m like, where is this Go module used in my estate? I can’t go and interrogate all these Go binaries all over the place. Like I just, want like a text file that tells me these things, right? So I mean, like for some upstreams, SBOMs would be super, super handy. For some of them probably doesn’t matter so much. Really critical for the producers.
And also really important that, you know, end users actually start using them. Like there was a promise for SBOMs that has never been realized. And it’s been like a decade plus, right? Certainly the impetus on it for the last five, six years, right? And I don’t see that it’s actually been well used other than saying, I have my SBOM, my compliance box is ticked, but we’re not actually using it for anything.
CRob (32:06)
And that’s been, I would say we’ve been involved with this SBOM space for almost a decade now. There’s been a lot of focus on producing it. Make the SBOM and you’ve got a lot of different permutations and you can go down and do transitive and transitive dependencies. But there’s been very little focus on using these things, aggregating it and then kind of extracting wisdom as a operator. Like how do I find out exactly where I need to go jiggle the handle or deploy a fix somewhere.
Vincent Danen (32:37)
Yeah, when you said jiggle the handle, I was thinking of a toilet. Right?
CRob (32:40)
Right, exactly. We’ve got to flush out vulnerability.
Vincent Danen (32:43)
But it means it’s true. Like to date, topics around SBOMs have been largely engineering focused or academic. Nobody’s putting on their UX hat and going, how would a user use this? What are the tools for this? Like if I can give a personal story that will throw a Red Hat a little bit under the bus. Yeah, I mean, it’s okay. Like they’ll work for me, so it’s fine. Like when we were producing VEX documents,
CRob (33:11)
Mm-hmm.
Vincent Danen (33:11)
right? Like we created them to the CSAF VEX standard. We published them. We did all of these things. I’m like, this is awesome, you guys. I love this. What are we using that a customer could use to actually interpret these files? And the answer I got back was like, I don’t know. I mean, you could go to CSAF and see if they have any tools. I’m like, are you kidding me?
This is the answer, like you guys engineered a solution that, I mean, I was not a believer in it at first, but they proved me wrong, right? It happens, and I’m glad for it. But then they weren’t giving me the tool to use it. So then I went off and I was like, well, forget this, and I wrote my own, right? So now there’s a VEX reader Python module that I wrote
CRob (33:54)
Oh nice
Vincent Danen (33:54)
That would actually do that because I’m like, how do I tell a customer to use this? And we’re like, here’s a document. And they’re like, okay.
CRob (34:04)
and
Vincent Danen (34:05)
And it has all the information that you want, period, run away. That felt a lot like the way we thought about SBOMs. We’ll create the SBOM, like, yeah, you can go use the SPDX tooling for that. Okay, I mean, that’s great. So I can take some JSON and spit it into text. I can grep around it, I guess. What else can I do with it? And we just, never told them.
Right, which made it of limited use.
CRob (34:34)
Mmhm. Yet it got baked into a lot of international regulations and requirements selling to governments.
Vincent Danen (34:43)
Right, because it sounds good and it is the right thing to do, but there’s so much more that can be done with it other than just having it. Right, like the mental picture I always have is like, here Mr. Mrs. Customer, here’s your SBOMB. They’re like, thank you. They throw it in the filing cabinet and they walk away. Right, like we might as well just print it out on paper for all the good it’s gonna do most. Now, I mean.
I don’t want to brush every single customer with the same, or like some of them are probably using it, but like as an industry, I don’t see SBOM usage being anywhere near what it should be.
CRob (35:29)
No. I agree.
Vincent Danen (35:21)
Like we should, we should be able to tell as like users of enterprise software or open source software, like I know precisely where in my, especially with containers…
CRob (35:33)
Mm-hmm.
Vincent Danen (35:33)
Because that has exploded the size. And we used to talk about what our real estate looked like, right? And now It’s like tent cities, not skyscrapers, right? Like there’s things everywhere. And it’s like, where are all the things? Like I see this with vulnerability scans for like Kubernetes clusters. And they’re like, my God, there’s like thousands of vulnerabilities here. It’s like, well, actually there’s dozens. They just show up a hundred times because you have a hundred containers, but you don’t normalize your list, right? And like, those are the things that we…
You need to know where these things are and it becomes that much more important in this containerized world.
CRob (36:16)
Mmhmm. So an interesting adjacent topic. Let’s talk about exploitability. And I’m in a container, I’m on bare metal, I’m on a phone. Exploitability can mean different things to different people. Could you maybe think about from your perspective when you’re given instructions by global regulators or
questions from customers. When you’re thinking about exploitability, what does that mean to like you and your crew?
Vincent Danen (36:48)
Yeah, this is a topic of hot debate, right? Because in theory, in theory, academically, every vulnerability is technically exploitable, right? Because if it wasn’t technically exploitable, it wouldn’t be a vulnerability, it would be a weakness, or it would be a bug. Our philosophy, and we are by no means alone, is that they are not all created equal.
Right, so like this CVE, that CVE, and this CVE are not all the same. Some will be remote code execution without authentication. Some will be local privilege escalation. I need to have shell access first. Right, so we have to start there when it comes to exploitability. In some of these things, it doesn’t matter if it’s exploitable. Right, if there’s sufficient security context and constraints around it.
Like I always say, nobody runs their there’s banking systems naked on the internet, absent any web application firewalls or traditional firewalls or anything, because they wouldn’t be in business long, right? So we have to assume there’s a certain moat of some size. And I know that some people, I saw this on LinkedIn the other day, some people were like, well, the walls don’t work. We have to adopt zero trust architectures and assume that attackers are in there. I mean, that’s true to an extent, right? Because you have insider threats, you’ve got…secretaries clicking links that they shouldn’t be and fishing is still the number one source of breaches. So I mean, that tells you that that’s not a software vulnerability. That’s a human thing, human trickery and susceptibility and then misconfigurations. Those are probably the two biggest things for exploitation. Software exploitation is, I mean, it’s getting worse. If you look at like Verizon’s DBIR, the numbers are turning higher in terms of software exploitation, but it’s still not necessarily easy. And when you look at things like a local privilege escalation inside a container, right?
When you’re in a container, you’re not expected to have shell access, right? So if an attacker can get shell access into your container, isn’t that the problem you should be solving? More so than the privilege escalation within the container and provided it is not a route privileged container, still not gonna get you a ton of privileges, right? So like these are some of those security contexts that are around things or privileged container.
So when we’re looking at it from exploitability, there are so many different factors, right? Like bare metal versus container versus a virtual machine, right? Like how far down the stack can I go? Can I do a container escape or break out of a virtual machine if I’m already on the bare metal server itself? Like all of those different considerations. I think the worst thing in this is gonna be really something to pay attention to in the age of AI, right? Because…
If we do get a thousand vulnerabilities a week when we used to get a hundred, right? We’re looking at a 10 X increase. We can’t treat them all the same because nobody can patch a thousand every week, right? And I’m not talking about the vendors. I’m talking about downstream consumers, right? Cause everybody I talked to, they’re not, they don’t do updates in YOLO mode. They want to test this stuff first, which means people have to fire it up, download it, test it, do all their things.
and propagate it to production, monitor, make sure it works properly. And then they can’t just turn around and be like, there’s the next patch and do it again. Nobody’s business model is to apply patches. Their business model is to run their business. They have to do their security due diligence, of course, but that’s not their job. So we really have to be looking at things in terms of what is the actual impact of exploitation for this thing, and then decide what are we going to do…
Like the old promise of CVSS prioritization, not risk. Right? Exactly. Exactly. So like, it’s a it’s a guide. It’s useful from given 1000 things, which ones do I attack first, maybe CVSS is the right way to do it. Well, you know, within like, those that are critical, which ones have the highest CVSS, I work my way down, then I’m back to the importance or whatever. Or I’m looking at it in terms of my business context, which is another reason why SBOMS are helpful. Like,
CRob (40:56)
Mm-hmm.
Vincent Danen (40:57)
Prioritization, not risk. Right?
CRob (40:59)
Exactly. It’s the string, not the number.
Vincent Danen (41:02)
Exactly. Exactly. So like, it’s a it’s a guide. It’s useful from given 1000 things, which ones do I attack first, maybe CVSS is the right way to do it. Well, you know, within like, those that are critical, which ones have the highest CVSS, I work my way down, then I’m back to the importance or whatever. Or I’m looking at it in terms of my business context, which is another reason why SBOMS are helpful.
Like…What does this vulnerability mean to me? Where is it deployed? Is it only deployed in a testing environment or a QE environment? Maybe I don’t care as much unless it’s in production. Like I was talking with a customer the other day and there are probably going to be times when we have to apply these fixes in production as soon as they’re available.
CRob (41:43)
Yep
Vincent Danen (41:45)
How do we know which ones to do that for? And I’m like, your business context?
CRob (41:)
Mmhm
Vincent Danen (41:45)
You knowing what you’re running? And then I would be going for the ones that we label critical first. Like at the end of the day, that’s it, right?
CRob (41:59)
Yeah. It always amuses me that we still today there are, and it may be a researcher trying to break into the industry, or where people make a giant deal out of a vulnerability that you have to be root to do. A root user could install a malicious package. Yes. Right, they could do that anyway. They don’t care.
Vincent Danen (42:16)
Yep. It’s like, cool. They could do that anyways.
CRob (42:19)
Right. They could do that anyway and they don’t care.
Vincent Danen (42:21)
Oh yeah. No, mean, and it’s, and it’s funny, right? And we’ve, we’ve, you’ve been doing this long enough. I’ve been doing this long enough. There’s like, there was the one ghost script vulnerability that had the cool song and I can’t remember what the name of it was, but it had the, song and like the logo and you know, everyone’s trying to make a name for themselves. And I mean this with the greatest respect for the security researchers. Some of them do fantastic jobs and give you the facts.
CRob (42:47)
Mm-hmm.
Vincent Danen (42:47)
Some of them hype it up because it looks good on a resume if they found a dozen critical vulnerabilities. And then turns out that they’re actually low, right? But they’re gonna hype it up and they’re gonna get everybody’s attention because they don’t, a job interview depends on it or whatever, right? Like, I don’t know.
CRob (43:04)
And exactly as you described, the risk that a downstream user incurs is solely based on their context. They can have firewalls or processes and monitoring. They can have a lot of things that would reduce that down to a critical, yes, we need to fix, but it’s not something I need to shut the whole business down today to go install a bunch of patches.
Vincent Danen (43:28)
Totally, totally. And it’s really important to have that context. Otherwise, due to sheer panic, people will make poor decisions like, I read this thing on the internet and now I’m downloading some random code that purports to fix it. Because my vendor isn’t fixing it fast enough. Because you know your vendor is actually testing these things before they ship it out to you, right?
CRob (43:49)
Testing it exactly.
Vincent Danen (43:51)
So that you don’t have to do as much testing yourself. We’ll make sure that we don’t bork your system when you do the upgrade. But the alternative of like, oh, Billy Bob said, you know, if I run this Ansible playbook or this script, that’ll fix it. And I’m a panicked, unskilled person and I go run it. That’s the stuff of nightmares because you might have just created a bigger problem for yourself, right? So like, how do we used to call this? Like the sensibility, calm, bringing calm to panic and stuff like that seesaw program we had customer security awareness that we used to do.
CRob (44:28)
Mm-hmm. Yep, the seesaw, yeah, Again, giving people the information so they can make decisions on their own based off of their own context and data.
Vincent Danen (44:37)
Yeah, informed decisions.
CRob (44:40)
Mm-hmm. Speaking about interesting new big things, it’s 2026, and this is the year where our friend, the EU Cyber Resilience Act is coming into effect. Where there are some vulnerability reporting requirements that are in place for manufacturers like Red Hat and others. So, um thinking about that, how can PSIRTS or upstream, downstream, how can we all coordinate so that when there is something that falls into that threshold that you need to report and react to, that exploited vulnerability, how can we work together to try to help? Everybody in the supply chain to react to these things so that we all can achieve the regulatory goals that the EU set out, for example.
Vincent Danen (45:31)
Yeah, I mean, I don’t know. I mean, that’s it. That’s my honest answer, right? Because prior to all of the growth and vulnerabilities discovered by AI that we’re not slopping, we’re actually good, right? Like, it’s just like, we’ve to find the right avenue, you know, whether it’s like SIRTS, Vince platform, or there are some existing mechanisms to do this. So we should probably still try to use them as much as we can. But with the volume part included in that, like, I mean,
Can Vince, not me, the other Vince, search Vince, can it handle a thousand a week? 10,000 a week. Or 10,000 not every week, but one week, right? Like can it handle that level of coordination? I literally don’t know. So like there’s going to be challenges there that I think we’re, like we’re going to stumble and trip and figure some of these things out as we go because we never design most of these systems for scale.
CRob (46:30)
Yeah, correct. Well, and then again, the whole exploited aspect of it, we’ve all in IR, we’ve always thought about, you know, if something is being if we have evidence, something’s being actively exploited, you know, we throw all the switches, we all hands on deck, it’s an emergency. But, those were rare situations. had a couple a year maybe where something was being actively exploited. But now, theoretically, just the language, anything could be exploited, as you mentioned. And now we all, some of us are on the hook for more consequences than others of us. But how are we gonna work together to kind of, again, all participants in the ecosystem?
Vincent Danen (47:17)
So I’m gonna throw something out that’s maybe a little controversial.
CRob (47:20)
Woo!
Vincent Danen (47:21)
I think rather than putting the burden on vendors to go report to a government body that their software is being exploited, right? Because, I mean, we might know, but it literally depends on a customer telling us, right? And maybe they have other things that they’re focused on if they’re being exploited, right? If they’re suffering a breach because of some exploited software.
Services out there like CISIS, Kev is one. Vulncheck has a Kev that’s a little bit more timely, a little more comprehensive than CISIS, right? There are sources out there, right? Gray Noise, Accorded Future. There’s a bunch of other places that governments could be going to for that information like, hey, give us the signals of things being exploited, right? I don’t know that every vendor can do what…Those guys do, because that’s their business.
So for me, it feels like maybe you’re asking the wrong person to do this. And I understand there’s the accountability aspect that they want and is appropriate for vendors. But to tell you, the moment something is being exploited, I may literally not know. VulnCheck may know before me. And in fact, I might be finding out from VulnCheck, because it might not be my customer who’s using this piece of open source software. It might be some other customer of some other vendor that’s using it, but I still ship that piece of software and may be also vulnerable in that way. So I depend on these services to tell me whether or not I should be paying attention to something or fixing it faster. To me, that feels like those are the people that should be feeding those government agencies awareness from that perspective.
Now it’s a little different when it comes to a security incident, right? Because that’s very much like something happened to Red Hat or something happened to SUSE or something happened to Fedora or some upstream community, right? But if it happens to me, then I know about it and I can tell you. Exploidable open source that a whole bunch of people ship, I may not know and might not be the first person to tell you. And then are they going to sit there and take on that coordination effort to say like, hey guys, XYZ open source vendor is reporting exploitation of this piece of software, are they gonna go back to every other manufacturer and tell them? Like I don’t think they’re gonna take on that level of coordination. So it makes me wonder, why do we have to do that in the first place? There are better ways to do this, right? That’s my hot take.
CRob (49:55)
I like it. Well, and again, Vincent, this was an excellent conversation on some very timely topics. As we wind down, any final thoughts you want to share with either upstream maintainers, downstream consumers, or your peers?
Vincent Danen (50:12)
Yeah, yeah, one, please let’s not fracture the CV ecosystem, right? I have lived through all the different IDs, meaning the same thing. I never wanna go back to those. Those were the dark days, the dark ages of security disclosures. I don’t wanna go back to that, right? The other thing is I think we have to figure out how to make coordinated responsible disclosures a thing again, because…
It doesn’t matter. Like, again, it’s not a competitive nature between companies or even upstream downstream, right? Like we’re not trying to make upstream look bad. Upstream shouldn’t necessarily be trying to make us look bad. Open source is being used and we’re all happy, right? Like that’s why we do it. It shouldn’t be a competition because there are literally tens of millions of systems that depend on us being the adults in the room and doing this stuff the right way. And so I don’t know what that looks like, but I think we have to figure it out and I actually think we have to figure out.
CRob (51:19)
I agree. Well, I really look forward to collaborating with you and our other ecosystem friends as we kind of think through some of these things.
Vincent Danen (51:27)
Yeah, me too.
CRob (51:29)
And with that, this is a wrap. I want everyone to stay cyber safe and sound out there. Thank you, Vincent, for all the work you have done and all the work we’re gonna get to do together. And I hope everyone out there, happy open sourcing.
Vincent Danen (51:42)
Awesome, thanks, CRob. Thank you.
