vulnerability management

00:03 Welcome: Host CRob introduces Linux kernel legend Greg Kroah-Hartman.
01:04 From Printers to Richard Stallman: Greg shares his origin story in embedded engineering and his introduction to free software.
02:00 The Weekend Driver and the Dopamine Hit: How a quick weekend project turned Greg into a lifelong kernel contributor.
04:20 Realizing Linux is Critical Infrastructure: The moment the telcos and banks moved in, signaling that Linux was everywhere.
05:28 2005: The Year the Kernel Grew Up: Implementing stable releases, the security team, and the rule to never break user space.
07:05 What People Get Wrong About Kernel Security: Linus’s mantra that “a bug is a bug,” and the reality of handling 35 fixes a day.
09:32 The Historic Fear of Updating: Why lagging behind for “stability” is actually incurring massive risk.
12:17 Global Regulation and the Cyber Resilience Act (CRA): How upcoming laws are changing vendor responsibility and why the open source community is exempt.
13:51 How OpenSSF is Reducing the Burden: Applauding the cross-ecosystem collaboration that protected maintainers from onerous drafts.
17:05 Pay Your Employees to Contribute: Greg’s best piece of advice for downstream enterprises relying on open source.
18:39 Inside the Kernel Security Alias: How the ad-hoc security team handles triage and assigns CVEs.
21:11 The CVE Numbers Game: Why the kernel ranks #2 in CVE creation and the trouble with automated severity scores.
25:39 We are Already There: AI Slop and Static Analysis: Greg addresses the recent surge of AI-generated bug reports and patches.
31:20 Rapid Fire Round: Spicy food, coffee, Star Wars, and the ultimate call to action.

• Greg Kroah-Hartman’s LinkedIn page
• The Linux Foundation CRA Stewards Playbook
• OpenSSF Global Cyber Policy Working Group
• The Linux Kernel Archive (Keep your systems current with the latest stable kernel releases)
• The Value of Open Source Software
• Get involved with the OpenSSF
• Subscribe to the OpenSSF newsletter
• Follow the OpenSSF on LinkedIn

CRob (00:23)
Welcome, welcome, welcome to What’s in the SOSS, the OpenSSF podcast where I talk to maintainers Contributors security researchers and just anyone in and around the amazing world of open source and open source software security Today we have an amazing treat One of my open source idols somebody that I get the opportunity to work with pretty frequently now I’m quite excited today. We have Greg Kroah-Hartman one of the kernel maintainers with us today. So Greg, welcome to the show.

Greg Kroah-Hartman (00:55)
Hey, thanks for having me.

CRob (00:58)
Yeah, it’s amazing that we get to interact with folks such as yourself. This is a real treat for us.

Greg Kroah-Hartman (01:04)
I’ve been interacting with you quite a bit over the years.

CRob (01:07)
I know. So thinking about it, how did you get into open source? Back in days of yore when you were, I’m assuming you started off as a software engineer of some type. How did you get into open source proper in the kernel?

Greg Kroah-Hartman (01:23)
Yeah, I was embedded. I did embedded work. So I wrote firmware for printers and barcode scanners and ATMs that dispense drugs in hospitals, fun things like that. And we had to use Linux for some of our devices, some of the companies I worked for. But even earlier than that, when I was in university, my girlfriend at the time, who’s now my wife, had this class that would have a visiting speaker every class.

Greg Kroah-Hartman (01:50)
And she was a political science major. was computer science. And she came back. was like, I heard this guy came and spoke about something called free software. His name was Richard Stallman. This was the early 90s. And I was like, oh, that sounds very strange. And then I ran into the GCC developers at a conference.

Greg Kroah-Hartman (02:10)
And GCC was putting out a compiler for embedded systems. That was free. And I was like, how in the world is this business model working? This is crazy. I told them that, and they’re like, well, we’re selling it pretty well. And then I actually used it on some of my products that I had worked on. And I’m like, this compiler is better than the stuff we pay a lot of money for. So free software turned out to be working pretty well. But then years forward, I worked on the USB standard. And I was writing firmware for USB devices.

I had a barcode scanner that showed up as a keyboard. And my Linux had a little bit of support for USB, not much. So started testing my firmware and all sorts of different operating systems to try and make sure it worked. Under Windows, it worked fine because it had USB support. Linux, it showed up as 102 key mouse. I was like, what? It turns out that was my fault in the firmware because the kernel was being very strict on what accepted Windows would be very loose on what they accepted. So I was like, I’ve fixed some bugs on the stuff and then it was fun. I got the driver book and played around with it some more and then played with it on the weekends. One day my wife was going away with my daughter at the time to visit some family and she said, write that driver for that device you’ve always wanted to do. I’m like, okay. So I spent the weekend, a driver, submitted it. And then I instantly got back responses like, this is wrong, this is wrong, this is wrong. And if you heard of this thing called multiprocessors, I was like, what?

Greg Kroah-Hartman (03:34)
What do mean locking? Two processors at once? This is crazy. But it was great. It was like a dopamine hit of like, these are really good engineers telling me what to do and how to make my code better. And I was hooked at that. And then over the, I started writing drivers and eventually realized that more people were using the stuff I gave away for free on the weekends than were getting, using the product of the companies I worked for. Not that they weren’t that successful, but they weren’t that good.

CRob (03:45)
Yeah. That’s awesome.

Greg Kroah-Hartman (04:00)
Soon then I got a job in the early 90s, or late 90s, doing Linux full time, working on the security model for Linux. Funnily enough, I was one of the architects of that. And I’ve been paid ever since for that.

CRob (04:14)
Great. So you’ve been working on the kernel for how many years now?

Greg Kroah-Hartman (04:21)
I think 25, 24, that’s long time. Yeah, long time.

CRob (04:22)
20, yeah. So kind of reflecting back over that long time within this ecosystem, was there a specific moment where you kind of realized that, holy crap, what I’m doing is critical infrastructure for the planet?

Greg Kroah-Hartman (04:40)
Way too early. In the late 90s, the telcos adopted Linux first. They’re seen as like the stodgy companies, but they were the leading edge and they adopted Linux and they had this big telco initiatives through what was called OSDL at the time. And we’re butting heads with the telco people and we’re like, what are you using Linux for? They’re like, oh, it’s running our digital infrastructure. We’re like, what? So they were one of the first users and really showed that it could really be used for backbones and networking and stuff like that. And I was like, oh, wow. And then you hear things like, banks for throwing out all their old sun systems and using it and then the stock exchange and then air traffic control but uniquely it showed up like in a programmable like thermostat in my mother’s house once and I was like this is weird so then I realized yeah okay we’re really everywhere this is gonna stick around for a

CRob (05:36)
Yeah.

CRob (05:41)
Did that change how you approach security decisions as you were doing your work?

Greg Kroah-Hartman (05:48)
Yes and no. mean, we took, it made us take things more seriously. So once the banks and other people got involved, we we started taking things a lot more seriously. Like 2005 is like the year we grew up. We got instituted a security team. We instituted the stable way we did releases. We instituted the rule, maybe the year before that, we will never break user space again. We instituted time-based releases before that so people could rely on us. So we kind of grew up between 2003 to 2005 in that way. there’s a big tipping point at that point in time that we realized we got to do this right. People are relying on us and let’s come up with an engineering process that people can rely on. And that’s why I like Linux, because it’s one thing to write an operating system once or software once. It’s another thing to maintain it over decades so that people can rely on it and it will survive and grow and evolve over time because the world evolves. And that’s a fun engineering project and a fun engineering task. And I don’t think people… give us enough credit for Linux being something like that. It’s pretty unique as far as software systems go.

CRob (06:52)
Yeah, I would agree. So thinking about this kind of sense of responsibility that so many derpy things like thermostats and whatnot depend on you, but so many critical things are dependent on the Linux and the kernel. And securing things at this scale is very different than just securing a web server or a point of sale system.

From your perspective, what do most people get wrong about kernel security?

Greg Kroah-Hartman (07:22)
Two things to get wrong. They got wrong. One that they think we know what all the security bugs are. It turns out a bug in our area is a security bug. So Linus’s old mantra of a bug is a bug is a bug, let’s fix it and move on is more true than ever. I don’t, so open source, you listen to a few different things.

Open source is unique in that we can’t dictate use, or used in satellites to cow milking machines, right? Some are more critical to others, but those cows would be pretty upset if you got that wrong in there too. bugs at our level should be fixed and pushed out to users and they should update. The trick is I don’t know if this bug is applicable to you or not.

Greg Kroah-Hartman (08:09)
And that’s the thing. You use this, I’m not forcing you to use Linux, so you should just take our bug fixes and move on. do that. I remember sitting in a meeting once with some big company, they’re like, insisting, well, you need to tell us everything. All the bugs that you fixed that are applicable to us. I’m like, I don’t know what you do. I know you must take them all. You must take them all, or you must tell us. I’m like, I’m giving them to you for free. Somebody else corrected, like, the community is giving you what? 30 bug fixes a day for free and you want them to do more work for you? It’s like, this is a very odd thing to be demanding. That’s the way it is. So take our bug fixes because we know they fix something. And if they don’t fix something or if they break something, because people are always worried about regressions, we’ll fix that. But it’s much better to take a fix for a known issue than worry about a potential for an unknown issue. That’s the thing that people need to get the whole bitch to take.

And also look and test. mean, we are 40 million lines of code, but you only run a very small portion of it.

Your laptop runs two to maybe three million. Your server runs two million, maybe one and a half servers are the simplest, dumbest things ever. Your phone runs five and a half to six million. They’re the most complex beasts out there. So phones have tons and tons of issues, or tons of crazy hardware and stuff like that. So just take all our fixes and move on. And the phone companies, Android’s proven that you can update to every single RC release for the past four or five years on Linus’s tree and have a stable phone.

CRob (09:23)
Eesh.

Greg Kroah-Hartman (09:24)
So you can update these things and you can keep them up to date and you have to rely on that and luckily people are finally realizing that they have to update their systems and take the bug fixes and move on.

CRob (09:51)
And where do you think some of the hesitation is from being closer to head that organizations have? it kind of historically there wasn’t a lot of cross testing or?

Greg Kroah-Hartman (10:01)
Well, the old historic model is change is bad, right? And if it works for me now, great, I’ll just stick with it. But the world changes, and the world changes their ideas of their threat models and how bugs are found and things like that. Once a bug is found and is public, could be trivial to reproduce. But the trick is if it’s not known, then that’s another issue.

So just because you aren’t changing doesn’t mean the world isn’t changing. So you have to update and go on. So it’s an old model of that way. And also people will work. So you should have a testing infrastructure in place to accept this.

Famously, everyone’s like, we’ll just take tiny fixes. Well, Dijkstra famously said a one bit change could radically alter a program. Software is unique in that. It is not proportional to the size of the change, it’s to the size of the result. So very tiny changes can have major, major impacts. Giant changes can have minor impacts. So it all depends on your use. So you better have a pipeline in order to this stuff. Also, people used to have huge kernel forks out of the tree and they weren’t working upstream. Embedded devices were famously like that. Some companies made that decision. It’s a business decision. It costs money to keep code out of the kernel, which is fine. They had money to waste. Now they’ve realized it and they’re working upstream. IBM and Intel publicly said it saves us time and money to work upstream. Why not do that? So it takes time for companies to move their code forward over time.

Pixel, that Pixel device I mentioned, it still has 300 out of tree drivers, which is crazy how many drivers the phone has. And it still was able to update to each one along the way. Engineers adapted the APIs and kept it up to date. It worked pretty well. So you can do things out of tree.

And you can work that way, but you just have to have the testing infrastructure in place. And I’ll point at Android. I mean, it’s a few billion, many billions of devices, but they have a testing infrastructure in place that they can update, take the latest releases, take the latest state releases, run it through their testing system. I got a report back pretty fast, whether everything’s good or bad, and they push out to devices over time. They stage them and roll them out and things like that. But you need that testing infrastructure in place. But you need that testing infrastructure in place no matter what. So create it. Build it for your systems and then you’re good. You’re good to go.

CRob (12:24)
Do you have any other advice for organizations that might be hesitant for staying current and, you know, afraid to update anything else beyond having kind of a very thorough testing plan and infrastructure?

Greg Kroah-Hartman (12:38)
If you’re afraid to update, then you better partition and firewall the heck out of those devices. I’m just by virtue of also it’s going to be against the law in many companies now not to update. So it’s nice that the governments are finally realizing the US government, EU government, Japan. It’s all a new initiative of China for the way they’re treating their software and automotive. This is actually really good. So they are going to require updates and that’s a good thing. And so you should be updating things over time.

CRob (13:05)
So basically lagging behind for stability is actually incurring more risk now, right?

Greg Kroah-Hartman (13:10)
it’s incurring more instability because you have known issues that your systems have. So that is a known problem and that means your system has a known vulnerability, right? That’s, mean, on your risk management scale, that’s very high, right? As far as I think the way risk management scales go. So be aware of that.

CRob (13:35)
So then you’ve touched on it. I won’t say the magic three letters, but we’ve seen a lot of increasing pressure globally on enterprises from regulators. So how do you see organizations like the OpenSSF or other open source foundations, how do you see them helping kind of reduce that compliance burden on the upstream maintainers?

Greg Kroah-Hartman (13:56)
I’ll say the magic word, three magic letters, CRA, Cyber Resilience Act from the EU, which I actually think is a really good thing. And I might be biased, I live here in Europe, I’m also on the expert committee to do this stuff.

Greg Kroah-Hartman (14:10)
It is mandating that vendors update their software and report vulnerabilities and report bugs upstream to open source developers and maintainers. And that’s a good thing overall. There’s also other things with SBOMs and all, whole life cycle and risk management, which manufacturers will have some additional work, but it’s good. It’s just a list of ingredients, right? Like you have to do for food or for electrical devices. This is nothing out of the ordinary that good software engineering firms shouldn’t be doing.

It’s going to be interesting to see. But the openness to self, will call out. You guys have done a very good job in handling the paperwork, a lot of documentation you have, like here’s a one-pager on what an open source steward means and how to handle it. There’s really just two things you got to do. It’s pretty simple. Who is an open source steward? There’s a flowchart. So you guys have documentation. I’ve been giving talks for the past year and a half about this stuff.

So there’s tons and tons of resources. I think you’re doing stuff for actually manufacturers too, aren’t you? Okay.

CRob (15:07)
Yep, the manufacturers of we’re in full swing there. So we should have some guidance out for those types of folks who are the consumers of a lot of the work of all of you upstream. So hopefully we can coach them to treat and interact with upstream maintainers in a much more professional manner.

Greg Kroah-Hartman (15:28)
Yeah, and I’ve worked with a lot of companies over the years. That’s a lot of work I do. And some companies are great at this, right? And some companies are not. So it’s a learning experience on their half. And I want to see that happen. These companies rely on open source and they rely on the fact that open source works. So why not become part of the community to ensure it continues to work for them? And that’s my biggest pitch to them is like, great, if you want to use Linux as is or other open source projects, just take, and that’s wonderful. But if you want to make sure that it will continue to work properly for you in the future, become part of the community and work with the guidance and work with the developers to ensure that your use case still is valid. Because all we are are tools for you to do your business, right? And so you have your value added, whatever you’re building or making. So work with us to make sure that still works properly. And open SSF again, yeah, you’re doing stuff for manufacturers. That’s good. Open source developer. Open source developers and CRA and laws do not apply to it all. So I was happy and happy. don’t have to worry about that.

But a lot of the work that OpenSSF did with the CRA was to ensure that that was going to be the case because the way the original laws were being drafted were very onerous when it came to actually developing it. And I will call out there’s a really a lot of good job work and jobs you guys have done for that to make sure it didn’t harm us.

CRob (16:42)
That was a really nice kind of cross ecosystem effort. I’m pretty proud of that, that a lot of the foundations and big projects came together to express the really negative consequences of the first draft of the law.

Greg Kroah-Hartman (16:53)
Yeah, yeah, and I’ll call out Apache has done an amazing job also with us continuing to do that. Eclipse has done some work as well. So we’re not alone there by any means, but all the open source stuff is.

CRob (17:07)
So from your perspective, kind of looking at all this looming regulation, what types of support would you try to suggest for downstream to support maintainers and projects in their communities?

Greg Kroah-Hartman (17:23)
Kind of support, well just send us patches that work. So, I mean, or just become developers, or just, or better yet, have your employees work on open source projects and company time. That’s the best thing to do.

CRob (17:28)
Patches welcome.

Greg Kroah-Hartman (17:37)
Maintainers need to be able to do their work of these open source projects on company time. The kernel has this document that shows how open source friendly is your company with different levels, like level zero is they’re not allowed to work on anything upstream to level five as they pay you to be a maintainer and work on whatever you want. There’s a little gradient type thing there. And that’s good. When I’ve worked at other companies, we had in our employee reviews of, we mandated that you had to work X percentage of time on upstream development. And that was good, and that was an instant payback. But if you’re not actually measuring that at the end of the year for their employee review, they’re not going to do that. that always comes, that’s the first thing to give.

Smart companies know that this is actually, it pays off in huge, huge benefits. I think there was actually a report by the LF recently about that. showed dollar amounts. Yeah, Frank did good job with that.

CRob (18:31)
Yep. Yeah, Frank Nagel did it. Yeah. I agree. So thinking about your day job is helping lead security for the Colonel. When there’s a vulnerability discovered from your perspective, what’s the hardest part of getting that addressed? The technical patch writing and testing or the coordination to getting that fixed once it’s ready available to the consumers?

Greg Kroah-Hartman (18:57)
So I’m one of the members of the security team. I won’t say we lead it, it’s an ad hoc group. But our kernel team is, it’s an alias. You send us a bug report. And if the people on that alias isn’t, we don’t know how to fix it, the kernel is huge. We will grab the right maintainer first thing and grab them in and join them to the emails for added.

CRob (19:20)
Mm-hmm.

Greg Kroah-Hartman (19:21)
Go from there. And ideally, the submitter has submitted a proposed patch. If they have a proposed patch, even if it’s wrong, don’t be afraid to send us something that you think might fix it. Because then that’s something you can work off of. I’d much rather have a broken patch, because it’s like, this is the area I think this might work with. Let’s go from there. If we drag you in enough times, the security team, we then gently ask you, do you wish to be part of this alias? So it’s not really a good thing to be part of this alias. So that kind of means your subsystem has been little bit problems over time. That being said, yeah, just work with that. So the majority of bugs we have are not a big deal. Like, oh, we expose some uninitialized stack memory here, or here’s a way to bypass random number or random address of stuff, which is, as a local user, that’s not a really security issue or et cetera. We have lots of minor issues reported all the time.

We work on the fix. We usually post the fix publicly and don’t declare that, this is a security fix, it’s just a bug fix, and away it goes. Because if you look on our mailing list, we’re fixing bugs that end up being vulnerability fixes because they can crash the machine. Every day we have 35 bug fixes a day we seem to average. That goes into the stable kernels, that goes into the lenin-systery first. And about one-third of those are deemed of CVE. So we average about 10 to 13 CVEs a day.

which seems like a high number. But so the small number that we get sent to the security list are not very much, not very big. Very rarely do we have anything that’s major. And even then we just work on fixing it and get pushups to the tree and go. And we don’t assign a CVE until things are in a stable kernel release. So we give people who follow the stable kernels a chance to be up to date and secure before things are publicized to the world, which is good.

And we don’t do any disclosure. It’s up to the submitter if they want to do some disclosure or they want to write a report or have some paper or whatever they do. We just fix bugs and move on. And that’s all we do.

CRob (21:29)
We had chatted earlier in an earlier call today. You folks lost a pretty impressive title that you had for several years. You want to talk about your rankings within the CNA ecosystem? I thought you were number one for a while. Okay, my mistake.

Greg Kroah-Hartman (21:40)
No, we’ve never been number one. We’ve always been number two. No, well, we’re the number two creators of CBEs. Number one is WordPress. But it’s WordPress and all the different plugins somehow. I don’t know how those people keep up to date with all that stuff.

And then if you look at GitHub, we alternate between us and GitHub, but GitHub is a conglomeration of all open source projects that don’t really have the CNA that’s assigned bugs to. So it’s from a single entity, we are kind of still the largest one. But we don’t declare severity to bugs for the most part. I’ll talk about that in a minute. So other companies, if you look at Microsoft and Apple, they only report severe vulnerabilities. They do not report low or minor or medium or high or things like

So if you look at just quantities, you can’t go off of quantities. And that’s an issue that a number of people have realized slowly over the years. it’d be nice if the CBE group would require all vulnerabilities to be released as a CBE.

CRob (22:42)
I would like that quite a lot.

Greg Kroah-Hartman (22:44)
Yeah, so the fact that we have a lot is unusual. But the scoring of CVEs is tough because I don’t know your use case, as I mentioned before. So I don’t know if this is an issue or not. And there’s a famous researcher that said, even if I know a vulnerability or a bug, determining if I can actually exploit that is almost impossible. just depends on your use case, depends on this, depends on how your system’s set up, depends on all these random different things which is both good and bad, right? It’s good in that these things are usually very hard to exploit, it’s bad in that we have no idea if they are exploitable or not. So you see some groups like NVD from NIST will go through and give a random number for a severity score and that random number for the next kernel was 5.5 for everything last year.

That number we was like, why did they pick that number? And that number is just below the threshold that triggers some automatic tools for which you must upgrade. This isn’t a good thing. We’ve asked NVD to stop this. We asked CISA to stop this. They said they would stop this, and they did that. And we are starting to roll out. We just actually scored some CVEs the other day and pushed out updates.

CRob (23:55)
yeah?

Greg Kroah-Hartman (23:58)
We picked some that were like, okay, if you do have this network type of connection, this is a bad bug. So we went in that, scored that. There’s a group of companies that got together, like, take, started looking at different use cases. So like the cloud server use case. And they were like, here’s some rules and let’s try and score them all this way. That kind of fell down and they kind of fell apart. And we’re trying to drag them back because it’s hard to add this metadata to a CVE of what is the use case and what is the score for that use case. CVE wants to have, what is it, supplier? They call it something. They want to add.

Something starts with an A. They want to add different people can do things to a record.

CRob (24:36)
The ADP, the authoritative data provider.

Greg Kroah-Hartman (24:39)
ADP. Yes. Right. They want to do that. But that’s great. But you have to be a CNA to do that. And that’s not going to really work well for larger projects. So we’re thinking with the kernel, we’ll take groups of people, like the cloud people, and have a way to add to our record that we publish. Because we publish everything in the Git repo that is an addition to the CVE record that information.

And then we might be able to push into something else like OSV or things like that where we can define the data format better. And then maybe get the embedded people to add their own record and we’ll just keep adding these different use cases as time goes by. And I think that’s the only really way forward we have to do that. It’s like, OK, for this use case, this category of people are claiming that it affects it this way. And sadly, LLMs can actually grade these things based on the use case.

It actually kind of works kind of well if you say, here’s my use case. Is this bug applicable to me or not if I run this code? And they come back with a pretty good score. And then there’s just a pattern matching. It’s a pattern matching of use case to code. And OK, yeah, here’s a problem. So people can some LLMs at it. And I think one reason the cloud group fell down is everybody wanted to use their own specific LLM model. They didn’t trust anybody else’s LLM model, which is.

We’ll just run it on my local machine and we’ll take this as one and go from there. Local models are actually working really well these days.

CRob (25:57)
Good to hear. Well, since you said the other magic word, thinking ahead, are we prepared for AI generated vulnerabilities at the scale of AI?

Greg Kroah-Hartman (26:11)
We are already there. So something happened about a month or two ago and we used to get AI slop and I’ve said this before and those are easy to notice. About two months ago, we started getting real reports that were written nicely, actually reported a bug and sometimes came with a patch. But it’s real. I don’t know what triggered it and I don’t know what seem to do things well and so something happened. So in talking to the other open source security people because we have some meetings at times and we see each other, we’ve all hit it. GitHub’s hit it for their number of things, Curl’s hit it. All the different open source projects have hit it, HAProxy, the kernel, everybody. Kernel, we can handle this because we’re a big group and we can partition our farm it out to different maintainers.

But other smaller projects, I am actually worried about it. But it’s kind of like when fuzzers hit us, right? Fuzzers hit us and fuzzers were like, okay, the world is on fire. Here’s a thousand bugs. We’re like, And then we just grind through them. And we also say, hey, fuzzing companies, once you made this tool, you provide the gun, once you provide the resources to hold the gun properly. And to be fair, Google did. Google provided the resources to triage the bugs and say, okay, there’s serious, and they still do this. They run the fuzzing tools and they find the severe bugs first.

They fix them, send out the fixes before they actually let the rest of the fuzzing tools, reports go public. And the other reporting tools from the public fuzzing results are we have what, 400 open bugs right now that interns are really good to work on because they’re tiny bugs.

Hard to hit, and so let’s let the interns learn things, and that’s a good learning community.

CRob (27:48)
Yeah, cut their teeth. Yeah.

Greg Kroah-Hartman (27:50)
Yeah, same thing with these AI steps. So the AI code reporting tools, I mean, I’ve run a bunch of these over the past couple of months, and some of them are good. They’re good static analysis, because it’s pattern matching, right? And that’s what we actually have been using in the kernel for over a decade. Famously, for the stable kernels, we knew that when we went to backport patches, developers would say, this is a bug fix, and we backported.

Well, Julia Lowell, a professor in Paris, said, hey, why don’t we compare all the patches that we humans say are good with all the patches that we don’t know if we’re good or not and see if they match? Because LLMs are just a fuzzy statistical matching model. And it turned out to work really well. And she’s written a bunch of papers. And her and Lowell or Sasha, has come up with some great tools. they backport stuff. And they do some wonderful things. And we’ve been using this over a decade.

So this pattern matching of is this a bug and is this not a bug is there and that’s what Anthropic published like not so long ago they said we found 500 bugs and there basically was like we looked at all these old CVEs and saw where they weren’t fixed in the current two bases – code bases.

And I’ve run those same types of scripts. And it’s like, hey, this actually matches pretty well. Like, OK, it catches some things. And it out we forgot to cut a bunch of things. I was like, I thought we had tests for that, and we forgot to run our tests. So even as kernel developers, our infrastructure isn’t always working properly. So we have to go back and turn on a few tests that we had stopped running. Which is good. It cut it. So it’s just going to be another layer. It’s another static code that will grind through and fix up all the bugs and keep on moving. But it’s going to affect a lot of people for a while because fuzzers were kind of hard to run and set up.

These AI tools are a little bit easier for other people to hit and run. But the fun thing is some of these AI tools can spit out patches. We’ve taken security bugs sent to us and I run it through a local model and here’s a dumb, stupid patch to fix this problem. And it kind of works about two thirds at a time. But it gives you that base to go off of. And then there’s something to go for there. A few of those patches have actually been merged. So because they actually do fix the bug, which is kind of sad and funny in the same way.

CRob (29:52)
Yeah, exactly.

Greg Kroah-Hartman (30:04)
Yeah, I mean the old adage of all input is evil. Sometimes we forget to check the input. these things actually catch that well.

CRob (30:08)
Well, and it’s interesting. I’m not worried about teams like you or Apache Kubernetes, the folks that have a large group of security experts. It’s the single maintainer projects that might not have the expertise or the time or the tools. That does worry me quite a lot.

Greg Kroah-Hartman (30:26)
Yeah, it does work for me too. And they could get overwhelmed. But ideally, can also, I tell these people, just push back and say, come with a patch. And these tools can make a patch. And it does work that. But a lot of those tools and infrastructure aren’t security like.

facing, right? So say my LSUSB, which is like, it scans the number, the USB devices and prints out a little graph of what devices you have. People will try to report security bugs in that before. I’m like, yeah, that’s wonderful. It’ll crash. But that’s not a problem. So you physically told it to run as a user. It’s going to crash. OK. And if you look at the way Rust does memory safety, if it encounters problems in its code, Rust, by default, will just crash because that’s safe. It’s causing a safe way. Now using Rust in the kernel, you don’t want the crash when you do other things in there. But memory safetiness and crashing is kind of a very common thing when security issues come up, you just want to panic and go away. And it’s very good for users to do that. Because normally somebody can notice it, kick the box or kick the node and away you go, you start it up again.

CRob (31:37)
Well, this has been, I think, very illuminating, but let’s move on to the rapid fire part of the conversation. I have a couple of crazy questions. Just give me the first thing that comes off the top of your head. Mild or spicy food?

Greg Kroah-Hartman (31:57)
Spicy.

CRob (32:02)
So what’s one security myth you’d like to eliminate?

Greg Kroah-Hartman (32:06)
One security myth that I know if a bug is a security issue for you or not.

CRob (31:)
There you go. Very nice. What’s one thing that every company using Linux should do tomorrow?

Greg Kroah-Hartman (32:02.376)
Update to the latest version.

CRob (32:21)
When you’re doing your patch releases, what’s your favorite beverage of choice?

Greg Kroah-Hartman (32:27)
Coffee

CRob (32:29)
Yummy! And finally and most importantly, Star Trek or Star Wars?

Greg Kroah-Hartman (32:38)
Star Wars, sorry.

CRob (32:40)
There are no wrong answers. But Star Wars is very excellent. I appreciate that. So Greg, as we wind down, do you have a particular call to action you want to share with our audience outside of the amazing little snippets of advice you’ve given us during our time here?

Greg Kroah-Hartman (32:54)
Report bugs and report bugs to maintainers and developers. If something isn’t working, tell us because otherwise we don’t know. Linux has been working for me for 20 years, but obviously it’s been working for everybody. tell me and do that.

But also on the flip side I only see bugs, which is funny. I’m amazed that this stuff actually works. So my daughter, who is now a doctor, she’s like, it’s just like a doctor. Doctors only see problems. How does the human body ever work? The majority of the time, the software does work, and it is all good.

But if you do have a problem, please let me know. I’m here to fix it and help us. We want to know how to fix it. And tell us how you’re using it. We’re kind of curious sometimes. I know, like the cow milking machine, satellites, the Mars Rover, fun things like that. It’s kind of interesting to see where our code is at. It’s fun.

CRob (33:46)
Excellent. So amazing advice, incredible storied career, sir. Thank you for your time. And with that, we’re going to call this a wrap. I want everyone to stay Stiber safe and sound and have a great day and happy open sourcing.