CRob (00:11.23)
Welcome, welcome, welcome to What’s in the SOSS, where we talk to upstream maintainers, security experts, and just generally interesting luminaries throughout the upstream open source ecosystem. My name’s CRob. I’m the security architect for the OpenSSF, and I’m also your host for What’s in the SOSS. Today, we have two amazing friends of the show, amazing community members and developers and contributors across the open source ecosystem. So I want to welcome Eddie and Ben. Do you want to take a moment just to introduce yourselves and kind of explain what your deal is?
Eddie (01:02)
Yeah, my deal is I am in Amsterdam with you at 9 AM with a completely different energy level than you have right now. I am loving this. This is this is awesome. Eddie Knight from Sonatype. I do a lot of work across the Linux Foundation related to security compliance.
Ben (01:20)
I’m Ben Cotton. I’m the open source community lead at Kusari. I’m the leader of the OSPS Baseline SIG and a member of the Orbit Working Group.
CRob (01:29)
Awesome. Great talks today. We’re going to be diving into the OSPS Baseline, the catalog, and ORBIT, GUAC, and a whole bunch of different topics. So let’s set the stage here, gentlemen. The Baseline. Folks have been hearing about this off and on over the last few months, but maybe we could explain this in plain English, like what is the Open Source Project Security Baseline and talk about the catalog?
Eddie (01:57)
All right, I’ll let Ben give the official answer since he’s the project lead for it. Baseline’s a control catalog that helps maintainers and consumers of software have a clear definition of good for their projects and their project security. Ben, you want to give a more real answer?
Ben (02:16)
Yeah, I mean, it’s what it says on the tin, right? It’s a baseline for open source projects to follow in terms of their security hygiene. And it’s not really about the software they’re producing. It’s about the practices that are used to build that software. So the analogy that I recently came up with as we were going back and forth on this is it’s like health department regulations that say, “You have to wash your hands. You can’t pick up the food off the floor and then give it to the customer.” It doesn’t say that the quality of your recipe has to taste good. But you have to use secure practices. So we’ve developed a catalog of controls at three different tiers, the idea being that new projects, small projects, projects that are more trivial in nature just have like a sort of a bare minimum of like, yeah, everyone’s got to do this. Everyone needs to wash their hands before they start cooking food.
CRob (03:14)
I appreciate that.
Ben (03:15)
And that is important for SOSS, right?
CRob (03:18)
Right.
Ben (03:18)
As you go up the levels, know, go up to level three, like that’s really big projects that are, you know, lots of contributors, typically well resourced, at least relative to other open source projects and really important to the world of technology as we know it. And so those have to have the tightest security requirements because they’re rich targets for attackers.
With the baseline, the motivation is like, this is not a list of all the cool things you could do. It’s do this. One of the requirements we have is there is no should – there is only must. Because we don’t want to be having maintainers spending a lot of time chasing down all these things that they could do. We understand that open source maintainers, which we are, are often volunteers doing stuff in their spare time without necessarily any real security training. And so we need to give them straightforward things that are meaningful to enhancing their project security posture.
CRob (04:27)
This is, think, the first time ever on the planet anyone has ever referred to security as cool. Thank you, sir. I appreciate that as a longtime securityologist. So let me ask you, gentlemen both, why do you think it’s so important to define this baseline? And why is that important for open source projects?
Ben (04:47)
So I think the most important thing that’s been coming up in conversations I’ve had with people here in Amsterdam and other places is like, It gives us a common language. Go be more secure is not helpful. It doesn’t tell anyone anything.
And with baseline, especially with these different levels, you can say, our project is secure. We meet baseline level one. We meet baseline level two. Now there’s a common language. We all can know what that means because there is an explicitly defined catalog that says what these levels are. And then conversely, If I’m a vendor or manufacturer of some product and I use an open source project and I want them to be more secure because I have regulatory obligations, I can go to them and say, I really need you to be at baseline level two. We can help you with these specific tasks. And now we’re talking the same language. We have this common understanding of what this means as opposed to you’re not secure enough or you need to be more secure.
CRob (05:52)
Love that.
All right, so from your perspectives, and I think you might have touched on this a little bit, Ben, but how do we think that the baseline makes it easier for maintainers and developers who are already so busy with just their general feature requests and bug burn down?
Eddie (06:09)
So we started this journey a long time before we ever started saying baseline, right. My very first interaction with CNCF before I ever did any commits on OpenSSF, I was just kind of like, maybe attending a call here and there. We were doing this, it was like a security bug bash. And maybe we had called it the slam by this point. We wanted to solve for folks who were doing really cool stuff and everybody in the conversation knew that their stuff was being built well and properly and everybody’s washing their hands and stuff like that, right? But we didn’t have a way to demonstrate that outward and say like, hey, this project is running a clean kitchen. You should trust this more than just a random, you know street food vendor, whatever the open source equivalent of that is.
We want to boost confidence in the projects in our ecosystem. And, back then we had the CLO monitor because it was just for CNCF. And there was this set of rules of like, these are the things that we expect CNCF projects to do. And when we could go to a project and say, and I would pull out my phone on the KubeCon floor and be like, click through, type in your project name, pull it up, see like, this is where you’re scoring right now, right? And the scoring part brings all of its own baggage. But the point is like, there’s this list, right? And they’re like, that’s all you need? That’s it? That’s all you needed me to do, right? And so we had projects that were able to increase their own like personal maintainer confidence in their project. Like, oh man, I’m actually doing a really good job here.
All I needed to do was like shift this, rename this section header in a file so it could be detected. And now people see that I’m actually doing this stuff. And so you’re dramatically boosting our own like confidence in our work, but then you’re also boosting the public confidence in it. And this source is just having a list, right? Now that list for CNCF is not, it did not prove to be scalable and compatible with the broader ecosystem. It’s like, well, we don’t do this, we don’t do that.
So having baseline is a way of saying, let’s get that list, let’s get those wins that we experienced within the CNCF and make that possible for everybody by making it this like, not just CNCF, but agnostic baseline of controls that are good for projects.
CRob (08:53)
And those of us that have come from enterprise computing, the term baseline is very common practice as you’re deploying systems and applications. There’s a checklist of things you have to do before that system is able to go live. I love it. So thinking about the catalog, I realized that we have a lot of other assets within the OpenSSF, a lot of great projects and guidance. Could you maybe touch on some of the other projects within the OpenSSF that Baseline works with / integrates to?
Ben (09:21)
Yes. Yeah. So there is this whole working group called ORBIT in which Baseline sits. And it’s really about generating some of the tooling. So we use Gemara to sort of the scaffolding, I guess, for the control catalog. And it’s a software project that provides a framework to build control catalogs on. We do that. We’re working on tooling to automate some of the baseline evidence collection to make it easier for maintainers to you know, quickly evaluate where they are and what tasks they need to do to achieve their desired level. There’s a very smart man who has done a lot of mapping. This CRob guy has done a lot of work to map baseline to existing things like the best practices badge, as well as other external frameworks like the CRA.
CRob (10:29)
I’ve heard of that.
Ben (10:30)
Right?! Various NIST guidance, you know, really kind of make it so that, you know, baseline gives you not just, you know, confidence in your security posture, but then also gives you pointers to, you know, these more regulatory kind of control catalogs, where if you have a vendor coming to you and saying, hey, we need you to be secure, you can say, well, here, here’s what we meet. Here’s the list of things now you know. You know, so we really try, you know, we want to make sure that baseline is a part of an ecosystem and not just this really good idea that we have off in the corner that is sort of academic.
CRob (11:14)
That’s that’s excellent. That actually helps me pivot to my next set of questions. Let’s move out of the ethereal world of theory and talk about some real world applications of this. We just recently released a case study where we worked with another OpenSSF project named GUAC. And I just loved reading this case study. Could you maybe walk us through what the project was trying to prove and how baseline helped the GUAC project?
Eddie (11:44)
Yeah, that one was actually remarkably easy because all I had to do was yell at Ben and then it was suddenly done. [Crob laughing]
Ben (11:53)
So, you know, with that case study, we had the advantage of I’m a contributor to GUAC and then also as the baseline SIG Lead, like there’s some good overlap there. You know, so really what we were looking at is, you know, sort of a two pronged approach. One, you GUAC, the graph for understanding artifact composition, is a software supply chain security project. It would be really bad if it were, say, compromised in a supply chain incident, right? So, when you’re a security project, you have to have your own house in order. And so, you know, from the beginning, the project has really been done with that in mind. But we want to see, like, you know, validate our assumptions. Like, are we actually doing these things that, you know, are sort of the best practices
CRob (12:41)
Make sense.
Ben (12:43)
And then also, like, you know, from the baseline perspective, we want to get that real world, like here’s an actual project using it. What are the things that are unclear? What are the things that makes that don’t make sense? What are the things that are really easy? And so, you know, with that, we were, was able to use the Linux foundations, LFX insights, now has some automated
evidence collection. And so that, you know, was able to mark up a lot of boxes off right away. Some things are like, well, that’s just how GitHub works. So check, check, check. And so in the space of an hour or so, I was able to do…
CRob (13:35)
An hour?!
Ben (13:26)
An hour. I got level one and level two almost done. There were like four or five things where I was like, I’m not sure if we as a project would consider this sufficient or in a couple cases like we don’t actually document our dependency selection process. There is one, we don’t just add things William Nilliam, but you know we just need to write that down because as new contributors come on they need to know too and so like you know it was the amount of work that actually needed to be done to check the boxes off was really low. Which was very you know good news for me on both sides because I was gonna be the one doing the work.
And I’m the one trying to tell people like, you should use baseline to evaluate your project security. And so we really would love to have more projects do that sort of initial run and give us that feedback and help us. We spent a lot of time as a SIG with probably two dozen people at least have been involved over the last two years.
Coming up with these ideas, debating, you know, what needs to be included, what needs to be excluded. Eddie and I recently spent several hours coming up with definitions of words that were very precise so that we could be very clear and unambiguous. Like when we say project, this is what we mean in this context. and, we’ve tried very hard to keep this as a not a security wish list, but like a practical set of things for real world project maintainers. But, even with dozens of people involved, that’s only dozens of experiences. We want this to be something that’s useful to all eleventy billion open source projects out there.
So we need some more like real world do this, come back and tell us, hey, this doesn’t make sense. “This really is not a fit.” “My project can never do this because” – that kind of information.
CRob (15:36)
That’s awesome. From your perspective, as a maintainer, not that you’ve gone through this for GUAC, as a maintainer, how does that add value to you? What are you hoping to leverage out of that experience beyond the project itself, but as a GUAC maintainer, what are you hoping to gain from going through this baseline process?
Ben (15:58)
Well, I think the first thing is that it just gives confidence that like, yep, we’re doing the right things. We are doing what we can to reasonably protect ourselves from security incidents, particularly supply chain compromise, because GUAC isn’t running a service or anything.
And then, you know, being able to build on that. And then, you know, if, you know, we get emails like, Daniel Stenberg gets from, you know, car manufacturers and stuff like that, you know, we can, you know, just be like, yep, here’s our, our baseline level go have fun – (Daniel, if you’re listening, I would love for the cURL project to try out the baseline) and then you can just be like Yep, here’s my statement that we meet this baseline level as of this date. Have fun. If you want more, send me a check.
CRob (16:59)
So Eddie, we’ve talked about the maintainer a lot. But let’s talk about some other personas that are involved with the baseline and might care. From like an OSPO or an executive or security engineer perspective, what do you see the value of a project going through and having achieved some level of baseline.
Eddie (17:20)
Oh yeah. I mean, any metric badge check mark, right? It’s always helpful because going off of the number of GitHub stars only gets you so far.
CRob (17:35)
Right.
Eddie (17:36)
Especially now, we see that there’s actually an inverted relationship between stars and security for Hugging Face projects.
CRob (17:46)
Huh, really?
Eddie (17:47)
Yeah. Like there’s like somebody, well damn, now I’m gonna have to like find the research and actually show it to you to back my claim up. But yeah, was a little while ago somebody posted something where they found that it used to be more stars is more eyes. More eyes is more critiques. More critiques is more security, right? But for like ML projects, these kinds of things that you find on Hugging Face are the folks who are doing something fishy are pretty good at spoofing stars.
CRob (18:27)
Gaming the system, huh? I don’t like that. That makes me sad. And angry.
Eddie (18:33)
Yeah. And it’s like the more fishy that their thing is, the better their skill set is at spoofing stars. So it’s just kind of a weird thing. So when we have something like the best practices badge, Like, CNCF loves that, like the TOC loves that. Within TAG security and compliance, we obviously also love, it was not meaning to be a contrast statement. You like shook your head, you’re like, what, do you guys disagree? No, we don’t disagree. But there is also this desire to have something that is a little bit more fleshed out, right, which is why we were like, real big on CLO monitor and things like that. So the more fidelity that the badge has the more interesting it is. But I mean anything anything that can help accelerate that selection process is really helpful for the like The OSPO type of personality that you’re talking about.
CRob (19:37)
It’s been interesting kind of working with these projects and then being like a downstream consumer it there are many tasks within application development and app sec that are very difficult to measure. And some things are, I can verify what your settings are in your source forge. I can validate if you’re using multi-factor authentication or not. But there’s like just some tasks that are very difficult. And I’m excited that it’s not a solved problem yet, but the team has a lot of great ideas. And I think things like using security insights and other tools, to help provide that additional evidence showing that yes, here’s our policy. And a lot of the baseline encompasses some tasks that developers don’t always love, which is things like documentation.
Eddie (20:36)
Yeah, we have a lot of documentation checks. That is the number one question that we get, which is a fair question set. But one of the most common question sets is just like, what does documentation have to do with security?
CRob (20:49)
So Eddie, what does documentation have to do with security?
Eddie (20:53)
This is one of those situations where I actually struggle to answer at first. I have an answer. But the first 10 seconds is me going, why is this even? Isn’t it obvious? This is obvious, right? And then I look around the room and it’s like, it’s not obvious. OK. So there’s a couple different types of documentation that we need. So we need the things that you would put in a SECURITY.MD.
Just where do I go if I find a bug, if I find a security vulnerability? Who should I contact? Where should I post this information? What should I expect back from me? Those types of things. But then there’s also stuff if I’m using the project. If I need to run GUAC, Is GUAC secure by default? Is everything locked down when I turn it on? So it might be a little bit harder to turn on and deploy into my cloud infrastructure or whatever, but I don’t need to worry about it. Or is it the opposite? Is it insecure by default? Because almost all projects are insecure by default. The goal is to get more users. So you make it easy to deploy. And that means that when you turn this on, it’s going to have root access to something, it’s gonna have some network traffic that would not be recommended for prod, things like that. And so if we don’t have clear, easily accessible documentation with like a link that people know how to get to that kind of thing, like if this isn’t created and it’s not in a place that people know about it, then you’re actually deploying software that can be secure, but in practicality for users, there’s a high likeliness that they’re going to deploy it in securely. So you might have done your job, but people aren’t gonna be using it in the secure fashion because you haven’t documented it well enough or made it available or clear to them. And those are just like the two that come straight to mind. Like there’s a few different documentation needs that we have.
Ben (23:00)
And some of that, the documentation controls too are around like project policy in terms of, and I mentioned the dependency selection process. you can’t rely on, well, everyone knows this because one, people forget, two, if it’s not written down, everyone knows it, but everyone might know a slightly different thing. And then, you three, hopefully you’re bringing new contributors into your community. They need a place to learn about these things. And so, you know, having some of those things like, you know, we look for dependencies, you know, we prefer that they are actively maintained that they have, you know, maybe an OpenSSF Scorecard score above a certain threshold or like maybe there’s an advanced algorithm you use to mix a bunch of things together and then figure out, you know, maybe, you know, if it’s, you know, a project within an ecosystem, you don’t pull in just random things off of package repository, you have an internal repository that you mirror things into to protect from things like that right but if that’s not written down if that’s not you know clearly documented for the people who need it it’s not going to get followed.
CRob (24:15)
So let’s get out our crystal balls and look into the future. You know what do you guys see for orbit the catalog and just this general let’s work in general?
Eddie ()
What do we see for the future? So we’ve right now we’ve stabilized the control catalog, I would like to, I would like to make it a Gemara complete control catalog, right? So it lists out the capabilities, the threats and the controls, right? Because we’ve written a threat informed control catalog, but we haven’t captured and documented, what threats are we mitigating with this? So I think that’d be pretty cool. How close are we to doing that? I don’t know.
The other thing is just getting, more people to actually demonstrate compliance with the controls? think most projects, especially in the LF, are gonna be predominantly compliant already. Like you’ve already done all this stuff. We just want to be able to tell everybody that you’ve done it.
CRob (25:16)
Get credit for your homework.
Eddie (25:18)
Yeah, we wanna give you credit for this, right. And so that’s gonna be a big lift is going through and doing that hour of work with GUAC. Doing that hour of work with all of these different projects kind of adds up. So that’s gonna be something that I hope happens very soon. Within CNCF, we did it in the security slam fashion, right? So OpenTelemetry, Meshery, OSCAL-COMPASS, and Flux actually, were all part of that in the spring. And that went pretty well. Where the breakdown happened was on multi-repo projects like OpenTelemetry. I think it was 70 repos.
Yeah, like a lot of repos. think Kubernetes is double that, right? Yeah. So when you have so many different repos and we need to go in and say, here’s where the documentation is for this repo. Here’s where the policy is for this repo, right? It gets a little bit bumpy. And I think there’s still some room for improvement on how we’re capturing and doing those evaluations. say, I think I have a backlog. I know. There’s improvement on that.
But as more people are going about that and giving feedback, like Ben comes and says, this is where something took 20 minutes, but I expected it to take five. Then we can actually make those improvements and improve our backlog, refine our backlog a little bit.
Ben (26:51)
Yeah, and I would, know, to Eddie’s point and you mentioned earlier, CRob, but we do not have fully complete tooling to measure all the measurable things yet. And so that’s an area that the Orbit Working Group is working on as a group. We’ve also had some sort proto discussions about having a catalog of examples. What does a dependency selection policy look like? What does this documentation thing look like?
In baseline itself on my backlog includes like just going out real world example, you know, from Fedora, from curl, from Kubernetes, from wherever, like here are some things that look like what we would suggest you have. And then, you know, ideally, I think we’d also want to have a project that is just templates for each of these things that are templatizable. Like you don’t have, you know, so code of conduct licenses, those are pretty well established.
A lot of this other stuff like what what is sort of like the platonic ideal of a security MD file? What is you know the best dependency selection policy that people can just you know do a choose your own adventure? I want this this this put it together. This is what makes sense for my project. Here you go. It’s no it’s of no use to anyone to have everybody writing this from scratch over and over again, especially if they’ve not seen an example of it before.
CRob (28:21)
So as we wind down here. are the calls to action do you have for the community or whether it’s developers in the OpenSSF or just kind of unaffiliated maintainers? What would you like folks to take away from this?
Ben (28:37)
I would love them to look at the open source project security baseline, baseline.openssf.org and evaluate your project against it and give us feedback. What worked? What didn’t? What do you think? Why isn’t this there? We want this real world feedback on the control catalog so we can make sure it is actually fit for the purpose we’ve designed it for. So for me personally, that’s the biggest takeaway I want from people listening to this.
Eddie (29:09)
Complain loudly. That’s what I want. We are trying to create an accelerator. We’re trying to improve the ecosystem. We’re trying to improve the lives of maintainers. And any single place where this is slowing down a maintainer, that is outside of intent. That is a design flaw of some kind. If this is slowing you down, if this is confusing, if you’re getting pushback from some end user who now thinks that you’re doing worse than before you started, before baseline existed, right? We heard that feedback from somebody. It’s like, hey, LFX Insights turned on their scanner, and now I have a user who thinks that our project’s doing a bad job with security. And it’s like, oh, well, that didn’t meet expectations.
CRob (30:00)
That was an unintended consequence.
Eddie (30:02)
Yeah. And it was that perception was inaccurate. The tests were accurate but imprecise, right? They nailed exactly what the tests were trying to do. They were very, very, very much there, but not, they weren’t aiming in the right direction, right? And so we refined like, okay, let’s zone that in, move it closer to the bullseye on what we’re trying to achieve. And I think we’re getting a lot better at that. But that’s because somebody came and ruffled our feathers and was like, hey, you’re not doing what you said you’re trying to do. we thought we were.
CRob (30:43)
Right.
Ben (30:45)
Yeah. And I just want to point out that the baseline is itself an open source project with open public meetings, pull requests welcome. We truly do want feedback and contribution from people who have tried things out or don’t understand. I shared very early on on my social media accounts and a guy I know came back and was like, we could never meet this. And it turns out the wording was just awful. We did not make this clear at all. And yeah, we fixed that. it’s like, OK, we went back and forth a few times. All right, this is our intent. We have now captured it well. And I think the wording is a lot better on that because people were confused and asked questions.
CRob (31:30)
Well, and to your patches welcome comment, we’ve had decent engagement with open source maintainers. I would love to see us have more downstream GRC security people giving us feedback from your perspectives. What other compliance regimes or laws would you like to see? And did we get our compliance mapping right? Is it spot on? Does it speak to the needs you need to have to defend yourselves against auditors and regulators?
Well, Eddie and Ben, two amazing community members, friends of the show here. Thank you for your time. Thank you for all you do across your fleet of open source projects that you contribute to and maintain. And with that, we’re going to call it a day. I want to wish everybody a wonderful day here from sunny Amsterdam. And happy open sourcing. Bye, everybody.
Eddie (32:23)
Thanks CRob.
Ben (32:24)
Thanks, CRob.
