What’s in the SOSS? Podcast #4 – Eric Brewer and the Future of Open Source Security

In this episode, Omkhar talks to Eric Brewer, professor emeritus of computer science at the University of California, Berkeley and vice president of infrastructure at Google. He’s also on the Governing Board of the OpenSSF. His research interests include operating systems and distributed computing. He is known for formulating the CAP theorem about distributed network applications in the late 1990s.

• 01:15 – Eric discusses his background
• 03:14 – Improving security in a corporate vs. open source environment
• 05:58 – Advancements Eric has noticed in open source in recent years
• 07:17 – How to make software repositories more secure
• 08:58 – The next big hurdle in open source security
• 11:12 – Rapid-fire questions: Mild or spicy food? Vim or Emacs? Spaces or tabs?
• 12:42 – Eric’s advice for aspiring security professionals
• 14:45 – The importance of being active in security communities

• Eric Brewer’s LinkedIn profile
• OS3I report

Eric Brewer (soundbite) (00:01)
I do think we need to start tackling build services and automated testing. And the reason these are harder is because they cost money. Even if you’re willing to volunteer time to work on open source, you may not be willing to pay for extensive building and testing. 

Omkhar Arasaratnam (00:18)
Hi everyone. Welcome to What’s in the SOSS, a podcast by the OpenSSF. I’m the OpenSSF General Manager Omkar Arasaratnam. And with us this week, we have Eric Brewer from Google. Eric, can you give us a little bit about your title and what you do at Google?

Eric Brewer (00:38)
Happy to. Nice to be on this podcast. I am a VP and fellow at Google. I’ve been working on cloud-related things for a long time, really since the 90s, before we had a cloud. At Google, I’ve been working on things like Kubernetes and open source security, and that’s what led me to help start the OpenSSF.

Omkhar Arasaratnam (00:58)
And for our listeners, a long time ago, Eric and I both worked together. And in fact, how we got to know each other was through my interest in open source security. And of course, the leadership that you were providing in open source security at Google. Can you talk a little bit about that aspect for our listeners?

Eric Brewer (01:15)
Yes, so through my work in Kubernetes, I came to realize in around 2018 that, you know, this was obviously a successful project, but it was also very complicated. At the time, it had like 1,200 dependencies. So it touched a whole bunch of code and that if you just look around, it wasn’t hard to find things that we depended on that realistically weren’t all that trustworthy.

And people were not very careful at the time about what kinds of things got included or how they would get used. And so that was a bit of a wake-up call for me. Then I started to look at a whole bunch of other things. I’m like, well, this is not just a dependency problem. It’s actually a supply chain problem more broadly. 

I started giving internal talks on supply chain security, which by the way, that term didn’t exist at the time either. We didn’t really call it that. It’s the right term, but how do we build our software? Why do we know it’s built correctly? And especially at the time, lots of times people would include packages directly off the Internet like you don’t even know what’s in there. My friend Kelsey Hightower often said it’s kind of like picking up a USB drive off the street, sticking it in your laptop. It’s like everyone knows not to do that, but we don’t know not to take random packages off the Internet, even though it’s a very apt analogy. 

I started thinking about that in 2018. I started thinking about what Google could do, but at the end of the day, this is not a Google problem. As much impact as Google can have, this is an industry-wide problem and needs an industry-wide solution.

Omkhar Arasaratnam (02:41)
As a grumpy old guy that’s been doing security for about two decades, the very thought of somebody randomly plugging in a USB stick they found in the parking lot sends chills down my spine. And I tota lly get the equivalence in terms of, hey, what’s the software you’re consuming off the internet? 

Now, Eric, you’ve worked in large organizations and you’ve also done a lot of work in open source. Can you speak to the differences in how you would improve security in a corporate environment like when you’re shipping code at Google versus what’s possible in an open source project?

Eric Brewer (03:14)
They are pretty different, which is something I hope to change eventually. And I give lots of talks to Google Cloud customers on this kind of topic for the corporate view, at least, which is roughly, have control of the code you’re using, which often means private copies of things. The private copies give you two advantages. One of them is you’re not taking it directly off the internet, even if it came indirectly from the internet. Second, when you have some kind of problem, you actually know exactly what code you used. Because another problem with pulling stuff off the internet is that code changes and you may not know which version you have or when it was pulled. And so you may not even know exactly what you’re running. 

So there’s some basic things like know the code you’re using. And then of course, the next big one after that is use a build service, right? You’re not allowed to build production code on your laptop because that device is probably in most cases not trustworthy for its supply chain attributes. And so those are kind of two basic examples of things that corporations can easily do and should be expected to do. 

Open source maintainers don’t have it so easy. For example, clearly they’re gonna use dependencies off the internet, they’re not gonna have a proper copy of all the dependencies, that kind of misses the point. And they typically have to build and test some packages at their own expense using whatever means they have. So to ask them to pay for a build service to do that is pretty unreasonable. And in fact, one of the things I’m looking at is how we could maybe build or rebuild packages in a secure signed way for free for open source maintainers. Maybe not all the time, but when you want to do a major release or something like that. So unclear where that will go, but there is plenty we could do to make it easier to be a maintainer for open source package, especially one that’s critical to the rest of the world.

Omkhar Arasaratnam (05:02)”
Absolutely, the notion of these kind of hermetically sealed build environments that are kept absolutely pristine versus some of the horror stories that we can think of — like, you know random laptop mining bitcoin on three of the cores while trying to do a production build of a package — that definitely has some obvious benefits in terms of using the hermetically sealed build environment, of course. 

It does sound like we’ve got quite a few things that we need to be focusing on, but since your involvement, which even predates the creation of the OpenSSF way back in 2017, 2018, I’m now referring to your involvement versus the OpenSSF’s inception in 2020, what are some accomplishments that we’ve had in the field of open source security? I mean, it’s been about, you know five, six years. What have we done so far that you’ve been proud of?

Eric Brewer (05:58)
I think there’s a bunch of things. The biggest one is probably just awareness, right? The US now has an open source strategy, right? There’s a cybersecurity executive order. Both of those things I had some influence on, but I didn’t cause them, right? The importance of these issues caused them. And so that awareness is just symptomatic of broader awareness from all the ecosystems like Python or Rust where they’re now taking their role seriously in how to secure supply chains, how to, in general, improve the hygiene of development in open source and also how to make it easier on maintainers. Those are all good goals. And our best approach is to prove the tooling that maintainers use so they don’t have to do the right thing by some kind of behavioral change. That’s hard to communicate, hard to educate.

If they can use the tools they like and it happens to do the right thing, that would be the best outcome.

Omkhar Arasaratnam (06:55)
That sounds like some great accomplishments that we’ve had and completely agree with you. Let’s move now to talk a little bit about the watering holes of our community, the software repositories, the package repos. Can you talk a little bit about how improving the security around those software repositories can help improve the security of open source software in general?

Eric Brewer (07:17)
Those are critical, critical components. Now, source code for these things is mostly in GitHub, and that does help in the sense that GitHub is at least a group that cares about improving security issues, so I thank them for that. Package managers are more complicated because there’s many of them, and each ecosystem has its kind of own culture and philosophy around these things, and so we’re not gonna have some universal agreement on exactly how to do supply chain security. But I think we can agree on some basic things. 

Like you would like to know for which are your critical projects and which are not. We’re not trying to improve, you know, or maybe raise the security burden on all things open source. Most things open source, frankly, aren’t relevant to national security. But a surprising number are. So I do expect we’ll have some kind of national dichotomy over time, which is those projects that know and actively accept that they’re part of critical infrastructure and take it seriously. And those that really want to just make their own website and create a new programming language do various things that are for their exploration or fun or all the great reasons one might do open source, right? But they’re not committing to say this is a viable thing to use in critical infrastructure.

I’d kind of like to get to the point where most projects know which camp they’re in, right? Because we have a bunch in the middle right now where it’s mostly fun, but oh, by the way, it’s included in very important things, right? Not by their choice, by the way. Being included in something is often not the choice of the maintainer, and that also causes many problems.

Omkhar Arasaratnam (08:49)
Yeah, that dependency graph can be very interesting. Looking ahead, what do you see as the next big hurdle in open source security, and what can we do about it?

Eric Brewer (08:87)
It’s a great question. We’ve done a lot of focus recently on package managers and SBOM generation. There’s still plenty to do there. I do think we need to start tackling build services and automated testing. And the reason these are harder is because they cost money. Even if you’re willing to volunteer time to work on open source, you may not be willing to pay for extensive building and testing. And how that gets paid for is I think one of the great open questions that we need to sort out the next several years. 

The point though is that we can’t really quickly fix security bugs today in part because we don’t know when you make a change to an open source package if it’s gonna break users of that package or not. So there’s some uncertainty about even pure security changes about what impact they might have on the rest of the ecosystem. Now the good news is most of the time these changes are small, most of the time they don’t break the ecosystem. But we don’t really know because we don’t have that many test cases and, in general, we don’t have test cases for all the dependents either. All the things that use Log4j, they may not have test cases that can tell if the new version of Log4j is gonna break them or not. So we have to kind of put forward changes, hope they don’t break stuff and cross our fingers. And that’s not a great place to be. 

I think if we had more automated testing, more test cases, control builds, all those would improve our chance to deploy security patches quickly and effectively when the time came. So that is an important. medium-term agenda, for sure. And this is a place actually where machine learning may be a great benefit because it looks like we’ve had some success already with generating test harnesses for fuzzing, improving test coverage. I think we can generate test cases, or we can probably do more on the front of actually making simple patches easy for a maintainer to accept. So I kept this patch to do a version bump to get the new Log4j. Some of that already, but there’s, these are mundane things and if we can do mundane things automatically, that will save everyone a lot of unfun time.

Omkhar Arasaratnam (11:07)
Absolutely, reduce toil wherever possible, right?

Eric Brewer (11:11)
Absolutely.

Omkhar Arasaratnam (11:12)
So we’re going to shift gears a bit and now we’re going to go into what I hope are a bunch of fun rapid-fire questions for you. So first out the gate, and I’ll provide you two options. The third option of course is, Omkar, it’s neither of those. Here’s my answer. So the first question is, spicy food or mild food?

Eric Brewer (11:33)
Sadly mild.

Omkhar Arasaratnam (11:34)
Oh no. Oh no. Would you like to say more on that?

Eric Brewer (11:38)
Well, I’ve been to lots of countries where there is no mild. Hyderabad in India comes to mind. The mildest they have is near the top of my spice chart. I do have, for better or worse, a very sensitive palate, which is great for wine tasting and a few other things, but it’s not good for eating spicy food.

Omkhar Arasaratnam (11:58)
Okay, well, good to know. Second question for the rapid fire. Vim or Emacs or other?

Eric Brewer (12:06)
Emacs still. Picked it up in grad school in the 90s and know certain keyboard shortcuts that are just deep in my brain.

Omkhar Arasaratnam (12:08)
Oh boy.

Omkhar Arasaratnam (12:16)
Emacs is a great operating system that has an editor attached to it. All right. Number three in the rapid-fire, and the final one for the rapid-fire, Eric. Tabs or spaces?

Eric Brewer (12:26)Spaces.

Omkhar Arasaratnam (12:27)
All right! We’re moving to our closeout now. So you’ve been in the industry for quite a while. You’re very well-regarded and have accomplished a number of things. What advice do you have for somebody that’s entering our field today?

Eric Brewer (12:42)
Well, the good news is I think open source is a great way to boost a career in computer science or tech more broadly. And it’s amazing how often I’ve wanted to hire people because of what I saw them write in open source. And I’m not saying that’s an easy path, but boy, it is a good path because it’s very clear to tell what you care about by what you choose to work on, and I can see the code you write, how you feel about lots of different topics, for example, tabs versus spaces, which is not a criterion I would use for hiring, but I might notice. 

Omkhar Arasaratnam (13:17)
Good to know.

Eric Brewer (13:19)
So there’s that role of open source, which is as a proving ground or a way to increase visibility. And by the way, if you’re helping those projects, that’s a great thing to do, regardless of even if you’re doing it for reasons to get exposure to learn about a project or space.

The second thing I would say is that, you know, it’s worth learning how to interact with many different projects. The easy thing to do is pick a project and work on it, and there’s certainly value in that. But I also feel like I’ve learned more when I’m working on five or ten different things at a time. They have slightly different cultures or different rules about what you do to submit code or readability, things like that. And those choices are worth understanding. And I’m not even saying that they’re good or bad. Point is they are diverse and the communities are diverse, and it’s much more important, I think, to be able to contribute to someone else’s project than to your own project, Most of what the world needs is interconnections of projects and glue. And that is itself a skill that’s worth acquiring.

Omkhar Arasaratnam (14:26)
I completely agree and the old adage I suppose holds up which is we can’t solve tomorrow’s problems using yesterday’s thinking, so encouraging that diversity of thought is certainly paramount, especially in security. Last question for you Eric and thank you for having been so generous with your time. What’s your call to action for our listeners?

Eric Brewer (14:45)
It’s a great question. It depends a little bit on the listeners, but I think I would start with all of the communities typically based around languages like Python or Rust, and what can you do to make your community have better support for security? And that can be all kinds of things. It can be two-factor authentication. It can be helping with automation for SBOMs or for signing things correctly or being able to know for sure which source goes with the built artifact. There’s so many things that communities can do, and kind of need to do, that that’s an area of great need right now, where they aren’t hard things to do. They just need people to think about for their community, how do they want to solv d then go do it. And that would be a huge benefit to the community but also to the world.

Omkhar Arasaratnam (15:35)
Thank you so much, Eric, for your time. That’s it for What’s in the SOSS. Stay tuned for our next episode.

Announcer (15:40)
Thank you for listening to What’s in the SOSS? An OpenSSF podcast. Be sure to subscribe to our series of conversations on Spotify, Apple, Amazon or wherever you get your podcasts. And to keep up to date on the Open Source Security Foundation community, join us online at OpenSSF.org/getinvolved. We’ll talk to you next time on What’s in the SOSS?