Video: Overcoming Vulnerabilities in Hardened Container Images | Duration: 3148s | Summary: Overcoming Vulnerabilities in Hardened Container Images

Transcript for "Overcoming Vulnerabilities in Hardened Container Images": Are we live, Erin? Looks like we're live. Hello. Okay. So folks are joining right now, we'll give it a minute or 2. How's everyone feeling? Happy Wednesday. Yeah. It's morning for me, but, Michael, it's gotta be later for you. Yeah. It's 6 PM, so, actually, I I will not be on the on the computer anymore this time. Okay. This is a dinner a dinner chat for you. Yes. John Speed, lunchtime? A lunch chat. I live outside of the. So, I I quickly ran upstairs, ate a little bit of food so I wouldn't be low blood sugar and nonsensical and ran back down to my computer. So here I am. Good. Great. Great prep. I wish I had a bowl of cereal. It's it's morning for me in in San Diego, and I think a a bowl of cereal would be an excellent, you know, addition to this to this discussion. Alright. We've got we've got 10 folks here. We're at 2 minutes after the hour. Should I give it another another minute? You can also just do introductions and then and then anyone that wants to jump in after that. But up to you. However you'd like to handle it. Alright. Alright. I I will kick us off. Okay? Early bird gets the worm. So hello, everyone. Welcome to our webinar, Securing the Stack, Overcoming Vulnerabilities in Container Images. I'm Erin Glass, a senior product manager at Chainguard, and I am going to be the host of today's discussion. At Chainguard, I create educational materials on software supply chain security for traditional and AIML development. So I am really, really excited to talk to 2 experts in container security today, John Speed Myers and Michael Wagner. To start, I'd just love for both of you to introduce yourselves. Michael, would you like to start? Sure. Yeah. So name's Michael. I currently am I'm working as a consultant for cybersecurity specializing in, application security and all that. They have sec ops, having the secure development life cycle. Especially, I'm focusing on tool integration, capping teams, with tool integration and also with the findings of different security scanning tools like we have for container images, for example, and also static, scanners for source code and other knowledge supply chain stuff. And, yeah, before I joined cybersecurity, I I was working as a software engineer for around 10 years, so I have a lot of, engineering experience as well. And, yeah, I guess that's all for introduction. Yeah. Fascinating. If we have time later, I'm gonna ask you about that switch from engineering to, to cybersecurity. Yeah. Because as you mentioned in your paper, right, not all software engineers are very security. You're right. But we'll save that. We'll save that. Okay. John Speed. Who are you? I feel embarrassed because that's such a logical way to come to studying container security. I have a more illogical path, personally. I I'm actually a public policy researcher by background, at least academic education. But, while doing research at a organization called In Q Tel, it's something like a venture capital firm that works with the US intelligence community. I stumbled upon open source software security as a research topic and became obsessed. And so I started writing about software supply chain security and open source software security, really more from, almost an economics perspective or a statistics perspective and really accidentally fell down the container rabbit hole. I came to ChainGuard a couple years ago, and, I had only ever really casually used containers just as a Docker user, using Docker pull and building Docker files. And, over the past 2 years, I have, you know, in this fortunate situation, been at that, place that has a lot of, expertise on containers and have become a researcher of container security of vulnerabilities and Linux distribution and packages and containers. And I now manage a research lab, Chinkard Labs, where myself and others, do research on container security. So that's really, how I got here. And, John Speed, I have to say one of the reasons I love talking to you about open source is because of that unique background you have. You know, coming to open source from almost an economic, you know, perspective that those are the that's the interest that drives you here, I think you come up with, really unique observations about this space, so really excited to dive in. Okay, well, as I said I'm really excited to hear more about your research, you've both discovered some findings, that I think will be very interesting to our audience today, and anyone really who works with containers on a day to day basis. But first I really want to make sure that we are all on the same page with one of the key concepts of our discussion today, right, hardened containers. So this question is for either of you. What is a hardened container, and by contrast, what is an insecure container? Oh, good thing we have Michael on the call. Yeah. I mean, I can start. If you talk about hardened containers, I guess you mostly talk about component, though, like containers, which have, reduced attack surface through reduction of components inside. That's why we always talked about component reduction in my research. But whatever you call it, the tax surface and, therefore, the security of containment container images, relates to the to to the amount of stuff lying around in which we will go into detail later. But, yeah, I mean, I guess that would be the definition of hardened containers. You could you could go a little bit far more far and say, like, okay. What about the host hosting the containers? Like, there there are special host operating systems, which also on a pretty minimal base, so that if you break out of a container or if an attacker would get into your container, into a running container, and they could break out, it's called container escape, then they would be on the host. And, there's a there's a best practice when you build your Docker images. And by default, you are the root user, which is the the super user on the Linux system. And if you break out your the root user inside a container, it's the same as the root user on the host. So if you would break out, you would immediately have root rights on the host of this machine, and then from there, you could do a lot of a lot more, bad stuff. That sounds not so so great. Confident. Yes. Yeah. Yeah. No. That's great. I love asking this question because I find everyone answers it a little bit differently and it it sort of widens your perspective on, like, what what does hardened containers mean. John Speed, do you have anything to add to that? Or Yeah. I'll just say that for me, I I I sometimes observe in computing circles that there's a intense need to define terms, and the optimistic hope that it's computers. It's all zeros and ones for software. Shouldn't there just be a concrete definition that can be reduced to zeros and ones, to if then statements and loops and typical programming concepts. And the tricky thing with a term like secure containers or hardening containers is there's a variety of perspectives. And for me, there's a number of components to the definition that I think are there's a rough consensus on. I think component reduction is is is an essential, part of any hardened container. Having unnecessary components, bloat increases attack surface. I think, one positive benefit from that has been having a fewer number of vulnerabilities, especially known vulnerabilities or CVEs in that container associated with those components. I think, sometimes, harder to appreciate benefit of having these more minimal containers too is that as new vulnerabilities show up in software, which inevitably happens, it also means your rate of vulnerability accumulation is smaller. So it's not only about having fewer vulnerabilities at day 0, it's about having fewer vulnerabilities at day 10, when you often, can't update your software every every single hour. I think there's some other things that come up too. I I won't even begin to mention them as there are there are a range of definitions. I'll just submit very coroquially that the research I'll talk about later. I definitely focus on CVEs. And that's not because I don't care about component reduction, and it's not because I don't care about, permissions and another a lot of other important Kubernetes and containers concepts. It's just because I think it's one useful angle. And, component reduction also often serves CB reduction purposes, though it serves others too. So, it's a good question, and I only have, just bear some simple ways to answer it, but I think there's a lot. Awesome. Well, we'll we'll get more into that as the conversation goes along. Michael, you wrote a paper about component reduction as a technique for, container security. Tell us tell us about this paper. First of all, what motivated you to write it, and then what what did you find? Yeah. Okay. So regarding motivation, so I, actually, it was my master thesis, and I published it, because of for NDA reason. I I just made a smaller paper out of it. But maybe let's start with the motivation on it. I was consulting or still consulting in a large enterprise. In fact, it's the it's one of the largest telecommunication enterprises in Germany, and they're also part of critical infrastructure. And what they do is they have this radical, radical digitalization project where they where they want to radically move old legacy applications to a cloud, and then they they build up a central product security team where I'm part of, consulting. And now what what we saw there is imagine you have a Java application on, like, 10 years old, was running fine, and now should get containerized and moved to the cloud. And then also they are as as they brought up this new product security team, including new policies, for regarding security, then we we introduced container image scanning. And now imagine the team that that that what we had a lot the the issue with the teams. Like, they containerized this old application. We scanned it, and then you have these policies, these new security policies telling them you're not allowed to deploy to production if if there are any critical or highest severity findings from the scanners. And then they never scanned before, and then we scanned them, and then they have, like, one application had, like, 200 more than 200 critical and high findings. And that was and this is just one example we had with so many teams, that was that you you could call it, for them, it's like a WTF moment because because they they were like, hey, guys. They're coming back to us in the in the product security team and say, yo, guys. 1st, this has nothing to do with with us. This is all operating system related where where we like, John, I saw you. You brought up the same comparison, which I really like. We told them, hey, guys. This is the same as your open source components. You you are responsible. If you select that base image, no default node from Docker Hub, for example, or default JDK or whatever. And this has, as we see later, like, 200 security vulnerabilities in it, then it it's it's exactly the same as if you select Spring Boot framework or Express JS or what whatever open source library you bring in. If if if Spring Boot has a vulnerability, attackers don't care. You know? Like, they they will exploit their vulnerabilities. It doesn't matter from from where they come, if they come from the operating system files, in that case, the image or from the open source library. Yeah. So I like the comparison, and that was the first argument. So, guys, this has nothing to do with us. And and second, we we don't have time to fix that because we want to deploy tomorrow, something like that. And and the thing is we had that with so many teams to to make that scale. Like, this is what you you also have this nice blog post that chain got where you did the interviews with teams and, where you say thousands of, hours are wasted of of vulnerabilities, blah blah. And, yeah, this is what what we experienced with the with those teams because, then we said, hey. This won't get scale at all because we had so many discussions, and they then you have findings like curl, g lip c and all that complex all that stuff lying around in the image, which, of course, doesn't have really much to do with the application or not at all, but if this application run-in the cloud, and this could get exploited. Right? So we did some research, and we first we found who we did to list, which is, I guess, the the big biggest player in that component reduction area. They started, I think, already 2013 with their open source project for component reduction of container images. And we found that, and I I was still having my master running beside the job, and I I had to write the master thesis someday. And then we said, well, why not take this topic as as the research topic? And the professors were very good with it. So that was the motivation for me. Yeah. And so what did you what did you find? Like, what, you know, what were you searching? Or Yeah. Sure. So I had 3 research questions, 2 of them published in the the paper here. So the first one was if if a reduction of confidence would actually reduce the attack surface. And the and the goal to answer that question in a scientific way would would was to scan and compare, regular images with the component reduced ones to see to hope to see in the end a drastically reduction of the CDs or the findings of the scanners. But I would I was also interested in the second research question. I was also interested if, if typical findings are actually exploitable because, like, I was thinking before, like, okay, you have so many findings, but, if you do a lot of cybersecurity in the AppSec area, you see a lot of false positives anyway. All scanners bring you stuff which isn't relevant. So I wanted to check if, yeah, if they in indeed actually exploitable. And then the third one, which which we didn't publish, but, we we did a lot of interviews with teams, because some of them already use Google Distressed, and we wanted to check what would be the implications when introducing component reduced images compared to using regular ones. So, yeah, I mean, I can quickly say what we've what we've found. Like, research question 1 was definitely to be expected because, like, all these tools advertise with the reduction of of vulnerability findings and reduce the tax surface. But interesting is that chain guard by far was what made the best results because they had, like, 0 with with all. I think maybe even Node. Js had 1, but it got fixed quickly. So, as I said, you you check the we I checked the number of vulnerability findings for certain container images for certain runtimes relevant to the enterprise. So we had Java, Node. Js, Python, and PHP, I think. And, yeah, the finding was You're looking at a variety of right. You you looked at distroless, and I'd love for you to just quickly define distroless. But you looked at a few others. Right? Like, the chiseled images and yeah. Maybe give a quick overview of of what you looked at. Yeah. We looked at Google Distulas and Alpine. Alpine also, like, you can say it's a component reduced, container image. They're also providing, some runtime environments. Then we had a look at Ubuntu. It's called Ubuntu Chisel. It's also an open source project from Canonical, the nonprofit arc behind Ubuntu. And this is pretty new, project started 2030 2023 or something. And what they do is they're taking Ubuntu as the base image, and then that's why it's called chisel. They chisel only the images you need, only the components you need for your application. But this is quite on on a yeah. It's an open source project and new kid on the block, so to say, so it's it was not so easy to build, and they don't provide production ready runtime images like Google, this, and chain got 2. And the last one we checked is Red Hat UBI, especially micro. I saw you also did check that, but we I I just had a look at the the UBI micro because they there's this funny blog post from Red Hat. If you Google for Google Distulas, you will find pretty quickly in the first results this blog post from Red Hat bashing against Google Distulus. But if you read it till the end, you see that they this is just an advertisement for the UBI micro project. So that's why I came on this. And, yeah, these these were a few. And to be honest, I I didn't have chain dot on my on my map, in the beginning. So during the during the research, I did a talk in Hamburg in Germany. It's called container days. It's a very large con conference, regarding containers, and I already I applied there, and I I got, they they let me speak there, and I I had my talk there with Google Distress, RedEdge, Ubuntu, and Alpine. And then in the end, someone from the audience asked me if I checked Cheynga, and I said no. To be honest, no. And so I took Cheynga into the research, and then it, yeah, it made the best. So thanks to the guy asking me in the conference. I wonder if that was a a plant. No. Just kidding. Before I turn to John's speed, could you just quickly, define distro list just for folks who who might not be familiar with it? Yeah. So, basically, what what a container image is, it's basically just, you can you can imagine as as a file system, you're making the the file system structure, normally, Linux operating system, and it has stuff like SSL, stuff in it and a c library, normally, you need. And and then, and normally you want to run an application on it, so you need the runtime. For example, Node. Js or Java runtime environment. So these are basically the components you need in the container image, and and this to list like, if you have the base image for Deviant or the base image from Ubuntu, they take it from the default, Ubuntu distributions, and therefore, they have a lot of stuff which the application won't need in the end. And this is this was the idea for distillates. For example, like, there's a package manager inside. There's a shell inside. Like, you don't need a shell normally when you run an application. You don't of course, you don't need the package manager in in production. And that that that was the idea from Google Distodes, which, yeah, got used from Chengdu and all the others as well. So especially no shell, no package manager, and reduced components means distilless. So distroless is a way of taking out some of these standard components to The RTC. Yes. Yeah, okay, that's great. Alright, well thank you, Michael. John Speed, you, have a very relevant complimentary paper. Love to hear more about the state of hardened containers. Tell us tell us a little bit about it. Thank you so much, Erin. Well, what I'm about to present, and I'll try to share slides in a second, is really a culmination of research by myself, but usually others too. I, steal with credit and often co authorship others good ideas. And, I'm gonna talk about this state of hardened container images report that published just a couple months ago. And I really see it as, a way a a first cut, of trying to assess, well, what is this thing hard in container images? What are the options out there, and how do they do? And so let me see if I can share some slides right now. See if I can switch to 1. I'm actually gonna start with, something that's not in the state of art in container images report. It's actually a report that I worked on with a colleague, Zach Newman, a couple years ago. We called it this is a silly title. It's all about that base image. And this was before there was even Shangard images the product. And we asked ourselves this question, which is really popular container images, how many vulnerabilities do they typically have? And I grabbed just a couple here, Node and, Red Hat's universal base image, one of their and I I did all this a couple years ago. So, if you go and scan it today, I'm not seeing the same results. And I took a few different popular software composition analysis, CVE scanners, Trivy, Gripe, which are both open source, and Docker, Scan are under the hood. It was Snyk at the time. And I scanned these images. I scanned some others too. I just grabbed some. Long story short, a lot of popular container images, because of having, so many components like Michael described, have hundreds of known vulnerabilities. It varies on the container image. It varies on the tag. So that's why I put some asterisk in these next few slides to show you that, results will vary. But the short story is often developers, when they reach to describe a popular container image, end up with without knowing it often that they have a container image with lots of known vulnerabilities. And part of that is because they have lots of components in it, too many components from the runtime perspective. And part of it, as you'll see is it's often using packages that themselves can't be updated to fix a vulnerability. There just is no update available, depends upon the distro. So this really started a research agenda that I've been a part of, especially with Paul Jaber and Trevor Dunlap, who are also members of ChangeGuard Labs, to to study containers and Linux distributions and understand why why are there so many vulnerabilities in containers, and, what can be done to reduce them. So I'm gonna show a few I kinda think of them as common sense approaches to try to reduce vulnerabilities. I'm gonna show you the results. I'm not gonna show the entire hardware container images report just because I think it's it's too much. But let me skip ahead. So I think one thing I've often heard, I was, doing interviews that Michael mentioned earlier, of platform engineers and their experience dealing with containers and vulnerabilities. I heard I heard at least one of them say very clearly, you know, the good thing is oftentimes, you can simply update your operating system base packages and get rid of vulnerabilities. I said, oh, that's a good point. I wonder to what extent that's really helpful, though. And so, I did an analysis, or really Trevor Dunlap did one, and then it was incorporated in this report about taking a set of popular container images, about 10 here, and, doing a before and after comparison. If you have this popular container image, go into it, do essentially a APK update or the equivalent. These are all dead end based images. And, try to remove or update to the latest versions of all the packages inside these containers and get to as close as possible to 0 CVEs. And here's the problem. At least with these container images, these popular ones, there's no magic. Simply an a p t APT update. It really hardly reduces vulnerability. Some, let's say, 5 to 7% on average. It's not a magical wand. And that's because containers, one way to think about them, is really a bunch of packages that are placed into this, larger meta package. And if those packages themselves, they come from often come from Linux distributions like Debian, RHEL, and others. And if they themselves, don't have a fix in that package ecosystem available, it just hasn't been packaged or, it's not considered stable yet, well, then, you as a container user, you're just just out of luck, unfortunately. So that's what this analysis is supposed to show or show a couple others. Another technique, and it's very closely related to component reduction, but it's actually about debloating containers, is I I, with a colleague, Paul Jaber, we asked ourselves, well, is there a way to simply debloat these, quote, bloated containers, these popular containers? And so we took 30 containers, just looked at their off the shelf number of vulnerabilities before debloating and then post, debloating. And, here's the story. Once again, before you debug popular containers, they often have hundreds of known vulnerabilities. In this case, about a 100 on average, though it varies widely. And we took a a tool. We were able to analyze the results of using a debugging tool by the company RapidFort applied to these container images. They actually make available a set of open source container images where they provided, deep loaded versions of their containers using this tool. And the short story is it works. It does indeed reduce vulnerabilities. We found by about 65% on average. But, the problem is is once you've done that reduction, you run into the same problem you ran into earlier. You're still using packages that don't have fixes available often, and there's gonna be dozens in this in this sample, dozens of known vulnerabilities, CVEs that still exist. So, debugging is helpful. It doesn't get you down to 0. I'll share just one more and then say a couple more broader things about this research. I think another thing that shows up is there's a hope that you can simply use there are there are there's some sort of repository out of there with blessed, hardened, golden container images for everyone to use that will simply simplify the developer's job. And that is not the case, at least, once again, asterisk. But there is a popular, container image repository called Iron Bank, that's part of US military, and they, make open source a variety of container images, including some Cheyngar images. So this is not an either or thing. It's you can have you can have you can use Cheyngar through Iron Bank. And, I we analyzed some of the most in this report, some of the most, popular container images in Iron Bank, the non Cheynguard ones. And, while, Iron Bank is doing a variety of useful things for the military community and military contractor community, that isn't captured in this, the the the finding is that, once again, these containers often have, in this case, a 100 plus known vulnerabilities CVEs on average in them. So stop sharing, and I'll just say something slightly broader, which is I think, come back to the question that Erin raised, which is what is a hardened container? And this report takes opinionated stance. It says that, CVEs and the number of known vulnerabilities, and sometimes it focuses on critical and high, are our key measure of security. Not the only measure, but a key one. I certainly think there are ways in future reports to broaden that definition. Some obvious ones came up in the review process. People will say, well, I want containers that are also free of malware. Why isn't that in the definition? And I think that those sorts of things are certainly welcome. But there are so many problems. Reducing CVEs is very clearly a low hanging fruit in the c container security ecosystem. And so that's the, lens of this report, admittedly, simplistically embraces, because I do think it provides a lot of value and, usefulness to container users. Thanks, John Speed. One of the things that I found, most surprising about this report was your analysis of Iron Bank. Right? I mean, it was shocking to me to realize that a registry for the US military that prioritizes security, they still have a huge problem with CVEs. And so the question I have for you is, 1, you know, were you surprised by that finding? And 2, why is it hard even for something like Iron Bank to, resolve or remediate these CVEs? Why do CVEs still exist for for them? Yeah. I think in, with all due respect, it's not that surprising because there's actually a deep technical challenge with all of our that shows off. And then that is it. If you're using, often off the shelf convenient images, and, it is hard to get them to 0 CBEs. These, a lot of pocket containing images have a lot of stuff in it. And if you're not diligent in the way that Michael described, it's easy to just have a lot of components and have them be out of date. So it's, I think, without a radical rethinking of, how container images are built, the sort of thinking that, you know, Google Distributed and not changes do and others do too. I think it is it's hard to achieve it on technical grounds. It's not that, the the persons and military units associated with Iron Bank aren't trying hard. It's not that it's difficult. And and really why? I think it goes back, and this is why ChainGuard had to create its own Linux distribution in part Wolfie, to power ChainGuard images. It's you really need a full control of the supply chain of these container images to, affect the sort of radical reduction that is associated with, chain guard images and others. And so without that, it's just like, maybe you actually need to create this meme. It's really, like, just someone a developer drowning, with CVEs pouring onto them. There are so many components in these container images, popular container images, often hundreds of components, many of them not used, that, it's really a CV tsunami, unless you're very diligent and rigorous in what you put in your container and how you can contain it and update it. So, it's really no knock against the update, the, you know, the technical strategies of, Iron Bank. It's really that's a very difficult problem and requires a rethinking of what goes in the popular container images. Well, then what about, you know, why don't upstream maintainers just fix these CVEs? I think you mentioned in your report, was it Red Hat that labeled some of the CVEs will not fix? And I know it's not just Red Hat that does that. You know, from your perspective, why would an upstream maintainer decide to label something will not fix? Yeah. I think an interesting thing is that the vast majority and I don't have the stat at my fingertips. The vast majority of open source projects, popular projects that have vulnerabilities in them do release fixes for those vulnerabilities in that upstream code, in that actual Git repository or whatever it may be. But there's this this critical distribution layer of modern software goes into Linux package ecosystems or the popular package histories, Python package index, NPM, and others. And, if there are updates and changes in that upstream code and that upstream repository where the, the maintainers fix it, but the that middle component doesn't pick up those changes, doesn't update sufficiently rapidly, well, then you're really in some sense propagating a vulnerability throughout this chain of, packages and then ends up in the software we all use. So, really what's going on is that with without controlling that middle layer, that distribution of packages, there's just CDs for everyone. And what option would a, a developer have if they, you know, notice that there's, CVEs in their container that are labeled will not fix? Like, what what are they supposed to do then? Yeah. I think a lot of them, unfortunately, just live with that reality. They say, hey. I have this I have this finding. I'd be curious if Michael's had been doing this. Sometimes, unfortunately, what a lot of software developers or security teams or compliance teams have to do, they simply have to account for them. That turns into long, messy spreadsheet, so it can be in other software too. That where they simply say, hey. This vulnerability showed up in this component with this version. There is no fix available. You know, fortunately, there's this mitigation or that mitigation, and it creates tons of toil and work. And, usually, it's not the sort of work that these persons want to be doing. It's it's considered tedious and low value work, but necessary, to meet security or compliance goals. So that's what happens unfortunately is when you have a bunch of will not fixes, it creates a lot of work for the persons in that software supply chain. Mhmm. Yeah. One thing I mean, it it's first dependent if this component is used at all. So you have you have to check this if this component is is not used by the application and not needed, then it's just the the typical false positive finding or maybe there's an a privilege escalation vulnerability in g libc. So this application does not use g libc, but if an attacker would make it somehow in a container, then he could use that g libc to escalate privilege inside in container and get root access or something. So if it's not used, they could just remove it even though it has an no fix, no patches available. And if it's used and there's no no patch available, then there's typically, like, a risk process initiated, and the vulnerability management team will have that on their map, so that they track it until they have maybe be a fix. But I think, like, I think in the in the distribution of, like, the package managers, from the distributions like APK, for example, I think some sometimes they say they they won't update that package even though there is a patch on GitHub, for example, then you this is just what you mean, you know, right, right, with the with the distribution life cycle or something, I would call it. And then you have the distribution life cycle of the deviant image. So that's why because we also did that in the beginning, like, that common sense stuff, like doing app upgrade and doing app get remove and or whatever, delete files manually, and this was so error prone. And Docker if you do Docker build, then you get so many errors, and the team just then the team get back to us, and they just hate us. Like, we we tell them, like yeah. They hate the security team for this. This was one of the main points also, which motivated them the research in the direction of component reduction. I was one of those folks that once hated security teams. Not anymore. Not anymore. Michael, this ties into, you know, one of your other questions with your research, about exploitability. Okay. We know that we have all these CVEs, but are they exploitable? So I'd love for you to talk a little bit about that, why some of them may not be exploitable. But, also, at the end of your report, you seem to still recommend, okay, even though low percentage of these are exploitable, you seem to recommend that we we still need to address them nonetheless. So could you could you talk about exploitability and your thoughts about whether this changes how we should deal with with CVEs? Yeah. Sure. So, yeah, as I said, we wanted to have a look at the actual exploitability as well. So I did 2 things to answer this research question. The first thing was, like, picking picking manually picking some findings based on, the categories of, the tools. So for example, you have 2 base categories normally in container security scanner findings. You have operating system related and, runtime related findings. So operating system related could be the c library lying around or, like, some binaries lying around like curl or open SSL, stuff like this. And then you have runtime related, issues like global ins globally inside node packages, like or or sequel or whatever, stuff like this. So I did a manually, I did manually pick some of these, some from operating system rated, some from run time related, and did a manual analysis of the CVEs related to the findings. And it was, like, I don't know, 15 or so or 20. And for, like, most of them, I've I've found that you can have we looked at the CVSS, which is the common vulnerability scoring system, which, calculates the severity of a finding. And then you have these different vectors like, is it is it remotely exploitable or not? Is user interaction needed to be to to make it exploitative or not and stuff like this. And I I went through them, and also I only took critical and high findings because that was related to the policies of the enterprise. And, yeah, basically, I found that most of them are false positives. So for example, I found that often you have to have a look at them in the context of the application. If you just scan the container image like we did in the research, we we scan, like, node node 17 like you showed, John, or we we scanned, Java, JDK 17, whatever. But then then you have, then, for example, it it find it's a node run time, and it just to give you an example, it finds that there is a node package called tough cookie, which has a critical vulnerability, but you have to check, is the application using that library? If not, then false positive. And and then with the operating system related stuff, yeah, we we I also said, like, for example, there was a g libc privilege escalation finding, which could, could make could make an attacker elevate privilege, but it was not exploited remotely. So the attacker would first need to get somehow access inside your container. For example, a common a common thing is that they exploit your application itself. So vulnerability in the web application, for example, so which which enables a shell for the attacker, a reverse shell something, so they get direct shell access inside the container. And then there there are frameworks like Metasploit. And when you're on the system, you can you can run some exploit, so just us. And they would then, normally, they would probably find that, oh, there's this GDC. You could use it to, to escalate privilege from the regular user to the root user. But but also that one, it's it's highly unlikely, like, likely. And I said, most of you have to look at the context of the application to to really answer that question. But the second big thing I I did to answer the question is I found during the research and doing the related work research, I found that there's a, there's a project. It's called EPSS. It's the Exploit Prediction Scoring System, and it's it's from the same guys. I I just don't remember the name, but it's it's from the same group who does the CDEs. I just don't remember the name right now. But what it does, it it calculates per CVE the average exploitability within the last 24 hours. And how does it do that? It it has a look at the CVSS scores and the exploitation vectors here, but it also checks a lot more. It checks, for example, GitHub, security adversary database. It checks the vendor, adversary databases from Red Hat or Deviant, and it it even checks social media, mentioning CVEs and all that stuff. And it has, like, pretty scientific method of doing that calculation, and it provides an API. So you can send in CVE CVE to that API and gives you back the APSS. And what I did is I I run all CVEs, all findings I had in my, through the scans. I had, like, 500 and and a few, and I calculated EPS EPSS for all of them, and I plotted it. You can see it if you if you look at the paper. I plotted it on on the x. It was EPSS score, and on the y, it was, the how was it? No. On the y EPSS. On the x was the CDs, of course. Yeah. And what we found to for 5 a 100, and a few findings, we found one which has, like, 40% of exploitation probability within the next 24 hours, and we found one which has, like, 7%. And all the others, like, 510 or so had, like, 0.000 0 7%. You know? Also, that that was the main scientific method that I used to answer the the question. And so are typical vulnerabilities exploitable? I I it was a definite no, based on EPSS and based on that manual analysis. But as you said, Aaron, I said, even though they are not exploitable, in in the end, it's still better to to reduce it in the first place because you you save all that headaches and the discussions and the email chains and the meetings with, like, and sign of calls. So we we did we did some manual. Often we have to come in because the motivation, for developers to use Distru Less or Cheynga is not quite high in that enterprise, right now. So we still do manual vulnerability assessments of some findings where we then come out in the end. Okay. It's it's not relevant for you. Okay? We can we can accept that risk, and you are allowed to deploy. We do the sign off. But that we could we could just, in the first place, we could just, get rid of that, you know, like, all that complex work and discussion and yeah. So that's why I say even though most of the stuff is not it's probably not exportable and a lot of people in security space say that. So I I I recently saw a meme where they say, what, what makes, what makes everybody hate you in security, and then he said 9 90 99% of vulnerabilities are irrelevant. So this this is this is, this is definitely a thing, but, you have these policies and you have the managers and you have the findings, and the policies dictate no findings allowed, and then you have to check them. So it would be better to have them removed in the first place. Way better. Yeah. And I would I it's it's probably a little bit different in Germany than it is in the US, but there's also the question about compliance. Right? Even if your own organization determines, determines, hey. This isn't really a risk. You know, there might be external, reasons that you would need to get rid of them anyway, or at least that's something that that I'm thinking about here in in the US. Interesting. John Speed. Let's see. I'm I'm trying to tie this conversation together. One thing, John Speed, you bring up is, malware and how malware is actually different than CVEs, but it's still something we need to consider. I don't know if, if that distinction is is really clear to a lot of developers working with containers. Could you talk a little bit more about, the distinction between malware and CVEs and the different considerations we need to make with both of those. Yeah. Glad to. So, a lot of CVEs, though not all, are of this variety. They're a software developer has made some sort of unintentional mistake, has introduced a, quote, security related vulnerability into a piece of software, and a malicious attacker could take advantage of that hole, that gap, that flaw. But there's there, though and there is actually a CVE category for these. There are CVEs that are for malicious software. It's relatively few, especially in open source. That's when, an attacker maliciously injects code that, it, you know, does something, undesirable into a piece of software. So that could be, they, for instance, get permission rights to be able to write to a public repository, and they they add malicious code. It could be that they somehow get access to a registry that publishes, a package of some sort, and they publish a malicious version of code. So, I think, like you bring up, these are not the same, though they do overlap. There are some CDs that are malicious. Unfortunately, there are a lot of malicious open source software packages that never get there's never a CVE record for them. So, there's a really excellent dataset, actually called it's got a very graphic name. It's a trigger warning. Backstabber's knife collection that actually is a large data set of someone trying to, compile all these known malicious open source software packages and the code associated with them for other researchers to look at. And how this is relates back to, container images is that if you think of container images as often just a bunch of open source software packages rebundled and put together, in a convenient developer experience, well, then you should worry not only about the CVEs, which are mostly unintentional vulnerabilities that someone could exploit, but you should also worry about, well, are those packages malicious or not? And that's something, like I said, this this is a area that has definitely gotten more interest in the past 5 years, especially, But especially, in the wake of this so called, XZ Utils Near Miss that, some of the listeners might have seen, where a very prominent, open source, project was backdoored. And, this is a a definition of security that the state of hard container images does not, focus on. It's not because it's not important. It's just there's so much ground to cover. So I, I think this is a ripe area of container users since they're really open source package users, but they might not think of themselves right. We should care about this dimension of security too. And, so that's that's why I think container users should care. It sounds distant at first. You know, open source packages that are malware, what does that have to do with me? Well, since containers are, in some sense, mostly open source software packages, you should care if they're malicious or not. Well, that's great. On that note, let's, open it up to the listeners and see if there's any questions, that they have for for you. And, folks, feel free to just enter enter them into the chat. And, John, speaking to Michael, if you have questions for each other, feel free to Yeah. I'll ask one of Michael right now. Let's see if anybody else wants to. Are you thinking about any future research projects? And if so, what? Yeah. So because you mentioned supply chain, I'm currently thinking about some reachability research because that that's also a topic for us, when you have, especially findings in libraries, like, software software composition analysis stuff. And we want to know if, if if it's really, reachable. So for example, Sam grab is a SAS scanner, and I want to have a look a little bit in in that area. What about you? I just need to let my cat out of the room quickly. Well, so that happens. I'm surprised that, someone has jumped into the store over here too. I've got we've got a couple of things. We're actually doing a, a malware and Linux distro project right now. We're trying to, working with a couple of members of the Chingarn Labs team to try to benchmark different open source package malicious detection approaches. I did a paper with a couple of those. It has a silly title called bad snakes a couple years ago about, benchmarking, malicious detection approaches for Python packages, like the Python package index in particular. We're I I think of it this project as a similar version, but for Linux distributions and Linux distribution packages in particular. So that's something that's on our research agenda right now. So, should be lots of fun stuff there. Okay. Great. Well, I'm just looking at our, our chat here. It looks like there are no, additional questions, and I know we are almost at time. So unless anyone enters 1 in the next, 5, 4, 3, 2, 1. Okay. I just wanna thank the both of you for this excellent conversation. I learned so much from both of your papers. I really wanna recommend that the audience checks them out. There's so much more in both of those papers that we didn't have a chance to talk about. For example, Michael gets into the history of containers, which helps you understand why we are at this moment with container security, so please do check out those reports, and, thank you so much to our speakers, Michael and John Speed. Just a delight chatting with you, and, looking forward to securing those containers with you all in the future. Thanks so much. Thanks, Erin. You're welcome. Thank you, Edward, for hosting. My pleasure. K. Welcome. Take care. Have a nice day. See you. Bye. Bye.