Episode Summary
Welcome to the Jon Myer Podcast, with your host, Jon Myer! Today's episode is going to be an enlightening discussion on a crucial topic in the world of Kubernetes: "Kubernetes Secrets are not secret." We have the privilege of having a distinguished guest, Michael Levan, a renowned Kubernetes expert, joining us to shed light on this subject and help us understand the intricacies of securing Kubernetes secrets effectively. In the ever-evolving landscape of container orchestration, Kubernetes has emerged as a leading platform. However, the management of secrets within Kubernetes can be a challenge, and organizations often struggle to ensure the confidentiality and integrity of sensitive information.
That's why we have Michael Levan here today, with his vast expertise and experience, to guide us through this critical aspect of Kubernetes security. During this episode, we will explore the common pitfalls and misconceptions surrounding Kubernetes secrets, discuss the potential vulnerabilities that can arise, and most importantly, uncover the best practices to protect your secrets effectively. We're thrilled to have Michael Levan with us, as his deep understanding of Kubernetes and his knack for simplifying complex concepts will undoubtedly provide invaluable insights for our listeners.
So, let's dive right in and gain a deeper understanding of why Kubernetes's secrets are not as secret as they may seem!
Are you looking to elevate your brand through compelling and highly professional video content? Myer Media has the solution for you, from start to finish we offer complete video creation services that include Customer Case Studies, Creative Content, Podcasting, and more.
To learn more CLICK HERE:
About the Guest
Michael Levan is a seasoned engineer and consultant in the Kubernetes space who spends his time working with startups and enterprises around the globe on Kubernetes and containerization projects. He is also a member of the Kubernetes v1.28 Release Team.
#aws #awscloud #finops #cloudcomputing #costoptimization
Episode Show Notes & Transcript
Host: Jon
Hi everybody. My name's Jon Myer, and we're bringing you a quick, informative update. And the question today is, are Kubernetes Secrets secret? Well, joining me to discuss this important topic is Michael Levan. Michael, thank you for joining me.
Guest: Michael
Yeah, thank you so much for having me. Appreciate it.
Host: Jon
Michael, the other day you posted on Twitter, it didn't matter what you were using on HashiCorp Vault, AWS Secrets, or other stuff, that Kubernetes secrets are not secrets or still using Kubernetes Secret, which is a misnomer. How about you educate everybody on your thoughts?
Guest: Michael
Yeah, so this is kind of the interesting piece here, and funny enough, posting about it on social media and having conversations with people, people were kind of like, oh, yeah, right? And that's what kind of brought up this topic in general is the idea of, let's say you use something like Hash Go Vault, but you still need the ability to integrate secrets into Kubernetes. Well, whether you're using the operator, whether you're using a plugin like the Argo CD plugin, whatever the case may be, it is still creating a Kubernetes secret for the communication between whether it's the operator or whether it's Argo or whatever the case may be. It's still creating this one-in-one replica in a sense of, for example, let's say the secret is on Vault. It's going to create the secret on the Kubernetes cluster via Kubernetes Secrets, and that kind of defeats the purpose. So at least for me, as I think about it, it's one of those ironic things where let's say you have something like Vault where you're storing not only secrets but certs everything else. Is it worth setting up the integration inside of a Kubernetes cluster? And it's part of me says yes, the other part of me says no, because you know, might as well at that point just start to encrypt Kubernetes secrets via Kubernetes.
Host: Jon
Okay. Kubernetes Secrets, are they secret, or how are they stored?
Guest: Michael
Yeah, so by default they're not secrets. The API has a resource called Secret. So when you create it, it's supposed to be a secret. However, the way that it's stored is in Base 64 plain text via at cd and et e D is the Kubernetes data store. Kind of think about it like a database, so a key-value pair essentially, but that's how the database is set up for Kubernetes. Now, that's by default. There are methods of encrypting the data on ET c D and ET c D itself, but that's an extra step. It's an extra hop to be able to do that. It's not like that out of the box. Luckily, if you look at, for example, asks, when you log into weeks, and I believe it's under the overview pane, there is a little button there where it says you can encrypt et e d for secrets. So they make it straightforward to do, but it's not like that out of the box.
Host: Jon
Okay. Why would Kubernetes call it Secrets if by default it's in plain text? That seems so confusing to me.
Guest: Michael
Yeah, great question. I had no idea.
Host: Jon
I love that. He's like, great question. Why don't you ask that? Well, you posted on Twitter, and you're like, Hey, am I the only one thinking about this? What secrets are not that secret? It's like calling something that's not an intro, and it's not encrypted, but you can encrypt it if you follow a certain process. There's a good question. How do you even encrypt it, right? I don't want to use third-party tools. I have my non-SEC secrets, however, you want to put that. How do I encrypt that? What is something I could do?
Guest: Michael
Yes, you can encrypt the data within ET c D, you can encrypt ET c D itself. But to be honest, that's just one step in a sense. The other step is this. When you create a Kubernetes Secret, how do you create it? Well, two different, I mean, there are multiple methods. You could use the Go Client for Kubernetes, you could use various API calls, et cetera, whatever. But the two primary methods would be the imperative method, cube ctl Create Secret, and then you would pass in the Secret, or you would use the declarative way with the Kubernetes API resource for Secret. Here lies the problem with that as well, whether you use the imperative model or the declarative model, let's just, let's go with the declarative model. So we write it in yaml, just like we would our Kubernetes pods and services and ingress and yada yada.
Guest: Michael
What's the problem? The problem is, is that you got to put that secret in. So now you're storing that secret in a YAML file in plain text, and then where are you going to store that secret? You got to put the code somewhere where people can use it. So then are you putting it in your ci cd pipeline? I mean in your Git repo, but then how are you calling upon that secret? Are you putting that secret as a secret variable within your repo, whether you're using GitLab, GitHub, or whatever? But then it's like, how do you remember what the secret is? Because there's no way to get to it. So it's this circle of confusion for what direction to go in. And because of that, my analysis is that there's no a hundred percent best way to do this.
Host: Jon
Michael, using HashiCorp Vault, you were saying, and other tools, I don't want to call a specific one out, but I have my secret in HashiCorp Vault. You said it duplicates that in Kubernetes Secret though, that it can utilize it, or is it a pass-through? How is it passing or communicating with these Men? I don't even want to call it a secret, because to me right now, it's just like, Hey, here's my password. I'm going to post it on the web,
Guest: Michael
Right? Yeah. So it's going to depend on whether you're using the Vault operator, the Vault plugin, or maybe you're using a little bit of get ops to deploy it. You're using the Argo CD vault plugin. It varies based on what version you're using, or not version, but what tool you're using or what plugin you're using. But really what it comes down to is at the end of the day, it's creating some secret on Kubernetes versus storing the secret in the vault and then calling upon it. Now, what could you do? Realistically, you could call upon the Vault API to pull the secret for you, but it's like then you have other issues doing all of that work just to call a secret to get it into Kubernetes. There are multiple API calls. You're working with multiple clients, so you're working with the Vault client, you're working with the Kubernetes client, and then you have to figure out how you're getting the data from one place to another.
Guest: Michael
Where are you running that? Are you running that in a pipeline? Are you running that locally? How are you getting the secret from Kubernetes or from Vault to Kubernetes? And then the other piece of it as well as if the application needs to use it later on, that secret still needs to be stored regardless of if you're making an API call or not. And then the problem again comes back to where are you storing it, which in this case would be something like Kubernetes Secrets, but then you kind of have the same problem starting all over again. So yeah, there's really no, I mean, to my knowledge, I've looked at this pretty extensively, and I have not found a solution other than making an API call like that to mitigate this risk.
Host: Jon
Margo seems very complex for the Kubernetes secrets to use it. Why would somebody use Kubernetes if I can't keep their secrets? Secret, man, this is just getting very complex and unencrypted. Let's just put it that way.
Guest: Michael
Yeah, well, I would say that you could say the same thing for GitHub. You can store a password in plain text if you want to. So somebody could say, Hey, why would I do that type of thing? So it honestly does come down to two things. Number one, Kubernetes isn't supposed to be secure out of the box, right? That's not its primary objective. It's up to the engineer to be able to do that. But then the problem, what occurs is like, well, what if you don't have a realistic way to do that? And that then comes down to the idea of Kubernetes as a platform is still pretty new in terms of adoption, in terms of how people are using it, in terms of what different various plugins somebody would need. So yeah, I mean, nothing out there right now just yet, but hopefully there'll be something that will make this process a little bit better.
Host: Jon
Margaret, you said Kubernetes is still new and not secure out of the box, but it's designed that way, and you have an engineer, say you have an engineer does not have this expert knowledge and experience. You would need someone like yourself to bring this to light and help them train and educate them on the process and things that they can do to secure their environment, correct?
Guest: Michael
Yeah. So the problem is that a lot of people think Kubernetes is just out of the box. It's this easy button. You write a little bit of code or run the code, boom, you have this environment up and you don't have to do anything with it ever again. But that's not the reality. The reality is, is that to make Kubernetes work in a specific way, in a production-ready way, you need to have the in-depth knowledge to be able to do that. And the problem is, is that engineers, whether they're DevOps engineers, platform engineers, or SREs, that are setting this up, they're already dealing with a multitude of other things. So they can't go as in-depth as I can, for example, because this is what I do every day, right? Day in and day out. I'm focusing on everything and anything Kubernetes. But not a lot of people have that luxury, unfortunately. So yeah, I mean, bringing somebody in myself, we can get down to the nitty-gritty and dive as deep as possible to figure out the right solution for you.
Host: Jon
All right, you're saying that you obviously need the expertise to implement a secure Kubernetes environment. And today we're talking about Kubernetes not being secrets, not being so secret, and some of the things you can do about it. Michael, thank you so much for joining me.
Guest: Michael
Thank you for having me on, Jon. Appreciate it.
Host: Jon
All right, everybody. My name's Jon Myer. You've been watching a quick tutorial on the Jon Myer podcast and we've been talking with Michael Levan about Kubernetes Secrets Not Being So Secrets, but some of the things that you can do to make it secret. And I think you need to count how many times we said secrets on this episode, and until next time, don't forget to hit that, like subscribe and notify, because as always, we're out of here.