Kubernetes makes it easy to deploy applications, but it also makes it easy to make a recurring mistake: assuming that a native Secret is “secure” because its value appears as Base64. Base64 is not encryption; it is a representation. And that difference, in companies, ends up materializing in credentials exposed in repos, in CI/CD, in tickets, or in incidents due to poorly scoped permissions.
This article focuses on the practical impact of that detail (Base64) and on an operational pattern to get out of the problem using External Secrets Operator integrated with Azure Key Vault or AWS Secrets Manager, without relying on “good intentions” or manual processes.
What it really means for a Secret to be Base64 (and why it matters in an audit)
A Kubernetes Secret in YAML format stores its values in Base64 inside the data field. That avoids problematic characters in YAML/JSON, but it does not add confidentiality. Any person or process with read access to the object can decode it in seconds. In audits it is common to see this confusion: it is assessed “whether there are Secrets” in the cluster, but not who can read them, how they are logged, and whether they are protected at rest.
In corporate environments, the risk is rarely “someone gets into the cluster and sees a Secret.” The typical case is a chain of small exposures: a role with excessive read permissions, an observability console that captures manifests, an operator who runs kubectl get secret -o yaml during troubleshooting and pastes it into an internal chat, or a pipeline that prints environment variables in logs. Base64 turns those hygiene failures into a real leak, because there is no cryptographic barrier.
A very common example: stringData is used for convenience (plain text in YAML) and the manifest is committed by mistake. Even if data looks “obfuscated,” the repository ends up containing the equivalent secret. From a compliance standpoint (ISO 27001, SOC 2, PCI depending on context), this is often indistinguishable from having credentials in plain text, because the effort to recover them is trivial.
How Secrets leak in real life: RBAC, etcd, logs, and backups
The first practical vector is RBAC. Many organizations grant broad read permissions to speed up operations (“so the support team can see everything”). The result: roles with get/list/watch over secrets in application namespaces. With Base64, that permission is equivalent to handing over credentials in cleartext. In internal incidents, I have seen more leaks due to excessive permissions than due to sophisticated attacks.
The second vector is persistence. Secrets live in etcd; if encryption at rest (encryption at rest) and key management are not configured, any access to etcd backups or snapshots becomes access to secrets. This shows up especially when backups are externalized to centralized storage and access control is not aligned with the cluster’s least-privilege model.
- Logs and troubleshooting tools: if a team dumps complete Kubernetes objects to logs (for example, upon a reconciliation error from an operator), it can end up logging the
datafield and, therefore, the Base64 secret. - Backups and DR: a recovery plan that replicates etcd or manifests to another environment without encryption/segregation can propagate credentials to less controlled environments.
- CI/CD: pipelines that render Helm/Kustomize and store artifacts (manifests) as deployment evidence may retain the Secrets “obfuscated” but recoverable.
In all cases, the pattern is the same: Base64 does not slow anything down. If your compensating control is “only trusted people will see it,” you have already shifted the risk to permissions, processes, and tools that are not always designed for secret custody.
Operational decision: what changes when moving secrets to an external manager
The practical way out in companies is not usually “let’s encrypt the YAML better,” but to change the control point: Kubernetes should not be the system of record for the secret, but a temporary consumer. That is where Azure Key Vault or AWS Secrets Manager come in as the source, and External Secrets Operator (ESO) as the mechanism to materialize the secret where the workload needs it.
This does not eliminate Secret objects inside the cluster (many apps still require them), but it does change governance: rotation, access auditing, versioning, and policies about who can read the secret are managed in the vault, not in etcd. In practice, the security team can enforce controls (for example, MFA, conditions, centralized logging) on the cloud plane rather than chasing scattered YAML manifests.
- Availability trade-off: you introduce dependency on the secrets service (Key Vault/Secrets Manager). This forces you to design caching, refresh, and operator fault tolerance.
- Permissions trade-off: you move from pure RBAC to a combination of RBAC + cloud identity (workload identity/IRSA). If this is done poorly, you can worsen the blast radius.
In environments with multiple clusters and teams, the biggest benefit is usually organizational: you reduce secrets sprawl and centralize responsibility. The biggest risk is usually operational: a poor identity configuration (for example, an overly open federation) that allows unauthorized pods to request secrets from the vault.
How to do it in practice with External Secrets Operator + Azure Key Vault / AWS Secrets Manager
The operational goal is for the pod to have a verifiable identity (without static secrets), for ESO to use that identity to read from the vault, and for a Secret to be created/updated in the target namespace. The key: remove “bootstrap” credentials from YAML and minimize access both in Kubernetes and in the external manager.
Concrete steps (common): install ESO, define a SecretStore/ClusterSecretStore pointing to the provider, and create an ExternalSecret that maps remote keys to a local Secret. From there, validate that: (a) the identity used is the expected one, (b) the operator cannot read more than necessary, and (c) the secret refreshes according to policy (rotation/TTL).
AWS (IRSA + Secrets Manager): the recommended pattern is to associate the ESO ServiceAccount (or the corresponding namespace’s, depending on your model) to an IAM role via OIDC. The policy must limit access to specific ARNs.
- Minimum IAM policy (example):
Limit access to read of a specific secret. Avoid broad wildcards, because they turn a mistake into a lateral incident.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret"
],
"Resource": "arn:aws:secretsmanager:REGION:ACCOUNT_ID:secret:app/prod/db-??????"
}
]
} - Role trust policy (OIDC) scoped:
Restrict by
subto the exact ServiceAccount. In corporate environments, this is the point that most often gets “opened too much” due to haste.{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/OIDC_PROVIDER"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"OIDC_PROVIDER:sub": "system:serviceaccount:NAMESPACE:external-secrets"
}
}
}
]
}
Validation in AWS: check in CloudTrail for GetSecretValue events with the expected role; verify that the role cannot list secrets; and confirm that the OIDC sub does not allow other ServiceAccounts. In the cluster, validate with kubectl describe externalsecret and events that the refresh works without authorization errors.
Azure (Workload Identity + Key Vault): the equivalent pattern is to federate the ServiceAccount with a Managed Identity / App Registration and grant minimum permissions in Key Vault (for example, access only to get of specific secrets). Practical validation often relies on Key Vault logs (diagnostic settings) to confirm which identity accessed which secret and from what context, in addition to the operator’s events in Kubernetes.
Recommendations for corporate environments
The Base64 problem in Kubernetes is not theoretical: it turns any excessive permissions or poor operational practice (logs, backups, CI/CD artifacts) into direct exposure. Treating native Secret objects as “encrypted” is one of the most frequent causes of repeated findings in audits and internal reviews.
External Secrets Operator with Azure Key Vault or AWS Secrets Manager is often the most realistic way to regain control: the secret lives and is audited in the vault; Kubernetes consumes it. For it to work in a company, the critical piece is identity (Workload Identity/IRSA) with minimum permissions and scoped trust, plus continuous validation (CloudTrail/Key Vault logs + operator events) that demonstrates that only what is expected can read what is necessary.
If you have to prioritize: first cut RBAC access to secrets, avoid manifests with secrets circulating through CI/CD and repositories, and move the source of truth to an external manager with auditing. Base64 is not the enemy; the false sense of security is.
Interested in Cloud Security?
Technical analysis, hands-on labs and real-world cloud security insights.