OWASP Kubernetes Top Ten – K08: Secrets Management Failures
Introduction
In the fast-evolving world of cloud-native applications, Kubernetes has emerged as the de facto standard for container orchestration. While its robust architecture streamlines deployment, scaling, and management of applications, Kubernetes introduces a unique set of security challenges. Among these, secrets management failures pose a significant risk, often leading to data breaches, unauthorised access, and compliance violations.
The OWASP Kubernetes Top Ten (K8s Top 10) highlights the most critical security risks in Kubernetes environments. K08: Secrets Management Failures underscores the common pitfalls software developers and software architects encounter when handling sensitive data such as API keys, credentials, and encryption keys.
In this in-depth analysis, we will explore:
- What constitutes secrets management failures in Kubernetes
- Common mistakes that lead to security vulnerabilities
- Real-world examples of breaches caused by poor secrets management
- Best practices for securely handling secrets in Kubernetes
- Business impact, ROI considerations, and risk mitigation strategies for C-Suite executives
By the end of this blog, you will have a comprehensive understanding of Kubernetes secrets management and how to safeguard your organisation from potential threats.
Understanding Secrets in Kubernetes
What Are Kubernetes Secrets?
Kubernetes Secrets are a native mechanism for storing and managing sensitive information. They prevent embedding credentials in container images or application code, thereby reducing exposure. Secrets in Kubernetes are stored as Base64-encoded strings and can be referenced by pods during runtime.
Common types of secrets in Kubernetes include:
- Opaque Secrets – Arbitrary key-value pairs (e.g., API keys, database credentials)
- Docker Config Secrets – Credentials for private container registries
- TLS Secrets – Certificates and private keys for secure communication
- Service Account Tokens – Used for authentication within the cluster
Despite these built-in capabilities, mismanagement of secrets remains a prevalent security risk.
Common Secrets Management Failures
1. Storing Secrets in Plaintext Configuration Files
A frequent mistake developers make is storing sensitive information in plaintext within YAML manifests, ConfigMaps, or environment variables. Unlike Kubernetes Secrets, ConfigMaps are not designed to handle sensitive data, as they do not offer any additional security mechanisms.
Example
apiVersion: v1
kind: ConfigMap
metadata:
name: db-config
data:
DATABASE_URL: “postgres://admin:SuperSecret123@db-service:5432/mydb”
Security Risk: If an attacker gains access to the cluster, they can easily retrieve secrets stored in ConfigMaps or plaintext files, leading to data exposure.
2. Hardcoding Secrets in Container Images
Embedding credentials directly into container images is a major security misstep. Since container images are often shared across multiple environments or pushed to public/private registries, any embedded secret could be exposed to unauthorised users.
Example
ENV DATABASE_PASSWORD=SuperSecret123
Security Risk: Anyone with access to the image can extract credentials, leading to unauthorised access to critical infrastructure components.
3. Improperly Configured RBAC Policies for Secrets
Kubernetes Role-Based Access Control (RBAC) governs who can access which resources. A common misconfiguration is assigning overly permissive roles, allowing unauthorised users or service accounts to retrieve secrets.
Example
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: secrets-reader
rules:
– apiGroups: [“”]
resources: [“secrets”]
verbs: [“get”, “list”]
Security Risk: If an attacker compromises a pod with this role, they can extract all stored secrets and escalate their privileges.
4. Lack of Encryption for Secrets at Rest
By default, Kubernetes stores secrets in etcd, the cluster’s key-value store, without encryption. If etcd is not properly secured, an attacker with access can retrieve all stored secrets.
Security Risk: A compromised etcd database exposes all secrets in plaintext, leading to massive data leaks.
5. Secrets Injection into Logs
If secrets are passed as environment variables or command-line arguments, they can inadvertently be logged by debugging tools or monitoring systems.
Example
kubectl logs my-pod | grep “DATABASE_PASSWORD”
Security Risk: Attackers monitoring logs can extract sensitive information, leading to data breaches.
6. Poor Lifecycle Management of Secrets
Many organisations fail to rotate secrets periodically, leaving old credentials valid indefinitely. Additionally, unused or orphaned secrets often remain in the cluster, increasing the attack surface.
Security Risk: If an attacker compromises an old credential that has not been revoked, they can persist undetected within the system.
Real-World Examples of Secrets Management Failures
Capital One Data Breach (2019)
A former Amazon Web Services (AWS) employee exploited a misconfigured web application firewall (WAF) to gain access to an AWS S3 bucket containing sensitive data. Poor secrets management practices contributed to the breach, exposing personal information of over 100 million customers.
Uber Secrets Exposure (2016 & 2022)
In 2016, Uber’s hardcoded AWS credentials were leaked in a public GitHub repository, leading to an unauthorised data breach. In 2022, an attacker used compromised credentials to infiltrate Uber’s internal systems, underscoring the risks of inadequate secrets management.
Real-World Cyber Incidents of K08: Secrets Management Failures
To fully grasp the severity of secrets management failures in Kubernetes, let’s examine high-profile cyber incidents where poor secrets handling led to significant security breaches. These real-world cases highlight the devastating consequences of mismanaging secrets, from financial losses to reputational damage.
1. Uber Data Breach (2022) – Compromised Hardcoded Credentials
What Happened?
In September 2022, a hacker gained full administrative access to Uber’s internal systems by leveraging exposed credentials. The attacker used social engineering tactics to compromise an employee’s Slack account, leading to the discovery of hardcoded secrets in a PowerShell script.
K08 Failure Points
- Hardcoded secrets in scripts, making them easily accessible once a system was breached.
- Overly permissive access within internal systems, allowing a single compromised credential to expose critical infrastructure.
Business Impact
- Loss of customer trust: Uber faced severe reputational damage.
- Legal and compliance repercussions: Exposure of sensitive information led to regulatory scrutiny.
Key Takeaway
Never store secrets in code or scripts. Use dedicated secrets management solutions to prevent unauthorised access.
2. Capital One Data Breach (2019) – AWS Credentials Exposure
What Happened?
A former AWS employee exploited a misconfigured firewall to gain access to an Amazon S3 bucket containing over 100 million customer records. The attacker leveraged embedded AWS credentials within a web application to escalate privileges and exfiltrate sensitive data.
K08 Failure Points
- Secrets stored in application configurations, making them easy to extract once a system vulnerability was exploited.
- Lack of secrets rotation, allowing the attacker to persist within the network undetected.
Business Impact
- Massive regulatory fines: Capital One was fined $80 million by US regulators.
- Loss of customer data: The breach exposed Social Security numbers, bank account details, and personal information.
Key Takeaway
Regularly rotate secrets and never store them in application configurations. Use environment variables or a secrets management system instead.
3. Tesla Kubernetes Hack (2018) – Exposed Kubernetes Dashboard
What Happened?
Attackers gained unauthorised access to Tesla’s Kubernetes environment by exploiting an unprotected Kubernetes dashboard. This allowed them to extract API keys and credentials stored in plaintext, ultimately using Tesla’s cloud resources for cryptojacking (mining cryptocurrency).
K08 Failure Points
- Publicly exposed Kubernetes dashboard, making secrets easily retrievable.
- Lack of authentication controls, allowing unauthorised users to access sensitive information.
Business Impact
- Financial loss: Attackers leveraged Tesla’s cloud computing power for illegal mining operations.
- Operational disruption: Tesla had to perform a full security audit and incident response.
Key Takeaway
Never expose Kubernetes dashboards publicly. Use RBAC, authentication, and network security controls to restrict access.
4. GitHub Secrets Leak (2021) – Accidental Credential Exposure
What Happened?
Researchers found over 100,000 exposed secrets in public GitHub repositories, including AWS keys, database credentials, and API tokens. Developers often mistakenly commit secrets to version control, making them publicly accessible.
K08 Failure Points
- Secrets stored in source code repositories, making them vulnerable to accidental leaks.
- Lack of secrets scanning tools, failing to detect and remediate exposed credentials.
Business Impact
- Security breaches: Attackers frequently scan GitHub for leaked secrets to launch targeted attacks.
- Compromised cloud resources: Exposed cloud credentials can lead to unauthorised data access and financial losses.
Key Takeaway
Use tools like GitHub Secret Scanning and TruffleHog to detect and prevent secrets exposure in repositories.
5. Marriott International Data Breach (2018) – Weak Secrets Management
What Happened?
Hackers maintained access to Marriott’s network for four years, extracting 500 million customer records. The breach occurred due to poor secrets management, including excessive access permissions and unmonitored credentials in internal systems.
K08 Failure Points
- Lack of access control and monitoring, allowing long-term unauthorised access.
- Failure to rotate secrets, making old credentials valuable to attackers.
Business Impact
- $124 million GDPR fine imposed on Marriott.
- Severe reputational damage due to prolonged breach exposure.
Key Takeaway
Monitor and rotate secrets regularly. Implement zero-trust principles to limit lateral movement.
To further illustrate the critical impact of secrets management failures in Kubernetes environments, here are additional high-profile cyber incidents where poor handling of secrets led to severe security breaches. These cases serve as cautionary tales for software developers and architects.
6. SolarWinds Supply Chain Attack (2020) – Weak Secrets Protection
What Happened?
The SolarWinds supply chain attack was one of the most sophisticated cyberattacks in history. Attackers compromised SolarWinds Orion software by injecting a backdoor into its update mechanism. Part of the compromise involved stealing and abusing privileged credentials stored within SolarWinds’ internal systems.
K08 Failure Points
- Inadequate protection of internal credentials, which attackers used to move laterally within the network.
- Failure to restrict privileged access, enabling hackers to manipulate software updates.
Business Impact
- Breach of multiple US government agencies and Fortune 500 companies.
- Estimated damages in billions due to espionage, intellectual property theft, and operational disruptions.
Key Takeaway
Enforce strict access controls and encryption on all secrets. Use secrets rotation and monitoring to detect unauthorised access.
7. CircleCI Security Breach (2023) – Compromised Developer Secrets
What Happened?
In early 2023, CircleCI suffered a security breach when an attacker gained access to customer secrets stored in their infrastructure. The breach was linked to a stolen employee session token, allowing the attacker to extract API keys, environment variables, and other secrets stored in CircleCI’s systems.
K08 Failure Points
- Unsecured developer secrets, which were not properly encrypted or isolated.
- Lack of multi-factor authentication (MFA) enforcement on critical access points.
Business Impact
- Thousands of customer accounts compromised, leading to leaked API tokens and service credentials.
- Damage to CircleCI’s reputation, as customers had to rotate secrets across all their applications.
Key Takeaway
Use short-lived secrets and enforce MFA for accessing sensitive infrastructure. Implement audit logging to detect suspicious access.
8. Microsoft Azure Cosmos DB Vulnerability (2021) – Primary Key Exposure
What Happened?
Security researchers discovered a major vulnerability in Microsoft Azure’s Cosmos DB that exposed primary database keys of thousands of customers. The flaw allowed unauthorised access to customer databases, potentially exposing sensitive data across thousands of enterprises using the service.
K08 Failure Points
- Overly permissive access to secrets, allowing attackers to retrieve primary keys.
- Lack of automated secret rotation, which would have mitigated long-term access risks.
Business Impact
- Microsoft was forced to issue urgent guidance asking customers to rotate their keys immediately.
- Potential exposure of critical enterprise data from cloud-hosted services.
Key Takeaway
Automate secrets rotation and enforce strict permission controls on stored secrets.
9. Docker Hub Credential Leak (2019) – Compromised Access Tokens
What Happened?
Docker Hub suffered a security breach in 2019, affecting 190,000 user accounts. The attacker gained access to private repositories and leaked API tokens, which could have been used to compromise Docker images or distribute malware.
K08 Failure Points
- Stored credentials in plain text, making them easily retrievable during a breach.
- Failure to notify users quickly, delaying their ability to rotate affected secrets.
Business Impact
- Risk of compromised container images being deployed in production environments.
- Security concerns for enterprises relying on Docker Hub for container distribution.
Key Takeaway
Use encrypted secrets storage and enforce least privilege for accessing repository credentials.
10. Shopify Insider Threat (2020) – Abused API Credentials
What Happened?
Two malicious insiders at Shopify abused API credentials to steal customer transaction data from over 200 online stores. The attackers were able to extract names, addresses, and order details by exploiting Shopify’s internal APIs.
K08 Failure Points
- Overly broad API access permissions, allowing unnecessary data exposure.
- Lack of strict monitoring on secrets usage, enabling the attack to persist undetected.
Business Impact
- Reputational damage for Shopify as an e-commerce platform.
- Data breach investigations, affecting multiple businesses relying on Shopify’s infrastructure.
Key Takeaway
Limit API access using role-based access control (RBAC) and audit all API interactions for suspicious activity.
Strengthening Secrets Management in Kubernetes
These real-world incidents demonstrate that poor secrets management can have catastrophic consequences for businesses, ranging from financial penalties to full-scale data breaches. The common thread in all these cases is failing to enforce proper security controls over sensitive credentials.
Key Lessons for Software Developers and Architects
✅ Never hardcode secrets – Use environment variables or a secrets management system.
✅ Implement RBAC and access controls – Limit who can access secrets.
✅ Rotate and expire secrets regularly – Reduce the risk of long-term exposure.
✅ Encrypt secrets at rest and in transit – Prevent unauthorised retrieval.
✅ Monitor and audit secrets usage – Detect leaks before they become breaches.
By proactively addressing K08: Secrets Management Failures, organisations can significantly reduce security risks, ensuring their Kubernetes environments remain resilient against cyber threats.
Best Practices for Secure Secrets Management in Kubernetes
1. Use a Dedicated Secrets Management Solution
Instead of relying solely on Kubernetes Secrets, use enterprise-grade solutions such as:
- HashiCorp Vault
- AWS Secrets Manager
- Azure Key Vault
- Google Cloud Secret Manager
These tools provide advanced security features like automatic secret rotation, access auditing, and encryption.
2. Encrypt Secrets at Rest and in Transit
Enable Encryption at Rest for Kubernetes Secrets by configuring encryption providers:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
– resources:
– secrets
providers:
– aescbc:
keys:
– name: key1
secret: <base64-encoded-key>
– identity: {}
Use TLS encryption for transmitting secrets over networks.
3. Implement Least Privilege Access Controls
Configure strict RBAC policies to ensure only authorised users and workloads can access secrets.
Example:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: limited-secret-access
rules:
– apiGroups: [“”]
resources: [“secrets”]
verbs: [“get”]
Assign roles to only specific service accounts that require access.
4. Use Environment Variables Securely
Instead of hardcoding secrets, inject them dynamically at runtime using Kubernetes Secrets:
env:
– name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: db-secret
key: password
5. Rotate and Revoke Secrets Regularly
Set up automated secret rotation policies using tools like:
- HashiCorp Vault Dynamic Secrets
- AWS Secrets Manager automatic rotation
- GitOps-based secret updates
Ensure revoked secrets are immediately invalidated to minimise security risks.
Business Impact, ROI, and Risk Mitigation
For C-Suite Executives, poor secrets management can lead to:
- Financial Losses – Data breaches result in multi-million-dollar penalties and lawsuits.
- Reputation Damage – Loss of customer trust and brand degradation.
- Regulatory Non-Compliance – Violations of GDPR, CCPA, HIPAA, and PCI-DSS leading to legal actions.
Investing in robust secrets management enhances business continuity, reduces financial risks, and ensures compliance.
Final Thoughts
Kubernetes Secrets are a powerful mechanism for securing sensitive data, but misconfigurations and poor practices can lead to catastrophic security failures. By implementing encryption, RBAC policies, automated secret rotation, and secure storage solutions, organisations can mitigate risks and safeguard their cloud-native environments.
For software developers and architects, proactive secrets management is non-negotiable—it is the foundation of secure and resilient Kubernetes deployments.