Security | Oct 29, 2021

Kubernetes security: Preventing secrets exfiltration (CVE-2021-25742)

Kubernetes security is especially important because the CNCF Security tag team performs audits on various CNCF projects, and Kubernetes consistently tops the list in terms of usage as well as vulnerabilities.  That’s not to say that Kubernetes is insecure, but it is complex and popular and presents a large opportunity for security mistakes.

The CNCF Security team recently disclosed the presence of a high impact vulnerability in the Kubernetes NGINX Ingress Controller (CVE-2021-25742) which can allow inappropriate access to secrets across all namespaces.  This blog post discusses how policy as code can be used to detect many misconfigurations before deployment, using that vulnerability as an example.

Accurics by Tenable has already added policies which identify this vulnerability.  If you want to know whether you are vulnerable to this problem, simply sign up for the Accurics platform or leverage the Terrascan open source project to scan your Kubernetes projects.  More complete instructions for scanning with Terrascan follow the description of the vulnerability and mitigation.

What is CVE-2021-25742?

Affected versions of ingress-nginx: V1.0.0, <=V0.49
Impact: Exfiltration of secrets across all namespaces
CVSS Score: 7.6 HIGH (3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:L/A:L)
Explaining this vulnerability is perhaps best accomplished by exploring one of the attack paths.  Say your organization is running a multi-tenant Kubernetes cluster where non-admin users have permission to create or update Ingress resources, and:
  1. Your deployment uses a vulnerable version of Ingress-Nginx controller, with a configuration that exposes the vulnerability:
    • Snippet annotations are enabled by setting allow-snippet-annotations to 'true' in the Ingress-Nginx ConfigMap
    • And the Ingress-Nginx controller’s ServiceAccount token is attached to a ClusterRole which includes list permission on the cluster wide resource secrets.
  2. A malicious non-admin user creates an Ingress resource which includes *-snippet annotations with arbitrary code referencing “/secrets/kubernetes.io” 
The following manifest is an example of a misconfigured Ingress controller’s ConfigMap and malicious Ingress resource: 
apiVersion: v1
kind: ConfigMap
metadata:
 labels:
   helm.sh/chart: ingress-nginx-4.0.6
   app.kubernetes.io/name: ingress-nginx
   app.kubernetes.io/instance: ingress-nginx
   app.kubernetes.io/version: 1.0.0
   app.kubernetes.io/managed-by: Helm
   app.kubernetes.io/component: controller
 name: ingress-nginx-controller
 namespace: ingress-nginx
data:
 allow-snippet-annotations: 'true'
----
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: cafe-ingress-with-snippets
 annotations:
   nginx.org/*-snippets: |  <arbitrary-code-to-access-service-account-token/secrets>
 
This example configuration is an abbreviated version of the full configuration available here to deploy Ingress-Nginx controller.  You can see that the ConfigMap is configured to set allow-snippet-annotations to 'true'.  This compromises Kubernetes security by eventually allowing privileged users to run arbitrary code in Ingress resources through *-snippet annotations.  This can be leveraged to access the controller’s service account token, or to directly access Kubernetes secrets. 

Securing Kubernetes against CVE-2021-25742

Engineering discussions about this issue are tracked in an issue in the NGINX Ingress Controller’s GitHub.  They suggest mitigation starts with upgrading the controller to one of the following versions:
  • >=V1.0.1
  • >=v0.49.1
In addition to upgrading, you will also need to prevent execution of arbitrary code via snippets in the controller.  That can be done with one of the following commands, depending how the Ingress-Nginx resource is created:
  • For controllers created through default YAML manifests:
    • kubectl edit configmap -n ingress-nginx ingress-nginx-controller
    • (In the editor) Set allow-snippet-annotation to 'false'.
  • For controllers created using Helm:
    • helm install [RELEASE_NAME] --set controller.allowSnippetAnnotations=false ingress-nginx/ingress-nginx
Based on the discussion in the issue, there seems to be a class of users that cannot simply disable these snippet annotations because it will break their apps.  There are some suggestions that appropriate namespacing can reduce the need to disable snippet annotations, if you need that functionality.  You can read the issue to better understand that approach.  At the time this blog was written, the developers had not endorsed that approach.

How Terrascan can Improve Kubernetes Security

As mentioned earlier, Terrascan and the Accurics platform include policies to identify whether you are exposed to CVE-2021-25742.  They are flexible tools and there are actually a few different ways to detect and respond to the presence of this vulnerability in Kubernetes workflows.

Terrascan policies for CVE-2021-25742

To detect this issue, we have written two policies.  The GitHub issue includes discussion of users that cannot, for various reasons, disable allow-snippet-annotations.  Rather than force users to customize the policy themselves in the event they need to leave that enabled, we provide a two-part policy that looks for the use of vulnerable versions and for potentially exploitable Ingress configurations.  
AC_K8S_0050 This policy identifies a violation if the version of the NGINX Ingress Controller is <=0.49 or == 1.0.0 and its ConfigMap has allow-snippet-annotations enabled. This policy identifies the use of versions in your cluster that cannot be mitigated.
AC_K8S_0051 This policy identifies a violation if the version of the NGINX Ingress Controller is <=0.49 or == 1.0.0 and an Ingress object is created which includes *-snippet annotations and the code in the annotations refers to /secrets/kubernetes.io.
Taken together, these policies enable you to protect against the use of insecure versions of the NGINX Ingress Controller, as well as introduction of ingress objects that try to exploit the vulnerability by including snippet annotations that access secrets.

Enforcing Kubernetes Security Policy with an Admission Controller

Because there is a high likelihood that changes to the cluster can be made by tools other than your CD tool, either by mistake or by malice, we recommend implementing proactive controls in runtime that prevent introduction of vulnerable changes. To protect against such imperative changes, Terrascan can be configured as a validating admission controller for Kubernetes You can use this Helm chart to deploy Terrascan as a server in your cluster along with a validating webhook. Before installing the Helm chart ensure you have created a config.toml in the path deploy/helm/data/config.toml with an appropriate configuration to identify the violations discussed in this post. An example Toml configuration:
[severity]
  level = "high"
[rules]
  scan-rules = [
    "AC_K8S_0050",
    “AC_K8S_0051”
  ]
[k8s-admission-control]
  denied-categories = [
    "Configuration and Vulnerability Analysis"
  ]
  denied-severity = "high"
 

The configuration denies admission of configuration changes that introduce high severity issues from the specified categories (basically meaning they violate the vulnerability discussed in this blog).  Full instructions for running Terrascan as an admission controller is available in this blog.

Enforcing Kubernetes Security Policy in Development and GitOps Workflows

Implementing runtime controls is important, but it is also important to find problems during development, before problems are introduced into runtime and when they are easier to fix.  As a result, Terrascan offers a few different integrations to simplify use in development workflows, including a universal command line interface.  Because we already include policies to identify this vulnerability, you only need to use Terrascan’s latest Rego based policies to statically scan the IaC (YAML/Helm) repositories and detect this issue.  Our getting started document provides a good overview.

If you use a GitOps workflow to deploy workloads in your Kubernetes cluster, you can ​​configure ArgoCD with Terrascan as a PreSync hook to scan the remote repository.  Instructions for configuring Terrascan as a PreSync hook are available in the documentation here.

In either case, the Terrascan output will look something like this if problems are found:

Terrascan output identifying CVE-2021-25742
Terrascan output identifying CVE-2021-25742

With the help of Policy as Code tools such as Terrascan, Cloud native misconfigurations leading to exploitable attack paths can be detected early in the life cycle and potential exploits blocked in runtime.

Event Recap: Key Takeaways from Accurics 2021 Code to Cloud Security Summit

Kubernetes Security Enhanced with NSA and CISA Guidance

Halloween Edition: Stories from the Trenches

We use cookies to ensure you get the best experience on our website. By continuing to browse this site, you acknowledge the use of cookies.