Security | Dec 11, 2020

Kubernetes security: preventing man in the middle with policy as code

Kubernetes security

Kubernetes security is complex to manage and is constantly changing. In our last blog, we addressed how a security vulnerability related to the Kubernetes snapshot controller can be proactively mitigated using policy as code. This time we discuss a different vulnerability covered in this issue, which describes how an adversary can intercept traffic in a multitenant cluster by exploiting the features of LoadBalancer or ClusterIP service types.

This vulnerability affects all versions of Kubernetes and there is no patch. It is only exploitable by users that can create or update services and pods in the cluster. Such users may be able to implement a “man in the middle” attack. As discussed in the issue comments, the only mitigation is to restrict access to the exploitable features.  This is another situation where policy as code can help, by enabling you to identify insecure configurations as described below.

Kubernetes security primer

Kubernetes uses the term “pod” to describe a manageable unit of the app.  To increase capacity, pods can be replicated so that several (essentially) identical pods are handling requests instead of just one. Pods are ephemeral, meaning that they are created and destroyed according to the needs of the app, and every new pod created by a replication controller is assigned a different IP address. This makes it difficult to keep track of pods and their associated IP addresses, so the Kubernetes service manages and exposes this pod information internally within the cluster so that related services can communicate with each other.

Depending on what is needed, organizations can choose from a few different types of services to access this information, such as ClusterIP, NodePort, LoadBalancer, and ExternalName. The vulnerability discussed in this document affects ClusterIP and LoadBalancer services and is assigned CVE id CVE-2020-8554.

CVE-2020-8554

Affected versions of Kubernetes: ALL

Impact: Man in the Middle Attack (MITM)

CVSS Score: 6.3 (Medium) (CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:L).

For this vulnerability to be exploited, an attacker must have access to the control plane of your multitenant cluster, and have privileges to create, update, or edit the pods and services.  Note that the fact that it is a multitenant cluster may make it easier for attackers to gain the necessary access.

Let’s assume that an attacker already has the necessary access and permissions in your cluster. What exactly can they do?

  1. They can create a ClusterIP service and set the spec.externalIP field to route traffic destined for that IP address through the service.
  2. They can issue a PATCH request to the API server to replace the status.loadBalancer.ingress.ip of a LoadBalancer service with the IP address of a service they control.  The net result is similar: they have effectively intercepted any traffic coming into the cluster.

Imagine an adversary who already has access to your cluster creates a ClusterIP service like the following:

apiVersion: v1
kind: Service
metadata:
    name: insecure service
spec:
    type: ClusterIP
    selector:
      app: MyApp
    ports:
      - name: http
	    protocol: TCP
        port: 80
        targetPort: 8080
    externalIPs:
      - 1.1.1.1
      - 8.8.8.8 

By applying such a configuration, an attacker would be able to redirect any cluster traffic destined for TCP port 80 on IP address 1.1.1.1 or 8.8.8.8 to the MyApp service on port 8080.  It is possible to intercept traffic using protocols other than TCP, by the way.  For example, an attacker could redirect DNS traffic using the UDP protocol in order to intercept additional communications. 

Similarly, an adversary present in your cluster could use the following command to initiate a patch request to a LoadBalancer service status and set the ingress.ip to intercept traffic destined for IP address 8.8.8.8.

curl -k -v -XPATCH \
    -H "Accept: application/json" \
    -H "Content-Type: application/merge-patch+json" \
    'http://127.0.0.1:8080/api/v1/namespaces/example-namespace/services/load-balancer-service/status' \
    -d '{"status":{"loadBalancer":{"ingress":[{"ip":"8.8.8.8"}]}}}'
 

NOTE: Have a look at this amazing blog for a proof of concept of these vulnerable features being exploited.

Mitigation options

Although there will be no fix in Kubernetes for this issue, it is possible to mitigate the risk of exploitation on a couple of fronts.  We recommend using both approaches to ensure you recognize unsafe configurations before deployment and you protect the runtime against exploitation. 

  1. In your automated CI/CD pipelines, ensure that no ClusterIP service is allowed to be created which contains an externalIPs spec.  The use of externalIPs is discouraged in general, so this policy should not be excessively noisy.  Policy as code is a great fit for this.  You can do something similar for the LoadBalancer, but you should also implement a runtime control to protect dynamic updates.
  2. Because the LoadBalancer service is widely used and commonly subjected to dynamic updates, organizations can implement an admission controller webhook to ensure that every namespace’s service account is scoped to a role that does not allow PATCH requests to the LoadBalancer service. This helps protect and audit the cluster during runtime.

Policy as code in pipelines

We recently added a policy to our policy pack for Kubernetes in open source Terrascan and our Accurics offerings to address option 1 above. Many thanks to my awesome colleague Gaurav for coming up with this simple yet effective policy.

Terrascan policies leverage the Open Policy Agent, and we added Rego code like this to ensure that ClusterIP services don’t specify externalIPs:

externalIpUsed[service.id] {
    service := input.kubernetes_service[_]
    type_check(service.config.spec)
    object.get(service.config.spec, "externalIPs", "undefined") != "undefined"
}
 
type_check(spec) {
	spec.type == "ClusterIP"
}
 
type_check(spec) {
    object.get(spec, "type", "undefined") == "undefined"
}   

The above policy will throw a violation whenever the manifest includes a service whose type is ClusterIP or undefined, and which specifies externalIPs.  Because Terrascan is open source, all Kubernetes developers can immediately benefit from this policy.  You can view the full policy in our repo if desired.

If you run Terrascan on vulnerable Kubernetes code, you would see a violation similar to the following:

Kubernetes security terrascan policy as code violation for CVE-2020-8554

Terrascan can be easily added to your pipelines as a CLI tool, or via other pipeline integrations, ensuring that violations are detected at build or deploy time–before insecure infrastructure is exposed to the world.

Protection in runtime

As discussed earlier, this policy helps to address vulnerable configurations before deployment.  Because Kubernetes provides a dynamic runtime environment, it is also important to prevent unsafe configurations from being introduced at run time.  The approach recommended by the Kubernetes team, in the GitHub issue linked toward the beginning of this article, is to use a Kubernetes admission webhook to prevent creation of services that specify externalIPs outside a range set by the cluster administrators.  They have already created a container for this purpose, with full instructions available via the link above.

Securing dynamic runtime environments such as Kubernetes requires not only careful architectural choices but automated controls that help identify and eliminate risks early and often.  Tools like policy as code can be integrated into your pipeline to help eliminate risks before deployment, and can be built into runtime controls such as admission controllers to programmatically enforce security policy at runtime.  

Kubernetes Security: Protect Internal Traffic with Policy as Code (CVE-2021-25737 and CVE-2021-25740)

Accurics and GitLab: Contextualizing Risk for Effective DevSecOps

Kubernetes Security in Four Straightforward Steps

We use cookies to ensure you get the best experience on our website. By continuing to browse this site, you acknowledge the use of cookies.