Security | Dec 17, 2020

Cloud Native Security with Kubernetes Mutating Admission Controller

Cloud native security with Kubernetes mutating admission controller webhook

Cloud native security is complex, with heterogeneous services, polyglot applications, and multi-cloud architectures. Thankfully, the ubiquity of Kubernetes for orchestration provides some consistency and automation around the deployment, management, and scaling of these complex, container-based microservices. Point solutions and piecemeal approaches to security are simply not effective in these complex dynamic environments.  Improving the security of cloud native apps requires holistic security controls and solutions which enable apps to protect themselves.

Cloud native security: holistic controls

There are two aspects of securing Kubernetes apps:

  1. Securing the Infrastructure: We need to ensure the architectural components of the master node and worker nodes are secure. Tools such as policy as code help here, as a way to assess the compliance of the architectural components (expressed in the Kubernetes configs) to security policies such as the CIS Benchmarks.
  2. Securing the Workload: In addition to securing the infrastructure components, we need to secure the runtime.  Common approaches include running a cron job to periodically scan the runtime environment, or using a privileged container to perform security checks or policy evaluation. Neither of these are ideal.  For example, running a privileged container introduces threats and increases the attack surface; if an attacker could exploit the privileged container they could escalate and gain access to the host machine.

Rather than introducing the risk that comes with deploying a privileged container to perform security checks on the running configuration, organizations can leverage a Kubernetes capability called an admission controller.

What are Kubernetes Admission Controllers?

Admission controllers are responsible for governing and enforcing security principles or constraints on the usage of clusters. They act as a gatekeeper for an authenticated API request.

There are generally two phases which are applied before an external request is handled by the API:

  1. Mutation, in which a controller may modify the request object
  2. Validation, in which a controller may reject the request object

Any given controller may operate in one or both of those phases.  There are a number of built-in controllers, including generic MutatingAdmissionWebhook and ValidatingAdmissionWebhook controllers that make use of webhooks via a REST endpoint to inspect the request object. Any such webhooks run as a service in the cluster which the master controller can communicate with.

Now, what is the benefit of this? It allows your developers to implement custom runtime constraints and policies which affect the way that request objects are processed by the controller.  You can use admission controllers to improve security, governance and configuration management in your cluster.

Let’s look at how we can leverage admission controllers to enforce security controls when resources are created, deleted, and updated.

Securing the workload using MutatingAdmissionWebhook

When a request arrives at the API server, it will first run through the configured admission controllers before being processed by the API server itself.  In the case of the webhook controllers, the request is sent to the webhook REST endpoint and the response sent back through that same connection.  The response indicates how or if the API server should continue processing the request.

Consider a scenario where you want to ensure that container images are pulled only from a specific registry.  Perhaps you maintain that registry and are confident that all the images have been thoroughly tested and secured.  To enforce this policy, you can add a custom MutatingAdmissionWebhookConfiguration to intercept any pod creation request.

We can start by adding a MutatingWebhookConfiguration which sends pod creation requests to our admission controller, which is available as the service image-registry-check-server:

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
    name: webhook
webhooks:
    - name: webhook-server.webhook-example.svc
      clientConfig:
  	  service:
    	    name: image-registry-check-server
    	    namespace: webhook-configuration
    	    path: "/mutate"
  	  caBundle: ${CA_PEM_B64}
      rules:
  	  - operations: [ "CREATE" ]
    	    apiGroups: [""]
    	    apiVersions: ["v1"]
    	    resources: ["pods"] 

Now, our admission controller has an opportunity to inspect any pod creation request before it is processed.  If the request is trying to use an image from an unauthorized registry, the image-registry-check-server service can decide whether to modify the request to use an authorized registry or to reject the request outright.

We still need to create the app that handles the REST API for the webhook endpoint. Our service config specifies the path /mutate, so that’s where we will handle requests.  In Go, the server might look something like this:

mux := http.NewServeMux()
mux.Handle("/mutate", requireSecureImageRegistryHandler)
server := &http.Server{
  Addr:	":8443",
  Handler: mux,
} 

We listen for requests on port 8443, and requests on the path /mutate will be handled by requireSecureImageRegistryHandler.  This blog doesn’t go into the details of that handler, but there are plenty of examples available online if you’d like a buildable example.  Suffice it to say that the handler is provided the input request, and it will inspect that request and send back a response that tells the API server how to proceed.  

If the handler decides that the request needs to be modified, those modifications are specified using JSON patches. In our example scenario, if the request uses an image from an unapproved registry then we need to specify patches to the AdmissionReview request which enable use of an approved registry. 

Say we receive a request that includes something like the following (unrelated content removed for clarity):

spec:
  containers:
  - name: nginx
    image: nginx:1.9 

To ensure we use an approved registry, we’d like to add an explicit reference to the image value.  

Our service will respond with an AdmissionReview response which specifies how to patch the request.  The patch itself will look something like this:

{
  "op": "replace",
  "path":  "/spec/container/image",
  "value": "my-secure-registry/nginx:1.9",
} 

This just indicates that the patch will `replace` the field /spec/container/image with the value my-secure-registry/nginx:1.9, where my-secure-registry is the approved registry that we would like to use.

The response sent from the webhook handler needs to adhere to the Kubernetes admission API and includes some additional content and might look something like this:

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": <value from request.uid>,
    "allowed": true,
    "patchType": "JSONPatch",
    "patch": "eyJvcCI6InJlcGxhY2UiLCJwYXRoIjoiL3NwZWMvY29udGFpbmVycy9pbWFnZSIsInZhbHVlIjoibXktc2VjdXJlLXJlZ2lzdHJ5L25naW54OjEuOSJ9"
  }
} 

Where the value of patch is the base64 encoded JSON patch described above.  To reject the request, we could have set the allowed field to false.

The overall processing flow would look something like this:

Cloud native security: Kubernetes MutatingAdmissionWebhook processing flow
Cloud native security: Kubernetes MutatingAdmissionWebhook processing flow

Note that we also need to pass the option --enable-admission-plugins=MutatingAdmissionWebhook to the Kube-API-Server in order to ensure that the MutatingAdmissionWebhook plugin is enabled.

Self-healing cloud native security

Cloud native teams can implement holistic security controls in their applications through the use of policy as code to enforce security policies prior to deployment, and Kubernetes admission controllers to enforce security policy at runtime.  It’s even possible to leverage the same policy as code from within the admission controller, to ensure consistency in the policies that are enforced throughout the application lifecycle.

Even better, cloud native security technologies like admission controllers allow for proactive security controls that can be codified at build time and enforced at runtime.  This avoids the need for brittle cron jobs and privileged containers and sidecars that increase the attack surface of the application.  In essence, this approach creates self-healing applications that are able to defend against malicious use and ensure that operational parameters remain within the constraints that the application was designed to handle.

Kubernetes Security: Stop Blind SSRF with Policy as Code (CVE-2020-8555)

Kubernetes security: preventing man in the middle with policy as code

AWS Cloud Security – Protecting Against SSRF

We use cookies to ensure you get the best experience on our website. By continuing to browse this site, you acknowledge the use of cookies.