Kubernetes Security: Terrascan as a Validating Admission Controller
TT – Kubernetes security is becoming vitally important. As more and more critical applications are managing their workloads in Kubernetes, the security controls of the ecosystem must also evolve. Terrascan‘s newest execution mode, available in Terrascan v1.5, empowers engineers to add an additional layer of security to their cluster at runtime. Coupled with Terrascan’s traditional static IaC scanning, Terrascan now enables teams to enforce the same security policies across the application lifecycle – in development and in runtime. This is the best approach for GitOps pipelines, where the code repository should represent the running environment.
Moreover, with the recent deprecation of PodSecurityPolicy, Terrascan’s admission controller is an example of an external admission controller that is powerful and flexible to enforce policy at the cluster. For static code security analysis, Terrascan has the ability to read the Kubernetes deployment files and highlight vulnerabilities threatening the security posture of a Kubernetes cluster. Terrascan also supports scanning K8s, Helm and Kustomize.
Kubernetes Security Requires Runtime Controls
If we have static security scanning, why do we need to worry about Kubernetes security at runtime, too? In short, just because a team uses secure IaC to deploy its infrastructure, doesn’t mean that it can only be deployed from there. You may accidentally pull an insecure image, or an attacker may intentionally inject malicious configurations. The cluster must enforce security controls to ensure that vulnerable resources can’t make their way into the cluster.
Let’s imagine an extreme, hypothetical situation: a security analyst is evaluating the security posture of a running Kubernetes cluster, which was originally deployed with a safe and secure IaC. Contrary to what is defined in the IaC, they find a pod with privileged access to the underlying host and ssh access allowed to port 22 from public IPs. Adding insult to injury, there is also a user with a role allowing permissions to list Kubernetes secrets.
We will stop adding drama here; I think we all get the point. It’s possible to change the cluster in runtime – in fact, that’s kind of the whole point of K8s – even if the initial configuration is well tested and secure. We also need to test and secure configuration changes applied in runtime, to avoid potentially dire consequences.
Terrascan as a Kubernetes Runtime Control
To address such scenarios, Terrascan can be configured to run as a validating admission controller, providing runtime Kubernetes security. Admission controllers give you a way to validate configuration changes and new resources as they are applied to a Kubernetes cluster. With the help of dynamic admission control, users can block resources that violate security policies from being created in a Kubernetes cluster. If you’d like more information about them, my colleague Harkirat wrote a whole blog post about them.
Terrascan is unique in that it includes more than 100 security policies for Kubernetes which can be applied both in development and runtime. Many K8s admission controllers require users to write their own policies, which only address runtime concerns. Terrascan provides a library of policies designed by experts to address the most common problems, and has an open, extensible architecture based on Open Policy Agent if you need more.
Configuring Terrascan to run as an admission controller for your cluster is similar to configuring other admission controllers:
- Obtain SSL certificates that are trusted by your cluster. You can use an official signing authority, something like Let’s Encrypt, or create self signed certificates if you have configured your cluster to trust those.
- Create a Terrascan config file
- Use the SSL certificates to start a Terrascan server. Yes! Terrascan can also run as a HTTP[S] server.
- Make sure Terrascan is accessible via HTTPS from the Kubernetes API server.
- Configure a ValidatingWebhookConfiguration resource in the Kubernetes cluster which points to the Terrascan server
Creating the certificates
For the purpose of this guide, we will create self signed certificates. In production please use certificates signed by a trusted CA.
- Create a
[req] default_bits = 2048 prompt = no default_md = sha256 x509_extensions = v3_req distinguished_name = dn [dn] C = IN ST = Maharashtra L = Pune O = MyOrganization emailAddress = [email protected] CN = my-terrascan-server.com [v3_req] subjectAltName = @alt_names [alt_names] DNS.1 = my-terrascan-server.com
- Create self signed certificates using the above
$ mkdir data $ openssl req -x509 -sha256 -nodes -newkey rsa:2048 -keyout data/server.key -out data/server.crt -config domain.cnf
Now, the file
server.key contains our private key and the
server.crt file contains our public key.
Creating the Terrascan Configuration
You should provide a config file that specifies which policies to use and which violations should lead to rejection. Policies below the
[severity] level will be ignored. Policies below the
[k8s-admission-control] denied-severity will be logged and displayed by Terrascan, but will not lead to a rejected admission response to the K8s API server.
A config file example (
[severity] level = "medium" [rules] skip-rules = [ "accurics.kubernetes.IAM.107" ] [k8s-admission-control] denied-categories = [ "Network Ports Security" ] denied-severity = "high"
You can specify the following configurations:
scan-rules: one or more rules to scan
skip-rules: one or more rules to skip while scanning
severity: the minimal level of severity of the policies to be scanned
k8s-admission-control section applies to Kubernetes admission control and recognizes two settings:
denied-categories– one or more policy categories whose presence in the detected violations will trigger rejection of the request
denied-severity– the severity level of detected findings, at or above which requests will be rejected
Note: If specifying both
denied-severity, Terrascan will reject if it matches either (OR logic).
Running the Terrascan Server
- Option 1: Running Terrascan server locally:
$ export K8S_WEBHOOK_API_KEY=somekey $ terrascan server --key-path ./data/server.key --cert-path ./data/server.crt -p 443 ./data/config.toml
- Option 2: Running Terrascan using docker image (quick and dirty; please be careful running containers as root):
$ docker run -d --name terrascan -p 443:9443 -v "$(pwd)/data:/data" -u root -e K8S_WEBHOOK_API_KEY=somekey accurics/terrascan server -p 9443 --cert-path /data/server.crt --key-path /data/server.key -c /data/config.toml
For purposes of the following steps, assume that our server is running on the host
Configuring the ValidatingWebhookConfiguration resource
- Create a CA bundle from the public key created before. The output will go to the shell, where you can copy it into the clipboard to be used in the next step.
$ cat data/server.crt | base64
Note: On Linux, you may need to add the arguments “
-w 0” to prevent line wrapping in the output and make it easier to copy/paste the output.
- Create a Kubernetes YAML with a ValidatingWebhookConfiguration resource. Copy the CA bundle created above in the
apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: name: my-terrascan-server.com webhooks: - name: my-terrascan-server.com rules: - apiGroups: - "" apiVersions: - v1 operations: - CREATE - UPDATE resources: - pods - services failurePolicy: Fail clientConfig: url: https://my-terrascan-server.com/v1/k8s/webhooks/somekey/scan/validate caBundle: "<paste the output from the above step here>" sideEffects: None admissionReviewVersions: ["v1"]
Testing Kubernetes runtime security with Terrascan admission controller
- Try to create a pod with security violations in the Kubernetes cluster. We’ll use an NGINX image because it violates our policies with excessive container privileges:
$ kubectl run not-so-good --image=nginx Error from server: admission webhook "my-terrascan-server.com" denied the request: https://my-terrascan-server.com/k8s/webhooks/logs/e6d55d86-bbfb-42b3-ac04-37440600b53c
- Note the URL received in the response. This URL gives a brief summary of the admission request received from the Kubernetes API server and the section “Deniable Violations” point to the violations which caused the admission request to be rejected by the Terrascan admission controller.
As you can see, the API server leveraged Terrascan’s validating admission controller to evaluate and reject the request to run a new pod based on a non-compliant image.