It will come as no surprise that Kubernetes deployments are growing exponentially across many industries. According to the Cloud Native Computing Foundation 2023 Annual Survey, 84% of organizations are now using or evaluating Kubernetes.
The popularity of Kubernetes, an open-source platform that automates the deployment, scaling, and management of containerized applications, is largely due to the fact that it’s essential in helping organizations manage their highly complex and dynamic container environments. Kubernetes helps them ensure high availability, resilience, and infrastructure cost management for their cloud native applications – a task that would be very cumbersome and inefficient without it.
However, the management of complex containerization environments also introduces significant security challenges. The intricate configuration options, extensive permissions, and numerous components can create vulnerabilities if not managed correctly.
This is why the Orca Research Pod is introducing KTE, an open-source Kubernetes Testing Environment for EKS (Amazon Elastic Kubernetes Service), GKE (Google Kubernetes Engine) and AKS (Azure Kubernetes Service), designed to help organizations improve their Kubernetes security by providing a safe and controlled space to identify and address potential vulnerabilities before they impact production systems. Using KTE, organizations can simulate various attack scenarios, test security patches, and evaluate the effectiveness of security configurations and policies.
Executive Summary:
- Highly complex and dynamic containerization environments introduce significant security challenges, such as misconfigurations, over-privileged roles, and vulnerabilities in components.
- To reduce risk, it is highly recommended to test the configurations and settings of clusters in a staging environment, prior to deploying to production. However, it often requires a tedious overhead of setup and research.
- This is why the Orca Research Pod has created KTE, an open-source Kubernetes Testing Environment for AWS (EKS), Azure (AKS), and Google Cloud (GKE), to help organizations improve their Kubernetes security by providing a safe and controlled space to identify and address potential vulnerabilities before they impact production systems.
- Orca is providing KTE to the open source community, under the Apache 2.0 License, on Orca’s main GitHub repository. KTE creates a free and agnostic KSPM (Kubernetes Security Posture Management) experience by utilizing open-source offerings.
- Orca will further maintain KTE to support the K8s security community and assist their staging endeavors, so they can confidently pinpoint and triage security threats. Users are encouraged to replace the default helm chart with one of their own, so they can use this project to test a real, staging environment.
What is Kubernetes Testing Environment (KTE)?
KTE stands for Kubernetes Testing Environment. In simple words, with the use of this GitHub repository, users are able to seamlessly test several security products on their K8s environment. Whether managed on AWS, GCP or Azure, participants can utilize KTE to deploy and scan their clusters. Most importantly, they are provided with clean visibility into their scan results, via intuitive web-based dashboards.
Orca has created and will further maintain KTE to support the K8s security community and assist their staging endeavors, so they could confidently pinpoint and triage security threats. Not through a single, but many open-source offerings, with the aim to include all. This will guarantee a robust and powerful approach to identify K8s misconfigurations and security weaknesses.
git clone https://github.com/orcasecurity-research/kte.gitIntroduction to Kubernetes Testing Environment (KTE)
KTE consists of several Terraform projects. Under deployment/clusters you’ll find a directory project for every vendor: EKS (Amazon Elastic Kubernetes Service), GKE (Google Kubernetes Engine (GKE)) and AKS (Azure Kubernetes Service). Each project deploys a Kubernetes cluster in the corresponding CSP, together with a generic helm chart described under deployment/clusters/helm/kte.
As we mention in the docs, we encourage users to modify the helm chart with one of their own, so they could use this project to test a real, staging environment.
The project currently supports all tools under the tools/ directory. As part of their scan, most tools require a deployment, and host a terraform project for each vendor. Some tools, such as kubesec and kube-score are static, and will run against the yaml files of the helm chart.
As an example, under polaris we can see the following files and directories tree:
/polaris
├── eks
└── main.tf
└── outputs.tf
└── variables.tf
├── gke
└── main.tf
└── outputs.tf
└── variables.tf
├── aks
└── main.tf
└── outputs.tf
└── variables.tf
Each project will deploy and execute polaris on a dedicated namespace, and export its results via a K8s service with type: ClusterIP, that will serve as a dashboard. This process is true to all tools that are provisioned with Terraform. Note: some tools, like polaris, have a native dashboard, while others don’t. You’ll see Grafana dashboards for some tools, as well as more improvised dashboards for others.
Static tools, on the other hand, don’t require a deployment. They statically scan the k8s resource files and export the results to html files. For example, kubesec is executed locally using Go over all the yaml files under deployment/clusters/helm/kte/templates and outputs the results to an html file under dashboards/kubesec.html.
All IPs of all dashboards are recorded as anchor tags inside the main dashboard page: dashboards/index.html. This page contains a numbered table with all the tools, links to their GitHub projects, links to their Results dashboards and stars count. Links to dashboards are transparent and lead to the same page, until a successful scan. Once the hyperlink is blue with an underline, it means the tool has been scanned successfully and the results are accessible.
Each IP is part of an internal cluster networking. To access the dashboard, we’ve set up terraform projects under deployment/addons/ssh that create a secure tunnel. You’ll need to enable dynamic tunneling and configure your browser with a proxy to reach the dashboards.
How to use Kubernetes Testing Environment (KTE)
Usage: ./kte.sh <command> [options]
Commands:
apply Execute the apply script
state Execute the state script
destroy [OPTIONS] Execute the destroy script with options
tunnel [OPTIONS] Execute the tunnel script with options
Options:
-h, --help Display this help manual and exit.
Examples:
./kte.sh apply
All activities related to this project are managed by the main CLI tool in the root directory. This tool is a wrapper for files under scripts/. It can provision terraform resources and subsequently execute the tool scans, present the current state of the environment, destroy it and tunnel through the cluster networking.
The apply command, which is used without any options, depends on the configuration of config.yaml. This aforementioned yaml file dictates what tools on what vendors should be used and executed. Assigning true near a tool, whether it is static or dynamic, will execute it and link to its results dashboard in the main dashboard html page.
kubesec: false
kube-score: false
polaris:
eks: false
gke: false
aks: false
kor:
eks: false
gke: false
aks: false
trivy-operator:
eks: false
gke: false
aks: false
checkov:
eks: false
gke: false
aks: falseYou can read more information about configuration and usage here.
Kubernetes Testing Environment (KTE): Community
This project was created with the community in mind. We hope that you find it useful for your own endeavors, but also feel empowered to add value to it. If you find this initiative valuable, we welcome you to participate in our discussions and contribute. Our goal is to make this project a powerful and free, one stop shop for securing staging Kubernetes environments.
Collaboration principles, guidelines and rules can be found under the contributing manual, and our code of conduct on GitHub.
Learn more about Orca’s other open-source tools for developers and security teams, designed to help make the cloud a safer place for all.
About Orca Security
We’re on a mission to provide the world’s most comprehensive cloud security platform while adhering to what we believe in: frictionless security and contextual insights, so you can prioritize your most critical risks and operate in the cloud with confidence.
Unlike other solutions, Orca is agentless-first and fully deploys in minutes with 100% coverage, providing wide and deep visibility into risks across every layer of your cloud estate. This includes cloud configurations, container images, the Kubernetes control plane, as well as your applications. Orca combines all this information in a Unified Data Model to effectively prioritize risks and recognize when seemingly unrelated issues can be combined to create dangerous attack paths. To learn more, see the Orca Platform in action.
