Table of contents

Container security across the SDLC

Gartner predicts that by 2027, 90% of global organizations will be running containerized applications in production. Furthermore, according to the Cloud Native Computing Foundation, 96% of organizations are currently using or evaluating Kubernetes. 

Cloud-native applications, built on modern architectures like containers and Kubernetes, are composed of many interconnected microservices that run on cloud infrastructure. Not surprisingly, these architectures continue to rise in adoption, because they are a lightweight, flexible, and efficient way to package and deploy applications. However, they also come with unique security challenges that traditional security measures, in most cases, cannot address.

In this article, we provide the optimal reference architecture for container security at different stages of the application development lifecycle, from build to deployment to runtime, and back again. Intended for DevSecOps and cloud security practitioners, this article will explain the most important and effective security measures that organizations should implement. Using Kubernetes and container security as our example, we’ll explore what an ideal DevSecOps journey should look like. 

Benefits of Kubernetes and Containers

The fast adoption of cloud-native applications is driven by the many advantages that Kubernetes and containers bring, including shorter release timeframes, consistent and reproducible environments, lower IT costs, portability, increased scalability, and flexibility in multi-cloud environments. Additionally, the fact that both Docker—the most popular containerization platform—and Kubernetes—the leading container orchestration platform for the management of containerized applications—are open-source projects with a wide support ecosystem, is another important driver for the wide adoption.

Kubernetes and containers also bring several important security benefits, including workload isolation in runtime, granular access controls, deep visibility and monitoring potential, and the ability to scan container images to identify and address vulnerabilities before deployment. 

That is, at least security should be a primary part of container development lifecycles. It’s not as simple as it sounds though. A recent report found that 67% of DevOps and security practitioners delayed an application release due to Kubernetes security issues. Container deployments face a constant risk of security breaches due to vulnerabilities that can exist throughout the software supply chain, as well as misconfigured cloud and container environments. Further exacerbating security woes, is the fact that an exploitable vulnerability in a container image can compromise a significant portion of your cloud infrastructure. 

Infrastructure-as-Code

In addition to container and Kubernetes adoption, another technology that enterprises are leveraging is Infrastructure-as-Code (IaC), to quickly and repeatedly provision cloud infrastructure. Infrastructure-as-Code is the process of managing and provisioning infrastructure through code. For example, rather than manually provisioning a virtual machine on AWS or Azure, the developer uses code within software (e.g., Terraform, AWS CloudFormation, or Azure Resource Manager) to perform this function.

The benefits IaC brings organizations include managing IT infrastructure needs while also improving consistency and reducing errors and manual configuration. Using an automated and repeatable application provisioning and deployment process, IaC helps you to align development, operations, and DevOps teams, and accelerate application delivery.  

Shifting security left

In the days when application deployment processes were slower and less agile, implementing security monitoring and risk management processes in the later stages made sense. However, as cloud-native application deployment is considerably faster (and continuous) with these newer automation technologies, changes were needed. We’ve seen in recent years increased adoption of approaches such as DevSecOps, which aims to integrate security at every stage of the software supply chain, effectively shifting security “left” towards earlier stages in the development lifecycle. This approach focuses on testing and fixing issues when development costs are the lowest – before code and applications are deployed in production. Shifting left reduces the risks and costs associated with fixing production security problems.

“Securing containers requires a comprehensive approach spanning many points in the software supply chain. Security and risk management technical professionals must use DevSecOps processes and techniques to effectively secure container environments.” – Gartner, Inc., Container Supply Chain: 10 Security Vulnerabilities and How to Address Them, William Dupre, March 24, 2023

Cloud provider services for container development and security

Shared Responsibility Model

Before diving further into this paper, an important concept to be familiar with is the shared responsibility model. The basic idea is that a public cloud provider keeps their platforms secure, but the customers are responsible for securing the workloads, data, and processes they run inside the cloud. In other words, cloud service providers (CSPs) own the security of the cloud infrastructure and services, and the customer is responsible for the security of the resources they run in the cloud (See Azure’s shared responsibility model for example).

Container orchestration to optimize development and security

Within the realm of container security, CSPs provide several services related to container and Kubernetes orchestration:

SDLC Stage 1: Build stage security best practices

Securing containers during the build stage of the software development lifecycle is crucial for establishing a strong foundation for container security. 

Visualization of the build, deploy, cloud to dev, and runtime cycle for the cloud security building stage

The following are essential security practices to implement during the build stage:

Use secure base images

In Docker containerization, a container image is a blueprint for creating a containerized environment, consisting of a set of instructions, as well as a collection of files such as installations, application code, and dependencies.. Developers can launch multiple containers using the same image, where the image can be built upon a previous image layer. The lowest layer in that pyramid is referred to as the “parent” or “base” image. The base image provides a foundation for the subsequent layers and enables users to exert complete authority over the contents of the derived images. However, handling the base images requires careful attention and diligence.

Begin by selecting a minimum amount of base container images from trusted registries and repositories that are regularly updated and maintained (in addition to the cloud provider registries mentioned above, other popular options include Docker Hub, GitHub’s Container Registry, and JFrog Container Registry). Ensure the images are scanned for vulnerabilities and come from reputable registries. In addition, it’s recommended to keep your image as small as possible. The more unnecessary components there are in the image, the larger the possible attack surface will be. 

Another recommendation is to maintain a set of internal golden base images that use an internally shared set of images. Rather than importing from external registries, internal images allow you to follow a common process to deliver images with best practices for security, authorization, package management, and operations pre-configured for the rest of the organization.

Scan container images and open source components

Securing open source components, which make up a majority of modern applications, is a critical part of security in the build stage. Therefore it is absolutely essential to employ automated tools to scan container images for known vulnerabilities. Regularly update the vulnerability database and promptly address any identified issues.

While it’s important that developers understand security best practices for internally written source code, scanning code from external sources is important to note here as well. Software composition analysis (SCA) involves scanning source code to detect vulnerabilities that stem from third-party modules or libraries, including those imported from open-source projects. SCA tools can examine your source repositories at the onset of the CI/CD pipeline, notifying you about dependencies that, if unattended, could lead to security vulnerabilities in your live application.

Apply image hardening techniques and image signing 

Apply security best practices to harden container images, including removing unnecessary packages, disabling unnecessary services, and minimizing the attack surface.

An additional way to strengthen security around container images is by signing them using digital signatures to verify their authenticity and integrity. Use a secure and trusted key management system to store and manage the signing keys. This can involve using checksums or cryptographic hashes to ensure the images have not been tampered with.

Practice secure configuration management

Establish secure configuration management processes for containers. Follow the principle of least privilege and configure container runtime settings, network configurations, and access controls appropriately. Kubernetes has a list of configuration best practices. 

Implement secrets management

Safely handle sensitive information such as passwords, API keys, and certificates by utilizing secrets management tools. Avoid hard-coding secrets within the container image and instead retrieve them securely at runtime. Note that secrets management is needed at multiple stages of the development lifecycle to integrate or build code, deploy applications in testing or production, etc.

The major cloud providers offer secrets management services, which include AWS Secrets Manager, Azure Key Vault, and Secret Manager from Google Cloud. These services help securely store and manage sensitive information and integrate with container services, allowing you to retrieve secrets securely at runtime without hard-coding them in the container images.

Regularly update and patch container images

Stay updated with the latest security patches and updates for both the base images and the software packages within the container. Establish a process for regularly scanning and updating containers to address newly discovered vulnerabilities.

Implement network segmentation

Utilize network segmentation techniques, such as network policies and firewalls, to control and secure container-to-container and container-to-external communications. This helps prevent unauthorized access and data exfiltration.

Incorporate IaC template scanning

As organizations adopt container deployments, the utilization of Infrastructure-as-Code (IaC) solutions has become prevalent for provisioning resources in downstream environments. These tools define crucial aspects of application resources, including service specifications, exposed ports, resource labels, environment variables, and other infrastructure components. 

As a result, a comprehensive representation of the application infrastructure is now stored in a code repository. This necessitates the need for end-to-end traceability, accountability, and robust security measures to ensure the integrity and protection of the infrastructure code throughout its lifecycle. Ensuring that IaC artifacts undergo thorough validation is essential to maintaining a secure deployment environment. IaC auditing includes scanning for vulnerabilities and misconfigurations on the developer desktop or as part of regular, continuous integration and continuous delivery (CI/CD) workflows.

The Orca Security Infrastructure as Code Policy dashboard

IaC scanning tools can parse commonly used cloud native formats such as Dockerfiles and Kubernetes YAML and then apply a set of rules that encode robust security practices, giving the user a view of where they can apply additional hardening to improve their environment’s security.

For example, they can detect Kubernetes misconfigurations in which the container image doesn’t have a user specified in it, resulting in the creation of containers that run as root. In Linux, “root” is the user that has access to everything on that host. So you definitely want to avoid that unless it’s necessary. Similarly, IaC scanning can detect containers running as privileged, meaning that Kubernetes runs that container as part of the underlying operating system. It has access to everything – the underlying host, etc. It is good practice to always give a container the minimum requirements it needs.

In general, container image and IaC scanning capabilities should include:

  • Vulnerability scanning, to include in every layer of a container image
  • Support for security checks specific to containers and IaC solutions, such as HashiCorp Terraform,  AWS CloudFormation, Azure Resource Manager, and Kubernetes YAML
  • Support for CIS benchmark controls and the ability to customize compliance policies
  • Ability to detect misconfigurations in both development images and templates
  • Results should be available both in the cloud security platform and in developer and DevOps tools.
  • Seamless integration with common repository managers such as GitHub and GitLab, as well as CI/CD tools like Jenkins, CircleCI, and others.

By adopting these practices, the security posture of containers can be greatly enhanced during the build stage, reducing the potential for vulnerabilities and establishing a solid foundation for secure container deployments.

SDLC Stage 2: Deployment challenges and best practices

The components of your containers have been built, and now it’s time to deploy them to production. The infrastructure supporting the CI/CD pipeline presents an attractive target for attackers seeking to inject malicious code and components into software artifacts. Although the code repository and container registry may hold legitimate components, attackers have demonstrated their ability to disrupt the build process and introduce malicious payloads before deploying an artifact. 

Securing Kubernetes and containers during the deployment stage of the software development lifecycle is essential to maintain a robust and protected environment. 

NOTE: several of the security best practices mentioned during the build stage section also play significant roles during the deployment stage as well. These include scanning for secrets, vulnerabilities, and misconfigurations.  

Here are some additional important practices to consider during deployment:

Strategically set controls on registry images using rules-based image pulls

Container image registries serve as repositories for images used in runtime environments. When it comes to scanning these registries, certain best practices should be followed.

Automated container scanning tools often include registry scanning as a crucial feature. However, it is important to exercise caution. The focus should be on scanning images in the pipeline and during deployment, while disregarding the images at rest in the registry. These registry images can consist of outdated images, unused images, and more. 

To address this challenge, utilizing rules-based image pulls can be beneficial. This approach involves automatically pulling images from registries and scheduling scans or rescans at specific intervals. Another effective strategy is to implement image scanning on a Kubernetes admission controller. This allows for the application of policies that require Kubernetes context and creates an additional layer of defense for your cluster.

Implement Role-Based Access Controls (RBAC)

Apply RBAC principles to ensure that only authorized individuals or entities have access to deploy and manage containers. Assign appropriate roles and permissions to different users, limiting access to sensitive resources.

Harden container hosts

Configure container runtimes with security best practices, such as using read-only file systems, enforcing resource limitations, and restricting privileged access within containers. 

Tighten Kubernetes security and monitor container activity

Implement comprehensive monitoring and logging mechanisms to track container activity and detect any suspicious or unauthorized behavior. Utilize Kubernetes to gain visibility into container performance and security metrics. It’s important to enforce least-privilege access to Kubernetes, and to use a cloud security platform for identifying and prioritizing risks inside the Kubernetes environment. Center for Internet Security (CIS) Kubernetes benchmarks — both general (CIS K8S) and cloud platform specific (CIS EKS, CIS AKS, and CIS GKE)—are a good source for Kubernetes best practice guidelines to follow.

The Orca Security CIS KBS EKS compliance dashboard

Block risky builds

Ideally, a cloud security solution tailored for DevSecOps should allow you to trigger alerts or block builds as part of your workflows. This capability is commonly referred to as guardrails. Guardrails work by establishing policies for critical issues that if they occur, trigger a build to be blocked from being deployed, making it difficult to make large security errors in the deployment process.

The biggest benefit of using guardrails is that it speeds up the deployment process, while ensuring high security quality. With a good guardrail system, developers and DevSecOps teams can be much more confident in deploying systems without introducing workloads into production that contain serious security risks.

By following these practices, organizations can significantly enhance the security of containers during the deployment stage, reducing the risk of security breaches and ensuring a more resilient and secure containerized environment.

SDLC Stage 3: Visibility and security during runtime

Securing containers during runtime and in production environments, while still partially focused on detecting vulnerabilities and misconfigurations, should also dedicate resources to detecting active and immediate security incidents. Production environments are monitored for risks with contextual alerts and risk prioritization, as well as integrations with ticketing and notification tools.

Best practices during this stage—in addition to the previously mentioned enforcing least privilege policies and using trusted container images—include the following:

Monitor container behavior

Implement runtime monitoring and logging to detect anomalies and suspicious activities within containers. Continuously monitor resource usage, network connections, and system logs to identify potential security incidents. These include log monitoring of Linux system calls, Kubernetes audit logs, and cloud activity logs (tools like AWS CloudTrail, Azure Monitor, and Google’s Cloud Audit Logs help).

Encrypt sensitive data

Implement encryption for sensitive data within containers, both at rest and in transit. Utilize encryption protocols and secure storage mechanisms to protect sensitive information from unauthorized access.

Regularly patch and update

Keep container runtimes, underlying operating systems, and software components up to date with the latest security patches and updates. Regularly apply updates to address known vulnerabilities and protect against emerging threats. Even more, it’s best (budget permitting) to have a rapid-response team in place to remediate critical out-of-band vulnerabilities when they are discovered.

However, it’s critical to remember that containers are ephemeral entities. The focus should not be on patching the container itself, but rather to build a new image and replace the outdated container deployments. 

Implement authentication and authorization

Enforce strong authentication mechanisms for container management interfaces and APIs. Implement RBAC policies to ensure proper authorization and limit access to sensitive container resources.

Leverage API visibility and security

API misconfigurations are a common risk that attackers exploit to gain unauthorized access into container and Kubernetes environments. Maintain a fully updated inventory of APIs in your cloud accounts, and use API Security tools to identify and prioritize API-related risks. 

Incident response and recovery

Develop and regularly test incident response plans specific to container security incidents. Establish procedures for detecting, responding to, and recovering from security breaches or unauthorized activities within containers.

By following these practices, organizations can enhance the security of containers during runtime or in production, reducing the risk of security breaches and ensuring a more resilient and secure containerized environment.

Integrate alerts with existing developer tools

Alerts to security incidents and risks are only as valuable if DevSecOps teams respond quickly to them. It’s highly recommended to integrate alerts into incident response tools such as PagerDuty or OpsGenie, communication platforms like email or Slack, and enable automated ticketing with Jira or ServiceNow. 

It’s also important that your cloud security platform is able to provide accurate origin and attribution of specific risks, in order for security teams to be able to notify the appropriate development teams. When the team is notified, they can quickly investigate the incident and respond. 

Adopt a CNAPP to protect containers and Kubernetes environments

“Use a cloud-native application protection platform (CNAPP) to protect container environments from workload attacks and cloud misconfigurations.”  – Gartner, Inc., Container Supply Chain: 10 Security Vulnerabilities and How to Address Them, William Dupre, March 24, 2023

One of the biggest benefits that a DevSecOps approach to container security should bring is the ability to view development and production risks holistically in a single, unified data model instead of as a disparate collection of siloed risks. This allows for a highly contextual view of all the different risks in the cloud environment that can then be prioritized based on severity, exploitability, radius, and business impact—allowing security organizations to understand and remediate their most critical issues first.

As seen in the quote, Gartner recommends adopting a cloud-native application protection platform (CNAPP) for better container security. Combining capabilities from previously separate tools, a CNAPP like the Orca Cloud Security Platform includes cloud workload protection, monitoring of cloud infrastructure for misconfigurations and compliance, cloud entitlements management, sensitive data protection, API security, and Kubernetes security. CNAPPs should also include “shift left” security capabilities including container image scanning and IaC scanning. If you can’t scan and detect risk in these development artifacts, you will have significant blind spots.

The Orca Security 5.2 File Integrity dashboard

SDLC Stage 4: From Cloud to Dev

Even once container-based applications are in production, there can be new vulnerabilities and other risks that are discovered that were previously not known. While security teams may be able to identify a risk, they cannot just fix it in production since it will simply be overwritten the next time DevOps deploys a new build. Instead, the original code needs to be fixed and redeployed. Finding the origin of the risk – the piece of code that needs to be adjusted, and the attribution – the developer that owns it, can be a painstaking process, sometimes taking days or even weeks. 

While cloud-native applications empower developers and DevOps teams to effortlessly deploy code to the cloud, security teams are left figuring out who to assign the issue to and start fixing issues detected in production. This causes delays in remediation and leaves organizations exposed for longer. To fix issues in running containers and cloud infrastructure, the security team is dependent on the developers and DevOps teams that created the original container images or IaC templates.

Orca Security released industry-first capabilities that automatically link cloud risk detections in production with the development pipeline, dramatically simplifying and accelerating the remediation process for security incidents. 

Orca’s capabilities support Infrastructure-as-Code templates and containers, with the ability to trace a production misconfiguration or vulnerability directly to the original source code repository that was the source of the alert or risk, even down to the exact line of code that is at the root of the identified risk. Thus, Orca eliminates the need for security teams to spend time tracing the source artifacts and their respective owners, greatly reducing Mean Time to Resolution (MTTR) and freeing up valuable time for higher-value activities. 

In general, it’s important that a DevSecOps framework has a process in place to complete the circle from cloud applications in production back to the dev and build stages. In a CI/CD process, where speed and quality is of the essence, runtime should most definitely not be seen as the end of the process, but as a turning point.

Conclusion: The Value of DevSecOps

The software development ecosystem has become increasingly intricate, encompassing various platforms, processes, and technologies. Within this environment, containers coexist with numerous other elements. To navigate this complexity, organizations are adopting DevSecOps practices and forming dedicated teams to ensure secure operations. By doing so, organizations can offer their development and product teams consistent, integrated, and secure platforms that facilitate efficient and protected workflows.

DevSecOps brings organizations several benefits when it comes to application development and security. It helps organizations to:

  • Detect risks earlier – Identify security problems in both the application and IaC early, and, as a result, empower developers to identify the root cause of issues and change application architecture or modify underlying components to improve application quality.
  • Build preventive controls – Implement guardrails and ensure compliance with security best practices.
  • Accelerate software delivery – As security is integrated throughout the development lifecycle, there is less time wasted patching and remediating at a late stage.
  • Lower costs – Reduce operating costs by identifying vulnerabilities and other risks before deployment.
  • Enable greater overall business success – DevSecOps instills greater confidence in the security of software development and facilitating the adoption of emerging technologies. This, in turn, drives revenue growth and empowers organizations to expand their range of business offerings.

How Orca Security helps strengthen cloud visibility and security at every stage

Orca helps organizations secure their Kubernetes and container environments by performing extensive security checks at every cloud layer. With the Orca Cloud Security Platform, security and DevOps teams can partner to integrate security from pipeline to production—scan container images and IaC templates pre-deployment, continuously scan container registries, and monitor vulnerabilities, compliance issues, and advanced threats at runtime.

Unlike other solutions, Orca is completely agentless and fully deploys in minutes with 100% coverage, providing wide and deep visibility into risks across every layer of your cloud estate. This includes cloud configurations, container images, the Kubernetes control plane, as well as your applications. Orca combines all this information in a Unified Data Model to effectively prioritize risks and recognize when seemingly unrelated issues can be combined to create dangerous attack paths.

The Orca Security Inventory dashboard featuring asset categories

Strengthening container security every step of the way

Orca brings critical security capabilities to each stage of the software development lifecycle:

  • Build – Container images and IaC templates are scanned on the developer desktop or as part of regular, continuous integration (CI) / continuous delivery (CD) workflows.
  • Deploy – Registries are continually monitored to ensure application images are secure before deployment, with guardrail policies in place to prevent insecure deployments.
  • Runtime – Production environments are monitored for risks with contextual alerts and risk prioritization, as well as integrations with ticketing and notification tools.
  • Cloud to Dev – Origin and attribution of container vulnerabilities discovered in production applications that allow security teams to instantly see which container image or IaC template is the source of the detected risk, along with its owner(s).

Additionally, Orca’s GitHub app and GitLab app allows security teams to quickly onboard GitHub and GitLab repositories, accounts, and organizations without manual configuration. This automates scanning of application code and IaC for embedded secrets, vulnerabilities in dependencies, and violations of security policy and best practice.

Orca can determine the risk posture of container workloads in cloud Kubernetes clusters or container hosts without requiring agents. The innovative approach that Orca uses to scan workloads is called SideScanning™ and utilizes cloud provider snapshot mechanisms. With SideScanning, Orca can perform a deep inspection on nodes that host containerized workloads and peer into the containers that execute on these hosts. 

The Orca Platform brings differentiation to cloud-native organizations with the ability to combine runtime context with vulnerability and compliance details across the application lifecycle. For example, organizations can now detect critical risks that often arise in development, including the following issues:

  • Sensitive data exposure: Context-aware controls, where an IaC template may configure an insecure storage bucket that stores PII.
  • Secrets detection: Identifying secrets where a private key as part of a CI scan could allow lateral movement risks.
  • Production risks connected to CI: Connecting the origin of artifacts and the exact CI that scanned them and auto-assigning new issues directly to the proper development or DevOps team.

Orca also helps to validate compliance with support for 125+ out-of-the-box compliance templates enabling fast, effective compliance and reporting. These compliance templates include vital industry frameworks, such as PCI-DSS, HIPAA, GDPR, SOC 2, as well as widely used CIS Benchmarks, including Docker, Kubernetes, Linux and more.

The Orca Security Compliance dashboards featuring AWS and CIS KBS frameworks

“Orca consolidates everything we have, and it looks deep down into our technology stack—including within containers. It’s important to have something that can instantly tell us what we have, if there are any impacts, what needs to be done, and what’s the practicality—that is, what’s the size of that impact across our estate? Orca provides that and gives us additional mileage as to what we’re looking at across our infrastructure and cloud. No other technology was able to provide us with that on the cloud infrastructure side.”

Leo Cunningham, 
Chief Information Security Officer, Flo Health

“Orca is huge for helping us work with DevOps. My sys admin can now show and explain to DevOps what we’ve found. We’re now more collaborative and helpful to them. It’s a big step toward DevSecOps—the organizational friction between DevOps and my security team is gone.”

Nir Rothenberg, 
Chief Information Security Officer, Rapyd

Table of contents