Security Advisory: Insufficient Tenant Separation in Azure Synapse Service

UPDATE – Jun-13-2022

Microsoft has implemented several improvements as they continue to work towards comprehensive tenant isolation. As such, we are removing alerting on our customers’ usage of Azure Synapse within the Orca Cloud Security Platform. We’ve now published the full technical details on our SynLapse discovery.

TL;DR

Orca Security is issuing this security advisory for CVE-2022-29972 to address hazards in the use of the Microsoft Azure Synapse service. We believe the tenant separation in this service is insufficiently robust to protect secrets against other tenants. This belief is based on Orca Security researchers bypassing tenant separation multiple times, beginning in January, and again bypassing fixes deployed by Microsoft in March and April. Based on our understanding of the architecture of the service, and our repeated bypasses of fixes, we think that the architecture contains underlying weaknesses that should be addressed with a more robust tenant separation mechanism. Until a better solution is implemented, we advise that all customers assess their usage of the service and refrain from storing sensitive data or keys in it. The Orca Cloud Security Platform will alert our mutual customers on the usage of this service.

Previous cloud provider vulnerabilities were fixed quickly

The Orca Research Pod is always on the lookout for new cloud risks and has found numerous novel vulnerabilities in the major cloud providers. We dedicate time to finding vulnerabilities to make the cloud a safer place – we believe that it is better if we find them, disclose them in a responsible way, and have them fixed – before the bad actors do. In previous cases, vulnerabilities were fixed by the cloud providers within a few days of our disclosure to the affected vendor. Superglue, for example, was fixed within a day, BreakingFormation was fixed within 26 hours, and AutoWarp was fixed within a few days. 

SynLapse discovery timeline

On January 4, Tzah Pahima, one of our researchers, reported a critical vulnerability in Azure Synapse which he named SynLapse. This vulnerability allows an attacker to access and control other customers’ Synapse workspaces, and leak sensitive data stored in the service including Azure’s service keys, API tokens, and passwords to other services.

Below is a demo of the original situation. In the video, a “customer” (not a real customer) uses Azure Synapse Analytics to store credentials to an external service (HTTP server in this example). The attacker then manages to easily extract said credentials while executing code on the customer’s machine.

We reported SynLapse to Microsoft Security Response Center (MSRC), in accordance with the industry-standard 90-day coordinated vulnerability disclosure process. The report included our concerns about the weak implementation of tenant separation in this service, as well as the fact that we were able to download highly privileged internal Microsoft keys. 

It took until late March, after several escalations, for MSRC to notify us that they had fixed the reported issue. We were disappointed to see that while the specific vulnerability was fixed, the tenant separation was still extremely weak, and we were able to demonstrate a different attack vector that also allowed a bypass of Synapse tenant separation on March 30. At this time, we also reminded Microsoft that the keys we downloaded were still valid and allowed access to sensitive data of other customers.

On April 10, we received notification from Microsoft that the issue had been fixed, and that the keys that we downloaded in January had finally been revoked. While the keys were successfully revoked, the fix was unfortunately still partial, and in under 24 hours, we were able to demonstrate another attack vector that allows bypassing tenant separation in this service to allow access to other tenants’ data. We extended the disclosure window by another month, to May 4, hoping a more robust solution would be implemented in the additional time.

Microsoft has since implemented additional mitigation measures that make exploitation much harder. Unfortunately, our research leads us to believe that the underlying architectural weakness is still present. There are areas in the service where a huge amount of Microsoft and 3rd party code, runs with SYSTEM permissions, processing customer controlled input. This runs on shared machines with access to Azure service keys and sensitive data of other customers. These areas of the service only have application-level separation and lack sandbox or hypervisor-level isolation. This is a major attack surface and not consistent with the level of security that public cloud customers expect. 

Microsoft response

We shared a draft of this blog post with Microsoft on May 5 asking for a public response to include in this blog. They chose not to provide one. Today MSRC issued the following advisory: Vulnerability mitigated in the third-party Data Connector used in Azure Synapse pipelines and Azure Data Factory (CVE-2022-29972). Microsoft confirms that additional defensive measures are required, as they stated they are still “ensuring that Cloud processes and workloads, including third-party data connectors, run in a Zero Trust architecture that advance cross tenant isolation. Specifically, we are implementing virtualization of third-party connector execution to achieve per-tenant isolation.”

We are a partner of Microsoft and have followed the coordinated disclosure process. Unfortunately, there is still no clear ETA for a robust implementation of tenant separation in this service. Until robust tenant separation is present, we believe that the current mitigations merely prevent specific attack vectors, and that other attack vectors remain present.

Reconsider your use of Azure Synapse

At this time, our recommendation is to reconsider the usage of the Azure Synapse service and consider deleting any keys, secrets, or sensitive data that exist on it, until the root issue is resolved, and more robust tenant separation is implemented and its details are shared publicly by Microsoft. 

We are going to hold off on publishing technical details of the exploits we have found until June 14, for two reasons. First, the vulnerabilities are also present in the on-premises version of Synapse, and this will provide Microsoft’s customers some additional time to deploy and remediate the existing mitigations in their on-premises environments. Second, we believe that the technical details of the exploit will make it easier for attackers to find more open attack vectors, and the delay will allow time for organizations to reconsider their usage of Synapse.

On a personal note, writing this was not an easy decision

Yoav Alon, our CTO and a seasoned vulnerability researcher, characterized our difficult decision as “we realized there would be a negative impact whether we published now or waited on a complete fix to become available from Microsoft.” Transparency and disclosure are key pillars of security, which is why we made the difficult decision to share these details with the wider security community now.

I empathize with the practitioners that will have to work hard around the world to handle this vulnerability. I’m also concerned that publishing this will send adversaries looking into this service, maybe being able to exploit it before Microsoft manages to harden it enough, or before practitioners will be able to stop using it.

 Having said that, coordinated disclosure processes exist for a reason – they balance the time needed for the vendor to fix the issue, and the ability of security practitioners to defend against risks. At Orca Security we have a commitment to alert our customers and the industry to cloud risks – and waiting any longer would, in our opinion, be a violation of this responsibility.