Rancher stored sensitive values in plaintext exposed Kubernetes clusters to takeover The Daily Swig
Rancher stored sensitive values in plaintext, exposed Kubernetes clusters to takeover
Ben Dickson 28 September 2022 at 14:08 UTC
Updated: 30 September 2022 at 12:16 UTC
Cloud Security Vulnerabilities Open Source Software
Maintainers patch vulnerability and offer mitigation advice over bug that affects Rancher-owned objects
Rancher stored sensitive values in plaintext, exposed Kubernetes clusters to takeover
UPDATED A now-patched version of Rancher, an open source Kubernetes management tool, stored sensitive values in plaintext, a pair of software developers have discovered.
Exploitation could have enabled attackers to gain privileged access to various Rancher-owned Kubernetes objects, they found.
Rancher, which was acquired by German software provider SUSE in 2020, is popular among the DevOps and Kubernetes communities.
Catch up with the latest cloud security news
The platform allows developers to deploy and run Kubernetes container clusters from different providers. It also adds value by centralizing authentication and role-based access control to clusters, which allows admins to control cluster access from one location.
The bug was reported by Linux system engineer Marco Stuurman and developer Florian Struck. Stuurman stumbled on the flaw while investigating Rancher’s service tokens. “I'm not a security researcher, but I keep my eyes open for things like this,” Stuurman told The Daily Swig.
“I fetched information from one of our Rancher set-ups and became suspicious of the token. I've looked into similar tokens before, so it caught my attention.”
‘Problem lies in low privileges’
According to Stuurman’s findings, Rancher stored sensitive fields like passwords, API keys, and account tokens in unencrypted plaintext directly on Kubernetes objects.
“Storing secrets in plaintext is indeed bad practice but sometimes needed. In this case, you can't choose to hash the secret as this is the access key to the cluster,” Stuurman said. “The problem lies in the low privileges needed to access this key.”
According to the bug report, the plaintext data would be available to anyone who has access to certain Rancher-owned Kubernetes objects.
“The attacker only needed the least possible privileges to a cluster Rancher manages. For example, our monitoring robot user’s only privilege was to proxy HTTP requests from rancher to the monitoring instance running in the target cluster,” Stuurman said.
Protecting your cluster
The service account token used to provision clusters is particularly critical since it has the highest privileges. In this case, the exploit could allow attackers to escalate their privileges and completely take over the cluster if there weren’t any other advanced prevention measures.
In comments made to The Daily Swig, the Rancher security team said that not storing that data as secrets objects in Kubernetes came from some initial architectural and scaling decisions made when Rancher was still under early development. It was reviewed and addressed in the latest version of the platform.
“The remediation of the issue is a direct result of improvements that we started to implement, like creating a security team composed of a security engineer and developers to focus on security related issues and improvements,” the security team said.
RECOMMENDED #AttachMe Oracle cloud bug exposed volumes to data theft, hijack
“We also changed our release process to have security architectural reviews and tests before releasing new features, as well as enabling more security focused tools in our development pipeline.”
The issue has been addressed in the latest version of Rancher. The maintainers of the project have provided a script to rotate Rancher service account tokens. They also advised admins to limit access to Rancher instances, to check their downstream clusters for potential signs of a breach, and to change credentials that might have leaked.
“The best way to protect your cluster is to limit your cluster management tool access to people you trust,” Stuurman said. “However, that doesn't mean it shouldn't be as secure as possible.”
This article was updated on September 30 with comments and corrections from Rancher, in particular to reflect the fact that sensitive values, rather than secrets, were exposed, and that exploitation could lead to attackers gaining privileged access to Rancher-owned objects.
Ben Dickson 28 September 2022 at 14:08 UTC
Updated: 30 September 2022 at 12:16 UTC
Cloud Security Vulnerabilities Open Source Software
Maintainers patch vulnerability and offer mitigation advice over bug that affects Rancher-owned objects
Rancher stored sensitive values in plaintext, exposed Kubernetes clusters to takeover
UPDATED A now-patched version of Rancher, an open source Kubernetes management tool, stored sensitive values in plaintext, a pair of software developers have discovered.
Exploitation could have enabled attackers to gain privileged access to various Rancher-owned Kubernetes objects, they found.
Rancher, which was acquired by German software provider SUSE in 2020, is popular among the DevOps and Kubernetes communities.
Catch up with the latest cloud security news
The platform allows developers to deploy and run Kubernetes container clusters from different providers. It also adds value by centralizing authentication and role-based access control to clusters, which allows admins to control cluster access from one location.
The bug was reported by Linux system engineer Marco Stuurman and developer Florian Struck. Stuurman stumbled on the flaw while investigating Rancher’s service tokens. “I'm not a security researcher, but I keep my eyes open for things like this,” Stuurman told The Daily Swig.
“I fetched information from one of our Rancher set-ups and became suspicious of the token. I've looked into similar tokens before, so it caught my attention.”
‘Problem lies in low privileges’
According to Stuurman’s findings, Rancher stored sensitive fields like passwords, API keys, and account tokens in unencrypted plaintext directly on Kubernetes objects.
“Storing secrets in plaintext is indeed bad practice but sometimes needed. In this case, you can't choose to hash the secret as this is the access key to the cluster,” Stuurman said. “The problem lies in the low privileges needed to access this key.”
According to the bug report, the plaintext data would be available to anyone who has access to certain Rancher-owned Kubernetes objects.
“The attacker only needed the least possible privileges to a cluster Rancher manages. For example, our monitoring robot user’s only privilege was to proxy HTTP requests from rancher to the monitoring instance running in the target cluster,” Stuurman said.
Protecting your cluster
The service account token used to provision clusters is particularly critical since it has the highest privileges. In this case, the exploit could allow attackers to escalate their privileges and completely take over the cluster if there weren’t any other advanced prevention measures.
In comments made to The Daily Swig, the Rancher security team said that not storing that data as secrets objects in Kubernetes came from some initial architectural and scaling decisions made when Rancher was still under early development. It was reviewed and addressed in the latest version of the platform.
“The remediation of the issue is a direct result of improvements that we started to implement, like creating a security team composed of a security engineer and developers to focus on security related issues and improvements,” the security team said.
RECOMMENDED #AttachMe Oracle cloud bug exposed volumes to data theft, hijack
“We also changed our release process to have security architectural reviews and tests before releasing new features, as well as enabling more security focused tools in our development pipeline.”
The issue has been addressed in the latest version of Rancher. The maintainers of the project have provided a script to rotate Rancher service account tokens. They also advised admins to limit access to Rancher instances, to check their downstream clusters for potential signs of a breach, and to change credentials that might have leaked.
“The best way to protect your cluster is to limit your cluster management tool access to people you trust,” Stuurman said. “However, that doesn't mean it shouldn't be as secure as possible.”
This article was updated on September 30 with comments and corrections from Rancher, in particular to reflect the fact that sensitive values, rather than secrets, were exposed, and that exploitation could lead to attackers gaining privileged access to Rancher-owned objects.
