The Apache CloudStack Blog
Dnsmasq Vulnerabilities Advisory for CloudStack
Recently, a number of security flaws were recently found in the DNSMasq tool. This tool is used by many systems to provide DNS and DHCP services, including by the CloudStack System VMs.
According to Google’s investigation into the software, out of seven issues, three — CVE-2017-14491, CVE-2017-14492, and CVE-2017-14493 — are remote code execution flaws caused by heap buffer overflow and stack buffer overflow errors through DHCP and DNS vectors.
Another issue, CVE-2017-14494, can be exploited to bypass the Address space layout randomization (ASLR) memory protection function, leading to information leaks.
In addition, three more bugs, CVE-2017-14495, CVE-2017-14496, and CVE-2017-13704, can lead to denial-of-service (DoS) attacks caused by invalid boundary checks, bug collision, and a coding issue.
Affect On CloudStack
CloudStack’s System VMs use DNSMasq to provide DNS and DHCP services to the guest VMs from the virtual routers. These services are only exposed on the internal guest interface(s) of the virtual routers. Therefore a malicious user could compromise a virtual router to which they have a guest instance attached.
On 9th October, an updated version of DNSMasq was released by the authors of DNSMasq for the Debian Wheezy Operating System which the CloudStack System VMs use. We have created new versions of the System VM templates which should be used to replace your existing System VMs using the procedure described below.
A short-term fix for currently running System VMs (if they have internet access) is to log into the System VMs and run:
apt-get upgrade dnsmasq -y
For information on logging into System VMs please see: http://docs.cloudstack.apache.org/en/latest/administration_guide.html?#accessing-system-vms
The above procedure will patch existing virtual routers, but should a virtual router be destroyed and recreated or a new virtual router created, the subsequent virtual router will no longer be patched.
The full fix is to replace the existing System VM template(s) with the latest patched versions as well as recreating or patch existing virtual routers.
System VM Patching Procedure
New System VM templates with updated DNSMasq for major CloudStack versions for XenServer, VMware and KVM hypervisors have been built. We advise CloudStack users to upgrade to the appropriate System VM template and either;
Patch all existing virtual routers using the procedure above or recreate all virtual routers using the procedure detailed in the link for updating system VM templates (below)
For ACS 4.10+: http://download.cloudstack.org/systemvm/4.10/dnsmasq/
For ACS 4.6-4.9: http://download.cloudstack.org/systemvm/4.6/dnsmasq/
The procedure for updating the system VM templates can be found at http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/latest/systemvm.html#enhanced-upgrade-for-virtual-routers.
Apache CloudStack registerUserKeys authorization vulnerability
The CloudStack security team recently received notice of a significant vulnerability in a CloudStack API call - registerUserKeys. The original intention for this call was for it to only be exposed for integration work - eg not to the public network in general. A weakness in the API call's implementation allows a malicious user to reset the API keys for other users on the system, thus accessing resources and services available to that user. We have released CloudStack versions 220.127.116.11 and 18.104.22.168 with patches for this issue. More details about the release can be read on the official announcement post.
Some users may be protected from this weakness already, if they have configured their commands.properties file to limit access to this api call from the integration API port, instead of general API port. This can be accomplished by setting registerUserKeys to 1.
Users of Apache CloudStack version 4.9 whom are using the dynamic roles feature can delete the "Allow" rule for "registerUserKeys" for each non-administrator role under the Roles/Rules section of the user interface.Credit:
This vulnerability was reported by Marc-Aurèle Brothier from Exoscale.
Two late-announced security advisories
Today I sent out two CloudStack-related security advisories: CVE-2015-3251 (related to VM credential exposure) and CVE-2015-3252 (related to VNC authentication). Details about these issues can be found on the CloudStack user and dev mailing lists, as well as on the Full Disclosure and BUGTRAQ security mailing lists.
While these vulnerabilities are of moderate and low severity (respectively), the reason for this post is because the advisories were announced approximately 5 months after the first release of the patches in 4.5.2. This is personally embarrassing, unacceptable, and in a more severe case could be downright dangerous.
The CloudStack security team worked through the related vulnerabilities through the summer of 2015. We had advisory drafts, patches, and mitigations all ready well before the release. Far enough ahead, actually, that we forgot about the release and weren't paying attention to the release (at least I wasn't - I know others were), and didn't send out the advisories at the appropriate time. Part of this is due to me having become an unofficial lead/spokesperson for the security team; In the past there has been at least one occasion when others released advisories when I was not available, but usually I'm coordinating issues and publishing announcements.
Luckily, the CloudStack Security Team works with and under the direction of the ASF security team. During one of their periodic reviews, they noticed CloudStack had loose ends on these two advisories, and asked for an update. Earlier today I realized the advisories had not been released, so here we are.
How will we improve?
Obviously, we don't want to be in this situation again. Here's some steps we're taking to minimize the chance of a repeat performance:
- I've modified the Release Procedure to specifically request the release manager give the security team a heads up that a release is about to be announced. This can be a simple non-blocking email that shouldn't slow down the release process, but still ensure that we're aware of the upcoming release.
- I'll be ensuring that other members of the security team feel comfortable crafting and releasing advisories. Like the rest of CloudStack and other ASF projects, the CloudStack security team does not have a named leader and should be able to operate if I or others are unavailable.
In the past I've referred to CloudStack as "critical infrastructure" - CloudStack powers infrastructure clouds for many large cloud providers. We take information security seriously, realizing that many depend upon our work. Vulnerabilities happen in most software at some point in time - the important part is how they are responded to. While in this case we did respond quickly to the issues and created and applied patches, we let the community down by not quickly releasing the advisories. This is an unfortunate chink in our armor, but we'll be taking steps to ensure it doesn't happen again.
Apache CloudStack 4.6 is released
The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today the availability of Apache™ CloudStack™ v4.6, the turnkey Open Source cloud computing software platform used for creating private-, public-, and hybrid cloud environments.
Apache CloudStack clouds enable billions of dollars' worth of business transactions annually across their clouds, and its maturity and stability has led it to has become the Open Source platform for many service providers to set up on-demand, elastic public cloud computing services, as well as enterprises and others to set up a private or hybrid cloud for use by their own employees.
"This 4.6 release of Apache CloudStack marks a significant shift in how we release CloudStack," said Sebastien Goasguen, Vice President of Apache CloudStack. "With a focus on quality and speed of releasing software, we implemented a new release workflow which allows us to have a production-ready release branch all the time, and allows us to quickly release new features. From now on, CloudStack will be released much faster without regression and with increased quality in each version."
Recognized as the Cloud orchestration platform that "just works", CloudStack includes an intuitive user interface and rich APIs for managing the compute, networking, software, and storage infrastructure resources.
Under the Hood
CloudStack v4.6 reflects dozens of new features and improvements, including:
- NuageVsp Network Plugin
- Bind integration with Globo DNSAPI
- SAML 2.0 Plugin
- Managed storage for KVM
- Improved CloudByte Storage Plugin
- Use SSH for commands sent to Virtual-Router
- Baremetal Advanced Networking Support
- Instance Password Generation length can now be changed
A complete overview of all new enhancements are detailed in the project release notes
CloudStack v4.6 reflects more than 200 bug fixes from previous releases.
Apache CloudStack is in use/production at thousands of organizations worldwide that includes BIT.Group GmbH, BT Cloud, China Telecom, CloudOps, DATACENTER Services, DataCentrix, Datapipe, EVRY, Exaserve, Exoscale, IDC Frontier, iKoula, Imperial College, INRIA, KDDI, Korea Telecom, LeaseWeb, M5 Hosting Inc., Melbourne University, Reliable Networks, Redbridge, SafeSwiss Cloud, Schuberg Philis, ShapeBlue, Tranquil Hosting, Trader Media Group, University of Cologne, and the University of Sao Paulo, among others.
"With the 4.6 release the Apache CloudStack continues to mature adding important new functionality that will benefit CloudCentral's customers," said Kristoffer Sheather, Founder & Chief of Australian cloud services provider CloudCentral, who have been using Apache CloudStack since 2010. "The new Redundant Routers for Virtual Private Cloud (VPC) networks feature will ensure continuous availability of customer VPC networks, and Browser Based Template & Volume Upload will make it easier for our customers to import and use their choice of operating system ISO images and import VM templates from other cloud systems."
"CloudOps is very excited about the release of Apache CloudStack 4.6, which represents significant improvements in feature set and quality," said Ian Rae, CEO of CloudOps. "We are proud of our involvement in this landmark release and look forward to supporting our customers achieve operational success in upgrading to and operating clouds based on this release. Apache CloudStack is the best kept secret in open source cloud computing and has a global user base of cloud operators many of whom contribute to the project."
"The 4.6 release of Apache CloudStack brings new features and fixes bugs which are critical for our Aurora cloud offering at PCextreme," said Wido den Hollander, CEO of PCextreme. "We've worked hard with the community to get 4.6 released. The committers working at PCextreme resolved multiple issues and also introduced new features in CloudStack including a new StatsCollector output to Graphite and better support for CEPH. This new release allows us to grow our cloud even further."
"I'm very excited with launch of the Apache CloudStack version 4.6," said Cyrano Rizzo, CIO of the University of Sao-Paulo. "This version brought many new features and benefits, such as the case in resilience with the new redundant router for VPC, the capability to rapid deployment, demo and test to run the Apache CloudStack inside Docker that will speed the growth, the possibility to manage the resources with Graphite, the ease of upload templates and volumes, among many others, this version also brought many improvements, I'm very happy with one in particular that makes SAML plugin to production grade, this functionality is helping me to build a huge project called interCloud that intend to federate many public universities across the Brazil with Single Sign On."
Apache CloudStack welcomes contribution and community participation through mailing lists as well as attending face-to-face MeetUps, developer trainings, and user events. Catch Apache CloudStack in action at the next CloudStack European User Group on 3 March 2016 in London
CloudStack and OpenSSL CVE-2015-1793
Updated July 11th, 2015:
After reviewing CloudStack components and seeing Debian's advisory on CVE-2015-1793 (CloudStack's "system VM" is Debian based), it looks like CloudStack is not affected by this vulnerability.
Original post follows...
On the 9th of July, the OpenSSL project announced a high severity vulnerability within the OpenSSL library. While this particular vulnerability does not seem to affect SSL servers, there are security issues with SSL clients powered by OpenSSL. Because of this, we suspect there may be issues with parts of CloudStack which initiate SSL connections.
At this point we are still reviewing which particular versions of OpenSSL are used by different versions of CloudStack. Once this review is complete, we will further update the community and this post as to our next steps.
CloudStack and the "Ghost" glibc vulnerability
UPDATE: mitigation instructions have been improved (don't update openswan) and we forgot to mention rebooting.
UPDATE: Links to updated System VM templates are now below
Yesterday, a buffer overflow vulnerability was announced in glibc that affects most current Linux distributions. In CloudStack, the system VMs contain a vulnerable version of glibc.
CloudStack community members have built an updated system VM template, which ShapeBlue is hosting at http://packages.shapeblue.com/systemvmtemplate/ (More information on the packages at http://shapeblue.com/packages).
For instructions on how to update the SystemVM template in CloudStack, see here.
For those who wish to patch their running system VMs, ssh into each one and run:
apt-mark hold openswan apt-get clean apt-get update && apt-get upgradeAfter updating glibc, the system will need to be rebooted.
Information about how to connect to your System VMs is available here.
Other CloudStack-related systems may be affected!
Please review security updates from Linux distributions you use on your management server, storage systems, hypervisors, as well as other Linux VMs and bare-metal systems running in your environments. This post provides instructions for determining if a system is vulnerable, as well as patching directions for common Linux distributions.
[ANNOUNCE] Announcing Apache CloudStack 4.3.2
The Apache CloudStack project is pleased to announce the 4.3.2 release of the CloudStack cloud orchestration platform. This is a minor release of the 4.3 branch which released on March 25, 2014. The 4.3.2 release contains more than 100 bug fixes since the 4.3.1 release. As a bug fix release, no new features are included in 4.3.2.
As a minor release it is a simple upgrade from 4.3.0 or 4.3.1 with no architectural changes.
The 4.3.2 release notes includes full list of corrected issues as well as upgrade instructions from previous versions of Apache CloudStack. Please see the Release Notes for a full list of corrected issues and upgrade instructions.
The official installation, administration and API documentation for each release are available on our Documentation Page.
The official source code for the 4.3.2 release can be downloaded from our Downloads Page.
About Apache CloudStack
Apache CloudStack is an integrated Infrastructure-as-a-Service (IaaS) software platform that allows users to build feature-rich public and private cloud environments. CloudStack includes an intuitive user interface and rich APIs for managing the compute, networking, software, and storage infrastructure resources. The project became an Apache top level project in March 2013.
For additional marketing or communications information, please contact the marketing mailing list: firstname.lastname@example.org
To learn how to join and contribute to the Apache CloudStack community please visit our website: cloudstack.apache.org
[CVE-2014-7807] Apache CloudStack unauthenticated LDAP binds
CVE-2014-7807: Apache CloudStack unauthenticated LDAP binds
The Apache Software Foundation
Apache CloudStack 4.3, 4.4
Apache CloudStack may be configured to authenticate LDAP users. When so configured, it performs a simple LDAP bind with the name and password provided by a user. Simple LDAP binds are defined with three mechanisms (RFC 4513): 1) username and password; 2) unauthenticated if only a username is specified; and 3) anonymous if neither username or password is specified. Currently, Apache CloudStack does not check if the password was provided which could allow an attacker to bind as an unauthenticated user.
Users of Apache CloudStack 4.4 and derivatives should update to the latest version (4.4.2)
An updated release for Apache CloudStack 4.3.2 is in testing. Until that is released, we recommend following the mitigation below:
By default, many LDAP servers are not configured to allow unauthenticated binds. If the LDAP server in use allow this behaviour, a potential interim solution would be to consider disabling unauthenticated binds.
This issue was identified by the Citrix Security Team.
The Apache CloudStack Project Announces Apache™ CloudMonkey™ v5.3.0
The Apache CloudStack project is pleased to announce the 5.3.0 release of CloudMonkey, the command line interface tool for Apache CloudStack configuration and management.[Read More]
Announcing Apache™ CloudStack™ v4.4.1
The Apache CloudStack project is pleased to announce the 4.4.1 release of CloudStack, turnkey Open Source cloud computing software platform used for creating private-, public-, and hybrid cloud environments.[Read More]
CloudStack's realhostip service to retire in less than a week!
Citrix is reporting that they are still seeing DNS queries against the domain; Those who have not reconfigured their CloudStack installations will find part of their installations breaking once the realhostip service is retired on September 30th.
If you are running a version of CloudStack older than 4.3 and you have not reconfigured your installation to not use realhostip.com, please take the time to do so now before users are affected. Instructions are available in the CloudStack Wiki as well as other blogs on the Internet.
Realhostip Reprieve for CloudStack Users
As mentioned previously, the realhostip.com dynamic DNS resolver service is being retired this summer. During testing of Apache CloudStack version 4.3, we found a few more issues related to realhostip.com that have been addressed for the 4.4 release.
In order to give everybody a reasonable window to update their CloudStack installations to use the updated code, the retirement date for the realhostip.com service has been pushed back to September 30th, 2014. This provides an additional 3 months from the original June 30th date.
Any questions related to the retirement of the realhostip.com service and it's affect on CloudStack installations should be send to the CloudStack Users or Development mailing lists. Further information about how to subscribe and interact with the mailing lists is available at https://cloudstack.apache.org/mailing-lists.html.
ASF Mailing List Problems (Update: we are back to normal)
Update: The email infrastructure is back to normal, although it may take some time for the queued messages to completely flush out of the backlog. See the linked post from the ASF infrastructure team below for more details.
There are ongoing problems with the ASF's email infrastructure that mean no mail delivery is happening for our project's lists. Please check the official ASF infrastructure blog post and twitter feed for updates.