Apache Infrastructure Team
apache.org downtime - initial report
This is a short overview of what happened on Friday August 28 2009 to the apache.org services. A more detailed post will come at a later time after we complete the audit of all machines involved.
On August 27th, starting at
about 18:00 UTC an account used for automated backups for the ApacheCon
website hosted on a 3rd party hosting provider was used to upload files
to minotaur.apache.org. The account was accessed using SSH key
authentication from this host.
To the best of our knowledge at this time, no end users were affected by this incident, and the attackers were not able to escalate their privileges on any machines.
While we have no evidence that downloads were affected, users are always advised to check digital signatures where provided.minotaur.apache.org runs FreeBSD 7-STABLE and is more widely known as people.apache.org. Minotaur serves as the seed host for most apache.org websites, in addition to providing shell accounts for all Apache committers.
The attackers created several files in the directory containing files for www.apache.org, including several CGI scripts. These files were then rsynced to our production webservers by automated processes. At about 07:00 on August 28 2009 the attackers accessed these CGI scripts over HTTP, which spawned processes on our production web services.
At about 07:45 UTC we noticed these rogue processes on eos.apache.org, the Solaris 10 machine that normally serves our websites.
Within the next 10 minutes we decided to shutdown all machines involved as a precaution.
After an initial investigation we changed DNS for most apache.org services to eris.apache.org, a machine not affected and provided a basic downtime message.
After investigation, we determined that our European fallover and backup machine, aurora.apache.org, was not affected. While the some files had been copied to the machine by automated rsync processes, none of them were executed on the host, and we restored from a ZFS snapshot to a version of all our websites before any accounts were compromised.
At this time several machines remain offline, but most user facing websites and services are now available.
We will provide more information as we can.
Posted at 12:33PM Aug 28, 2009
by pquerna in General |
Comments [24]
|
Posted by Steve 'Ashcrow' Milner on August 28, 2009 at 01:09 PM UTC #
Posted by Justin R on August 28, 2009 at 01:58 PM UTC #
Posted by Phil Rigby on August 28, 2009 at 03:10 PM UTC #
Posted by 127.0.0.1 on August 28, 2009 at 03:33 PM UTC #
Posted by Network Sentry on August 28, 2009 at 04:21 PM UTC #
Posted by Tom on August 28, 2009 at 04:53 PM UTC #
Posted by Narasimha on August 28, 2009 at 06:25 PM UTC #
Posted by send9 on August 28, 2009 at 07:41 PM UTC #
Posted by whoppie on August 28, 2009 at 08:01 PM UTC #
Posted by Ross Gardler on August 28, 2009 at 08:15 PM UTC #
Posted by marcc on August 29, 2009 at 01:47 AM UTC #
Posted by H4TIM on August 29, 2009 at 02:03 AM UTC #
Posted by Sameer on August 29, 2009 at 02:44 AM UTC #
Posted by foo on August 29, 2009 at 03:21 AM UTC #
Posted by Bart Van der Avort on August 29, 2009 at 07:19 AM UTC #
Posted by theodore on August 29, 2009 at 11:15 AM UTC #
Posted by zaher on August 29, 2009 at 05:24 PM UTC #
Posted by 127.0.0.1 on August 30, 2009 at 07:04 AM UTC #
Posted by tashen on August 31, 2009 at 07:55 AM UTC #
Posted by Macedonia IT Pros on August 31, 2009 at 01:56 PM UTC #
Posted by Jose_X on August 31, 2009 at 09:11 PM UTC #
Posted by Snake on September 01, 2009 at 09:23 AM UTC #
Posted by Blind Injection on September 01, 2009 at 09:34 AM UTC #
Posted by Joe Poniatowski on September 02, 2009 at 08:42 PM UTC #