Apache Infrastructure Team

Friday August 28, 2009

apache.org downtime - initial report

This is a short overview of what happened on Friday August 28 2009 to the apache.org services.  A more detailed post will come at a later time after we complete the audit of all machines involved.

On August 27th, starting at about 18:00 UTC an account used for automated backups for the ApacheCon website hosted on a 3rd party hosting provider was used to upload files to minotaur.apache.org.  The account was accessed using SSH key authentication from this host.

To the best of our knowledge at this time, no end users were affected by this incident,  and the attackers were not able to escalate their privileges on any machines.

While we have no evidence that downloads were affected, users are always advised to check digital signatures where provided.

minotaur.apache.org runs FreeBSD 7-STABLE and is more widely known as people.apache.org.  Minotaur serves as the seed host for most apache.org websites, in addition to providing shell accounts for all Apache committers.

The attackers created several files in the directory containing files for www.apache.org, including several CGI scripts.  These files were then rsynced to our production webservers by automated processes.  At about 07:00 on August 28 2009 the attackers accessed these CGI scripts over HTTP, which spawned processes on our production web services.

At about 07:45 UTC we noticed these rogue processes on eos.apache.org, the Solaris 10 machine that normally serves our websites.

Within the next 10 minutes we decided to shutdown all machines involved as a precaution.

After an initial investigation we changed DNS for most apache.org services to eris.apache.org, a machine not affected and provided a basic downtime message.

After investigation, we determined that our European fallover and backup machine, aurora.apache.org, was not affected.   While the some files had been copied to the machine by automated rsync processes, none of them were executed on the host, and we restored from a ZFS snapshot to a version of all our websites before any accounts were compromised.

At this time several machines remain offline, but most user facing websites and services are now available.

We will provide more information as we can.



Thank you for being open about this!

Posted by Steve 'Ashcrow' Milner on August 28, 2009 at 01:09 PM UTC #

You need to circulate this post and the ones to come more - there's already misleading articles related to this.

Posted by Justin R on August 28, 2009 at 01:58 PM UTC #

Sucks that you got hit, but great that you're telling people about it and being upfront. Would love to know more about the compromised key though, how that happened...

Posted by Phil Rigby on August 28, 2009 at 03:10 PM UTC #

yeah...apache.org deals better with this things than Dan Kaminsky or Matasano...

Posted by on August 28, 2009 at 03:33 PM UTC #

@ Ouch! @pquerna Yes - thanks for being open and honest. Makes us feel more comfortable than when companies 'fog' the facts.

Posted by Network Sentry on August 28, 2009 at 04:21 PM UTC #

Doh! Time to switch hosting providers eh?

Posted by Tom on August 28, 2009 at 04:53 PM UTC #

Is this a DDOS attack

Posted by Narasimha on August 28, 2009 at 06:25 PM UTC #

@Narasimha: Are you joking or did you not bothering reading at all?

Posted by send9 on August 28, 2009 at 07:41 PM UTC #

Why are the attacked guys the good ones... this sucks, go chase those idiots - hunt them down and make'em pay!

Posted by whoppie on August 28, 2009 at 08:01 PM UTC #

The ASF infra team rocks! Thanks.

Posted by Ross Gardler on August 28, 2009 at 08:15 PM UTC #

Hi Guys - Yes, kudos for posting this info. Question: Any guidance on what tools you were using to become aware of these 'rogue' processes running on your servers? SIEM?? Many thanks! Marc

Posted by marcc on August 29, 2009 at 01:47 AM UTC #

???? ???? ?? ?????? ??? ?? ??????

Posted by H4TIM on August 29, 2009 at 02:03 AM UTC #

Thanks for the information, actually I got this link from one of the user group mails. How can SSH authentication from a host be used by an hacker unless he/she has some access to that host itself ?

Posted by Sameer on August 29, 2009 at 02:44 AM UTC #

Perhaps it is time to consider a policy of off-by-default for CGI, PHP, WSGI etc and only whitelist specific directories that need to run scripts? Please do this change and push it down to all your distributors.

Posted by foo on August 29, 2009 at 03:21 AM UTC #

Way to go. Let other companies take a lesson here and adopt your openness when/if they get hit by a hack. Thumbs up, good luck in finding the guilty party. Stay Secure!

Posted by Bart Van der Avort on August 29, 2009 at 07:19 AM UTC #

Bad that you got hacked... good that you've come clean. When you have all of your systems back on-line, can you post a followup that details (if you can) 1) how you determined you had a problem (tripwire?) and 2) how you recovered (restore from backup?) and 3) what you did to prevent it from happening again (chmod?) Thanks, and keep up the good work. (ps, when i add 6 + 28 and get 2D, i get an error)

Posted by theodore on August 29, 2009 at 11:15 AM UTC #

Thanks for the information

Posted by zaher on August 29, 2009 at 05:24 PM UTC #


Posted by on August 30, 2009 at 07:04 AM UTC #

Thank you for both your information and work. Additionally, I want to know below. 1. It affected to maven.apache.org? 2. same thing of repo01.maven.org was? thanks.

Posted by tashen on August 31, 2009 at 07:55 AM UTC #

[Trackback] ?? ??? ????????? ???????: ???? 28/8/09 hackers ???????? ?? ??????? website ??? Apache, ?? www.apache.org

Posted by Macedonia IT Pros on August 31, 2009 at 01:56 PM UTC #

Have the devs given insight over their workflows and security measures to new players recently? Going partially open can be a problem if you pick the wrong sub population. Hope no one is being bribed. OK, I am over-reacting.

Posted by Jose_X on August 31, 2009 at 09:11 PM UTC #

Hi,guys.. You`ve hacked - its true,but: 0 files deleted 0 files modified 0 trojans uploaded This hack was done to ensure that you become more secure.Also, your server is running FreeBSD 7.2 and SunOS 5.10 with patches,which do not have public exploits at the moment. So, i`m a white hat..no black. ;-) ps: Thank you for your understanding. Any questions - write here, i will try to answer. 4moke)))

Posted by Snake on September 01, 2009 at 09:23 AM UTC #

[Trackback] https://blogs.apache.org/infra/entry/apache_org_downtime_initial_report

Posted by Blind Injection on September 01, 2009 at 09:34 AM UTC #

[Trackback] A compromised SSH key lets hackers into Apache.org. ...

Posted by Joe Poniatowski on September 02, 2009 at 08:42 PM UTC #

Post a Comment:
Comments are closed for this entry.



Hot Blogs (today's hits)

Tag Cloud