Apache Infrastructure Team

Friday August 28, 2009

apache.org downtime - initial report

This is a short overview of what happened on Friday August 28 2009 to the apache.org services.  A more detailed post will come at a later time after we complete the audit of all machines involved.

On August 27th, starting at about 18:00 UTC an account used for automated backups for the ApacheCon website hosted on a 3rd party hosting provider was used to upload files to minotaur.apache.org.  The account was accessed using SSH key authentication from this host.

To the best of our knowledge at this time, no end users were affected by this incident,  and the attackers were not able to escalate their privileges on any machines.

While we have no evidence that downloads were affected, users are always advised to check digital signatures where provided.

minotaur.apache.org runs FreeBSD 7-STABLE and is more widely known as people.apache.org.  Minotaur serves as the seed host for most apache.org websites, in addition to providing shell accounts for all Apache committers.

The attackers created several files in the directory containing files for www.apache.org, including several CGI scripts.  These files were then rsynced to our production webservers by automated processes.  At about 07:00 on August 28 2009 the attackers accessed these CGI scripts over HTTP, which spawned processes on our production web services.

At about 07:45 UTC we noticed these rogue processes on eos.apache.org, the Solaris 10 machine that normally serves our websites.

Within the next 10 minutes we decided to shutdown all machines involved as a precaution.

After an initial investigation we changed DNS for most apache.org services to eris.apache.org, a machine not affected and provided a basic downtime message.

After investigation, we determined that our European fallover and backup machine, aurora.apache.org, was not affected.   While the some files had been copied to the machine by automated rsync processes, none of them were executed on the host, and we restored from a ZFS snapshot to a version of all our websites before any accounts were compromised.

At this time several machines remain offline, but most user facing websites and services are now available.

We will provide more information as we can.


Saturday August 01, 2009

Relaying mail from apache.org.

One of the more common issues committers face at Apache is in trying to send mail from their apache.org account.  We've just made that process a whole lot easier by setting up an SSL-enabled, smtp-auth based mail submission service on people.apache.org port 465; which is compatible with gmail's recently announced feature to allow outbound mail from your apache.org address to be directed to people.apache.org, instead of to a gmail server, for delivery.  Say goodbye to all the ezmlm moderation battles: your SMTP envelope sender will now match your From header!

In the future we may wish to tighten up the SPF records for apache.org, so please take advantage of this new service for all outbound delivery of your personal apache.org email.

Wednesday July 15, 2009

Public Preview of Drafts feature added to ASF Roller instance

Previously, to be able to preview a draft post by any Roller Blog, one had to be a member user of that blog.

For those that would like an easy way to post previews of drafts for lazy consensus or voting, a script has been setup to allow the preview url that Roller generates to be shared publicly.  For example:

   (roller preview url)

   (public preview url)

A typical process is to create the blog post, set it up to publish in 3-4 days via the "Advanced Settings", then post the modified preview URL to your dev@ list with the anticipated publish date for lazy consensus.

Projects must opt-in by adding the "preview" user with "Limited" access.

Details here:


Thursday May 21, 2009

It's official, we now have LDAP running!

Earlier this week the Infrastructure team rolled out phase one of the planned LDAP services.  

We are using LDAP for authentication of shell accounts.  For now this is the extent of the implementation, however the next phase should follow this quite quickly.

The next phase will involve moving to LDAP to manage access to our subversion repositories. This is a slightly more complicated migration as we currently use an SVNAuthz file, that contains the appropriate groups and their memberships.  We are currently working on a new template system where by changes to LDAP will trigger a build of the SVNAuthz file based on groups in LDAP.  This means we must watch LDAP changes, work on a template system, and if a new version of the template is checked into Subversion we need to trigger a build again.  This is a work in progress at the moment. 

If you find yourself in the position of needing to change your shell account password you can do it by doing this on the command line "ldappasswd -W -S -A -D uid=availid,ou=people,dc=apache,dc=org"  -- Where availid is your ASF username.   For example  "ldappasswd -W -S -A -D uid=pctony,ou=people,dc=apache,dc=org".  This is far from an elegant solution, but for now it works.  You will be required to enter and confirm your current password, and then enter and confirm your new password choice, followed by your LDAP password (this is your old password) .

We are working on a web portal that will allow users to edit attributes, such as forwarding address, password, etc.  This will be made available as soon as it is ready.  If you don't know your current password, then you will need to email  root@ as per usual. 

You can follow the trials and tribulations of the rollout on my personal blog  

Sunday May 03, 2009

Git support at Apache

Git is a new version control system that has been getting increasingly popular during the past few years. Many Apache contributors have also expressed interested in using Git for working with Apache codebases. While the canonical location of all Apache source code is our Subversion repository, we also want to support developers who prefer to use Git as their version control tool.

Based on work by volunteers on the infrastructure-dev@ mailing list, we have recently set up read-only Git mirrors of many Apache codebases at http://git.apache.org/. These mirrors contain the full version histories (including all branches and tags) of the mirrored codebases and are updated in near real time based on the latest svn commits.

See the documentation and wiki pages for more details about this service and how to best use it. We are also open to good ideas on how to extend or improve this service. Please join the infrastructure-dev@ mailing list for the ongoing discussion!

Monday April 06, 2009

New mailing list for CI Build Services

Established today, we now have a dedicated mailing list to talk about and work out all things to do with our build services. Currently infrastructure provides projects with use of Hudson, Continuum, Gump and now we have another option in Buildbot. Buildbot is a new service here at Apache Infrastructure, currently in its last stages of testing , more info coming soon. 

 All these services and all the projects that use them, are welcome to meet together on the new mailing list. Maybe your project is looking to use one or more of these CI's to build & test their code, build their site, publish to Nexus. Maybe you are already using a CI and want some configuration additions/changes or extra jobs run.

Also look out for us poor souls looking after these instances and the machines they run on - we might need more information from you projects, clarification or updating of build requirements, builds taking too long that needs investigation.

Failing builds are of course for each project to solve code-wise, but be sure that whichever CI(s) you choose, they are there to inform and will give you constant reminders of build failures ;)

 Sign up to the new mailing list - builds-subscribe-AT-apache-DOT-org.

 See you there! 


Thursday April 02, 2009

Improving our Subversion Services

This week the ASF Infrastructure Team deployed one of the first major changes to how svn.apache.org works since it was launched, 6 years ago.

We now distribute Subversion traffic to our servers based on the geographic region of a client.

We are using pgeodns, the same software that powers CPAN Search and the NTP Pool.  With pgeodns we can give out different DNS entries to clients, depending on where they are connecting from.  It isn't an exact science, but for most clients it is good enough to find the closer Subversion Server.

If you are connecting from Europe, your client will connect to Harmonia.  Harmonia is a Sun x4150 running FreeBSD 7.0, using ZFS raid2z over  6 disks, hosted in Amsterdam at SURFnet.

Users in North America are directed to Eris, our traditional Subversion Master Server.  Eris is a Dell 2950 also running FreeBSD 7.0, using ZFS raid2z over 4 disks, hosted in Corvallis, Oregon at OSUOSL.

Using svnsync as described in Norman's ApacheCon EU 2009 Talk, we replicate all commits to the master to the slave in real time.  If a commit is made to the slave, we proxy the commit to the master.

Read operations are handled on the nearest mirror, and are much faster for everything from the initial checking out to running an update due to the decreased latency.

While this change should improve the experience significantly, we have some other changes coming up soon for svn.apache.org:

  • Upgrade to Subversion 1.6: Representation Sharing, inode packing, and memcached support should help make our SVN servers even faster.
  • Upgrade both Eris and Harmonia to FreeBSD 7.2-STABLE: The ZFS filesystem is experimental in FreeBSD 7, and there are many stability and performance enhancements available in newer versions.
  • Adding more Geographic Mirrors: Once we are comfortable with the current setup, we would like to expand to another mirror location, hopefully in Australia or Asia. 

Subversion on-the-fly Replication Talk

Last week (at ApacheCon 2009 EU) I gave a session talk about "Subversion on-the-fly Replication" and how we (ASF) deployed such an setup last year with in the Apache Software Foundation.  So check out the slides if you are interested in how it works, why you should do it, what are the known problems and the solutions etc..


 Thanks to all the people who attended the session I (we) are still open for good suggestions and feedback ;-) 

There is no need for being sad if you missed the session talk, it was recorded as part of the "HTTP Server Administration" track. You can register for it HERE

Ps: Thanks to Tony Stevenson to act as my session chair and to Paul Querna to help out on answering questions ;)

Wednesday March 25, 2009

New faces in Infrastructure

Over the past year the Infrastructure Team has grown to meet new challenges.  Here is a list of the new folks on the team:

  • Gavin McDonald (gmcdonald)
  • Norman Maurer (norman)
  • Tony Stevenson (pctony)
  • Wendy Smoak (wsmoak)
  • Mark Thomas (markt)
  • Chris J. Davis (chrisjdavis)
  • Jukka Zitting (jukka)

Congratulate these people on the hard work they have done for the ASF the next time you bump into one or two of them- or better yet, buy 'em a beer!



Hot Blogs (today's hits)

Tag Cloud