Apache Infrastructure Team

Wednesday May 28, 2014

Mail outage post-mortem

During the afternoon of May 6th we began experiencing delays in mail delivery of 1-2 hours. Initial efforts at remediation seemed to clear this up but on the morning of May 7th the problem worsened and we proactively disabled mail service to deal with the failure. This outage affected all ASF mailing lists and mail forwarding. The service remained unavailable until May 10th, and it took almost 5 additional days to fully flush the backlog of messages.

You can find a timeline here that was kept during the incident: https://blogs.apache.org/infra/entry/mail_outage

This was a catastrophic failure for the Apache Software Foundation as email is core to virtually every operation and is our primary communication medium.  

What happened:

The mail service at the ASF is composed of three physical servers. Two of these are external facing mail exchangers that receive mail. The final server handles mailing list expansion, alias forwarding and mail delivery in general. That latter server had two volumes that experienced a disk outage each. This degraded performance substantially and led to the mail delays seen on May 6th and 7th. The service was proactively disabled on May 7th in an attempt to let the arrays rebuild without the significant disk I/O overhead caused by processing the large mail backlog. Ultimately multiple attempts to rebuild the underlying arrays failed and eventually other drives in the array where the data volume was stored failed rendering recovery a hopeless task on May 8th. We began working to restore backups from our offsite backup location to our primary US datacenter. When this began to take longer than expected, additional concurrent efforts began to restore service in one of our secondary datacenters as well as in a public cloud instance. Ultimately we ended up completing the restoration to our primary US datacenter first and were able to bring the service online. When the service resumed, we had an estimated 10 million message backlog in addition to our normal 1.7-2 million ongoing daily message flow. The amount of backlogged mail taxed the existing infrastructure and architecture of the mail service and took almost 5 days to completely clear.

What worked:

Our backups were sufficient to allow us to restore the service in good working order.
Early precautions taken when we discovered the problem combined with our backups resulted in no data loss from the incident.
Our mail exchangers continued to work during the outage and held incoming mail until the service was restored.

What didn't work:

Our monitoring was not sufficient to identify the problem or alert us to the symptoms.
No spare hard drives for this class of machine were on-hand in our primary datacenter. 
The restore time from our remote backups took an excessively long time. This was partially due to the large size of the restore data, and partially due to the transport method used for the data.
After the service was restored we had approximately a 10M message backlog that took days to clear.
The primary administrator of the service was on vacation, and the remaining infrastructure contractors were not intimately familiar with the service. 
Our documentation was insufficient to easily restore the service in a rapid manner by folks without intimate knowledge. 

Remediation plan:

Our immediate action items:

  • Update the documentation to be current/diagram mail flow.
  • Improve the monitoring of the mail service itself as well as the hardware.
  • Insure we have adequate spares on hand for the majority of our core services.
  • Place our mail server under configuration management to reduce our MTTR

Medium-to-Long term initiatives.
  • Crosstraining contractors in all critical services
  • Work on moving to a more fault-tolerant/redundant architecture
  • More fully deploy our config management and automated provisioning across our infrastructure so MTTR is reduced.


Friday May 23, 2014

New monitoring system: nagios is dead long live circonus

23 may 2014 the old monitoring system "nagios" was put to sleep, and "circonus" was given production status.

The new monitoring system is sponsored by circonus and most of the monitoring as well as the central database runs on www.circonus.com. The infrastructure team have built and deployed logic around the standard circonus system:
- A private broker, to monitor internal services  without exposing them on internet
- A dedicated broker (inhouse development) that monitor special ASF systems (like svn compare US - EU)
- A configuration system, that are based on svn.
- A new status page status.apache.org
- A new team structure (all committers with sudo karma on a vm, get an email when something happens with the vm)

The new system is a lot faster and we can therefore offer projects monitoring of project URLs, of course the project also need to have a team that handles the alerts.

The current version has approx. the same facilities as Nagios, but we are planning (and actively programming) a version.2 that will allow us to better predict problems before they occur.

Some of the upcoming features are:
- disk monitoring
- vital data statistic from core system (like size of mail queues)

The change of monitoring system is a vital component in our transition to automate services and thereby enable infra to more effectively secure the stability of the infrastructure as well as make early detection of potential problems.

The system was presented in Apachecon denver 2014, slides can be found  here. We hope to present the live version at apachecon budapest 2014.

On behalf of the infrastructure team

jan I.

Wednesday May 07, 2014

Mail outage

During the afternoon of May 6th we began experiencing delays in mail delivery of 1-2 hours. Initial efforts at remediation seemed to clear this up but on the morning of May 7th the problem worsened and we proactively disabled mail service to deal with the failure. The underlying hardware suffered failures on multiple disks. This outage effects all ASF mailing lists and mail forwarding.

 This service is housed at OSUOSL, and we are currently waiting on smart hands to help with replacing hardware. Our expectation at this point is that we still have multiple hours worth of outage. 

 Incoming mail is currently being received and held in queue by our mail exchangers. We also have a copy of the existing queue that hasn't been processed; so we expect no mail or data loss.  

ASF Infra's twitter bot will provide updates as we have them for the duration of the outage. Feel free to follow @infrabot on Twitter. There will be an update on this post as well as the situation progresses.

UPDATE 7 May 19:27 UTC - Drives have been replaced, array is attempting to rebuild. As indicated earlier on twitter, there likely remains hours of outage.  

UPDATE 7 May 20:44 UTC - The disk array is still in the process of repairing. Several hundred mails were processed during a reboot, but more work remains before service is restored.  Mail service has been disabled again as the array repair process is CPU-bound. The plan going forward is to allow the disk arrays to finish repairs. Once that is complete, we'll reenable the mail service and flush what is currently in the queue. Finally, once the queue is empty we'll begin receiving mail again.

UPDATE 8 May 05:00 UTC - The disk array failed to repair itself. The disks have been replaced and a new installation has been completed. Progress continues to be made towards resolution, but nothing firm enough yet for us to predict an time for restoration.

UPDATE 8 May 15:45 UTC - No material change of status has occurred. Infra worked in shifts around the clock last night and continue to do so to restore service. More updates as they become available.  

UPDATE 9 May 11:20 UTC - We are working on temporarily restoring the most essential email aliases. In the meantime, inquiries may be made to infrastructure@apache.pw or on our IRC channel, #asfinfra on Freenode. The work on restoring the service in full is still ongoing.

UPDATE 9 May 17:20 UTC - We've successfully restored a host from backups and will be starting testing soon. Based on the progress made in those tests we'll try and provide expectations around restoration of service timeline.

UPDATE 10 May 15:45 UTC - We've started pushing live mails through the system - you'll begin to see them trickle in as we gradually open the floodgates to restore service. Expect intermittent spurts for a while. 

UPDATE 10 May 21:55 UTC -  The floodgates have been opened.  As we have a significant amount of backlog to catch up on, please be patient as the service does this.  As always feel free to contact us if you have any questions. In the immediate short term (next day or so, we suggest you continue to use infrastructure@apache.pw and our IRC channel, #asfinfra on Freenode.  We would like to thank you for your patience during this extremely busy time. 

UPDATE 12 May 16:04 UTC - Clarification - we have opened the floodgates, but have a substantial amount of backlog; and with the sudden rush of mail are being throttled by various mail services. With the addition of mail thats coming through anyway; it may take us from 2-5 days to fully flush the backlog. This time is so wide because of a wide variety of factors that are largely outside of our control, such as new mail coming in and mail services individual throttling policies.  

Friday Apr 11, 2014

heartbleed fallout for apache

Remain calm.

What we've learned about the heartbleed incident is that it is hard, in the sense of perhaps only viable to a well-funded blackhat operation, to steal a private certificate and key from a vulnerable service.  Nevertheless, the central role Apache projects play in the modern software development world require us to mitigate against that circumstance.  Given the length of time and exposure window for this bug's existence, we have to assume that some/many Apache passwords may have been compromised, and perhaps even our private wildcard cert and key, so we've taken a few steps as of today:

  1. We fixed the vulnerability in our openssl installations to prevent further damage,
  2. We've acquired a new wildcard cert for apache.org that we have rolled out prior to this blog entry,
  3. We will require that all committers rotate their LDAP passwords (committers visit id.apache.org to reset LDAP passwords once they've been forcibly reset),
  4. We are encouraging all service administrators to all non-LDAP service like jira to rotate those passwords as well.

Regarding the cert change for svn users- we'd also like to suggest that you remove your existing apache.org certs from your .subversion cache to prevent potential MITM attacks using the old cert.  Fortunately it is relatively painless to do this:

 % grep -l apache.org ~/.subversion/auth/svn.ssl.server/* | xargs rm

NOTE: our openoffice wildcard cert was never vulnerable to this issue as it was served from an openssl-1.0.0 host. 

Tuesday Mar 25, 2014

Scaling down the CMS to modest but intricate websites

The original focus of the CMS was to provide the tools necessary for handling http://www.apache.org/ and similar Anakia-based sites.  The scope quickly changed when Apache OpenOffice was accepted into the incubator... handling over 9GB of content well was quite an undertaking and will be soon discussed at Apachecon in Denver during Dave Fisher's talk.  From there the build system was extended to allow builds using multiple technologies and programming languages.

Since that time in late 2012 the CMS codebase has sat still, but recently we've upped the ante and decided to offer features aimed at parity with other site building technologies like jekyll, nanoc and middleman.  You can see some of the new additions to the Apache CMS in action at http://thrift.apache.org/. The Apache Thrift website was originally written to use nanoc before being ported to the newly improved Apache CMS. They kept the YAML headers for their markdown pages and converted from a custom preprocessing script used for inserting code snippets to using a fully-supported snippet-fetching feature in the Apache CMS. 

"The new improvements to the Apache CMS allowed us to quickly standardize the build process and guarantee repeatable results as well as integrate direct code snippets into the website from our source repository."
- Jake Farrell, Apache Thrift PMC Chair

Check out the Apache Thrift website cms sources for sample uses of the new features found in ASF::View and ASF::Value::Snippet.

Wednesday Feb 12, 2014

Improved integration between Apache and GitHub

After a few weeks of hard work and mind-boggling debugging, we are pleased to announce tighter and smarter integration between GitHub and the Apache Software Foundation's infrastructure.

These new features mean a much higher level of replication and retention of what goes on on GitHub, which in turns both help projects maintain control over what goes on within their project, as well as keeping a record of everything that's happening in the development of a project, whether it be on ASF hardware or off-site on GitHub.

To be more precise, these new features allows for the following:

  • Any Pull Request that gets opened, closed, reopened or commented on now gets recorded on the project's mailing list
  • If a project has a JIRA instance, any PRs or comments on PRs that include a JIRA ticket ID will trigger an update on that specific ticket
  • Replying to a GitHub comment on the dev@ mailing list will trigger a comment being placed on GitHub (yes, it works both ways!)
  • GitHub activity can now be relayed to IRC channels on the Freenode network.

As with most of our things, this is an opt-in feature. If you are in a project that would like to take advantage of these new features, please contact infrastructure, preferably by filing a JIRA ticket with the component set to Git, and specifying which of the new features you would like to see enabled for your project.

On behalf of the Infrastructure Team, I hope you will find these new features useful and be mindful in your use of them.

Wednesday Mar 06, 2013

paste.apache.org sees the light of day

Today, the Apache Infrastructure team launched http://paste.apache.org, a new ASF-driven site for posting snippets, scripts, logging output, configurations and much more and sharing them with the world.

 Why yet another paste bin, you ask?

Well, for starters, this site is different in that is it run by the ASF, for the ASF, in that we fully control what happens to your data when you post it, or perhaps more important, what does NOT happen to it. The site enforces a "from committers to everyone" policy, meaning only committers may post new data on the site, but everyone is invited to watch the result. While this is not a blanket guarantee that the data is accurate or true, it is nonetheless a guarantee that what you see is data posted by an Apache committer.

Secondly, committers have the option to post something as being "committers only", meaning only committers within the ASF can see the paste. This is much like the "private" pastes offered by many other sites, but with the added benefit that it prevents anyone snooping around from watching whatever you paste, unless they are actually a committer.

 Great, so how does it work?

It works like most other paste sites, in that you go to http://paste.apache.org,  paste your data, select which type of highlighting to use, and you get an URL with your paste. For text-only clients, raw data will be displayed, while regular browsers will enjoy a full web page with the ability to download or edit a paste. Currently we have support for httpd configurations, C/C++, Java, Lua, Erlang, XML/HTML, PHP, Shell scripts, Diff/Patch, Python and Perl syntax highlighting. If you want to have any other type of highlighting added, don't hesitate to ask!

Since this site enforces the "from committers to everyone" policy, you are required to use your LDAP credentials when making a paste. To allow for the use of the service within console applications (shells etc) that might not (or should not) provide authentication credentials (on public machines you'd want to avoid storing your committer credentials for instance!), we have equipped the site with a token generator, that both allows you to pipe any output you may have directly to the site as well as gives you some hints on how you may achieve this.

Imagine you have a directory listing that you'd only want your fellow committers to see. Publishing this, using the token system, is as easy as doing:
$> ls -la | privpaste               

And there you have it, the command returns a URL ready for sharing with your fellow committers. Had you wanted for eveyone to be able to see it, you could have used the pubpaste alias instead (click on "generate token" on the site to get more information about tokens and the useful aliases).

We hope you'll enjoy this new service, and use it wisely as well as often. Should you have any questions or suggestions, we'd be most happy to receive them through any infra channel you want to use.

Thursday Jul 26, 2012

New Infra Team Members

Since out last update over a year ago, the Infra Team has expanded by another NINE (9) members!

Congrats and our warmest thanks go to:

Niklas Gustavsson - (ngn)
Jeremy Thomerson - (jrthomerson)
Mark Struberg - (struberg)
Eric Evans - (eevans)
Brandon Williams - (brandonwilliams)
Mohammad Nour El-Din - (mnour)
David Nalley - (ke4qqq)
Yang Shih-Ching - (imacat)
Daniel Gruno - (humbedooh)

The rest of the Infra team look forward to continuing to work with you all.

There are now a total of 80 infrastructure members with another 36 in the infrastructure-interest group.

Monday Jul 09, 2012

ASF Comments System Live!

Daniel Gruno has recently developed a comments system for Apache projects to use.  The purpose of the system is to enable public commentary on project webpages and is already in production use in the httpd and trafficserver projects.  This new system nicely complements the ASF CMS system and trivially integrates with it- see http://comments.apache.org/help.html for details.

The comment system is now open- enjoy!  Please file a jira ticket with INFRA to get started today.

Sunday Jun 24, 2012

Apache CMS: New features for anonymous users

Two new features have recently been added to the CMS, courtesy of David Blevins.  These features are geared towards streamlining the user experience for anonymous users.  The first feature is "Quick Mail", which is the analog of "Quick Commit" but for anonymous users who cannot otherwise commit their changes directly.  Quick Mail, which is enabled by default, will take the immediate submission of an anonymous Edit session and post it directly to the project's dev list, saving several steps that might be hard for a new user to walk through.

The second feature is a natural result of that known as anonymous clones.  In the subsequent mailout from "Quick Mail", there will be an url for committers to use to effectively clone the working copy of the anonymous user who generated the patch.  This makes review and subsequent commit operations much more convenient than directly applying the emailed patch to a local working copy.  In fact it is possible for users to clone a non-anonymous user's working copy, so anyone experiencing chronic problems with their working copy on the CMS can get help from other committers by simply using the "Mail Diff" feature to contact either the dev list or another apache committer with details of their problem.

We have added these features in the hopes this will considerably lower the bar for anonymous users in particular to take advantage of the CMS.  Please let your community know about them!

Saturday Jun 09, 2012

The value of taint checks in CGI scripts

Consider the following snippet taken from a live CGI script running on the host that serves www.apache.org:


use strict;
use warnings;

print "Content-Type: text/html\n\n";
my $artifact = "/apache-tomee/1.0.1-SNAPSHOT/";
$artifact = $ENV{PATH_INFO} if $ENV{PATH_INFO};

$artifact = "/$artifact/";
$artifact =~ s,/+,/,g;
$artifact =~ s,[^a-zA-Z.[0-9]-],,g;
$artifact =~ s,\.\./,,g;

my $content = `wget -q -O - http://repository.apache.org/snapshots/org/apache/openejb$artifact`;

Looks pretty good right?  Any questionable characters are removed from $artifact before exposing it to the shell via backticks... hmm, well turns out that's not so easy to determine.

The first warning sign that was given to the author of this script was that he hadn't enabled taint checks- if he had this is how things probably would have looked:

#!/usr/bin/perl -T

use strict;
use warnings;

print "Content-Type: text/html\n\n";
my $artifact = "/apache-tomee/1.0.1-SNAPSHOT/";
$artifact = $ENV{PATH_INFO} if $ENV{PATH_INFO};

$artifact = "/$artifact/";
$artifact =~ s,/+,/,g;
$artifact =~ m,^([a-zA-Z.[0-9]-]*)$, or die "Detainting regexp failed!";
$artifact = $1;
$artifact =~ s,\.\./,,g;

my $content = `wget -q -O - http://repository.apache.org/snapshots/org/apache/openejb$artifact`;

Which doesn't look like much of a change, but the impact on the actual logic is massive: we've gone from a substitution that strips unwanted chars to a fully-anchored pattern that matches only a string full of wanted chars only, and dies on pattern match failure.  Sadly the developer in question did not heed this early advice.

As it turns out, there is a bug (well several) in the core pattern that renders the original substitution ineffective.  However the impact on the taint-checked version causes the detainting match to fail and renders the script harmless!  The practical difference is that instead of a script with a working remote shell exploit, we have script that serves no useful purpose.  To the Apache sysadmins this is a superior outcome, even though to the developer the original, essentially working script is preferable- worlds are colliding here, but guess who wins?

At the ASF the sysadmins almost invariably refuse to run perl or ruby CGI scripts without taint-checking enabled, and will always prefer CGI scripts be written in languages that support taint checks as they tend to enforce good practice in dealing with untrusted input.  This example, which is in fact one of the first times we've even considered allowing Apache devs to deploy non-download CGI scripts on the www.apache.org  server, serves as a useful reminder to Apache devs as to why using languages that support taint checks is an essential component of scripting on the web.

Tuesday May 29, 2012

apache.org incident report for 05/29/2012

Last week, internal audit activity discovered that the access logs of some committer-only Apache services contained passwords but had been available to every Apache committer.

The problem

The httpd logs of several ASF services are aggregated and archived on minotaur.apache.org.  Minotaur is also people.apache.org, the shell host for committers, and committers were encouraged to analyse the logs and produce aggregated data.

However, for two services, the archived logs included forensic logs, which are extra-verbose logs that include all HTTP request headers.  (The logs are never encrypted, even if the HTTP connection was wrapped by SSL encryption.)  Both of these services --- http://s.apache.org and http://svn.apache.org --- allow anyone to use them in a read-only manner anonymously, and allow further operations (such as creating shortlinks) to LDAP-authenticated committers.  Authentication is usually done by embedding the username and password, encoded in base64, in the "Authorization:" HTTP header, under SSL encryption.

Base64 is a reversible transform.  (It is an encoding, not a cipher.)

Consequently, any Apache committer could learn the passwords of any other committer by reading the log files and reversing the base64 encoding.

Shutting the barn door

The logs archive directory was made readable by the root user only.  Forensic logging was disabled, and past forensic logs deleted.  ZFS snapshots containing those logs were destroyed, too.

Finding the horse

We know that several committers had on one occasion or another copied the logs in order to analyse them, so we operated on the assumption that copies of the sensitive forensic logs were circulating on hardware we do not control.  We therefore opted to have all passwords changed, or reset.

Several Apache committers whose passwords grant very high access were advised privately to change their passwords.  The root@ team ensured the follow-through and, before announcing the vulnerability any further, changed the passwords of those whom had not done so themselves.  The root@ team also changed the passwords of all non-human (role) accounts on those services.

The vulnerability was then announced to all Apache committers with the same instructions: 'Your passwords may be compromised; change them "now"; we will explain the problem later.'.  This notice was authenticated via a PGP signature and via acknowledging it in a root-owned file on people.apache.org.

Finally, passwords that have not been changed after forensic logs had been disabled --- and, therefore, were presumed to be contained in compromised forensic logs --- were changed by the root@ team to random strings.


Were some committer to have compromised another Apache account using this vulnerability prior to these steps being taken, note that root access to all apache.org hosts is only available using one-time-passwords (otp) for certain privileged sudo users.  Such account holders have been instructed not to use the same password for otp as for LDAP, so this would not have resulted in an attacker gaining root privileges without our knowledge.  All of our commit activity is peer-reviewed and logged to various commit lists, and no reports of unusual commit activity have been received during the time frame in which this exposure was effective.  In fact no unusual activity has ever been reported regarding any of our LDAP-based services, so there is no reason for us to suspect malicious activity has occurred as a result of this vulnerability.

Preventing recurrence

No code changes were needed to the software that s.apache.org and svn.apache.org run; the software was behaving correctly according to its configuration, but the configuration itself --- and the in-house log archiving scripts --- were incorrect.

A member of the infrastructure team will be approaching the Apache HTTPD PMC with a documentation patch for mod_log_forensic.


There were no malicious parties involved here (to our knowledge); we just made a configuration error.  The nature of the error meant we had to assume all passwords were compromised, and that was costly to fix.

We hope our disclosure has been as open as possible and true to the ASF spirit.  Hopefully others can learn from our mistakes.  See our prior incident reports from the Apache Infrastructure Team.

Committers --- please address questions to root@apache.org only.

Queries from the press should be sent to press@apache.org.

Happy hacking!

Saturday Mar 10, 2012

Apache CMS and external build support

Recently we've been working with the maven team to facilitate migration of maven.apache.org to the Apache CMS, using maven as the core build system instead of the standard perl build scripts.  A mockup has been created at maventest.apache.org  to see how this will work.   Once the site is completed, there will be roughly 5GB of data to service, spanning dozens of maven components.  Each component will be self-contained and managed externally from the CMS site using a local maven svnpubsub plugin written mainly by Benson Margulies.  The CMS will glue all the components together into a single common site using the extpaths.txt file to configure the paths.

The doxia subproject requires special treatment as an independent CMS subproject which is also using maven as it's core build system.  Special logic was introduced into the CMS to properly redirect subproject links based on maven source tree layouts, and the system has worked seamlessly so far.

Other recent news includes the migration of the main incubator.apache.org site to the CMS.  There the CMS relies on Ant/Anakia to produce site builds instead of the standard perl build scripts, providing an easy migration path for folks accustomed to the old way of building the site.

Essentially we've made good on the promise that the CMS is simply CI for websites with an easy way of editing pages within your browser.  Support for forrest builds is planned but hasn't been fleshed out with any live examples to date.  That would round out the major java site-building technologies currently deployed by Apache projects- volunteers welcome!

Sunday Feb 26, 2012

Apache CMS: latest new feature is SPEED!

Over the past few months the Apache CMS has seen lots of new improvements, all under the general theme of making the system more performant.  Supporting very large sites like the Apache OpenOffice User Site with almost 10 GB of content has presented new challenges, met largely with the introduction of zfs clones for generating per-user server-side working copies, changing what was an O(N) rsync job to an O(1) operation.  We've also moved the update processing out-of-band to further cut down on the time it takes for the bookmarklet to produce a page, eliminating all O(N) algorithms from the process.

 More recent work focuses on the merge-based publication process, which for large changesets took a considerable amount of time to process.  That too has been recoded based on svnmucc and is now another O(1) operation- essentially a perfect copy of staging with a few adjustments for "external" paths.

Combine that with the activity around parallelizing the build system and you have a completely different performance profile compared to the way the system worked in 2011.  In short, if you haven't tried the CMS lately, and were a bit offput by the page rendering times or build speeds, have another look!

Next up: describing the work done around external build support, focusing first on maven based sites.

Sunday Dec 11, 2011

translate service now open!

A few projects have requested it, now it is here! Check out https://translate.apache.org and get your project added.

See also https://cwiki.apache.org/confluence/display/INFRA/translate+pootle+service+auth+levels for more information - you will see that general public non-logged in users can submit translate requests whilst any logged in user (i.e. - committers) can process those submissions.

Enjoy! - Any queries to the infra team please or file a INFRA Jira ticket.



Hot Blogs (today's hits)

Tag Cloud