Apache Infrastructure Team

Friday November 21, 2014

MoinMoin Service - User Account Tidy Up

In recent months we have become increasingly aware of a slowing down of our MoinMoin wiki service.  We have attributed this, at least in part, due to the way MoinMoin stores some data about user accounts.  

Across all of our wiki instances (in the farm) we had a little over 1.08 million distinct user accounts.  Many of which have never been used (spam etc).  So we have decided to archive all users who have not accessed any of the wiki sites they were registered for in more than 128 days.  

This has resulted in us being able to archive a little over 800k users.  This leaves us with around 200k users across 77 wikis. This still feels very high, and in the coming weeks we will investigate further still in how we can better understand if those remaining accounts are making valid changes, or are they just link farm home pages.

If you think your account was affected by this, and you would like to have your account restored, then please contact the Infra team using this page http://www.apache.org/dev/infra-contact

ASF Infra Team

Monday October 06, 2014

Code signing service now available

The ASF Infrastructure team is pleased to announce the availability of a new code signing service for Java, Windows and Android applications. This service is available to any Apache project to use to sign their releases. Traditionally, Apache projects have shipped source code. The code tarballs are signed with a GPG signature to allow users and providers to verify the code's authenticity, but users have either compiled their own applications or some projects have provided convenience binaries. With projects like Apache OpenOffice, users expect to receive binaries that are ready to run. Today's desktop and mobile operating systems expect that binaries will be signed by the vendor -- which had left a gap to be filled for Apache projects.  

After a great deal of research, we have chosen Symantec's Secure App Service offering to provide code signing service. This allows us to granularly permit access; and each PMC will have their own certificate(s) for signing. The per-project nature of certificate issuance allows us to revoke a signature without disrupting other projects. 

This service will permit projects to sign artifacts either via a web GUI or a SOAP API. In addition a Java client and an ant task for signing have been written and a maven plugin is under development.

This service results in a 'pay for what you use' scenario, so PMCs are asked to use the service responsibly. To that end, projects will have access to a test environment to ensure that they have their process working correctly before consuming actual credits.

Thus far, we've had two projects who have helped testing this and working out process for which we are very grateful. Those projects, Commons and Tomcat, have successfully released signed artifacts recently. (Commons Daemon 1.0.15 and Tomcat 8.0.14)

Projects that wish to use this service should open an Infra JIRA ticket under the Codesigning component. Further information for projects using the service is also maintained by the infra team

Thursday October 02, 2014

GitHub pull request builds now available on builds.apache.org

The ASF Infrastructure team is happy to announce that you can now set up jobs on builds.apache.org to listen for pull requests to github.com/apache repositories, build that pull request’s changes, and then comment on the pull request with the build’s results. This is done using the Jenkins Enterprise GitHub pull request builder plugin, generously provided to the ASF by our friends at CloudBees. We've set up the necessary hooks on all github.com/apache repositories that are up as of Wednesday, Oct 1, 2014, and will be adding the hooks to all new repositories going forward.

Here’s what you need to do to set it up:

  • Create a new job, probably copied from an existing job.
  • Make sure you’re not doing any “mvn deploy” or equivalent in the new job - this job shouldn’t be deploying any artifacts to Nexus, etc.
  • Check the "Enable Git validated merge support” box - you can leave the first few fields set to their default, since we’re not actually pushing anything. This is just required to get the pull request builder to register correctly.
  • Set the “GitHub project” field to the HTTP URL for your repository - i.e.,"http://github.com/apache/incubator-brooklyn/"- make sure it ends with that trailing slash and doesn’t include .git, etc.
  • In the Git SCM section of the job configuration, set the repository URL to point to the GitHub git:// URL for your repository - i.e., git://github.com/apache/incubator-brooklyn.git.
  • You should be able to leave the “Branches to build” field as is - this won’t be relevant anyway.
  • Click the “Add” button in “Additional Behaviors” and choose "Strategy for choosing what to build”. Make sure the choosing strategy is set to “Build commits submitted for validated merge”.
  • Uncheck any existing build triggers - this shouldn’t be running on a schedule, polling, running when SNAPSHOT dependencies are built, etc.
  • Check the “Build pull requests to the repository” option in the build triggers.
  • Optionally change anything else in the job that you’d like to be different for a pull request build than for a normal build - i.e., any downstream build triggers should probably be removed,  you may want to change email recipients, etc.
  • Save, and you’re done!

Now when a pull request is opened or new changes are pushed to an existing pull request to your repository, this job will be triggered, and it will build the pull request. A link will be added to the pull request in the list of builds for the job, and when the build completes, Jenkins will comment on the pull request with the build result and a link to the build at builds.apache.org

In addition, you can also use the "Build when a change is pushed to GitHub" option in the build triggers for non-pull request jobs, instead of polling - Jenkins receives notifications from GitHub whenever one of our repositories has been pushed to. Jenkins can then determine which jobs use that repository and the branch that was pushed to, and trigger the appropriate build.

If you have any questions or problems, please email builds@apache.org or open a BUILDS JIRA at issues.apache.org

Thursday September 04, 2014

On-demand slaves from Rackspace added to builds.apache.org

A couple weeks ago, Apache's Infrastructure team added a new feature to our Jenkins server, builds.apache.org. In order to help deal with the at times overwhelming queues of builds waiting for an executor. While this has been improved dramatically by the increase in slaves generously provided by Yahoo! on physical hosts, we're always trying to look forward and be prepared for increased usage, etc in the future. 

To that end, we've set up slave images on Rackspace, generated using the fantastic tool Packer. Using the Apache jclouds plugin for Jenkins, Ubuntu slaves will be spun up dynamically on Rackspace using those images when there's a queue of pending builds that are able to run on the “ubuntu” label. Up to five of these slaves can be going at a time, and they're automatically removed from Jenkins and destroyed on Rackspace once they've been idle a set period of time. This burst capacity will help us prevent a long wait for builds to run on builds.apache.org.

We're able to do this thanks to Rackspace generously donating resources to the Apache Software Foundation. We're extremely grateful for this, and if any other public cloud providers are also interested in donating compute cycles to the Foundation, please contact the Infrastructure team.

One thing to note - the slave image we're using is still new and may have bugs in it. If you see your build suddenly failing for mysterious reasons, please take a look at the slave at ran on - if it's a slave named something like “jenkins-ubuntu-1404-4gb-abc”, please open a BUILDS JIRA at issues.apache.org with a link to the failing build and we'll investigate.

We've got more improvements for builds.apache.org planned for the future, and we're looking forward to sharing them with all of you - there'll be a talk at ApacheCon EU this November on the current status of Jenkins at the ASF, what we've done to stabilize and improve the developer experience on builds.apache.org this year, and what's planned for the future - hope to see you there!

Monday August 18, 2014

Infrastructure Team Adopting an On-Call Rotation

As the Apache Software Foundation (ASF) has grown, the infrastructure required to support its diverse set of projects has grown as well. To care for the infrastructure that the ASF depends on, the foundation has hired several contractors to supplement the dedicated cadre of volunteers who help maintain the ASFs hardware and services. To best utilize the time of our paid contractors and volunteers, the Infrastructure team will be adopting an on-call rotation to meet requests and resolve outages in a timely fashion. 

Why We're Establishing an On-Call Rotation

In groups, especially groups that are charged with overlapping duties, there's occasionally a sense of diffusion of responsibility. There tends to be a good number of tasks or incidents that routinely occur that need a clear owner. We've also tried to set expectations around our service levels relative to the importance of a service. In example, a new mailing list can be set up as convenient, but a failing mail service needs to be addressed immediately.

The technical side of this has been that we have historically alerted via email and/or SMS about any urgent issues that came up. Of course those alerts went to everyone on the team. If the alert occurs at an inconvenient time, either everyone responds, which is likely wasteful, or no one responds thinking someone else will.

At the Infrastructure team's face to face meeting in July we decided we'd adopt an on-call rotation for the contractors so that everyone wasn't responsible for everything all of the time. We then went looking for something to let us sanely (and without building it ourselves) deal with that.

We ended up choosing PagerDuty, which has a number of ways of receiving alerts. More importantly, it allows us to set a schedule, easily override it for holidays or illnesses, and do so programmatically. It also seamlessly integrates with HipChat, which Infrastructure is running a trial of and communicates with our mobile devices.

PagerDuty also supports a clear escalation path that begins alerting other people about issues if the person on-call fails to respond in a timely manner. Additionally, PagerDuty's mobile apps are built with Apache Cordova, which is an interesting circle. We've finished our trial and decided to adopt PagerDuty. PagerDuty was especially gracious and made our account gratis.

Adopting an on-call rotation will allow us to provide a better service and response time, while also clearly setting expectations around contractor availability so they can relax on their off weeks.

If you have questions or want to get involved, feel free to join us on the infrastructure mailing list infrastructure@apache.org or joining us in our Hipchat room.

Sunday June 15, 2014

Email from apache.org committer accounts bypasses moderation!

Good news!   We've finally laid the necessary groundwork to extend the bypassing of committer emails sent from their apache.org addresses, from commit lists to now all Apache mailing lists.  This feature was activated earlier today and represents a significant benefit for cross-collaboration between Apache mailing lists for committers, relieving moderators of needless burden.

Also we'd like to remind you of the SSL-enabled SMTP submission service we offer committers listening on people.apache.org port 465.  Gmail users in particular can enjoy a convenient way of sending email, to any recipient even outside apache.org, using their apache.org committer address.  For more on that please see our website's documentation.

To complement these features we'd also like to remind committers of the ability to request an "owner file" be added to their email forwarder by filing an appropriate INFRA jira ticket.  Owner files alleviate most of the problems associated with outside organizations, who may be running strict SPF policies, attempting to reach you at your apache.org address.  Without an owner file those messages will typically bounce back to those organizations instead of successfully reaching you at your target forwarding address.  For those familiar with SRS, this is a poor-man's version of that specification's feature set.  Please direct your detailed questions about owner files to the infrastructure-dev@apache.org mailing list.

NOTE: we've extended this bypass feature to include any committer email addresses listed in their personal LDAP record with Apache.

Tuesday June 03, 2014

DMARC filtering on lists that munge messages

Since Yahoo! switched their DMARC policy in mid-April, we've seen an increase in undeliverable messages sent from our mail server for Yahoo! accounts subscribed to our lists.   Roughly half of Apache's mailing lists do some form of message munging, whether it be Subject header prefixes, appended message trailers, or removed mime components.  Such actions are incompatible with Y!'s policy for its users, which has meant more bounces and more frustration trying to maintain inclusive discussions with Y! users.

Since Y!'s actions are likely just the beginning of a trend towards strong DMARC policies aimed at eliminating forged emails, we've taken the extraordinary step of munging Y! user's From headers to append a spec-compliant .INVALID marker on their address, and dropping the DKIM-Signature: header for such messages.  We are an ezmlm shop and maintain a heavily customized .ezmlmrc file, so carrying this action out was relatively straightforward with a 30-line perl header filter prepended to certain lines in the "editor" block of our .ezmlmrc file.  The filter does a dynamic lookup of DMARC "p=reject" policies to inform its actions, so we are prepared for future adopters beyond the early ones like Yahoo!, AOL, Facebook, LinkedIn, and Twitter.   Interested parties in our solution may visit this page for details and the Apache-licensed code.

Of course this filter only applies to half our lists- the remainder that do no munging are perfectly compatible with DMARC rejection policies without modification of our list software or configuration.  Apache projects that prefer to avoid munging may file a Jira ticket with infrastructure to ask that their lists be set to "ezmlm-make -+ -TXF" options.

Wednesday May 28, 2014

Mail outage post-mortem

During the afternoon of May 6th we began experiencing delays in mail delivery of 1-2 hours. Initial efforts at remediation seemed to clear this up but on the morning of May 7th the problem worsened and we proactively disabled mail service to deal with the failure. This outage affected all ASF mailing lists and mail forwarding. The service remained unavailable until May 10th, and it took almost 5 additional days to fully flush the backlog of messages.

You can find a timeline here that was kept during the incident: https://blogs.apache.org/infra/entry/mail_outage

This was a catastrophic failure for the Apache Software Foundation as email is core to virtually every operation and is our primary communication medium.  

What happened:

The mail service at the ASF is composed of three physical servers. Two of these are external facing mail exchangers that receive mail. The final server handles mailing list expansion, alias forwarding and mail delivery in general. That latter server had two volumes that experienced a disk outage each. This degraded performance substantially and led to the mail delays seen on May 6th and 7th. The service was proactively disabled on May 7th in an attempt to let the arrays rebuild without the significant disk I/O overhead caused by processing the large mail backlog. Ultimately multiple attempts to rebuild the underlying arrays failed and eventually other drives in the array where the data volume was stored failed rendering recovery a hopeless task on May 8th. We began working to restore backups from our offsite backup location to our primary US datacenter. When this began to take longer than expected, additional concurrent efforts began to restore service in one of our secondary datacenters as well as in a public cloud instance. Ultimately we ended up completing the restoration to our primary US datacenter first and were able to bring the service online. When the service resumed, we had an estimated 10 million message backlog in addition to our normal 1.7-2 million ongoing daily message flow. The amount of backlogged mail taxed the existing infrastructure and architecture of the mail service and took almost 5 days to completely clear.

What worked:

Our backups were sufficient to allow us to restore the service in good working order.
Early precautions taken when we discovered the problem combined with our backups resulted in no data loss from the incident.
Our mail exchangers continued to work during the outage and held incoming mail until the service was restored.

What didn't work:

Our monitoring was not sufficient to identify the problem or alert us to the symptoms.
No spare hard drives for this class of machine were on-hand in our primary datacenter. 
The restore time from our remote backups took an excessively long time. This was partially due to the large size of the restore data, and partially due to the transport method used for the data.
After the service was restored we had approximately a 10M message backlog that took days to clear.
The primary administrator of the service was on vacation, and the remaining infrastructure contractors were not intimately familiar with the service. 
Our documentation was insufficient to easily restore the service in a rapid manner by folks without intimate knowledge. 

Remediation plan:

Our immediate action items:

  • Update the documentation to be current/diagram mail flow.
  • Improve the monitoring of the mail service itself as well as the hardware.
  • Insure we have adequate spares on hand for the majority of our core services.
  • Place our mail server under configuration management to reduce our MTTR

Medium-to-Long term initiatives.
  • Crosstraining contractors in all critical services
  • Work on moving to a more fault-tolerant/redundant architecture
  • More fully deploy our config management and automated provisioning across our infrastructure so MTTR is reduced.


Friday April 11, 2014

heartbleed fallout for apache

Remain calm.

What we've learned about the heartbleed incident is that it is hard, in the sense of perhaps only viable to a well-funded blackhat operation, to steal a private certificate and key from a vulnerable service.  Nevertheless, the central role Apache projects play in the modern software development world require us to mitigate against that circumstance.  Given the length of time and exposure window for this bug's existence, we have to assume that some/many Apache passwords may have been compromised, and perhaps even our private wildcard cert and key, so we've taken a few steps as of today:

  1. We fixed the vulnerability in our openssl installations to prevent further damage,
  2. We've acquired a new wildcard cert for apache.org that we have rolled out prior to this blog entry,
  3. We will require that all committers rotate their LDAP passwords (committers visit id.apache.org to reset LDAP passwords once they've been forcibly reset),
  4. We are encouraging all service administrators to all non-LDAP service like jira to rotate those passwords as well.

Regarding the cert change for svn users- we'd also like to suggest that you remove your existing apache.org certs from your .subversion cache to prevent potential MITM attacks using the old cert.  Fortunately it is relatively painless to do this:

 % grep -l apache.org ~/.subversion/auth/svn.ssl.server/* | xargs rm

NOTE: our openoffice wildcard cert was never vulnerable to this issue as it was served from an openssl-1.0.0 host. 

Tuesday March 25, 2014

Scaling down the CMS to modest but intricate websites

The original focus of the CMS was to provide the tools necessary for handling http://www.apache.org/ and similar Anakia-based sites.  The scope quickly changed when Apache OpenOffice was accepted into the incubator... handling over 9GB of content well was quite an undertaking and will be soon discussed at Apachecon in Denver during Dave Fisher's talk.  From there the build system was extended to allow builds using multiple technologies and programming languages.

Since that time in late 2012 the CMS codebase has sat still, but recently we've upped the ante and decided to offer features aimed at parity with other site building technologies like jekyll, nanoc and middleman.  You can see some of the new additions to the Apache CMS in action at http://thrift.apache.org/. The Apache Thrift website was originally written to use nanoc before being ported to the newly improved Apache CMS. They kept the YAML headers for their markdown pages and converted from a custom preprocessing script used for inserting code snippets to using a fully-supported snippet-fetching feature in the Apache CMS. 

"The new improvements to the Apache CMS allowed us to quickly standardize the build process and guarantee repeatable results as well as integrate direct code snippets into the website from our source repository."
- Jake Farrell, Apache Thrift PMC Chair

Check out the Apache Thrift website cms sources for sample uses of the new features found in ASF::View and ASF::Value::Snippet.

Wednesday March 06, 2013

paste.apache.org sees the light of day

Today, the Apache Infrastructure team launched http://paste.apache.org, a new ASF-driven site for posting snippets, scripts, logging output, configurations and much more and sharing them with the world.

 Why yet another paste bin, you ask?

Well, for starters, this site is different in that is it run by the ASF, for the ASF, in that we fully control what happens to your data when you post it, or perhaps more important, what does NOT happen to it. The site enforces a "from committers to everyone" policy, meaning only committers may post new data on the site, but everyone is invited to watch the result. While this is not a blanket guarantee that the data is accurate or true, it is nonetheless a guarantee that what you see is data posted by an Apache committer.

Secondly, committers have the option to post something as being "committers only", meaning only committers within the ASF can see the paste. This is much like the "private" pastes offered by many other sites, but with the added benefit that it prevents anyone snooping around from watching whatever you paste, unless they are actually a committer.

 Great, so how does it work?

It works like most other paste sites, in that you go to http://paste.apache.org,  paste your data, select which type of highlighting to use, and you get an URL with your paste. For text-only clients, raw data will be displayed, while regular browsers will enjoy a full web page with the ability to download or edit a paste. Currently we have support for httpd configurations, C/C++, Java, Lua, Erlang, XML/HTML, PHP, Shell scripts, Diff/Patch, Python and Perl syntax highlighting. If you want to have any other type of highlighting added, don't hesitate to ask!

Since this site enforces the "from committers to everyone" policy, you are required to use your LDAP credentials when making a paste. To allow for the use of the service within console applications (shells etc) that might not (or should not) provide authentication credentials (on public machines you'd want to avoid storing your committer credentials for instance!), we have equipped the site with a token generator, that both allows you to pipe any output you may have directly to the site as well as gives you some hints on how you may achieve this.

Imagine you have a directory listing that you'd only want your fellow committers to see. Publishing this, using the token system, is as easy as doing:
$> ls -la | privpaste               

And there you have it, the command returns a URL ready for sharing with your fellow committers. Had you wanted for eveyone to be able to see it, you could have used the pubpaste alias instead (click on "generate token" on the site to get more information about tokens and the useful aliases).

We hope you'll enjoy this new service, and use it wisely as well as often. Should you have any questions or suggestions, we'd be most happy to receive them through any infra channel you want to use.

Monday July 09, 2012

ASF Comments System Live!

Daniel Gruno has recently developed a comments system for Apache projects to use.  The purpose of the system is to enable public commentary on project webpages and is already in production use in the httpd and trafficserver projects.  This new system nicely complements the ASF CMS system and trivially integrates with it- see http://comments.apache.org/help.html for details.

The comment system is now open- enjoy!  Please file a jira ticket with INFRA to get started today.

Sunday June 24, 2012

Apache CMS: New features for anonymous users

Two new features have recently been added to the CMS, courtesy of David Blevins.  These features are geared towards streamlining the user experience for anonymous users.  The first feature is "Quick Mail", which is the analog of "Quick Commit" but for anonymous users who cannot otherwise commit their changes directly.  Quick Mail, which is enabled by default, will take the immediate submission of an anonymous Edit session and post it directly to the project's dev list, saving several steps that might be hard for a new user to walk through.

The second feature is a natural result of that known as anonymous clones.  In the subsequent mailout from "Quick Mail", there will be an url for committers to use to effectively clone the working copy of the anonymous user who generated the patch.  This makes review and subsequent commit operations much more convenient than directly applying the emailed patch to a local working copy.  In fact it is possible for users to clone a non-anonymous user's working copy, so anyone experiencing chronic problems with their working copy on the CMS can get help from other committers by simply using the "Mail Diff" feature to contact either the dev list or another apache committer with details of their problem.

We have added these features in the hopes this will considerably lower the bar for anonymous users in particular to take advantage of the CMS.  Please let your community know about them!

Saturday June 09, 2012

The value of taint checks in CGI scripts

Consider the following snippet taken from a live CGI script running on the host that serves www.apache.org:


use strict;
use warnings;

print "Content-Type: text/html\n\n";
my $artifact = "/apache-tomee/1.0.1-SNAPSHOT/";
$artifact = $ENV{PATH_INFO} if $ENV{PATH_INFO};

$artifact = "/$artifact/";
$artifact =~ s,/+,/,g;
$artifact =~ s,[^a-zA-Z.[0-9]-],,g;
$artifact =~ s,\.\./,,g;

my $content = `wget -q -O - http://repository.apache.org/snapshots/org/apache/openejb$artifact`;

Looks pretty good right?  Any questionable characters are removed from $artifact before exposing it to the shell via backticks... hmm, well turns out that's not so easy to determine.

The first warning sign that was given to the author of this script was that he hadn't enabled taint checks- if he had this is how things probably would have looked:

#!/usr/bin/perl -T

use strict;
use warnings;

print "Content-Type: text/html\n\n";
my $artifact = "/apache-tomee/1.0.1-SNAPSHOT/";
$artifact = $ENV{PATH_INFO} if $ENV{PATH_INFO};

$artifact = "/$artifact/";
$artifact =~ s,/+,/,g;
$artifact =~ m,^([a-zA-Z.[0-9]-]*)$, or die "Detainting regexp failed!";
$artifact = $1;
$artifact =~ s,\.\./,,g;

my $content = `wget -q -O - http://repository.apache.org/snapshots/org/apache/openejb$artifact`;

Which doesn't look like much of a change, but the impact on the actual logic is massive: we've gone from a substitution that strips unwanted chars to a fully-anchored pattern that matches only a string full of wanted chars only, and dies on pattern match failure.  Sadly the developer in question did not heed this early advice.

As it turns out, there is a bug (well several) in the core pattern that renders the original substitution ineffective.  However the impact on the taint-checked version causes the detainting match to fail and renders the script harmless!  The practical difference is that instead of a script with a working remote shell exploit, we have script that serves no useful purpose.  To the Apache sysadmins this is a superior outcome, even though to the developer the original, essentially working script is preferable- worlds are colliding here, but guess who wins?

At the ASF the sysadmins almost invariably refuse to run perl or ruby CGI scripts without taint-checking enabled, and will always prefer CGI scripts be written in languages that support taint checks as they tend to enforce good practice in dealing with untrusted input.  This example, which is in fact one of the first times we've even considered allowing Apache devs to deploy non-download CGI scripts on the www.apache.org  server, serves as a useful reminder to Apache devs as to why using languages that support taint checks is an essential component of scripting on the web.

Tuesday May 29, 2012

apache.org incident report for 05/29/2012

Last week, internal audit activity discovered that the access logs of some committer-only Apache services contained passwords but had been available to every Apache committer.

The problem

The httpd logs of several ASF services are aggregated and archived on minotaur.apache.org.  Minotaur is also people.apache.org, the shell host for committers, and committers were encouraged to analyse the logs and produce aggregated data.

However, for two services, the archived logs included forensic logs, which are extra-verbose logs that include all HTTP request headers.  (The logs are never encrypted, even if the HTTP connection was wrapped by SSL encryption.)  Both of these services --- http://s.apache.org and http://svn.apache.org --- allow anyone to use them in a read-only manner anonymously, and allow further operations (such as creating shortlinks) to LDAP-authenticated committers.  Authentication is usually done by embedding the username and password, encoded in base64, in the "Authorization:" HTTP header, under SSL encryption.

Base64 is a reversible transform.  (It is an encoding, not a cipher.)

Consequently, any Apache committer could learn the passwords of any other committer by reading the log files and reversing the base64 encoding.

Shutting the barn door

The logs archive directory was made readable by the root user only.  Forensic logging was disabled, and past forensic logs deleted.  ZFS snapshots containing those logs were destroyed, too.

Finding the horse

We know that several committers had on one occasion or another copied the logs in order to analyse them, so we operated on the assumption that copies of the sensitive forensic logs were circulating on hardware we do not control.  We therefore opted to have all passwords changed, or reset.

Several Apache committers whose passwords grant very high access were advised privately to change their passwords.  The root@ team ensured the follow-through and, before announcing the vulnerability any further, changed the passwords of those whom had not done so themselves.  The root@ team also changed the passwords of all non-human (role) accounts on those services.

The vulnerability was then announced to all Apache committers with the same instructions: 'Your passwords may be compromised; change them "now"; we will explain the problem later.'.  This notice was authenticated via a PGP signature and via acknowledging it in a root-owned file on people.apache.org.

Finally, passwords that have not been changed after forensic logs had been disabled --- and, therefore, were presumed to be contained in compromised forensic logs --- were changed by the root@ team to random strings.


Were some committer to have compromised another Apache account using this vulnerability prior to these steps being taken, note that root access to all apache.org hosts is only available using one-time-passwords (otp) for certain privileged sudo users.  Such account holders have been instructed not to use the same password for otp as for LDAP, so this would not have resulted in an attacker gaining root privileges without our knowledge.  All of our commit activity is peer-reviewed and logged to various commit lists, and no reports of unusual commit activity have been received during the time frame in which this exposure was effective.  In fact no unusual activity has ever been reported regarding any of our LDAP-based services, so there is no reason for us to suspect malicious activity has occurred as a result of this vulnerability.

Preventing recurrence

No code changes were needed to the software that s.apache.org and svn.apache.org run; the software was behaving correctly according to its configuration, but the configuration itself --- and the in-house log archiving scripts --- were incorrect.

A member of the infrastructure team will be approaching the Apache HTTPD PMC with a documentation patch for mod_log_forensic.


There were no malicious parties involved here (to our knowledge); we just made a configuration error.  The nature of the error meant we had to assume all passwords were compromised, and that was costly to fix.

We hope our disclosure has been as open as possible and true to the ASF spirit.  Hopefully others can learn from our mistakes.  See our prior incident reports from the Apache Infrastructure Team.

Committers --- please address questions to root@apache.org only.

Queries from the press should be sent to press@apache.org.

Happy hacking!



Hot Blogs (today's hits)

Tag Cloud