Apache Infrastructure Team

Friday Apr 11, 2014

heartbleed fallout for apache

Remain calm.

What we've learned about the heartbleed incident is that it is hard, in the sense of perhaps only viable to a well-funded blackhat operation, to steal a private certificate and key from a vulnerable service.  Nevertheless, the central role Apache projects play in the modern software development world require us to mitigate against that circumstance.  Given the length of time and exposure window for this bug's existence, we have to assume that some/many Apache passwords may have been compromised, and perhaps even our private wildcard cert and key, so we've taken a few steps as of today:

  1. We fixed the vulnerability in our openssl installations to prevent further damage,
  2. We've acquired a new wildcard cert for apache.org that we have rolled out prior to this blog entry,
  3. We will require that all committers rotate their LDAP passwords (committers visit id.apache.org to reset LDAP passwords once they've been forcibly reset),
  4. We are encouraging all service administrators to all non-LDAP service like jira to rotate those passwords as well.

Regarding the cert change for svn users- we'd also like to suggest that you remove your existing apache.org certs from your .subversion cache to prevent potential MITM attacks using the old cert.  Fortunately it is relatively painless to do this:

 % grep -l apache.org ~/.subversion/auth/svn.ssl.server/* | xargs rm

NOTE: our openoffice wildcard cert was never vulnerable to this issue as it was served from an openssl-1.0.0 host. 

Tuesday Mar 25, 2014

Scaling down the CMS to modest but intricate websites

The original focus of the CMS was to provide the tools necessary for handling http://www.apache.org/ and similar Anakia-based sites.  The scope quickly changed when Apache OpenOffice was accepted into the incubator... handling over 9GB of content well was quite an undertaking and will be soon discussed at Apachecon in Denver during Dave Fisher's talk.  From there the build system was extended to allow builds using multiple technologies and programming languages.

Since that time in late 2012 the CMS codebase has sat still, but recently we've upped the ante and decided to offer features aimed at parity with other site building technologies like jekyll, nanoc and middleman.  You can see some of the new additions to the Apache CMS in action at http://thrift.apache.org/. The Apache Thrift website was originally written to use nanoc before being ported to the newly improved Apache CMS. They kept the YAML headers for their markdown pages and converted from a custom preprocessing script used for inserting code snippets to using a fully-supported snippet-fetching feature in the Apache CMS. 

"The new improvements to the Apache CMS allowed us to quickly standardize the build process and guarantee repeatable results as well as integrate direct code snippets into the website from our source repository."
- Jake Farrell, Apache Thrift PMC Chair

Check out the Apache Thrift website cms sources for sample uses of the new features found in ASF::View and ASF::Value::Snippet.

Wednesday Feb 12, 2014

Improved integration between Apache and GitHub

After a few weeks of hard work and mind-boggling debugging, we are pleased to announce tighter and smarter integration between GitHub and the Apache Software Foundation's infrastructure.

These new features mean a much higher level of replication and retention of what goes on on GitHub, which in turns both help projects maintain control over what goes on within their project, as well as keeping a record of everything that's happening in the development of a project, whether it be on ASF hardware or off-site on GitHub.

To be more precise, these new features allows for the following:

  • Any Pull Request that gets opened, closed, reopened or commented on now gets recorded on the project's mailing list
  • If a project has a JIRA instance, any PRs or comments on PRs that include a JIRA ticket ID will trigger an update on that specific ticket
  • Replying to a GitHub comment on the dev@ mailing list will trigger a comment being placed on GitHub (yes, it works both ways!)
  • GitHub activity can now be relayed to IRC channels on the Freenode network.

As with most of our things, this is an opt-in feature. If you are in a project that would like to take advantage of these new features, please contact infrastructure, preferably by filing a JIRA ticket with the component set to Git, and specifying which of the new features you would like to see enabled for your project.

On behalf of the Infrastructure Team, I hope you will find these new features useful and be mindful in your use of them.

Wednesday Mar 06, 2013

paste.apache.org sees the light of day

Today, the Apache Infrastructure team launched http://paste.apache.org, a new ASF-driven site for posting snippets, scripts, logging output, configurations and much more and sharing them with the world.


 Why yet another paste bin, you ask?

Well, for starters, this site is different in that is it run by the ASF, for the ASF, in that we fully control what happens to your data when you post it, or perhaps more important, what does NOT happen to it. The site enforces a "from committers to everyone" policy, meaning only committers may post new data on the site, but everyone is invited to watch the result. While this is not a blanket guarantee that the data is accurate or true, it is nonetheless a guarantee that what you see is data posted by an Apache committer.

Secondly, committers have the option to post something as being "committers only", meaning only committers within the ASF can see the paste. This is much like the "private" pastes offered by many other sites, but with the added benefit that it prevents anyone snooping around from watching whatever you paste, unless they are actually a committer.

 Great, so how does it work?

It works like most other paste sites, in that you go to http://paste.apache.org,  paste your data, select which type of highlighting to use, and you get an URL with your paste. For text-only clients, raw data will be displayed, while regular browsers will enjoy a full web page with the ability to download or edit a paste. Currently we have support for httpd configurations, C/C++, Java, Lua, Erlang, XML/HTML, PHP, Shell scripts, Diff/Patch, Python and Perl syntax highlighting. If you want to have any other type of highlighting added, don't hesitate to ask!

Since this site enforces the "from committers to everyone" policy, you are required to use your LDAP credentials when making a paste. To allow for the use of the service within console applications (shells etc) that might not (or should not) provide authentication credentials (on public machines you'd want to avoid storing your committer credentials for instance!), we have equipped the site with a token generator, that both allows you to pipe any output you may have directly to the site as well as gives you some hints on how you may achieve this.

Imagine you have a directory listing that you'd only want your fellow committers to see. Publishing this, using the token system, is as easy as doing:
$> ls -la | privpaste               
http://paste.apache.org/p/1234

And there you have it, the command returns a URL ready for sharing with your fellow committers. Had you wanted for eveyone to be able to see it, you could have used the pubpaste alias instead (click on "generate token" on the site to get more information about tokens and the useful aliases).

We hope you'll enjoy this new service, and use it wisely as well as often. Should you have any questions or suggestions, we'd be most happy to receive them through any infra channel you want to use.



Thursday Jul 26, 2012

New Infra Team Members

Since out last update over a year ago, the Infra Team has expanded by another NINE (9) members!

Congrats and our warmest thanks go to:


Niklas Gustavsson - (ngn)
Jeremy Thomerson - (jrthomerson)
Mark Struberg - (struberg)
Eric Evans - (eevans)
Brandon Williams - (brandonwilliams)
Mohammad Nour El-Din - (mnour)
David Nalley - (ke4qqq)
Yang Shih-Ching - (imacat)
Daniel Gruno - (humbedooh)

The rest of the Infra team look forward to continuing to work with you all.

There are now a total of 80 infrastructure members with another 36 in the infrastructure-interest group.

Monday Jul 09, 2012

ASF Comments System Live!

Daniel Gruno has recently developed a comments system for Apache projects to use.  The purpose of the system is to enable public commentary on project webpages and is already in production use in the httpd and trafficserver projects.  This new system nicely complements the ASF CMS system and trivially integrates with it- see http://comments.apache.org/help.html for details.

The comment system is now open- enjoy!  Please file a jira ticket with INFRA to get started today.


Sunday Jun 24, 2012

Apache CMS: New features for anonymous users

Two new features have recently been added to the CMS, courtesy of David Blevins.  These features are geared towards streamlining the user experience for anonymous users.  The first feature is "Quick Mail", which is the analog of "Quick Commit" but for anonymous users who cannot otherwise commit their changes directly.  Quick Mail, which is enabled by default, will take the immediate submission of an anonymous Edit session and post it directly to the project's dev list, saving several steps that might be hard for a new user to walk through.

The second feature is a natural result of that known as anonymous clones.  In the subsequent mailout from "Quick Mail", there will be an url for committers to use to effectively clone the working copy of the anonymous user who generated the patch.  This makes review and subsequent commit operations much more convenient than directly applying the emailed patch to a local working copy.  In fact it is possible for users to clone a non-anonymous user's working copy, so anyone experiencing chronic problems with their working copy on the CMS can get help from other committers by simply using the "Mail Diff" feature to contact either the dev list or another apache committer with details of their problem.

We have added these features in the hopes this will considerably lower the bar for anonymous users in particular to take advantage of the CMS.  Please let your community know about them!



Saturday Jun 09, 2012

The value of taint checks in CGI scripts

Consider the following snippet taken from a live CGI script running on the host that serves www.apache.org:

#!/usr/bin/perl

use strict;
use warnings;

print "Content-Type: text/html\n\n";
my $artifact = "/apache-tomee/1.0.1-SNAPSHOT/";
$artifact = $ENV{PATH_INFO} if $ENV{PATH_INFO};

$artifact = "/$artifact/";
$artifact =~ s,/+,/,g;
$artifact =~ s,[^a-zA-Z.[0-9]-],,g;
$artifact =~ s,\.\./,,g;

my $content = `wget -q -O - http://repository.apache.org/snapshots/org/apache/openejb$artifact`;
... 


Looks pretty good right?  Any questionable characters are removed from $artifact before exposing it to the shell via backticks... hmm, well turns out that's not so easy to determine.

The first warning sign that was given to the author of this script was that he hadn't enabled taint checks- if he had this is how things probably would have looked:

#!/usr/bin/perl -T

use strict;
use warnings;

print "Content-Type: text/html\n\n";
my $artifact = "/apache-tomee/1.0.1-SNAPSHOT/";
$artifact = $ENV{PATH_INFO} if $ENV{PATH_INFO};

$artifact = "/$artifact/";
$artifact =~ s,/+,/,g;
$artifact =~ m,^([a-zA-Z.[0-9]-]*)$, or die "Detainting regexp failed!";
$artifact = $1;
$artifact =~ s,\.\./,,g;

my $content = `wget -q -O - http://repository.apache.org/snapshots/org/apache/openejb$artifact`;
... 

Which doesn't look like much of a change, but the impact on the actual logic is massive: we've gone from a substitution that strips unwanted chars to a fully-anchored pattern that matches only a string full of wanted chars only, and dies on pattern match failure.  Sadly the developer in question did not heed this early advice.

As it turns out, there is a bug (well several) in the core pattern that renders the original substitution ineffective.  However the impact on the taint-checked version causes the detainting match to fail and renders the script harmless!  The practical difference is that instead of a script with a working remote shell exploit, we have script that serves no useful purpose.  To the Apache sysadmins this is a superior outcome, even though to the developer the original, essentially working script is preferable- worlds are colliding here, but guess who wins?

At the ASF the sysadmins almost invariably refuse to run perl or ruby CGI scripts without taint-checking enabled, and will always prefer CGI scripts be written in languages that support taint checks as they tend to enforce good practice in dealing with untrusted input.  This example, which is in fact one of the first times we've even considered allowing Apache devs to deploy non-download CGI scripts on the www.apache.org  server, serves as a useful reminder to Apache devs as to why using languages that support taint checks is an essential component of scripting on the web.



Tuesday May 29, 2012

apache.org incident report for 05/29/2012

Last week, internal audit activity discovered that the access logs of some committer-only Apache services contained passwords but had been available to every Apache committer.

The problem

The httpd logs of several ASF services are aggregated and archived on minotaur.apache.org.  Minotaur is also people.apache.org, the shell host for committers, and committers were encouraged to analyse the logs and produce aggregated data.

However, for two services, the archived logs included forensic logs, which are extra-verbose logs that include all HTTP request headers.  (The logs are never encrypted, even if the HTTP connection was wrapped by SSL encryption.)  Both of these services --- http://s.apache.org and http://svn.apache.org --- allow anyone to use them in a read-only manner anonymously, and allow further operations (such as creating shortlinks) to LDAP-authenticated committers.  Authentication is usually done by embedding the username and password, encoded in base64, in the "Authorization:" HTTP header, under SSL encryption.

Base64 is a reversible transform.  (It is an encoding, not a cipher.)

Consequently, any Apache committer could learn the passwords of any other committer by reading the log files and reversing the base64 encoding.

Shutting the barn door

The logs archive directory was made readable by the root user only.  Forensic logging was disabled, and past forensic logs deleted.  ZFS snapshots containing those logs were destroyed, too.

Finding the horse

We know that several committers had on one occasion or another copied the logs in order to analyse them, so we operated on the assumption that copies of the sensitive forensic logs were circulating on hardware we do not control.  We therefore opted to have all passwords changed, or reset.

Several Apache committers whose passwords grant very high access were advised privately to change their passwords.  The root@ team ensured the follow-through and, before announcing the vulnerability any further, changed the passwords of those whom had not done so themselves.  The root@ team also changed the passwords of all non-human (role) accounts on those services.

The vulnerability was then announced to all Apache committers with the same instructions: 'Your passwords may be compromised; change them "now"; we will explain the problem later.'.  This notice was authenticated via a PGP signature and via acknowledging it in a root-owned file on people.apache.org.

Finally, passwords that have not been changed after forensic logs had been disabled --- and, therefore, were presumed to be contained in compromised forensic logs --- were changed by the root@ team to random strings.

Implications

Were some committer to have compromised another Apache account using this vulnerability prior to these steps being taken, note that root access to all apache.org hosts is only available using one-time-passwords (otp) for certain privileged sudo users.  Such account holders have been instructed not to use the same password for otp as for LDAP, so this would not have resulted in an attacker gaining root privileges without our knowledge.  All of our commit activity is peer-reviewed and logged to various commit lists, and no reports of unusual commit activity have been received during the time frame in which this exposure was effective.  In fact no unusual activity has ever been reported regarding any of our LDAP-based services, so there is no reason for us to suspect malicious activity has occurred as a result of this vulnerability.

Preventing recurrence

No code changes were needed to the software that s.apache.org and svn.apache.org run; the software was behaving correctly according to its configuration, but the configuration itself --- and the in-house log archiving scripts --- were incorrect.

A member of the infrastructure team will be approaching the Apache HTTPD PMC with a documentation patch for mod_log_forensic.

Epilogue

There were no malicious parties involved here (to our knowledge); we just made a configuration error.  The nature of the error meant we had to assume all passwords were compromised, and that was costly to fix.

We hope our disclosure has been as open as possible and true to the ASF spirit.  Hopefully others can learn from our mistakes.  See our prior incident reports from the Apache Infrastructure Team.

Committers --- please address questions to root@apache.org only.

Queries from the press should be sent to press@apache.org.

Happy hacking!

Saturday Mar 10, 2012

Apache CMS and external build support

Recently we've been working with the maven team to facilitate migration of maven.apache.org to the Apache CMS, using maven as the core build system instead of the standard perl build scripts.  A mockup has been created at maventest.apache.org  to see how this will work.   Once the site is completed, there will be roughly 5GB of data to service, spanning dozens of maven components.  Each component will be self-contained and managed externally from the CMS site using a local maven svnpubsub plugin written mainly by Benson Margulies.  The CMS will glue all the components together into a single common site using the extpaths.txt file to configure the paths.

The doxia subproject requires special treatment as an independent CMS subproject which is also using maven as it's core build system.  Special logic was introduced into the CMS to properly redirect subproject links based on maven source tree layouts, and the system has worked seamlessly so far.

Other recent news includes the migration of the main incubator.apache.org site to the CMS.  There the CMS relies on Ant/Anakia to produce site builds instead of the standard perl build scripts, providing an easy migration path for folks accustomed to the old way of building the site.

Essentially we've made good on the promise that the CMS is simply CI for websites with an easy way of editing pages within your browser.  Support for forrest builds is planned but hasn't been fleshed out with any live examples to date.  That would round out the major java site-building technologies currently deployed by Apache projects- volunteers welcome!

Sunday Feb 26, 2012

Apache CMS: latest new feature is SPEED!

Over the past few months the Apache CMS has seen lots of new improvements, all under the general theme of making the system more performant.  Supporting very large sites like the Apache OpenOffice User Site with almost 10 GB of content has presented new challenges, met largely with the introduction of zfs clones for generating per-user server-side working copies, changing what was an O(N) rsync job to an O(1) operation.  We've also moved the update processing out-of-band to further cut down on the time it takes for the bookmarklet to produce a page, eliminating all O(N) algorithms from the process.

 More recent work focuses on the merge-based publication process, which for large changesets took a considerable amount of time to process.  That too has been recoded based on svnmucc and is now another O(1) operation- essentially a perfect copy of staging with a few adjustments for "external" paths.

Combine that with the activity around parallelizing the build system and you have a completely different performance profile compared to the way the system worked in 2011.  In short, if you haven't tried the CMS lately, and were a bit offput by the page rendering times or build speeds, have another look!

Next up: describing the work done around external build support, focusing first on maven based sites.


Sunday Dec 11, 2011

translate service now open!

A few projects have requested it, now it is here! Check out https://translate.apache.org and get your project added.

See also https://cwiki.apache.org/confluence/display/INFRA/translate+pootle+service+auth+levels for more information - you will see that general public non-logged in users can submit translate requests whilst any logged in user (i.e. - committers) can process those submissions.

Enjoy! - Any queries to the infra team please or file a INFRA Jira ticket.

Friday Apr 15, 2011

PEAR package hosting available

Any projects in the position of being able to release via PEAR packages can now do so hosted officially on ASF servers.

http://pear.apache.org is now up and running and ready to serve!

Tuesday Mar 22, 2011

Welcome new members of the infra team

Well, some are not exactly new faces, but since our last blog update of new infra members in 2009 , we have conned with promises of fame, fortune and beer the following new additions to the infra team:

  • Chris Rhodes: (arreyder)
  • Brian Fox: (brianf)
  • Matt Benson: (mbenson)
  • David Blevins: (dblevins)
  • Rudiger Pluem: (rpluem)
  • Noirin Plunkett: (noirin)
  • Ulrich Stärk: (uli)
  • Daniel Shahaf: (danielsh)
  • Paul Davis: (davisp)

Infra work is not your normal volunteer work, and it is greatly appreciated when any of these folks get to help.

Thursday Feb 24, 2011

Changes to email service for all committers

In the near future the Infrastructure team will be implementing a change to the way we handle emails for all committers.

Historically we have allowed users to choose how to handle their apache.org email. However we will be making the following changes:

  1. Making LDAP authoritative for all mail forwarding addresses.
  2. Users will no longer be allowed to store their apache.org email locally on people.apache.org (minotaur)
  3. The Infra team will take the mail address currently held in either your .qmail or .forward file (.qmail is authoritative if they both exist) and inject this into LDAP
  4. We will no longer allow users to configure mail filtering, but you can configure your SpamAssassin threshold as per our recent blog post.
  5. We will make committers ~/.forward and ~/.qmail files read-only, there will still be at least one of these files, but it will be owned by the mail daemon user.

This means that all committers will be required to forward their apache.org email to an email address outside of the foundation.

We are doing this to simplify the email infrastructure, and to help reduce the current level of complexity of maintaining people.apache.org. Also, making LDAP authoritative means we can move some of the work straight out to the MXs, and thus avoid sending it through several mail servers. In the new architecture if someone emails you directly at your apache.org mail address it will only be handled by one apache.org MX.

Of course, we wont delete any email you currently have on people.apache.org. Should you want to edit your LDAP record you should use https://id.apache.org to do this.

Calendar

Search

Hot Blogs (today's hits)

Tag Cloud

Categories

Feeds

Links

Navigation