Apache Infrastructure Team
paste.apache.org sees the light of day
Today, the Apache Infrastructure team launched http://paste.apache.org, a new ASF-driven site for posting snippets, scripts, logging output, configurations and much more and sharing them with the world.
Why yet another paste bin, you ask?
Well, for starters, this site is different in that is it run by the ASF, for the ASF, in that we fully control what happens to your data when you post it, or perhaps more important, what does NOT happen to it. The site enforces a "from committers to everyone" policy, meaning only committers may post new data on the site, but everyone is invited to watch the result. While this is not a blanket guarantee that the data is accurate or true, it is nonetheless a guarantee that what you see is data posted by an Apache committer.
Secondly, committers have the option to post something as being "committers only", meaning only committers within the ASF can see the paste. This is much like the "private" pastes offered by many other sites, but with the added benefit that it prevents anyone snooping around from watching whatever you paste, unless they are actually a committer.
Great, so how does it work?
It works like most other paste sites, in that you go to http://paste.apache.org, paste your data, select which type of highlighting to use, and you get an URL with your paste. For text-only clients, raw data will be displayed, while regular browsers will enjoy a full web page with the ability to download or edit a paste. Currently we have support for httpd configurations, C/C++, Java, Lua, Erlang, XML/HTML, PHP, Shell scripts, Diff/Patch, Python and Perl syntax highlighting. If you want to have any other type of highlighting added, don't hesitate to ask!
Since this site enforces the "from committers to everyone" policy, you are required to use your LDAP credentials when making a paste. To allow for the use of the service within console applications (shells etc) that might not (or should not) provide authentication credentials (on public machines you'd want to avoid storing your committer credentials for instance!), we have equipped the site with a token generator, that both allows you to pipe any output you may have directly to the site as well as gives you some hints on how you may achieve this.
Imagine you have a directory listing that you'd only want your fellow committers to see. Publishing this, using the token system, is as easy as doing:
$> ls -la | privpaste
http://paste.apache.org/p/1234
And there you have it, the command returns a URL ready for sharing with your fellow committers. Had you wanted for eveyone to be able to see it, you could have used the pubpaste alias instead (click on "generate token" on the site to get more information about tokens and the useful aliases).
We hope you'll enjoy this new service, and use it wisely as well as often. Should you have any questions or suggestions, we'd be most happy to receive them through any infra channel you want to use.
Posted at 06:37PM Mar 06, 2013
by humbedooh in General |
|
New Infra Team Members
Since out last update over a year ago, the Infra Team has expanded by another NINE (9) members!
Congrats and our warmest thanks go to:
Niklas Gustavsson - (ngn)
Jeremy Thomerson - (jrthomerson)
Mark Struberg - (struberg)
Eric Evans - (eevans)
Brandon Williams - (brandonwilliams)
Mohammad Nour El-Din - (mnour)
David Nalley - (ke4qqq)
Yang Shih-Ching - (imacat)
Daniel Gruno - (humbedooh)
The rest of the Infra team look forward to continuing to work with you all.
There are now a total of 80 infrastructure members with another 36 in the infrastructure-interest group.
Posted at 02:35AM Jul 26, 2012
by administrator in Status |
Comments[1]
|
ASF Comments System Live!
Daniel Gruno has recently developed a comments system for Apache projects to use. The purpose of the system is to enable public commentary on project webpages and is already in production use in the httpd and trafficserver projects. This new system nicely complements the ASF CMS system and trivially integrates with it- see http://comments.apache.org/help.html for details.
The comment system is now open- enjoy! Please file a jira ticket with INFRA to get started today.
Posted at 04:49PM Jul 09, 2012
by joes in General |
|
Apache CMS: New features for anonymous users
Two new features have recently been added to the CMS, courtesy of David Blevins. These features are geared towards streamlining the user experience for anonymous users. The first feature is "Quick Mail", which is the analog of "Quick Commit" but for anonymous users who cannot otherwise commit their changes directly. Quick Mail, which is enabled by default, will take the immediate submission of an anonymous Edit session and post it directly to the project's dev list, saving several steps that might be hard for a new user to walk through.
The second feature is a natural result of that known as anonymous clones. In the subsequent mailout from "Quick Mail", there will be an url for committers to use to effectively clone the working copy of the anonymous user who generated the patch. This makes review and subsequent commit operations much more convenient than directly applying the emailed patch to a local working copy. In fact it is possible for users to clone a non-anonymous user's working copy, so anyone experiencing chronic problems with their working copy on the CMS can get help from other committers by simply using the "Mail Diff" feature to contact either the dev list or another apache committer with details of their problem.
We have added these features in the hopes this will considerably lower the bar for anonymous users in particular to take advantage of the CMS. Please let your community know about them!
Posted at 01:37PM Jun 24, 2012
by joes in General |
|
The value of taint checks in CGI scripts
Consider the following snippet taken from a live CGI script running on the host that serves www.apache.org:
#!/usr/bin/perl
use strict;
use warnings;
print "Content-Type: text/html\n\n";
my $artifact = "/apache-tomee/1.0.1-SNAPSHOT/";
$artifact = $ENV{PATH_INFO} if $ENV{PATH_INFO};
$artifact = "/$artifact/";
$artifact =~ s,/+,/,g;
$artifact =~ s,[^a-zA-Z.[0-9]-],,g;
$artifact =~ s,\.\./,,g;
my $content = `wget -q -O - http://repository.apache.org/snapshots/org/apache/openejb$artifact`;
...
Looks pretty good right? Any questionable characters are removed from $artifact before exposing it to the shell via backticks... hmm, well turns out that's not so easy to determine.
The first warning sign that was given to the author of this script was that he hadn't enabled taint checks- if he had this is how things probably would have looked:
#!/usr/bin/perl -T
use strict;
use warnings;
print "Content-Type: text/html\n\n";
my $artifact = "/apache-tomee/1.0.1-SNAPSHOT/";
$artifact = $ENV{PATH_INFO} if $ENV{PATH_INFO};
$artifact = "/$artifact/";
$artifact =~ s,/+,/,g;
$artifact =~ m,^([a-zA-Z.[0-9]-]*)$, or die "Detainting regexp failed!";
$artifact = $1;
$artifact =~ s,\.\./,,g;
my $content = `wget -q -O - http://repository.apache.org/snapshots/org/apache/openejb$artifact`;
... Which doesn't look like much of a change, but the impact on the actual logic is massive: we've gone from a substitution that strips unwanted chars to a fully-anchored pattern that matches only a string full of wanted chars only, and dies on pattern match failure. Sadly the developer in question did not heed this early advice.
As it turns out, there is a bug (well several) in the core pattern that renders the original substitution ineffective. However the impact on the taint-checked version causes the detainting match to fail and renders the script harmless! The practical difference is that instead of a script with a working remote shell exploit, we have script that serves no useful purpose. To the Apache sysadmins this is a superior outcome, even though to the developer the original, essentially working script is preferable- worlds are colliding here, but guess who wins?
At the ASF the sysadmins almost invariably refuse to run perl or ruby CGI scripts without taint-checking enabled, and will always prefer CGI scripts be written in languages that support taint checks as they tend to enforce good practice in dealing with untrusted input. This example, which is in fact one of the first times we've even considered allowing Apache devs to deploy non-download CGI scripts on the www.apache.org server, serves as a useful reminder to Apache devs as to why using languages that support taint checks is an essential component of scripting on the web.
Posted at 09:45PM Jun 09, 2012
by joes in General |
|
apache.org incident report for 05/29/2012
Last week, internal audit activity discovered that the access logs of some committer-only Apache services contained passwords but had been available to every Apache committer.
The problem
The httpd logs of several ASF services are aggregated and archived on minotaur.apache.org. Minotaur is also people.apache.org, the shell host for committers, and committers were encouraged to analyse the logs and produce aggregated data.
However, for two services, the archived logs included forensic logs, which are extra-verbose logs that include all HTTP request headers. (The logs are never encrypted, even if the HTTP connection was wrapped by SSL encryption.) Both of these services --- http://s.apache.org and http://svn.apache.org --- allow anyone to use them in a read-only manner anonymously, and allow further operations (such as creating shortlinks) to LDAP-authenticated committers. Authentication is usually done by embedding the username and password, encoded in base64, in the "Authorization:" HTTP header, under SSL encryption.
Base64 is a reversible transform. (It is an encoding, not a cipher.)
Consequently, any Apache committer could learn the passwords of any other committer by reading the log files and reversing the base64 encoding.
Shutting the barn door
The logs archive directory was made readable by the root user only. Forensic logging was disabled, and past forensic logs deleted. ZFS snapshots containing those logs were destroyed, too.
Finding the horse
We know that several committers had on one occasion or another copied the logs in order to analyse them, so we operated on the assumption that copies of the sensitive forensic logs were circulating on hardware we do not control. We therefore opted to have all passwords changed, or reset.
Several Apache committers whose passwords grant very high access were advised privately to change their passwords. The root@ team ensured the follow-through and, before announcing the vulnerability any further, changed the passwords of those whom had not done so themselves. The root@ team also changed the passwords of all non-human (role) accounts on those services.
The vulnerability was then announced to all Apache committers with the same instructions: 'Your passwords may be compromised; change them "now"; we will explain the problem later.'. This notice was authenticated via a PGP signature and via acknowledging it in a root-owned file on people.apache.org.
Finally, passwords that have not been changed after forensic logs had been disabled --- and, therefore, were presumed to be contained in compromised forensic logs --- were changed by the root@ team to random strings.
Implications
Were some committer to have compromised another Apache account using this vulnerability prior to these steps being taken, note that root access to all apache.org hosts is only available using one-time-passwords (otp) for certain privileged sudo users. Such account holders have been instructed not to use the same password for otp as for LDAP, so this would not have resulted in an attacker gaining root privileges without our knowledge. All of our commit activity is peer-reviewed and logged to various commit lists, and no reports of unusual commit activity have been received during the time frame in which this exposure was effective. In fact no unusual activity has ever been reported regarding any of our LDAP-based services, so there is no reason for us to suspect malicious activity has occurred as a result of this vulnerability.
Preventing recurrence
No code changes were needed to the software that s.apache.org and
svn.apache.org run; the software was behaving correctly according to
its configuration, but the configuration itself --- and the in-house
log archiving scripts --- were incorrect.
A member of the infrastructure team will be approaching the Apache HTTPD PMC with a documentation patch for mod_log_forensic.
Epilogue
There were no malicious parties involved here (to our knowledge); we just made a configuration error. The nature of the error meant we had to assume all passwords were compromised, and that was costly to fix.
We hope our disclosure has been as open as possible and true to the ASF spirit. Hopefully others can learn from our mistakes. See our prior incident reports from the Apache Infrastructure Team.
Committers --- please address questions to root@apache.org only.
Queries from the press should be sent to press@apache.org.
Happy hacking!
Posted at 04:59PM May 29, 2012
by joes in General |
|
Apache CMS and external build support
Recently we've been working with the maven team to facilitate migration of maven.apache.org to the Apache CMS, using maven as the core build system instead of the standard perl build scripts. A mockup has been created at maventest.apache.org to see how this will work. Once the site is completed, there will be roughly 5GB of data to service, spanning dozens of maven components. Each component will be self-contained and managed externally from the CMS site using a local maven svnpubsub plugin written mainly by Benson Margulies. The CMS will glue all the components together into a single common site using the extpaths.txt file to configure the paths.
The doxia subproject requires special treatment as an independent CMS subproject which is also using maven as it's core build system. Special logic was introduced into the CMS to properly redirect subproject links based on maven source tree layouts, and the system has worked seamlessly so far.
Other recent news includes the migration of the main incubator.apache.org site to the CMS. There the CMS relies on Ant/Anakia to produce site builds instead of the standard perl build scripts, providing an easy migration path for folks accustomed to the old way of building the site.
Essentially we've made good on the promise that the CMS is simply CI for websites with an easy way of editing pages within your browser. Support for forrest builds is planned but hasn't been fleshed out with any live examples to date. That would round out the major java site-building technologies currently deployed by Apache projects- volunteers welcome!
Posted at 05:28PM Mar 10, 2012
by joes in General |
Comments[1]
|
Apache CMS: latest new feature is SPEED!
Over the past few months the Apache CMS has seen lots of new improvements, all under the general theme of making the system more performant. Supporting very large sites like the Apache OpenOffice User Site with almost 10 GB of content has presented new challenges, met largely with the introduction of zfs clones for generating per-user server-side working copies, changing what was an O(N) rsync job to an O(1) operation. We've also moved the update processing out-of-band to further cut down on the time it takes for the bookmarklet to produce a page, eliminating all O(N) algorithms from the process.
More recent work focuses on the merge-based publication process, which for large changesets took a considerable amount of time to process. That too has been recoded based on svnmucc and is now another O(1) operation- essentially a perfect copy of staging with a few adjustments for "external" paths.
Combine that with the activity around parallelizing the build system and you have a completely different performance profile compared to the way the system worked in 2011. In short, if you haven't tried the CMS lately, and were a bit offput by the page rendering times or build speeds, have another look!
Next up: describing the work done around external build support, focusing first on maven based sites.
Posted at 02:23AM Feb 26, 2012
by joes in General |
Comments[1]
|
translate service now open!
A few projects have requested it, now it is here! Check out https://translate.apache.org and get your project added.
See also https://cwiki.apache.org/confluence/display/INFRA/translate+pootle+service+auth+levels for more information - you will see that general public non-logged in users can submit translate requests whilst any logged in user (i.e. - committers) can process those submissions.
Enjoy! - Any queries to the infra team please or file a INFRA Jira ticket.
Posted at 08:30PM Dec 11, 2011
by administrator in General |
|
PEAR package hosting available
Any projects in the position of being able to release via PEAR packages can now do so hosted officially on ASF servers.
http://pear.apache.org is now up and running and ready to serve!
Posted at 05:32AM Apr 15, 2011
by administrator in General |
|
Welcome new members of the infra team
Well, some are not exactly new faces, but since our last blog update of new infra members in 2009 , we have conned with promises of fame, fortune and beer the following new additions to the infra team:
- Chris Rhodes: (arreyder)
- Brian Fox: (brianf)
- Matt Benson: (mbenson)
- David Blevins: (dblevins)
- Rudiger Pluem: (rpluem)
- Noirin Plunkett: (noirin)
- Ulrich Stärk: (uli)
- Daniel Shahaf: (danielsh)
- Paul Davis: (davisp)
Infra work is not your normal volunteer work, and it is greatly appreciated when any of these folks get to help.
Posted at 10:09AM Mar 22, 2011
by administrator in General |
|
Changes to email service for all committers
In the near future the Infrastructure team will be implementing a change to the way we handle emails for all committers.
Historically we have allowed users to choose how to handle their apache.org email. However we will be making the following changes:
- Making LDAP authoritative for all mail forwarding addresses.
- Users will no longer be allowed to store their apache.org email locally on people.apache.org (minotaur)
- The Infra team will take the mail address currently held in either your .qmail or .forward file (.qmail is authoritative if they both exist) and inject this into LDAP
- We will no longer allow users to configure mail filtering, but you can configure your SpamAssassin threshold as per our recent blog post.
- We will make committers ~/.forward and ~/.qmail files read-only, there will still be at least one of these files, but it will be owned by the mail daemon user.
This means that all committers will be required to forward their apache.org email to an email address outside of the foundation.
We are doing this to simplify the email infrastructure, and to help reduce the current level of complexity of maintaining people.apache.org. Also, making LDAP authoritative means we can move some of the work straight out to the MXs, and thus avoid sending it through several mail servers. In the new architecture if someone emails you directly at your apache.org mail address it will only be handled by one apache.org MX.
Of course, we wont delete any email you currently have on people.apache.org. Should you want to edit your LDAP record you should use https://id.apache.org to do this.
Posted at 09:13PM Feb 24, 2011
by pctony in General |
Comments[2]
|
Controlling your SpamAssassin threshold
Committers,
The Infrastructure Team has just enabled a new feature to control your SpamAssassin Threshold for your apache.org account. The default score for user delivery has always remained at 10, but with this new feature you can lower that score to anything you want. Many people with older accounts will probably prefer a lower score, like 5, which is the default for all apache mailing lists.
To lower your score login to id.apache.org and change your 'SpamAssassin Threshold (asf-sascore)' attribute to your desired level. Don't forget to supply the form with your LDAP password.
Enjoy.
Posted at 03:37PM Jan 27, 2011
by joes in General |
Comments[3]
|
https://id.apache.org -- New Password Service
Folks,
The infrastructure team are pleased to announce the availability of id.apache.org the new password management tool for all ASF committers and members. This new service will allow users to:
- Reset forgotten LDAP passwords themselves, no need to contact the Infra team anymore.
- The ability to change their LDAP password.
- The ability to update your LDAP record, i.e. change forename, surname or mail attributes. [1].
Once logged in you will note that some fields are not editable, this is by design and are there merely to show you your LDAP entry. You are currently only allowed to edit your Surname, Given name (Forename), and Mail attributes. This list may be extended as we make more features available, and they will be announced as and when.
Users of this service should note that we have a few small bugs to iron out, and this will be done as soon as possible. For example if you attempt to modify your details and do no re-enter your password you will currently see a generic HTTP 500 error.
Thanks must go to Ian Boston (ieb), and Daniel Shahaf (danielsh) for making this work. Ian provided the initial code (his first ever attempt at Python too). Daniel then took it and implemented several changes and generally improved the backend.
[1] - It should be noted that updating your mail record in LDAP will not currently have any affect on where your apache.org email is forwarded on too. This is planned to take place later this year.
Posted at 04:36PM Jan 14, 2011
by pctony in Development |
|
LDAP and password policy
As of approximately 03:00 (UTC) today the infrastructure team have enabled a password policy for all LDAP accounts.
This policy has been implemented at the LDAP infrastructure level and will affect all users. It has been deployed using OpenLDAP's password policy schema, and overlay.
At the time of launch we will be enforcing the following policy.
- At the time of a given users 10th successive login failure the account will be locked.
- The account will then be automatically unlocked 24 hours later, or until a member of root@ unlocks it for you.
- If the user successfully completes a login before the tally reaches 10, the counter for failed logins is reset back to 0.
We are enabling this to try and prevent any brute force attempt at guessing passwords. It will also highlight potential issues with accounts.
As with all account related queries, you should be contacting root@ - We will be able to unlock your account for you, allowing you to gain access.
Posted at 06:38AM Dec 17, 2010
by pctony in Development |
|
