Entries tagged [subversion]

Thursday December 02, 2010


Over the past 3 months, the Infrastructure Team has developed and deployed a custom CMS for Apache projects to use to manage their websites. There is a document available which explains the rationale, role, and future plans for the CMS. We have opened up the ACLs for the www.apache.org site for all committers to now be able to edit content on the site using the cms (while still restricting live publication to the Apache membership and the Infrastructure Team).

The basic workflow for committers is easy to describe: first install the javascript bookmarklet on your browser toolbar. Next visit a webpage on the www.apache.org website. When you've located a page you'd like to edit, click on the installed bookmarklet: you'll be taken to a working copy of the markdown source for the page in question. To edit the content click on the [Edit] link. A markdown editor will show you a preview of your changes while you work. When you have finished, submit your changes and [Commit] them.

Your commit will trigger buildbot to build a staging version of your changes. You can follow the build while it is ongoing, and once it has completed you can click on the [Staged] link to see the results. Members and Infrastructure Team members can continue on and publish those changes once they are satisfied with them. Other committers may need to send a note to the site-dev@ mailing list to request publication of their changes.

The publication links in the CMS are essentially merge + commit operations in subversion which are tied into the live site via svnpubsub. That means publishing in the CMS is virtually instantaneous.

The CMS is now open to all top-level and incubating projects. Interested projects should contact the infrastructure@ mailing list or simply file an INFRA ticket against the CMS component. Early adopters are encouraged to collaborate on the wiki page for working out usage and adoption issues.

Thursday September 23, 2010

1 million commits and still going strong.

Yesterday, the main ASF SVN code repository passed the 1 million commit mark. Shortly thereafter one of the ASF members enquired as to how he could best grab the svn log entries for all of these commits. As always there were a bunch of useful replies, but they were all set to take quite some time; mainly because if anyone just simply runs

svn log http://svn.apache.org/repos/asf -r1:1000000 

It will not only take several hours, it will also cause high levels of load on one of the two geo-balanced SVN servers. Also, requesting that many log entries will likely result in your IP address being banned.

So I decided to create the log set locally on one of the SVN servers. This is now available for download [http://s.apache.org/1m-svnlog] [md5]
This is a 50Mb tar/gz file. It will uncompress to about 240Mb. The log 'only' contains the log entries from 1 -> 1,000,000 - if you want the rest you can run:

svn log http://svn.apache.org/repos/asf -r1000001:HEAD

This will give you all the log entries from 1M+1 to current

Monday February 22, 2010

LDAP, groups and SVN - Coupled together

The infrastructure team have now completed the next stage of the planned LDAP migration.
We have migrated our old SVN authorisation file, and POSIX groups into LDAP data.  SVN access control is now managed using these groups.

This means to change access the Subversion repositories is now as simple as changing group membership. We use some custom perl scripts that build the equivalent authorisation file meaning that we dont need to use the <location> blocks nasty hack to do this.  It also means that all changes, including adding new groups and extending access control is made simple.

ASF PMC chairs, are now able to make changes to their POSIX, and SVN groups whilst logged into people.apache.org - using a selection of scripts:

  • /usr/local/bin/list_unix_groups.pl
  • /usr/local/bin/list_committees.pl
  • /usr/local/bin/modify_unix_groups.pl
  • /usr/local/bin/modify_committees.pl

All of these scripts have a '--help' option to show you how to use them.

What's next?  We are now working on adding a custom ASF LDAP schema, that will allow us to record ASF specific data such as ICLA files and date of membership etc.
We will also be looking at adding support for 3rd party applications such as Hudson, and building an identity management portal where people can manage their own account.

Thursday April 02, 2009

Improving our Subversion Services

This week the ASF Infrastructure Team deployed one of the first major changes to how svn.apache.org works since it was launched, 6 years ago.

We now distribute Subversion traffic to our servers based on the geographic region of a client.

We are using pgeodns, the same software that powers CPAN Search and the NTP Pool.  With pgeodns we can give out different DNS entries to clients, depending on where they are connecting from.  It isn't an exact science, but for most clients it is good enough to find the closer Subversion Server.

If you are connecting from Europe, your client will connect to Harmonia.  Harmonia is a Sun x4150 running FreeBSD 7.0, using ZFS raid2z over  6 disks, hosted in Amsterdam at SURFnet.

Users in North America are directed to Eris, our traditional Subversion Master Server.  Eris is a Dell 2950 also running FreeBSD 7.0, using ZFS raid2z over 4 disks, hosted in Corvallis, Oregon at OSUOSL.

Using svnsync as described in Norman's ApacheCon EU 2009 Talk, we replicate all commits to the master to the slave in real time.  If a commit is made to the slave, we proxy the commit to the master.

Read operations are handled on the nearest mirror, and are much faster for everything from the initial checking out to running an update due to the decreased latency.

While this change should improve the experience significantly, we have some other changes coming up soon for svn.apache.org:

  • Upgrade to Subversion 1.6: Representation Sharing, inode packing, and memcached support should help make our SVN servers even faster.
  • Upgrade both Eris and Harmonia to FreeBSD 7.2-STABLE: The ZFS filesystem is experimental in FreeBSD 7, and there are many stability and performance enhancements available in newer versions.
  • Adding more Geographic Mirrors: Once we are comfortable with the current setup, we would like to expand to another mirror location, hopefully in Australia or Asia. 

Thursday March 26, 2009

LDAP - It's getting closer

As of this afternoon whilst at ApacheCon Europe 2009, we have gotten our initial LDAP platform in place ready for testing.  This will allow us to move to a centralized AAA system. 



Hot Blogs (today's hits)

Tag Cloud