The Apache Software Foundation Blog

Tuesday April 04, 2017

Success at Apache: JFDI --the unconditional love of contributors

by Daniel Gruno

In many respects, The Apache Software Foundation is like a dog.

Some people like dogs, some like cats. The ASF doesn't care, it still loves you. Some people are female, some are male, some don't share the narrow dichotomy of the masses. The ASF doesn't care, it still loves you. Some people have cool parents, and a car, and money, and are connected with the right friends, and some are "strange" loners that can count their friends on one hand (sometimes you don't even need a hand!). The ASF doesn't care, it loves you long as you feed it some code or documentation (or any of the other valuable contributions), you get love and respect in return.

But before I start telling my story about a dog named Apache, I should probably tell you a little about myself. I've been involved with Apache (both the foundation and the good ol' HTTP Server) for five years now. I am a Member of the foundation, and the Chair of the Apache STeVe project. The reason I got involved and why I'm still here, working for the ASF as an infrastructure architect, is that I had an itch (several in fact) to scratch and a yearning to show the world that I stuff!

I came to Apache from an academic background. I had been studying at various universities in Denmark, initially Statistics and Business Administration, and later on Human Resource Management, so my assumptions about how Open Source worked were that it probably worked like academia works: You have an idea, you present it, ask for feedback, start a collaborative process and have your peers review it, then implement said idea once they all say it's okay. Now, in some regards, this is accurate, but the method of execution is vastly different.

Academia is built around a mix of healthy and unhealthy inherent mistrust (in many respects akin to the notion of original sin in some beliefs).

The healthy part was in my experience --and in very(!) simple terms-- attributed to critical thoughts of Karl Popper and like-minded science philosophers who, argue against _proof_ as a means of advancing science, but rather seek out a _lack of evidence against_ as the proper way to corroborate a theory (and in doing so removing things like "is there a god?" or "are gnomes real?" from the sphere of academic theories, as you cannot disprove the existence of said figures, thus they are categorized as meta-theories and philosophical conundrums instead --the question of "do we really deserve dogs?" however is still valid!). Instead of proving that the sun does indeed rise every day, one would instead say "I have a theory that the sun won't rise tomorrow" and then debunk that. While proving that the sun rises tomorrow is a nigh impossible task (even with a 99.999% chance of it rising tomorrow, that means there's only a 16% chance of it rising in 500 years time), proving you were wrong tomorrow when the sun rises yet again is a simpler and more attainable goal that _has practical value_.

Now, while this works well in academia, the notion of having to _prove your code doesn’t work_ can be a bit much for someone just trying to fix a typo in a script. Still, things such as unit tests and fuzzing can be compared to the notion of _proving by failing to disprove_, in that we too in Open Source employ a notion of "if we can't break it, it must be working". This calculated and practical mistrust in our work is healthy.

The unhealthy part stems from our fellow human beings being human beings...and not dogs. Unlike Open Source communities like Apache, universities are, in my experience, just high school all over, with fancier charts and bigger books. It matters who you know, how well you can network, who foots the bills. While some schools do make you feel at home, in most cases you are left to fend for yourself and build up a network, both socially and scientific. If you did not have the proper social skill-sets, you were alone. There was no sense of inherent belonging, it was --in a sense-- the capitalist dream turned to a nightmare. Your future was yours to create, and yours alone. If you lacked the skills or mental capacity for socializing, you were simply left behind.

A Dog Named Apache
Looking back at my experiences with education institutes, imagine my surprise when I --the classic loner type with limited social skills-- asked if I could get some patches applied, and five minutes later, they were! The response was, to me, an overwhelming acceptance of the work I had done, and people saying "please do contribute more, we value this immensely". More than that, it was a sense of being invited to a community that didn't have any other reasons to invite me than "I have some patches". 

At Apache, it didn't matter who you had known for years, or what your social standing was, what you looked like or any other generic measurement we generally use in the outside world. If you have something to contribute, and it makes sense, you are welcomed with open arms. 

From the very beginning, I was gently nudged to contribute what I thought was interesting --not what THEY thought they needed, but what I had an interest in solving-- and I saw a profound interest in my skill-set and ideas. I saw people thinking what I did was cool, and I was cool, even though they had no idea who I was. It was like suddenly making friends in a platonic speed-dating séance where all that matters was being interested in doing SOMETHING --didn't matter what it was, as long as it made sense.

There is a general notion that Apache is a meritocracy: You contribute, and through your contributions earn merit that in turn affords you leverage and say in matters. I'll posit that not only is this true, but it's also a positive sum game with "original merit" applied. People are inherently trusted to have good intentions and are afforded an initial goodwill that you might not afford people in other circumstances. I didn't know these people, they didn't know me --and none of that mattered here.

Fast-tracking ideas
Within a week(!) of contributing patches to the HTTP Server project, I was voted in as an Apache Committer, much to my surprise. Even more surprising was the attitude at Apache, especially the Infrastructure Team: If you want to do something, go ahead and do it (with minimal supervision). Here's a server you can hack on, here's a place you can put your code, here's someone who will help review it! I had an idea of making a comment system for the HTTPd documentation, and (again) politely asked if it was at all possible that I could write it. At this point in time, I was expecting a bureaucratic NO, with some explanation of how they didn't know me (and thus, why would they entrust me with their hardware?). The answer was a very terse "JFDI, here's a FreeBSD jail for you". While I felt a bit scared of the general tone at the time, the notion that you could just do something without having to spend time gaining trust, requisitioning things, getting reviews prior to implementation and so on, was exhilarating to me: I could hack on something, I HAD A BOX TO EXPERIMENT ON, no strings attached! Again, the notion that you were inherently trusted was present. It didn't matter that I hadn't worked with infrastructure before, I had an idea that could solve a problem, and to them, that was all that mattered. Welcome to the team!

And so I wrote a comment system for our documentation. It got implemented in the documentation, other projects saw it and said "can we please use this too?". Not long after, I was neck deep in infrastructure business, having discovered that Apache was not only the HTTP Server...It was a plethora of interconnected projects all sharing the same notion of coming together to solve problems and help make the world a better place through advancing computer science. Everywhere I looked inside Apache, the notion seemed the same: If you can help us, you're one of us. Don't care who you are, where you come from, as long as you can contribute in some way, you're welcome in our community as a valued member.

So I joined another project, and another, and another. Today I am an official part of 10 Apache projects, on the Project Management Committee (PMC) in 8 of them. Do I know about these projects in extreme detail? Heck no! I can barely tell you what some of them really do, but that really doesn't matter at Apache. What matters is your willingness to contribute, no matter what your expertise level is, no matter what field of expertise is. If you have something to contribute, Apache will accept and love you. Just like a dog.

So put down your phone, stop facetweeting, and most importantly, stop thinking you can't possibly help or become a part of Apache. If you can write an email, you can help out. If you can fix typos, you can help out. If you can *sort* of code in a programming language, you can help out. If you can write newsletters, know how to fix a configuration error, help people on IRC, you can help out...and Apache will love you for it. And before you know it, you'll be a deeply integrated part of the Apache community.

If you are in doubt as to what project you would or could contribute to, Apache has an awesome Community Development project that helps mentor and engage people in projects, as well as teach about how the foundation and projects work. For more information, head on over to and check out the resources available to you. You can also check out and see if you know one of Apache’s projects, or perhaps discover a new project that fits what you are interested in --welcome aboard!

# # #

"Success at Apache" is a new monthly blog series that focuses on the processes behind why the ASF "just works":

1) Project Independence
2) All Carrot and No Stick
3) Asynchronous Decision Making
4) Rule of the Makers

Monday March 06, 2017

Success at Apache: Rule of the Makers.

By Nick Kew

I started working on a range of Web applications in the 1990s, the first of them internal to my (then) workplace where it provided an operator interface to the daily processing, archiving and distribution of satellite image data; the second a forerunner of what is now called social media, and my first use of an Apache server. The release of Apache HTTPD 2.0 drew me from server user to developer: in part because I needed to re-implement some existing functions, but more excitingly because I saw tremendous potential for the server itself to become a powerful applications platform. This led me to working on the core software and interacting with the Apache community alongside releasing my own modules and documentation. In 2003 I gave my first ApacheCon presentation, and sometime after that was invited into the Foundation first as a Committer, and became a Member in 2005. Since then my interests have encompassed not just the Web server and related projects, but also the Apache community and its dynamics. I’ve been involved in mentoring several projects through the Incubator. If you were to ask me today about the single goal I’d most like to accomplish, it’s a framework for Identity management that is not merely cryptographically strong, but sufficiently straightforward for the world to use, and robust against social engineering attacks such as phishing, while at the same time free of any centralised authority (such as government) whose motives might come under suspicion. An end to identity fraud, and to password management nightmares.

Much has been said and written about The Apache Way [1]. It typically focusses on the importance of community, and on the democratic and meritocratic elements of project governance. And on the role of an Apache project's formal governing body, the Project Management Committee (PMC), comprising contributors elected by the community on the basis of their track record of contributions and constructive engagement.

In practice, the role of the PMC is largely reactive. The big, interesting questions like "what direction does our project go from here" are discussed in public, where inputs from the wider community are welcomed. A lot of the nitty-gritty detail is determined by the needs of the community as measured by feedback received and seen, and by the needs of individual developers. The latter is often described as "scratch your own itch".

Setting aside the purely reactive, the detail of what actually happens is often determined as much by what a developer is prepared to work on than by any grand plan. If you do the work, then you control what happens. If you need something, you go ahead and make it happen. This has occasionally been described by a clumsy English-Greek hybrid word "do-ocracy". In the hope that it's not too late to nip that in the bud before it becomes as ubiquitous as the Greek/Latin hybrid of which CP Scott memorably told us no good would come [2], I shall instead call it Pratocracy. Rule of the Makers. It applies both within an Apache core team and in the broader community.

So how does a Pratocratic project work?  How does it avoid the chaos of a Tower of Babel?  How does it remain cohesive and focussed? And if an Apache project looks like a good basis for your company's needs, how best can you work with the Pratocracy towards your goals?

Taking the first question first, how does the Pratocracy work? Eric Raymond proposed the contrast of the Cathedral vs the Bazaar: the autocracy of the traditional project vs a free ecosystem. The Bazaar ecosystem is essentially Darwinian, in that the strongest projects prosper and evolve, while a Long Tail go nowhere very much and are forgotten as their authors move on to new things.

The Apache processes are all about ensuring our projects are fit to be among the winners in that ecosystem. An Apache project is a graduate of an incubation process [3] that requires it to build a broad development community, with sufficient diversity to survive the loss not merely of a key individual, but even of a key company team that might have been the originators of the project. The key principle is that if the community is healthy, the software will follow.

Thus the context of the Pratocracy is selective: it is the well-known Meritocracy element of Apache. Members of the core community are elected based on demonstrated commitment to the project. That means they understand the distinction between Apache work and company work, and are comfortable holding multiple allegiances and avoiding or resolving potential conflicts between them.

So, to the crucial question. Suppose I have identified an Apache project that doesn't fully meet my company's needs, but is the ideal starting point for our project. How do I best work with the Pratocracy to make it happen?

Let's consider some possible approaches:
  1. Ask via the development list and issues database.
  2. Approach developers individually to discuss the project.
  3. Just go ahead with the company's in-house developers.

Each of these approaches has merit in its own right. But in many cases, the ideal approach will encompass all the above. In an Apache community, the whole is greater than the sum of the parts. The closer you can work with the community, the more benefits you stand to derive, particularly if your project finds itself wanting to integrate more deeply with the Apache project than is provided for in stable public interfaces, when the alternative to working within the Community is a patched core and the risk of a maintenance nightmare. On the other hand, there are good and bad ways to engage with an Apache community: the right approach can make all the difference to the benefits you can derive from it.

First, it's always worth asking about your needs. Post a question about your requirements to the project's lists, or issues database. Engage in discussion. It may be that someone somewhere has already done the work, or that it's closely aligned with someone's work-in-progress. Even if you draw a blank, it's a start to engaging with the community.

Approaching individuals is most likely to be appropriate if you're looking to hire in expertise. General questions are likely to be redirected to public support fora, but some Apache developers work as consultants and may welcome a professional approach. They can help you figure out the best way to proceed, both technically and in terms of working with the Apache community. But bear in mind the potential for conflicts of interest. If you want your work to happen within the Apache project, your consultant will want to be satisfied that you and your project are indeed a good fit, and won't cause trouble such as unmaintainable software down the line. And of course, any NDA should make it clear you're not laying claim to the consultant's regular Apache work!

If you have the in-house developers for it, you might just task them to go ahead with the work. That's easiest when you can work with public, stable interfaces of the Apache project, harder and messy if you have to hack into the core. When you start that work, the Apache community becomes your team's best resource, so if your developers are any good (and not cowed into keeping their heads down and never talking to outsiders) they'll be interacting with the Apache community as they go along.

Now you're participating in the Pratocracy, albeit (usually) outside the core community. You have something to offer. If you share it with Apache, members of the Apache community will take a look, and may get involved. At best this can lead to a virtuous circle where your developers not merely benefit from the Apache team, but contribute to it, and earn themselves the merit to be invited to join. But this is by no means automatic: both developers and their managers need to be able to see outside corporate control structures. Crucially, individual developers have to be comfortable separating out their Apache hats from their professional allegiances, which means managers have to allow that to happen. But don't try to manage such a process from above: that's herding cats. Just facilitate!

That probably means that at some stage you'll have to sit down with the team and fix boundaries: what belongs in the Apache project vs what belongs in your company's own space. It helps immensely with developers' confidence if they're not forever looking over their shoulders and wrestling with ambiguity. This may be obvious if your project isn't destined to be shared openly with the world, but it's a discussion worth having even if everything is Apache-licensed. In the latter case, the boundaries will focus on what your company expects to sell and/or support commercially and wants to maintain full control on, vs what passes out of your direct control.

Even if your company merely clones or mirrors the Apache project and contributes everything, you'll want to separate out matters of your business model, and customer-facing vs Apache-facing matters.

Note that it's easy to get this kind of thing wrong. Something we sometimes see at Apache is developers who've been told by their managers to join an Apache community, but have no specific tasks and no motivation. That becomes painfully obvious, and leads nowhere useful.
The right approach is to give your developers real-world tasks and let them come to the Apache community - if their course takes them that way - in their own time. If close interaction with the Apache project is an important goal, you may want to approach and maybe hire individual developers having a strong track record with Apache or comparable Open Source environments.

To conclude, let's consider a real-world example from my own experience with HTTPD (the Web server still commonly called "Apache") and APR projects. The history of SQL database integration into the Web server spans many project teams, both within and outside Apache, both corporate and community-driven. Each developer or team just went ahead with something appropriate to its time, and (at least those you or I have heard of) achieved success in their respective contexts.

In the early days, there was a Web server with no SQL support. So any application that required it had to manage its own database connections. A range of third-party modules were developed for purposes such as authentication by SQL, and logging to SQL, both with popular databases such as MySQL and PostgreSQL and in commercially-supported products with commercial databases. Generalised frameworks for Web/SQL applications were provided by scripting modules, pioneered by mod_perl, giving developers a language's own database interfaces, together with a persistent database connection. 

In time, the server and its applications outgrew this ad-hoc approach, with multi-threaded processing and multiple applications in a server. Building on a range of ideas, a native SQL database connection framework was proposed and developed, offering some major efficiency gains to applications through dynamic connection pooling and sharing. It also promoted rationalisation, so in an area like authentication, a whole raft of different database-driven modules could be replaced by a single module for generic SQL authentication. Different SQL databases are handled by per-database drivers which serve all kinds of application in the server.
This work took place within the core Apache team, and propagated more widely as others such as the scripting languages incorporated it as an option for their users. It evolved from minimalist beginnings to provide more capabilities as required by developers "scratching an itch", and has benefited from contributions of database drivers from a range of different sources.

Having said that, it remains intentionally fairly small, and won’t meet the needs of every application. Some projects continue to use SQL their own way. The fact there is now a Right Way to access SQL doesn't mean that other ways are wrong! There are still third-party projects that, for their own reasons, use other ways to access SQL. The fact that someone has done the work and met a need gives them an entirely valid niche in a Pratocratic ecosystem.

The bottom line?  Many Apache users can just use our software off-the-shelf (perhaps after asking around for solutions to problems), others may need to add significantly to it to meet their needs. If you fall into the latter camp, just go ahead and do it: you can discuss and/or share your work at any point you choose! The basic management decision is the tradeoff between control of your work and community input. Interaction with the project's core people is down to how your developers work best. If leaving that to chance is not sufficient --perhaps because you would like to influence the project's core software or introduce new APIs-- that's the point when you'll need to consider your developers' track records in community-driven Open Source, and make sure you have that expertise to hand.


# # #

"Success at Apache" is a new monthly blog series that focuses on the processes behind why the ASF "just works":
1) Project Independence
2) All Carrot and No Stick
3) Asynchronous Decision Making

Monday February 06, 2017

Success at Apache: Asynchronous Decision Making

by Bertrand Delacretaz

Asynchronous decision making is a key enabler of our geographically and culturally distributed Open source teams. In this post I'll explain the ingredients that make it work at the ASF.

I became active in the ASF in 2001 via Gianugo Rabellino - he was the one who started the discussions with Apache Fop about me donating the jfor XLS-FO to RTF converter that I had developed earlier. It was already too late to uninvent RTF which is a terrible format, but I digress. I am currently a member of the Board of Directors of the ASF and have been doing a lot of thinking (and presentations) about what makes the ASF tick in terms of collaboration and Shared Neurons.

If synchronous decision-making meetings were required in ASF projects, even using remote channels like IRC or videoconference, we would move forward at a snail-like pace, as just finding a time where all stakeholders are available is almost impossible in an environment that has no managers and no central schedule.

Meetings are also very expensive when you are working on a maker's schedule, as described by Paul Graham [1]. Frequent meetings ruin the productivity of craftsmen, and there's lots of craftmanship in our industry, especially when you're building leading edge stuff.

So, what's needed to enable people to make collective decisions asynchronously, without requiring meetings?

The first thing you need is a central asynchronous communications channel. Which technology you use for that doesn't really matter, but it has to allow everybody to get the same information, and provide a usable way of having threaded discussions, where you can branch off on a topic while ignoring other topics being discussed on the same channel. This can be as simple as a whiteboard if people often visit the same place, or as elaborate as web-based forums, accessible from any mobile device so you can bother^H^H^H^H^H reach people everywhere. At the ASF we use plain mailing lists for that, very successfully when people use them with the right discipline (see the appendix below). Archiving this channel is very useful, to allow newcomers to get a feel for how things work as well as documenting the reasoning that led to each decision and avoid having to repeat things over and over.

The second required tool is a way to build consensus, where you avoid deadlocks and make sure decisions go forward. Unanimity in decisions is ideal of course, but the second best is consensus, defined as widespread agreement among people who have decision power. Requiring unanimity or allowing vetoes in decisions can block progress, so at the ASF vetoes only apply to a very limited set of decisions types, as defined by our voting rules [4]. In companies, decision power can be based on hierarchy to break deadlocks. That has to happen sometimes, but abusing it can cause employees to lose their autonomy and purpose, which kills your team in the long term.

To keep track of each decision, a case management system is ideal. You could work without that, depending on your team's size and the number of decisions that you take, but it's very convenient to be able to discuss the details of a given decision and keep associated information in a single place. You don't need complex software for that, at the ASF we use fairly simple issue trackers. Those are Web-based systems where each case is handled on a single page, with a history of comments and actions. Some non-urgent or very hard decisions can take a long time to reach closure, and it's very useful to keep their history in a single place, if only to avoid having to explain them again to new members of the team. In a low tech environment you could just use a single sheet of paper to briefly document each decision with the key points that led to it, and keep those in binders or physical files.

A nice side effect of using case management software is that each decision gets a simple unique identifier, like FOO-123 for the 123th ticket of the FOO project. This removes any ambiguity as to which issue one's discussing, by mentioning those identifiers in conversations.

So, in summary, the following should allow your group to make decisions asynchronously, without requiring meeting and with a written trace of everything that happens:

  • An archived asynchronous communications channel, where everybody can get the same information and threaded discussions can take place.
  • A way of building consensus, including fair rules for breaking deadlocks.
  • If possible, a case management system to keep track of each decision's details, in a much cleaner way than the often messy discussions that happen on the asynchronous channel. 

Semi-asynchronous decision making at the ASF

I've been a member of the ASF's Board of Directors for a few terms now and I'm still impressed by how efficient our monthly phone conferences are. The meeting regularly lasts only 60 to 90 minutes, during which we approve around 50 project reports, vote on a few resolutions and often address a few discussion items.

Besides a few simple things like good phone discipline and a side channel for less important comments (and jokes), the main reason this meeting is so efficient is that almost everything is decided in advance.

Board members are expected to read the project reports before the meeting, and a dead simple case management system (described below) helps discuss issues in advance, and find out which reports require a more extensive discussion.

Assuming the majority of board members have read the reports in advance, and flagged them as ok or requiring discussion, we don't need any housekeeping time during the meeting. Everybody shows up with a clear view of where difficult discussions might arise, so they have time to prepare for that, including asking others for clarification before the meeting so we can resolve any outstanding issues without delay.

The case management system that we use for this is extremely simple, but in terms of enabling asynchronous (or rather semi-asynchronous) decision making it fullfills its role. Our meeting agenda consists of a single text file in our source control system, with a simple structure that provides for a small discussion space for each report that we have to approve and each resolution that we need to vote on.

The agenda file structure looks roughly like this:

Call to order
Roll call
Officer reports
Project reports, headers and discussion space
Board Resolutions with discussion space
Appendix: Full Project reports and other supporting material

And a project report header and discussion space is as simple as this:

E. Apache Blazinator Project [Bob Blazer / Bertrand]
  See Attachment E
  [ Blazinator.
    approved: bd, mm, dd, db, jc, ldv
      bd:  Not sure why LEGAL-123 blocks their release
      ldv: They are waiting for the committer to supply
           an updated iCLA as the received one was 
      bd:  Ok, thanks, approving the report then.

This simple block of structured text builds a very simple "case management system" for the case of approving the Blazinator report.

The "approved" line indicates which board members have approved the report, on a single line so that simple text-based tools can validate and count the approvals.

The "comments" section allows stakeholders to comment on the report (which is found in an appendix later in the text file), and reply to each other's comments to hopefully reach closure before the meeting. If this happens, approving this report takes almost no time in the meeting, the chairman can just list the project names ("case identifiers" according to the above terminology) of such pre-approved reports, asking if anybody's opposed to approving them.

Combined with the ASF board's mailing list, this builds a very simple and very efficient system for semi-asynchronous decision making. Most decisions are taken before the meeting, and the participants can spend their time where it actually adds value as opposed to exchanging boring status information during the meeting.

Try it yourself!

Many ASF and other Open Source projects release world-changing software while having no or very few meetings, demonstrating that these techniques work.

If you're bogged down with inefficient or useless meetings, I suggest that you try applying these principles to a meaningful subset of your decision making activities. People will need to hone their skills to work efficiently in this way, but the rewards can be huge for distributed teams.

Appendix: Mailing lists at the ASF

At the ASF we use mailing lists as our central asynchronous communications channel, based on our if it didn't happen on the dev mailing list, it didn't happen rule [2]. Mailing lists might be seen as tools of the past when you compare them with the latest shiny tools, but they remain a ubiquitous way of communicating in loosely coupled remote groups, especially when used with a strong discipline of Precise Quoting [3] and paying attention to meaningful subject lines. Unfortunately I hear some "modern" email clients make a mess of that quoting - just stay away from them.


[1] - Paul Graham, Maker's Schedule, Manager's Schedule, July 2009

[2] - The Apache Project Maturity Model, ASF community development team, 2015.

[3] - Gianugo Rabellino "[OT/Rant] Quoting", message to the cocoon-dev mailing list, January 2002

[4] - ASF voting rules, created in 1999 probably, or even earlier among the Apache Group.

 # # #

"Success at Apache" is a new monthly blog series that focuses on the processes behind why The Apache Software Foundation (ASF) "just works". First article: Project Independence Second article: "All Carrot and No Stick"

Monday January 09, 2017

Success at Apache: "All Carrot and No Stick"

By Danny Angus

When the ASF launched their "Success at Apache" series I offered to share my own experiences. If you read on, remember that this is my personal experience and that others may disagree with me, but as you'll see, that's really part of the fun. 

For a bit background I’m currently the Project Management Committee (PMC) Chair of Apache Labs and in my day job I’m a "Divisional CTO" for a FTSE250 technology company. I first came to the ASF around 2000 when I was part of a startup - I was a CTO then too, it was the dot com boom, and it was just me and a couple of guys. We were considering a partnership with some researchers who wanted to commercialise their work, and were looking for a bit of software that we could use as the foundation for a product because a) we couldn’t afford to write it or buy it, and b) we didn’t have the knowledge anyway. What I found was Apache James , so I downloaded it, got it up and running, and did some prototyping, but we quickly realised that it needed work if we were going to be able to use it in production. I dug into it a little, subscribed to the mailing lists, asked questions and figured out what needed to be done to fix and extend what was already there, then started to modify it locally. Meantime I found myself answering other users’ questions on the user list, and one day noticed that I was actually answering more questions that I was asking. Shortly after that, that I was answering more questions than anyone else. Then I started submitting patches to the developer list (this was in the days of CVS: long before git!), which were reviewed and committed for me by the committers … but eventually they got bored with that and decided to extend commit privileges to me so I could do it all myself.

My experience illustrates an important characteristic of Apache projects: the fact that you can just turn up and get involved. Another very other important characteristic is that we are a meritocracy: demonstrating your capability is all you need to do in order to gain more responsibility; demonstrating your willingness and trustworthiness should be enough to get you the job. "Karma" is a word that is used to mean "access permission" in many Open Source projects, and we used to say that if you knew how to ask for karma properly, that was itself a sign that you could be trusted with it. Of course we were a much different organisation in those days, but the principles of a community built on merit and trust are still core to our identity. It's no coincidence that organisations cannot be part of our community: only individuals. Organisations are an important part of the world in which we exist, but we don't exist for their benefit, we only exist at all because as individuals we each bother to turn up and do stuff, from the guy who one time downloads and installs the Apache HTTP Web Server to Sam Ruby, our current (and can I just say excellent) President, everyone is contributing in their own way to the life of Apache and achieving benefits suited to their own, personal, motivations. So it was OK for me to focus on my own and my employer's priorities, which meant that I could learn from my new friends, develop the software we needed at work and become part of this amazing community all at the same time.

My experience of Apache is that it is what I would call "all carrot and no stick". I think that is the most healthy model of Open Source, as it is predicated on the fact that every participant will benefit from their participation without the need to contribute more than they are prepared to do. For me, focusing my contribution on the things I knew about was not only the most efficient use of my time, in terms of meeting our company's product goals, but it also allowed me to learn from others who had, and continue to have, way more knowledge and experience than I, and to benefit from their work. Mixing with these amazing people, many of whom are now real friends of mine, has taught me more than I would ever have learned any other way.

At this point in my involvement Apache went through a bit of what has diplomatically been described as "navel gazing", and settled on the idea that the organisational structure should be very very flat, and there should be no limit to our growth. As long as our standards were met by projects and people, we would welcome them both into our community. Those standards are partly about merit, partly about legal protection, one of the key roles Apache plays is to provide a degree of protection to projects and the people contributing to them, from the threat of bullies, trolls, and gorillas with expensive lawyers; and partly about ensuring that the behaviours and practices that define our identity and have contributed to the survival and the success of our organisation are continued by new generations of people in new projects using and creating technologies that we could hardly have dreamed about 16 years ago.

Before long the dust settled and I found myself voted to chair the Apache James Project , which was a whole new dimension of interesting. Chairing a project using only positive motivation teaches you a lot about people, including yourself, and I have a few observations about successful collaboration that I have found to be helpful both at work, where I strive to implement bottom-up decision making, and at the ASF where I want to make a positive contribution and see our communities flourish:

  • Free your mind.The collective sense of direction may not be what you expect, there have been times when I have been very sceptical about the reality of great sounding ideas, but I have also learned that it’s OK to go down the wrong road because most of the time it makes little difference in the end, usually you learn a lot regardless, and if people are really behind it you stand a much better chance of success than if the really good idea has all the fun of a death march. One phrase which is often used to summarise the spirit of Apache is “Community over code”, put the community first, and the code will follow.

  • Listen, and be supportive. There are a lot of different people involved in our projects with a lot of very different motivations. They are mostly all valid, and mostly all equally important if that even has an absolute scale. There are students studying our code, asking questions using our software and maybe fixing defects so that they can learn, there are employees of corporations who are being paid to protect their investment, to implement the product roadmap and maintain some predictable velocity, there are researchers who are pushing the boundaries of their chosen topic, there are people whose livelihood and success depends on a project, and those who are involved because it is a release from the pressure of things with names like "impact", "benefits", "deadlines" and "goals". Moderate or steer the discussion to ensure that all sides are heard, a meritocracy needs to listen to everyone not just the most vocal or assertive, and when I say listen that doesn’t mean formulating your own response while someone else is talking. Support people who you agree with, help to realise other people’s ideas, collaboration is only achieved by being truly committed to each other’s success, not just your own.

  • "A's hire A's B's hire C's". Find, support, and mentor the next generation, when your success depends upon the community it makes sense for you to put some effort into creating the best community you can.

  • Use Positive Language. When I was a kid being mean to my sister, adults used to say, "If you don't have anything nice to say, don't say anything at all". That's great advice if you’re involved in any collaborative venture, but doubly so when it is something like an Open Source project where you are usually communicating using written English, with people you don't know well, who might not have the same language skills as you do, who live in a different time zone and sometimes have very different cultural background than you. On top of all that you"re often debating the details of highly abstract technical concepts. The communication barrier itself can cause a kind of baseline of frustration so go easy on the negativity, one thing I like to do when I strongly disagree with someone is to write how I feel, then try to reword it using only positive language, it might sound like touchy-feely hippy nonsense to you, but you will be surprised how effective changing "I think you’re wrong and here’s why..." into "You have clearly thought a lot about this, I wonder if you have considered...". Alienating people is not the way to get your point across.

  • Learn to be a good loser. You don't own your projects, not here, and you're not the smartest person here either (OK so that’s not going to be 100% true, but there are 5,938 Committers today which makes it about 99.98%) recognising that and learning to embrace the collective view is hard for some people, but being able to step outside your subjective point of view and make a success of something you didn't believe in is a lesson in leadership that is definitely worth learning, because if not, your growth will be limited by the ideas that come from your own head, not accelerated by other people.

  • We are making it up as we go along . Yes, it sometimes seems from the outside like we have it all sorted and nailed down, and that we want to lawyer up and suck the life out of every fun thing (I mean we have a major software licence with our name on it, how grown up is that for goodness sakes?)  But the truth is that Apache, The Apache Software Foundation is, and probably always will be, a work in progress, hopefully will be at-least-good-enough to survive the next unexpected storm, and there have been several of those already, but the only way we ever find that out is when it hits us. Over a relatively long period we have figured out, adopted, borrowed, adapted, had donations of, and thunk out with nothing but our own brains, a whole load of ideas about effective Open Source collaboration, governance, legal shenanigans, marketing, community building, and so on. Things that work well, some that mostly work, and some that are sometimes rubbish, but better than nothing. We write these things down and propagate this good practice amongst projects because it is the bedrock on which our foundation rests, but that doesn’t mean that it can’t change, we correct, adapt and evolve our best practices all the time, this is how we adapt, this is how we have survived and remained relevant in a field that seems to change almost beyond recognition every four or five years. And, being a meritocracy, if you don’t agree with the way things are, if you think it is out of date or ineffective or pointless, don’t complain, stay and fix it. We have another saying which is that "you can scratch your own itch" - don’t be passive, if you care about it, do it.

    The important point about Apache is not that we have rules and committees but that we have these things because they have been shown to help us do the right thing, to help us to live by our principles and to provide a home for Open Source projects that will equip them to survive amongst the commercial sharks in the ocean of the software industry.

  • Finally: Define your own achievements. Whether you are doing it because you need some software, or because, like me, you just found it and it wasn't quite ready, whether you want to make friends, or to learn something new, whether it is because you are being paid to promote your employer's best interest, because you want to explore new ideas, or because you always wanted to write a book, Success at Apache is yours to define. Create your own measure of success and let us achieve it together.

# # #

"Success at Apache" is a new monthly blog series that focuses on the processes behind why The Apache Software Foundation (ASF) "just works". First article: Project Independence

Monday December 05, 2016

Success at Apache: Project Independence

By Mark Thomas

I've been involved in The Apache Software Foundation (ASF) since 2003. I was using Apache Tomcat at work and I hit a problem that needed a new feature to be implemented. There was already an enhancement request in Bugzilla so I submitted a patch. After some re-work by the project committers, the patch was applied and the feature available in the next release. I enjoy problem solving, so I started to take a look at the other open Tomcat bug reports and my involvement grew from there to include Apache Commons, the Infrastructure Team, the Security Team and, most recently, the Board of Directors to which I was elected in March 2016.

Apache Tomcat has always been at the heart of my involvement and is where I spend most of my time. Tomcat started with a donation to the ASF by Sun in 1999 and, some seven major versions later, the project continues to be very successful. A significant part that success is due to the involvement of a wide range of individuals from different companies. The reason those companies are happy co-operating on Tomcat is because of the importance the ASF places on project independence.

There are many aspects to project independence but, for me, the most important is that committers and Project Management Committee (PMC) members contribute to the project as individuals and do so with the intention of doing what is best for the community as a whole. Some committers contribute in their free time – I did for the first five years or so with Tomcat – and some are allowed /directed to spend time contributing to Apache projects by their employer. However, those committers contributing on their employer's time still need to act in the best interests of the community rather than the best interest of their employer.

To give a specific example, my employer has a product that is built around Apache Tomcat. The sales folks at my employer asked if I could add a feature to this product. The problem was that this feature required access to low-level Tomcat internals in order to implement it effectively. For this to be possible, I would have needed to make some ugly API changes to Tomcat to provide the integration points required. Rather than try and push those changes through, I persuaded my employer that it would better to donate the entire feature to the Apache Tomcat project.

This feature also demonstrates other important elements of a successful ASF project: the ability to make decisions in public and always aiming to achieve community consensus with those decisions. As the development of this new feature progressed, the design evolved as the community reviewed the commits and suggested improvements. This isn't always the quickest way of working but the quality of the end result – both technically but more importantly in terms of community health - more than makes up for that.

The perception of project independence is as important as projects actually being independent. It is a key factor in many projects choosing the ASF as their home so projects need to ensure that the perception agrees with reality.

Things can and do go wrong. With 350 projects it is pretty much a given that there will be a handful of ongoing issues at any given time. For example, there might be an attempt to push a project in a particular direction or to suggest that some external entity controls / leads / manages the project. Typically these are self-corrected by the PMC. Sometimes the PMC needs help to resolve the issue e.g. from V.P. Brand Management or possibly the ASF Board.

Being a board member is often viewed as more significant than it is. I have no more status in Apache Tomcat, Apache Commons or any other project as a board member than I did before my election to the board. I can still have bad ideas and my fellow community members still point it out when it happens. I don't get to always have my way just because I am board member. It is the board as a whole, rather than the individual board members, whose voice carries significant weight. It is fairly rare for any board member to speak on behalf of the board. To give that some context, I've probably done it no more than once a month since joining the board. It is sufficiently rare that board members always include an explicit "on behalf of the board" when speaking for the board rather than as an individual. Sometimes this point isn't appreciated and the views of an individual board member are incorrectly taken to be the views of the board.

The ASF board is also very different to a corporate board. The board manages the Foundation but it is the PMC that manages the project and sets the direction. The board has no role in the technical direction of a project. The board has responsibility for corporate governance, finance, legal etc., but its primary role is monitoring, mentoring and coaching our project communities to help keep them healthy. As part of this, the board reviews all projects on a regular basis. Newly graduated projects are reviewed monthly for typically 3 months before moving to quarterly reviews. The project V.P. (PMC Chair) is an important part of this. They are the eyes and ears of the board. While the board will look for warning signs as part of its regular review, the V.P. has much more in depth knowledge of the project and can flag specific issues early. Where issues are identified, the aim is to get the PMC to self-correct. The board will provide mentoring / coaching / guidance as necessary but it will be the PMC members who do the work to correct the issue.

As an example of the board working with a PMC, earlier this year the V.P. for a particular project became unavailable. The board became concerned because the regular reports were not being produced for the project. In this instance, no-else on the PMC had experience of being a project V.P so the board worked with the PMC to identify a new V.P. and to then mentor the new V.P. as they found their way in their new role.

For the last 17 years, the ASF has provided a home for a large and diverse set of open source projects. Key to this success has been the importance the ASF places on project independence as part of the Apache Way. By continuing to adhere to the principles of the Apache Way, I am confident that the ASF will continue to be successful for another 17 years and a long way beyond.



Hot Blogs (today's hits)

Tag Cloud