The Apache Software Foundation Blog
Success at Apache: Dip into the Apache Way
Like other recent contributors to this blog, I am not a developer by trade. My day job is as a Linux Systems Engineer and team manager, and, truth be told, my programming skills are not something I would rely on to make a living. Despite these facts, I've found something beyond acceptance in being a part of the Apache Guacamole project: mentoring.
= = =
"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works" https://blogs.apache.org/foundation/category/SuccessAtApache
# # #
Posted at 10:59AM May 07, 2018 by Sally in SuccessAtApache | |
Success at Apache: Am I there yet? A n00b's perspective
Let me start out by saying that I am not a developer. I do have a technical background, but I hadn't coded in Java for at least 10 years before I got involved in the Apache Drill project. One has to wonder how, as a non-developer, I ended up as a committer for the Drill project. In this blog post, I'd like to share with you how I came to be involved with the Drill project.
But first, why Drill?
I first heard about Drill at an industry conference several years ago. I was speaking with Dr. Ellen Friedman about some data issues we were having and she casually mentioned have I tried Drill? I had not heard of it at that point, so I did some research and it seemed as if Drill could solve a lot of problems that my clients were having. But then, I tried using it and kept getting stuck.
If you aren't familiar with Apache Drill, Drill is an SQL engine which allows you to query any kind of self-describing data. After experimenting with Drill for a while, I was impressed enough to thing that the tool had major potential in security. One of the biggest problems that Drill solves is the need to Extract, Transform, Load (ETL) data into an analytic tool before actually doing analysis of that data. This ETL process adds no value to anything really, and costs large enterprises literally millions of dollars as well as adding unnecessary delays between the time data is ingested and when the data is actually available for analysis. In security applications, this delay directly translates into risk. The longer it takes to make your data available, the more time it will take to potentially find malicious activity and hence, more risk. Therefore, if you're able to query the data without having to do any kind of ETL or ingestion, you are lowering your risk as well as potentially saving millions of dollars.
Unfortunately, when I started using Drill, I saw this potential, but I couldn't get it to work. My next step from here was to try to get assistance at my company. I pitched the ideas to my company leadership, but it proved very difficult to get the company to pull Java developers from revenue generating projects to work on this "pie-in-the-sky", unproven project. After spending several months on this, I got really frustrated and decided that I was going to try to do it myself, however, I really had no idea what I was doing. I hadn't coded in Java for at least 10 years at the time, and had zero experience with all the modern Java development tools such as Maven and Git. What I did have was persistence, so I started asking for help and decided that I was going to dive right in and start adding the functionality that I felt Drill needed to be useful in security applications. I started working on something that someone else started—the HTTPD format plugin for Drill. Most of the coding was done, but there was still enough there for me to get my hands dirty and start figuring things out.
What I learned
I still would not consider myself a developer, but after getting that particular item committed to the codebase, I learned a lot about how open source projects actually work as well as writing production quality code. Since then, I've tried to add at least one bit of new functionality to each Drill release. I would encourage anyone who is interested in contributing to an Open Source project at the Apache Software Foundation, to dive right in, and start. There are still a lot of ideas I have for Drill, and with time, I hope to have the time to see them through to implementation.
In conclusion, I'm fairly certain that my involvement with Drill and the Apache Software Foundation is really just beginning. I'm currently working on the O'Reilly book about Apache Drill with a fellow Drill committer. It is my hope that the book will spark additional interest in Apache Drill. Open Source software is at the heart of the ongoing data revolution which is dramatically expanding what is possible with data. I firmly believe that Apache Drill will have a role to play in this data revolution and I'm honored to have the opportunity to play a small role in developing Drill.
Charles Givre CISSP is a Lead Data Scientist at Deutsche Bank where he works in the Chief Information Security Office (CISO). Mr. Givre is an active data science instructor and regularly teaches classes about data science and security at various industry conferences, such as BlackHat. Mr. Givre is a committer for the Apache Drill project and together with Mr. Paul Rogers, is working on the forthcoming O’Reilly book about Apache Drill. He can be reached at cgivre(at)apache(dot)org.
= = =
"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works" https://blogs.apache.org/foundation/category/SuccessAtApache
# # #
Posted at 11:37PM Apr 10, 2018 by Sally in SuccessAtApache | |
Success at Apache: Open Innovation from a Non-native English Country
= = =
"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be 8) Meritocracy. https://s.apache.org/DiEo 9) Lowering Barriers to Open Innovation https://s.apache.org/dAlg 10) Scratch your own itch. https://s.apache.org/Apah 11) What a Long Strange (and Great) Trip It's Been https://s.apache.org/gVuN 12) A Newbie's Narrative https://s.apache.org/A72H 13) Contributing to Open Source even with a high-pressure job https://s.apache.org/lM9O 14) Open Innovation from a Non-native English Country https://s.apache.org/lh61
# # #
Posted at 03:00PM Mar 05, 2018 by Sally in SuccessAtApache | |
Success at Apache: Contributing to Open Source even with a high-pressure job
by Anthony Shaw
I believe in the mission of the ASF for many reasons, but the first is the reason why I got into open-source software- free and open access to knowledge.
For the past 4 years I've made around 1,000-2,000 contributions annually. These have consisted of bug fixes, submissions, and to around 50 projects.
The largest contributions I've made have been to Apache Libcloud, a multi-cloud abstraction library written in Python. Initially this was driven by a work commitment to contribute an integration with the cloud API we'd designed, but I soon realised the power of the library. Going back to my original goal of free and open access to knowledge, I'd seen an alarming trend in the computing world. Proprietary APIs were driving what is known in the industry as "stickiness" or to be frank, lock-in.
Cloud lock-in means that anyone without access to a reliable network, money or willing to sign up to these contracts is being pushed out of advances in technology. I know developers that are students, in remote areas such as rural Australia, Asia and Africa, or those who simply have little money.
You can easily sit until 3am banging your head against the wall trying to figure it out. This was my advice when I used to manage development teams. If you get stuck, take a break, ask for help and if that still doesn't work, move onto something else.
Look after your health, be smart with your time and contribute for a cause.
Anthony Shaw is the Group Director of Innovation and Talent Development at Dimension Data, an NTT company. Anthony is an open-source advocate, member of the Apache Software Foundation and Python Software Foundation and active contributor to over 20 open-source projects including Apache Libcloud and SaltStack. At Dimension Data, Anthony is driving digital transformation for Dimension Data’s global clients across 50 countries and 30,000 employees. Key initiatives are software skills, automation, DevOps and Cloud. Anthony is based in Sydney, Australia and blogs about skills, software and automation to 170,000 readers annually.
= = =
"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be 8) Meritocracy. https://s.apache.org/DiEo 9) Lowering Barriers to Open Innovation https://s.apache.org/dAlg 10) Scratch your own itch. https://s.apache.org/Apah 11) What a Long Strange (and Great) Trip It's Been https://s.apache.org/gVuN 12) A Newbie's Narrative https://s.apache.org/A72H 13) Contributing to Open Source even with a high-pressure job https://s.apache.org/lM9O
# # #
Posted at 11:02AM Feb 26, 2018 by Sally in SuccessAtApache | |
Success at Apache: A Newbie’s Narrative
On the third day of my job at Yahoo in 2015, I received a YouTube link on An Introduction to Apache Tez. I watched it carefully trying to keep up with all the questions I had and recognized a few names from my academic readings of Yarn ACM papers. I continued to ramp up on YARN and HDFS, the foundational Apache technologies Oath heavily contributes to even today. For the first few weeks I spent time picking out my favorite (necessary) mailing lists to subscribe to and getting started on setting up on a pseudo-distributed Hadoop cluster. I continued to find my footing with newbie contributions and being ever more careful with whitespaces in my patches. One thing was clear – Tez was the next big thing for us. By the time I could truly call myself a contributor in the Hadoop community nearly 80-90% of the Yahoo jobs were now running with Tez. But just like hiking up the Grand Canyon, the last 20% is where all the pain was. Being a part of the solution to this challenge was a happy prospect and thankfully contributing to Tez became a goal in my next quarter.
The next sprint planning meeting ended with me getting my first major Tez assignment – progress reporting. The progress reporting in Tez was non-existent – "Just needs an API fix," I thought. Like almost all bugs in this ecosystem, it was not easy. How do you define progress? How is it different for different kinds of outputs in a graph? The questions were many.
In 2018 as I move on to explore Hadoop 3.0 as our future release, I hope that if someone outside the Apache community is reading this, it will inspire and intrigue them to contribute to a project of their choice. As an astronomy aficionado, going from a newbie Apache contributor to a newbie Apache committer was very much like looking through my telescope － it has endless possibilities and challenges you to be your best.
Kuhu Shukla is a software engineer at Oath and did her Masters in Computer Science at North Carolina State University. She works on the Big Data Platforms team on Apache Tez, YARN and HDFS with a lot of talented Apache PMCs and Committers in Champaign, Illinois. A recent Apache Tez Committer herself she continues to contribute to YARN and HDFS and spoke at the 2017 Dataworks Hadoop Summit on "Tez Shuffle Handler : Shuffling At Scale With Apache Hadoop". Prior to that she worked on Juniper Networks' router and switch configuration APIs. She likes to participate in open source conferences and women in tech events. In her spare time she loves singing Indian classical and jazz, laughing, whale watching, hiking and peering through her Dobsonian telescope.
= = =
"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be 8) Meritocracy. https://s.apache.org/DiEo 9) Lowering Barriers to Open Innovation https://s.apache.org/dAlg 10) Scratch your own itch. https://s.apache.org/Apah 11) What a Long Strange (and Great) Trip It's Been https://s.apache.org/gVuN 12) A Newbie's Narrative https://s.apache.org/A72H
# # #
Posted at 11:00AM Feb 05, 2018 by Sally in SuccessAtApache | |
Success at Apache: What a Long Strange (and Great) Trip It's Been
By Jim Jagielski
It is normally during this time of year that people get awful retrospective. We look over the last 12 months and come to terms with what kind of year it has been. We congratulate ourselves on the good and (hopefully) learn from the bad. We basically assess the ending year and start planning, even a little bit, on the one to come.
In general, we reminisce.
I am thinking not about 2017, however, but instead of 1995 and the origins of The Apache Software Foundation. And what a long, strange, and great trip it's been. And how incredibly lucky I've been to be a part of it.
A common saying is that success is mostly about being there at the right place at the right time, and although I'm not sure about the "success" part, it certainly applies to me. At the time I was working at NASA and was starting off a side business as an ISP and Web Hoster, and using the old NCSA web-server. I had created a small reputation for myself as an "expert" on a flavor of UNIX called A/UX, which was Apple's UNIX offering at the time. In addition to being the editor of the FAQ for A/UX, I also ported a bunch of "free software" to that platform and that's how I got started with Apache, providing patches to support A/UX, which is what I used as my web hosting platform. It was really no different than what I did for other software projects at the time.
And then something wonderful happened. I got hooked.
I really, really enjoyed the people I was collaborating with. I wasn't an "outsider" providing patches, I was part of the inner circle. I was a full fledged member of the Apache Group. I started to really understand just how all this really could change the world, and how I could maybe be a small part of it.
As a result, Apache changed my life, literally. Instead of doing software development as a way of "getting my job done" (at NASA, I was a power system engineer, and so I would code modeling and simulation software for spacecraft solar arrays, batteries and orbital mechanics), I starting doing software development as my job, in addition to my hobby. Apache and Open Source became a huge part of my life, and my career changed to focus on Open Source almost primarily, a change that continues to this day.
During this time I've been fortunate enough to work with, and learn from, extremely talented people. Not only related to code, but legal matters, inter-personal skills, presentation skills, etc. I've had opportunities that I never imagined and met people I never would have had expected otherwise. I'm made great friends. I've been mentored by incredibly giving people and have mentored in return. And have seen my mentees become mentors themselves.
Over the years, I've seen Apache grow from a rag-tagged group of people working on a web server to one of the leading Open Source foundations in the world with more than 300 projects under our belt. I've been blessed to serve on the board of the ASF for every single year since we incorporated in 1999, seeing 2nd and now 3rd "generation" Apache Members take on the reins.
The Open Source movement, and especially Apache, have given more to me than I could ever pay back, and that is why I still volunteer and contribute. Of course, to be honest, I still get a kick out of it, and love what I am doing, and continue to enjoy the opportunities and, especially, the people that I get to work with.
But, you see, I'm nothing special. All this is also open and available to you. You too can change the world, and have your world changed in return. We all have talents that can be shared, talents that can be recognized and rewarded. Apache is a family, always looking for new family members.
So take that first step. Find a project and community you want to a part of. Jump in. Have fun. Grow. Learn. Teach. Live.
But just be prepared to get hooked, and have your life change.
Jim Jagielski is a well known and acknowledged expert and visionary in Open Source, an accomplished coder, and frequent engaging presenter on all things Open, Web and Cloud related. As a developer, he’s made substantial code contributions to just about every core technology behind the Internet and Web and in 2012 was awarded the O’Reilly Open Source Award and in 2015 received the Innovation Luminary Award from the EU. He is likely best known as one of the developers and co-founders of the Apache Software Foundation, where he has previously served as both Chairman and President and where he’s been on the Board Of Directors since day one. Currently he is Vice-Chairman. He's served as President of the Outercurve Foundation and was also a director of the Open Source Initiative (OSI). Up until recently, he worked at Capital One as a Sr. Director in the Tech Fellows program. He credits his wife Eileen in keeping him sane.
= = =
"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be 8) Meritocracy. https://s.apache.org/DiEo 9) Lowering Barriers to Open Innovation https://s.apache.org/dAlg 10) Scratch your own itch. https://s.apache.org/Apah 11) What a Long Strange (and Great) Trip It's Been https://s.apache.org/gVuN
# # #
Posted at 12:10PM Dec 12, 2017 by Sally in SuccessAtApache | |
Success at Apache: Scratch Your Own Itch.
By Ignasi Barrera
Recently I was at an industry conference and was happy to see many people stopping by the Apache booth. I was pleased that they were familiar with the Apache brand, yet puzzled to learn that so many were unfamiliar with The Apache Software Foundation (ASF).
It's important to recognize not just Apache's diverse projects and communities, but also the entity behind their success.
Gone are the days when software, and technology in general, was developed privately for the benefit of the few. As technology evolves, the challenges we face become more complex, and the only way to effectively move forward to create the technology of the future is to collaborate and work together. Open Source is a perfect framework for that, and organizations like the ASF carry out a decisive role in protecting its spirit and principles.
The ASF's mission is to provide software for the public good. We take it one step further, by giving all our Open Source software away for free. According to this mission, the foundation was established back in 1999 as a US 501(c)(3) non-profit charitable organization, and constitutes an independent legal entity to which companies and individuals can donate resources and be assured that those resources will be used for the public benefit. Its all-volunteer nature, along with the meritocracy model followed by its communities, are the pillars of the neutral, trusted space where Apache software is developed.
We strongly believe that good software is built by strong communities. Successful Open Source projects are the result of the work and collaboration in their communities and the people behind them. It is all about the people. Experience has shown us that helping people work together as peers is key in producing software in a sustainable way, and we have collected the lessons learned all these years in what we call "The Apache Way".
This Apache Way is a set of core behaviors all Apache projects follow that are designed to ensure projects are independent and diverse, and that anyone can participate no matter what gender, culture, time zone, employer, or even expertise they have. One can start collaborating with a project by contributing patches or implementing new features, but merit is not only measured by code contributions. Helping users, improving documentation, promoting the project, and other non-coding activities are very valuable and recognized as such, and the recognition of this merit and implication is expressed by granting more privileges in the project: from commit access, to invitations to join the Project Management Committee, to invitations to join the ASF Membership. One of the great differentiators between the ASF and other open source foundations is that the ASF does not dictate the technical direction of its projects: each Apache project is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides their respective project's day-to-day operations, including community development and product releases. Meritocracy drives the growth of the communities, and ensures anyone can contribute to projects that are ruled by the people who is involved and really cares about them.
Learning to work this way is not always easy, though. Projects come to the Foundation from very different backgrounds and whilst some of them already have communities that are used to collaborate in open ways, others find it challenging to embrace these core behaviors. The Apache Incubator is the main entry point for codebases and their communities wishing to officially become part of the Foundation, and is where they learn how to put all these principles in practice. Some will find this way of working a good way to rule a project and will graduate as an Apache top-level project, some may find that the Foundation is not the best option for them and choose to leave. Both options are good outcomes, as projects will have invested time in thinking about their community model and how they want governance to be, and this always benefits the Open Source world.
This Open Source model not only exists to create sustainable Open Source projects, but also to meet the expectations of the rest of the world. Software developed at Apache comes with a set of guarantees granted by the popular and business-friendly Apache License, but also with others that are the product of this open governance model, such as project independence or a well-defined project lifecycle. The ASF not only defines how projects operate while active, but also what happens when a project reaches its end-of-life, which is also important for adoption but often not considered by Open Source projects.
These guarantees, along with the reputation earned by many years of producing high-quality open source software, make the +300 freely available Apache projects, from Abdera to HTTP Server to Hadoop to Zookeeper, a trusted choice for individuals and companies looking for Open Source solutions.
The saying "Scratch Your Own Itch" is popular in the tech space, and is an integral principle at the ASF. Apache Committers have a responsibility to the community to help create a product that will outlive the interest of any particular volunteer, as well as for helping to grow and maintain the health of the Apache community.
As an ASF Member, I'm helping with project outreach and mentoring new individuals that make up the greater Apache community.
The Apache Software Foundation provides a safe place for Open Source development, and will keep evolving as technology evolves, welcoming all kinds of projects and communities, and helping people embrace Open Source. Let's see what the future holds for the Open Source world and how we can contribute to making it a better place. Scratch your own itch.
Ignasi Barrera is a long-term Open Source contributor and became involved with the ASF in 2013, when jclouds was first submitted to the Apache Incubator. He is a member of the Apache jclouds Project Management Committee and still actively contributes to the project. Ignasi became an ASF Member in 2015, and helps with community development activities and the promotion of Open Source.
= = =
"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be 8) Meritocracy. https://s.apache.org/DiEo 9) Lowering Barriers to Open Innovation https://s.apache.org/dAlg 10) Scratch your own itch. https://s.apache.org/Apah
# # #
Posted at 10:06PM Oct 25, 2017 by Sally in SuccessAtApache | |
Success at Apache: All My Roads Led to Apache
by Pat Ferrel
I became involved with Apache in 2011. After several years in startups where, as CTO, I felt too removed from building things. Looking for a change, I was keenly aware that the most interesting thing about the startups was our early use of Machine Learning techniques and I wanted to see if building ML solutions, for companies new to the field might not be more satisfying. I started by spending nearly a year in researching the type of applications we had needed in the startups: Natural Language Processing (NLP), text analysis, clustering, and classification. In those days Apache Mahout http://mahout.apache.org/ had several good solutions that were designed for Big Data and approachable by an individual. These ideas seem fairly commonplace now but were in early days only 6 years ago.
Welcome to Big Data
We need something new
One of the mentors of Apache Mahout, Ted Dunning, had suggested a new idea during this time. There was something about it that seemed very intriguing. He had proposed a way to use one type of user behavior to predict another. This was an aha moment for me because it codified intuition. I remember the first time he wrote in email on the Mahout user mailing list the equation that crystallized it all. I began to imagine the implications; all sorts of new data that could be useful, not just "views" but contextual data like location, and enrichment data like tag or category preferences. These all seem to obviously have a bearing on recommendations but now we had a beautiful simple equation to test the intuition.
Becoming a Committer
The hack was accepted into Mahout Examples and I was invited to become a committer. Then the world changed.
Apache Spark and Mahout-Samsara
Those were exciting times and though I helped with the DSL I remained fixed on implementing CCO, which was first included in Mahout 0.10.0 in October 2014.
I found a project that included everything I needed and was Apache licensed but was run by a small startup called PredictionIO. They had a Machine Learning Server that was a framework for Templates that could implement a wide range of Algorithms. The Server also included nice high-level integrations with Elasticsearch (Lucene server), Spark, and HBase. In May of 2015 I had the first running CCO Server build on Mahout and a whole list of other Apache projects.
Back to Apache
With the 3rd release of PIO from Apache we are now in the process of graduation to an Apache Top-Level Project, hatched by the Apache Incubator. I fully expect that we'll be celebrating soon.
This is a story of someone single mindedly following a goal over several years. There are many ways to do this in the Software Development world, but not all OSS projects are open to bringing people in. The Apache Software Foundation most certainly is and openly recruits as diverse a group of committers and members as possible. If you want to make a difference and influence the course of an OSS project Apache is a good place to look. Start by getting involved with a project of interest, make contributions, get involved in discussions. If the match is good you'll be invited in as a committer and move on from there. I think of Apache as a do-ocracy, if you do something of value it goes a long way towards being invited in.
Slides describing the CCO Algorithm: https://www.slideshare.net/pferrel/unified-recommender-39986309
IBM DevWorks Post on "Making one thing Predict Another": https://developer.ibm.com/dwblog/2017/mahout-spark-correlated-cross-occurences/
Apache Mahout CCO Implementation: http://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html
Apache PredictionIO: http://predictionio.incubator.apache.org/
The Universal Recommender Template: http://predictionio.incubator.apache.org/gallery/template-gallery/
Professional Support for the Universal Recommender: http://actionml.com/universal-recommender
# # #
"Success at Apache" focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be 8) Meritocracy. https://s.apache.org/DiEo 9) Lowering Barriers to Open Innovation https://s.apache.org/dAlg
Posted at 10:01AM Oct 02, 2017 by Sally in SuccessAtApache | |
Success at Apache: Lowering Barriers to Open Innovation
By Luke Han
Over the past decade, I was a Java developer using many Apache projects such as Tomcat, Jakarta, Struts, and Velocity. In 2010 I stepped into the Big Data field and started to actively participate in Apache projects, and became an ASF Member 3 years ago. In addition to being the VP of Apache Kylin, I helped projects such as Apache Eagle and CarbonData move to the ASF, and have been a mentor for Apache Superset, Weex, and RocketMQ. Today, I'm co-founder/CEO of Kyligence (prior to that, I was Big Data Product Lead of eBay, and Chief Consultant of Actuate China).
Apache Kylin, as its name may suggest, originated from China ("Kylin": A powerful yet gentle fire-breathing creature in eastern mythology. Also written as Qilin. "Apache Kylin": OLAP on Hadoop, capable of analyzing petabytes of data within seconds http://kylin.apache.org/ ). I started this project with a few members in early 2015.
As a pioneer of the first highly-recognized Apache project from the Eastern world, I was proud to see that, within 2 years, Kylin has helped over 500 organizations across the globe to solve their Big Data challenges.
Before Kylin graduated from the Apache Incubator, the Kylin team faced a lot of cultural challenges. Since a great number of projects from China had failed in the past, we too received many questions and doubts from both eastern and western worlds. As our native language is not English, communication with mentors did become difficult during the coaching process. Fortunately, by fully embracing The Apache Way, Kylin is able to succeed with strong support from the Apache community members. Much more beyond the Kylin software, our team has also worked with those talented people in a way to spread our Chinese voice to the world.
While developing high-quality software, we are engaging more Westerners to understand the Eastern culture. I had many chances to travel and meet people across the globe since I initiated Kylin. Some of them are Apache directors and mentors, some of them are developers and contributors. Some are from US, Australia, Canada and Chile; some are from Japan and Taiwan. Some are impressed with Kylin, some are curious about Easterners’ attitude toward Open Source software. I asked them a lot of questions about The Apache Way, and they all generously coached me and my team with lovely and detailed answers. We too could reach consensuses after intensive and open arguments. Kylin received much more encouragement and recognition than I expected.
As a VP of a Top-Level Project, my responsibility grew after Kylin graduated from the Apache Incubator. Kylin faced more opportunities as it has been bug-fixed quickly and tested frequently, with the nature of an Open Source software. In the China’s well-knowingly-big market, Apache Kylin has received many users’ feedback and evolved fast. We received many suggestions from both developers’ perspective and products’ perspective. Beyond my expectation, many community members are passionately writing tools for Kylin and helping users better understand and use Kylin. Assembling members’ ideas, we are also sharing our knowledge as a way to give back to the community.
Thanks to ASF and everyone involved in the Open Source community, I have the opportunity to work with people that I’ve always admired and make a difference in the world all together. I feel I and my team are deeply connected with such warm, global, open community.
= = =
"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be 8) Meritocracy. https://s.apache.org/DiEo
# # #
Posted at 12:45PM Sep 05, 2017 by Sally in SuccessAtApache | |
Success at Apache: Meritocracy.
Kevin A. McGrail is a cybersecurity expert and Open Source advocate who loves stopping spammers. He got involved with the ASF when the Apache SpamAssassin project joined the foundation in 2004. Today he still helps the SpamAssassin project while also serving as an executive officer and VP of Fundraising.
= = =
"Success at Apache" is a new monthly blog series that focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be
Posted at 01:06PM Aug 15, 2017 by Sally in SuccessAtApache | |
Success at Apache: Learning to Build a Stronger Community
by John Ament
As the next line in the series of "Success at Apache", I had to think about what kind of blog post I wanted to write. Given my personal focus, it made sense to focus on new projects coming in and the incubator. When I'm not busy dreaming up new ideas and working on personal projects, I'm helping new projects get in to Apache, keeping their goals in alignment with the Apache Way http://apache.org/foundation/governance/ . I'm a member of a few different PMCs here at Apache, notably the Incubator. I'm a mentor to five different podlings right now. While my primary programming focus is on programming models, my podlings are all over the place. Starting a new project here at Apache can be a daunting task: how do I get in? What if I don't build a diverse community? Becoming a podling has more to do with the community than it does the technical aspects of the project. We don't expect you to be experts in it, but we do expect new projects to be experts in how their own software works. We want to teach you, and we want you to be receptive to learning about The Apache Software Foundation and its best practices.
I'm not sure if everyone does it, but I build a lot of parallels between how an ASF project works and how an Agile team works. Agile teams start off as a bunch of people who don't really know each other but have assembled themselves into an informal team focused on solving a problem, or some number of problems, knowing that they can only do it together. They have common goals and objectives, but lack camaraderie early on to be able to work together smoothly. Over time, they get to know one another, figure out strengths and weaknesses and can resolve issues together. A well-functioning team isn't one at the beginning. It takes time and practice for them to work well - both together and as an outwardly facing unit.
The ASF is pretty big on open communication, wherever it's a sensible solution. We want to discuss with each other what we're doing, ideas around how to solve it and come up with a good solution together, as a team, in an open manner.
This all ties into agile practices. We host stand ups to talk about what we're doing and see if others have an opinion about what we're doing.
When a project comes to Apache, the original authors need to remember that they're bringing in a lot of experience, and the expectation is that those existing contributors must help get new contributors from the outside - outside their organization specifically - to contribute into the project. By driving towards open communication, outside of your own organization, you're encouraging more people to participate. This sort of governance model ensures that all parties who can participate are aware of decisions being made.
Turning Into a Well Oiled Machine
Once a project begins to grow, new people start to get attracted to it. As a community, you have to figure out how to work together. Building a community of diverse ideas and skills will ensure that new ideas keep flowing. Contributors can react quickly to a user's question on list and help them resolve the problem, put in an enhancement request or get a bug report squashed in a following commit. Time is of the essence right now because I have availability to work on this.
There can't be a long drawn out waterfall style process when dealing with Open Source. At the same time, making sure there's a documented decision process and in sometimes an in depth design is critical for both new contributors and existing alike to come to a shared understanding of what is being proposed.
Projects need to plan for longevity. Longevity comes in many forms. A strong backlog of features is important. Having a diverse set of committers is even more critical. You could even say that each helps create the other. Just like any feature set, we get to a point where the feature is complete enough that we can move on to another feature.
How do you get there?
Apache's main way to go to these points is to incubate http://incubator.apache.org/ . You can't get to this point by yourselves, experiencing with first-hand from existing Foundation members will help get your community to turn a new leaf and adopt this way of working. We want you to be successful, as long as your project can dedicate itself to the practices that have been set forth within the Foundation.
New projects may be comfortable with a champion http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Champion that can work with them closely, answering their questions up front. While a lot of the pre-incubation chatter will happen off list, it is important that potential new podlings subscribe to the incubator general list http://incubator.apache.org/guides/lists.html#general+at+incubator.apache.org and understand both the goings on of a podling as well as try to build their list of mentors http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Mentor in the open. Mentors are extremely important to a podling, and understanding their roles and why you need to pick great mentors is something your champion and the rest of the Incubator community can help explain. Participating in our public discussion lists is sometimes the first step to joining the foundation at a deeper level.
Where do we go next?
If you're a potential new project, feel free to reach out on the Incubator mailing lists http://incubator.apache.org/guides/lists.html#general+at+incubator.apache.org to get started. We'd love to hear from you and get you acquainted with The Apache Software Foundation.
If you're on an existing project, we want to hear your perspectives on how the Foundation works. You may want to reach out to dev@community http://community.apache.org/lists.html to let others know your thoughts, or even just subscribe and see what others have to say. We're all working together to make the foundation better. The more input we receive, both the positive and the negative, will help shape everyone's actions in the community.
= = =
"Success at Apache" is a new monthly blog series that focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh
Posted at 12:04PM Jun 05, 2017 by Sally in SuccessAtApache | |
Success at Apache: Meritocracy and Me.
by Tom Barber
When Sally asked for volunteers to help with a blog post series "Success at Apache" I realised there was a very human story to tell about how the ASF helped me get to where I am today and hopefully where I'll go tomorrow. Over the years I have worked on and run a number of Open Source projects whilst working with an awful lot of Open Source software. One day I was browsing Slashdot as you do, yeah I know a lot of people disparage it, but it's an awfully hard habit to kick, and without it I wouldn't have got involved in the ASF so I owe it a lot. Anyway, one day when browsing Slashdot I saw this article (https://it.slashdot.org/story/11/01/08/1544204/apache-to-steward-nasa-built-middleware), I had been working in the Open Source business intelligence industry for a few years at that point and I spent a lot of time hacking around and managing data systems, so I wondered how I could get some help out of OODT (http://oodt.apache.org/). Also as a kid I had always loved everything about space, I was a huge Apollo fan, had a small telescope, went to the total eclipse in the UK in 1999 and so on. I thought this OODT project would be a fun way for me to chat nonsense to a few NASA employees, find out how they did stuff and do a bit of Open Source hacking on the side, which would at least let me participate in some NASA related development work, and so it began.
As I mentioned at the start, this blog series is about success at Apache, hopefully this proves that success can come in a number of ways, the ASF was selected by NASA as the home for its data middleware platform, that proves that the NASA deemed the incubation process, the license and ecosystem acceptable, that is success the the Apache Foundation. Similarly the foundation has proved very successful in placing people into employment from a range of different walks of life into new lines of work, and that is exactly what happened to me and the reason I wanted to share my story about success at Apache.
= = =
"Success at Apache" is a new monthly blog series that focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM
Posted at 01:05PM May 01, 2017 by Sally in SuccessAtApache | |
Success at Apache: JFDI --the unconditional love of contributors
by Daniel Gruno
In many respects, The Apache Software Foundation is like a dog.
Some people like dogs, some like cats. The ASF doesn't care, it still loves you. Some people are female, some are male, some don't share the narrow dichotomy of the masses. The ASF doesn't care, it still loves you. Some people have cool parents, and a car, and money, and are connected with the right friends, and some are "strange" loners that can count their friends on one hand (sometimes you don't even need a hand!). The ASF doesn't care, it loves you anyway...as long as you feed it some code or documentation (or any of the other valuable contributions), you get love and respect in return.
But before I start telling my story about a dog named Apache, I should probably tell you a little about myself. I've been involved with Apache (both the foundation and the good ol' HTTP Server) for five years now. I am a Member of the foundation, and the Chair of the Apache STeVe project. The reason I got involved and why I'm still here, working for the ASF as an infrastructure architect, is that I had an itch (several in fact) to scratch and a yearning to show the world that I could...do stuff!
I came to Apache from an academic background. I had been studying at various universities in Denmark, initially Statistics and Business Administration, and later on Human Resource Management, so my assumptions about how Open Source worked were that it probably worked like academia works: You have an idea, you present it, ask for feedback, start a collaborative process and have your peers review it, then implement said idea once they all say it's okay. Now, in some regards, this is accurate, but the method of execution is vastly different.
Academia is built around a mix of healthy and unhealthy inherent mistrust (in many respects akin to the notion of original sin in some beliefs).
The healthy part was in my experience --and in very(!) simple terms-- attributed to critical thoughts of Karl Popper and like-minded science philosophers who, argue against _proof_ as a means of advancing science, but rather seek out a _lack of evidence against_ as the proper way to corroborate a theory (and in doing so removing things like "is there a god?" or "are gnomes real?" from the sphere of academic theories, as you cannot disprove the existence of said figures, thus they are categorized as meta-theories and philosophical conundrums instead --the question of "do we really deserve dogs?" however is still valid!). Instead of proving that the sun does indeed rise every day, one would instead say "I have a theory that the sun won't rise tomorrow" and then debunk that. While proving that the sun rises tomorrow is a nigh impossible task (even with a 99.999% chance of it rising tomorrow, that means there's only a 16% chance of it rising in 500 years time), proving you were wrong tomorrow when the sun rises yet again is a simpler and more attainable goal that _has practical value_.
Now, while this works well in academia, the notion of having to _prove your code doesn’t work_ can be a bit much for someone just trying to fix a typo in a script. Still, things such as unit tests and fuzzing can be compared to the notion of _proving by failing to disprove_, in that we too in Open Source employ a notion of "if we can't break it, it must be working". This calculated and practical mistrust in our work is healthy.
The unhealthy part stems from our fellow human beings being human beings...and not dogs. Unlike Open Source communities like Apache, universities are, in my experience, just high school all over, with fancier charts and bigger books. It matters who you know, how well you can network, who foots the bills. While some schools do make you feel at home, in most cases you are left to fend for yourself and build up a network, both socially and scientific. If you did not have the proper social skill-sets, you were alone. There was no sense of inherent belonging, it was --in a sense-- the capitalist dream turned to a nightmare. Your future was yours to create, and yours alone. If you lacked the skills or mental capacity for socializing, you were simply left behind.
A Dog Named Apache
Looking back at my experiences with education institutes, imagine my surprise when I --the classic loner type with limited social skills-- asked if I could get some patches applied, and five minutes later, they were! The response was, to me, an overwhelming acceptance of the work I had done, and people saying "please do contribute more, we value this immensely". More than that, it was a sense of being invited to a community that didn't have any other reasons to invite me than "I have some patches".
There is a general notion that Apache is a meritocracy: You contribute, and through your contributions earn merit that in turn affords you leverage and say in matters. I'll posit that not only is this true, but it's also a positive sum game with "original merit" applied. People are inherently trusted to have good intentions and are afforded an initial goodwill that you might not afford people in other circumstances. I didn't know these people, they didn't know me --and none of that mattered here.
And so I wrote a comment system for our documentation. It got implemented in the documentation, other projects saw it and said "can we please use this too?". Not long after, I was neck deep in infrastructure business, having discovered that Apache was not only the HTTP Server...It was a plethora of interconnected projects all sharing the same notion of coming together to solve problems and help make the world a better place through advancing computer science. Everywhere I looked inside Apache, the notion seemed the same: If you can help us, you're one of us. Don't care who you are, where you come from, as long as you can contribute in some way, you're welcome in our community as a valued member.
So I joined another project, and another, and another. Today I am an official part of 10 Apache projects, on the Project Management Committee (PMC) in 8 of them. Do I know about these projects in extreme detail? Heck no! I can barely tell you what some of them really do, but that really doesn't matter at Apache. What matters is your willingness to contribute, no matter what your expertise level is, no matter what field of expertise is. If you have something to contribute, Apache will accept and love you. Just like a dog.
So put down your phone, stop facetweeting, and most importantly, stop thinking you can't possibly help or become a part of Apache. If you can write an email, you can help out. If you can fix typos, you can help out. If you can *sort* of code in a programming language, you can help out. If you can write newsletters, know how to fix a configuration error, help people on IRC, you can help out...and Apache will love you for it. And before you know it, you'll be a deeply integrated part of the Apache community.
If you are in doubt as to what project you would or could contribute to, Apache has an awesome Community Development project that helps mentor and engage people in projects, as well as teach about how the foundation and projects work. For more information, head on over to https://community.apache.org and check out the resources available to you. You can also check out https://projects.apache.org and see if you know one of Apache’s projects, or perhaps discover a new project that fits what you are interested in --welcome aboard!
# # #
"Success at Apache" is a new monthly blog series that focuses on the processes behind why the ASF "just works":
1) Project Independence https://s.apache.org/CE0V
2) All Carrot and No Stick https://s.apache.org/ykoG
3) Asynchronous Decision Making https://s.apache.org/PMvk
4) Rule of the Makers https://s.apache.org/yFgQ
Posted at 03:07PM Apr 04, 2017 by Sally in SuccessAtApache | |
Success at Apache: Rule of the Makers.
- Ask via the development list and issues database.
- Approach developers individually to discuss the project.
- Just go ahead with the company's in-house developers.
Having said that, it remains intentionally fairly small, and won’t meet the needs of every application. Some projects continue to use SQL their own way. The fact there is now a Right Way to access SQL doesn't mean that other ways are wrong! There are still third-party projects that, for their own reasons, use other ways to access SQL. The fact that someone has done the work and met a need gives them an entirely valid niche in a Pratocratic ecosystem.
# # #
"Success at Apache" is a new monthly blog series that focuses on the processes behind why the ASF "just works":
1) Project Independence https://s.apache.org/CE0V
2) All Carrot and No Stick https://s.apache.org/ykoG
3) Asynchronous Decision Making https://s.apache.org/PMvk
Posted at 02:04PM Mar 06, 2017 by Sally in SuccessAtApache | |
Success at Apache: Asynchronous Decision Making
by Bertrand Delacretaz
Asynchronous decision making is a key enabler of our geographically and culturally distributed Open source teams. In this post I'll explain the ingredients that make it work at the ASF.
I became active in the ASF in 2001 via Gianugo Rabellino - he was the one who started the discussions with Apache Fop about me donating the jfor XLS-FO to RTF converter that I had developed earlier. It was already too late to uninvent RTF which is a terrible format, but I digress. I am currently a member of the Board of Directors of the ASF and have been doing a lot of thinking (and presentations) about what makes the ASF tick in terms of collaboration and Shared Neurons.
If synchronous decision-making meetings were required in ASF projects, even using remote channels like IRC or videoconference, we would move forward at a snail-like pace, as just finding a time where all stakeholders are available is almost impossible in an environment that has no managers and no central schedule.
Meetings are also very expensive when you are working on a maker's schedule, as described by Paul Graham . Frequent meetings ruin the productivity of craftsmen, and there's lots of craftmanship in our industry, especially when you're building leading edge stuff.
So, what's needed to enable people to make collective decisions asynchronously, without requiring meetings?
The first thing you need is a central asynchronous communications channel. Which technology you use for that doesn't really matter, but it has to allow everybody to get the same information, and provide a usable way of having threaded discussions, where you can branch off on a topic while ignoring other topics being discussed on the same channel. This can be as simple as a whiteboard if people often visit the same place, or as elaborate as web-based forums, accessible from any mobile device so you can bother^H^H^H^H^H reach people everywhere. At the ASF we use plain mailing lists for that, very successfully when people use them with the right discipline (see the appendix below). Archiving this channel is very useful, to allow newcomers to get a feel for how things work as well as documenting the reasoning that led to each decision and avoid having to repeat things over and over.
The second required tool is a way to build consensus, where you avoid deadlocks and make sure decisions go forward. Unanimity in decisions is ideal of course, but the second best is consensus, defined as widespread agreement among people who have decision power. Requiring unanimity or allowing vetoes in decisions can block progress, so at the ASF vetoes only apply to a very limited set of decisions types, as defined by our voting rules . In companies, decision power can be based on hierarchy to break deadlocks. That has to happen sometimes, but abusing it can cause employees to lose their autonomy and purpose, which kills your team in the long term.
To keep track of each decision, a case management system is ideal. You could work without that, depending on your team's size and the number of decisions that you take, but it's very convenient to be able to discuss the details of a given decision and keep associated information in a single place. You don't need complex software for that, at the ASF we use fairly simple issue trackers. Those are Web-based systems where each case is handled on a single page, with a history of comments and actions. Some non-urgent or very hard decisions can take a long time to reach closure, and it's very useful to keep their history in a single place, if only to avoid having to explain them again to new members of the team. In a low tech environment you could just use a single sheet of paper to briefly document each decision with the key points that led to it, and keep those in binders or physical files.
A nice side effect of using case management software is that each decision gets a simple unique identifier, like FOO-123 for the 123th ticket of the FOO project. This removes any ambiguity as to which issue one's discussing, by mentioning those identifiers in conversations.
So, in summary, the following should allow your group to make decisions asynchronously, without requiring meeting and with a written trace of everything that happens:
- An archived asynchronous communications channel, where everybody can get the same information and threaded discussions can take place.
- A way of building consensus, including fair rules for breaking deadlocks.
- If possible, a case management system to keep track of each decision's details, in a much cleaner way than the often messy discussions that happen on the asynchronous channel.
Semi-asynchronous decision making at the ASF
The agenda file structure looks roughly like this:
Call to order
Project reports, headers and discussion space
Board Resolutions with discussion space
Appendix: Full Project reports and other supporting material
And a project report header and discussion space is as simple as this:
E. Apache Blazinator Project [Bob Blazer / Bertrand]
See Attachment E
approved: bd, mm, dd, db, jc, ldv
bd: Not sure why LEGAL-123 blocks their release
ldv: They are waiting for the committer to supply
an updated iCLA as the received one was
bd: Ok, thanks, approving the report then.
This simple block of structured text builds a very simple "case management system" for the case of approving the Blazinator report.
The "approved" line indicates which board members have approved the report, on a single line so that simple text-based tools can validate and count the approvals.
The "comments" section allows stakeholders to comment on the report (which is found in an appendix later in the text file), and reply to each other's comments to hopefully reach closure before the meeting. If this happens, approving this report takes almost no time in the meeting, the chairman can just list the project names ("case identifiers" according to the above terminology) of such pre-approved reports, asking if anybody's opposed to approving them.
Combined with the ASF board's mailing list, this builds a very simple and very efficient system for semi-asynchronous decision making. Most decisions are taken before the meeting, and the participants can spend their time where it actually adds value as opposed to exchanging boring status information during the meeting.
Try it yourself!
Many ASF and other Open Source projects release world-changing software while having no or very few meetings, demonstrating that these techniques work.
If you're bogged down with inefficient or useless meetings, I suggest that you try applying these principles to a meaningful subset of your decision making activities. People will need to hone their skills to work efficiently in this way, but the rewards can be huge for distributed teams.
Appendix: Mailing lists at the ASF
At the ASF we use mailing lists as our central asynchronous communications channel, based on our if it didn't happen on the dev mailing list, it didn't happen rule . Mailing lists might be seen as tools of the past when you compare them with the latest shiny tools, but they remain a ubiquitous way of communicating in loosely coupled remote groups, especially when used with a strong discipline of Precise Quoting  and paying attention to meaningful subject lines. Unfortunately I hear some "modern" email clients make a mess of that quoting - just stay away from them.
 http://www.paulgraham.com/makersschedule.html - Paul Graham, Maker's Schedule, Manager's Schedule, July 2009
 https://community.apache.org/apache-way/apache-project-maturity-model.html - The Apache Project Maturity Model, ASF community development team, 2015.
 http://s.apache.org/gianugo_quoting_2002 - Gianugo Rabellino "[OT/Rant] Quoting", message to the cocoon-dev mailing list, January 2002
 http://www.apache.org/foundation/voting.html - ASF voting rules, created in 1999 probably, or even earlier among the Apache Group.
# # #
"Success at Apache" is a new monthly blog series that focuses on the processes behind why The Apache Software Foundation (ASF) "just works". First article: Project Independence https://s.apache.org/CE0V Second article: "All Carrot and No Stick" https://s.apache.org/ykoG
Posted at 12:42PM Feb 06, 2017 by Sally in SuccessAtApache | |