The Apache Software Foundation Blog

Monday October 12, 2020

Inside Infra: Daniel Gruno --Part II

The "Inside Infra" series with members of the ASF Infrastructure team continues with Part II of the interview with Daniel Gruno, who shares his experience with Sally Khudairi, ASF VP Marketing & Publicity.

" speaks of how tenaciously the Foundation guards its core values, one of which really is provenance, because it's the Apache seal of approval, means this has been thoroughly vetted. We know where every single piece of code comes from. And we know that it works."

What about "user demand" --what does it take for you collectively to decide, "OK, we'll support Kubernetes," as you mentioned it earlier, or whatever? Are there strategic technologies that you want to work with or plan to support, or is it all coming from the projects themselves? How does that process work? You're creating projects out of some kind of pain point or some kind of vision. So for you, is it a longer-term thing? Do you have an influence on this? What drives the growth of services delivered? It's a mix. It's a mix of, first of all, the Infrastructure team is paid by The Apache Software Foundation and it's paid by The Apache Software Foundation to help the projects. So what we do must first and foremost be something that helps the projects and not something that just helps Infra. I mean, of course, we can make tools and have services that will assist us in our work, but the ultimate goal must be supporting the projects. First and foremost, we listen for projects that come and tell us, "We would really like this or we would really like that." Having said that, we do not always say yes. We have costs to consider. We have maintainability to consider. So as a general rule of thumb we will say, "Okay, project A wants to use service foo. Does anyone else want to use service foo right now?" On occasion, you get, "Nope. No one else wants to use service foo." And then we go back to project A and say, "It doesn't seem like this is feasible for us economically to maintain if it's just you." But you can also have a situation where 10 projects suddenly say, "Yep, we really, really want to use this." Once you have a trend for something, we are usually not proactive, but reactive to these trends. So a project will come and tell us, "We really want you to use this." We will go out and see if anyone else wants to use this, and they will say, "Yes, please." That's when we'll add that feature or service. We also have ideas of our own that are, by and large, a result of either existing services not doing what they're supposed to, or they're being... Let's say you have... For example, there is Google and there are mail archives that we had in the olden days. At some point we wondered, "Why don't we combine it so you can search for emails in the archive?" That's how came to be. So we have both things that projects come and say, "We really want this," and we also have this crystal ball where we look at problems we're having with existing services, where we look at possible combinations between existing services and other existing services or new services that are emerging in the Web. Or we just have someone say, "Hey, wouldn't it be wonderful if something like this existed?" So it's really a mix of projects asking us and trends emerging and just blue skying, "Wouldn't it be cool if...?" Have you guys been in the situation where you found yourselves caught where there was this magical trend that everyone wanted, and it just didn't serve the Foundation, it failed? Were you guys in that situation where you had to back pedal? Or is that not part of your experience? I would say the most prominent or obvious feature or service would probably be GitHub where we started in 2010 with mirrors of our local Subversion and Git repositories. They would be mirrored to GitHub. That was actually a bit later, but around that time, they started mirroring stuff to get up, but you couldn't write to GitHub. We were adamantly against it. Because provenance, provenance, provenance: that is that thing that if you know Apache, you know that provenance is one of our key features. We like to be able to say, "Oh this came from that. This came from this. This came from that." We had concerns at Infra that we were not able to have the exact --emphasis on exact-- same provenance as we had on our own servers, and we got a lot of pushback for that. In the end, we figured that maybe we don't need this kind of providence that we had. Because we had very verbose logging going on for our own service that we couldn't get from GitHub because GitHub is a third party provider. They're not going to fork over sensitive data about their customers to us. So a) we were willing, at some point, to compromise, because it turned out that the data that we had been collecting was maybe not so important after all, and b) we came up with this linking utility that would actually allow us to go in and see who that person committing was on the ASF side. That is, if someone commits with a GitHub account, we can go in and see, "Oh, this is actually this specific ASF Committer," because we have this internal mapping going on with GitBox. And so with that, and then the realization that we didn't need all of this verbose logging, we finally decided that we're going to allow write access, but that was probably... It could have happened a year earlier. A year sooner. But I wouldn't say that it's a failing, of us, as Infrastructure. I think it's more it speaks of how tenaciously the Foundation guards its core values, one of which really is provenance, because it's the Apache seal of approval, means this has been thoroughly vetted. We know where every single piece of code comes from. And we know that it works. If you're suddenly letting go, even if it's not really the case, but if you're seemingly letting go of some of those core values, you are going to get pushback because we all, I want to say, love and cherish the Foundation. We all believe so powerfully in its mission that for a moment, we forget reason sometimes and we just push these core values without interpreting them, which is sometimes the right thing to do. If we have a core value that says, "We need to be able to see where the code comes from." That doesn't necessarily mean we need these five specific points of data from every single user. It just means we need to know where the code comes from. And if that means these four we know, plus this one new one, then that's just as good. That was a bit grandiose, sorry. No, no, no. There's a lot to it. And I love the angle that you're providing with your answers. That's very different from the other guys' perspectives and that's super helpful. It's important, because that's demonstrative of the diversity of the Foundation. We're people, we're not just machines. And so it's very cool to hear this. Moving on specifically with our growth, like how do you close your skills gap? Do you do that? Do you rely on the team? How do you cope with stuff that way? Oh that's a good question. I rely on mentors that I have. I'm not a bookworm, for example. I can't sit to read a book. I can barely watch a movie because I have a very low attention span. So what I'll do is I'll make some mistakes and I'll have some mentors that I have come in and tell me, "You're doing it wrong." And then I'll fix it or try to fix it and they'll say, "You're doing it wrong again." Eventually, well in a loving way, eventually they'll say that this is right. I love to learn by example. I have a lot of mirror neurons in my brain. I like to copy styles. I like to mishmash styles together. And I love to fall in love with new ways of --this is going to sound very nerdy-- I love to fall in love with new styles of programming. I recently discovered something called MyPy, which is a typing checker for Python. At first I was like, "Oh this is boring," and then I realized, "Oh, I can actually use this for checking whether what I write is going to always work." Then it changes into, "This is awesome. I love it." Which then changes into, "Everything I've ever written must now be written using this typing hint." And then suddenly I have Greg Stein yelling at me saying, "Yes, this is technically valid, but I really don't need this." Another mentor I have introduced me to this typing hint. And so I progress by observing other people doing their things and where they and I differ, there are basically two scenarios. Either they're worse than you are or they're better than you are. If they're better, or perceived better, I will usually try to study, "What are they doing that's different from what I am doing?" and if I like it, it tends to stick like a rash. Then suddenly, it's in everything I do, because I'm not a trained programmer. I never studied programming. I never studied computer science. I studied social economics and then human resource management which is very far from it. It was always just a hobby thing so I never really learned about unit testing. I never really learned about unit testing. I never really did learn about proper documentation outlines. And I never really learned this is the correct way to program in this language. This is how you style it. It was always just looking at some examples and then picking the parts that I thought was interesting. So what I initially want to start off as a program, what I wrote it, it would work, but it would be very ugly and it would be very error prone. So people would say, this is a cool piece of software, but it's very not pretty. This is what you should do to change it. So I've relied on people not telling me that I am good or bad, but telling me, this is the difference between what you do and what I do, and then having it be up to me to figure out is this something I want to adopt. Greg, for instance, has been a tremendous help in that Python department, not necessarily by saying, “you need to do this, you need to do that,” but by writing some examples. Commenting code on saying, “this could be” --emphasis on “could be”-- “could be this. Or you could use this instead.” Because he's got decades of experience in Python programming, for him to say there's a different, smarter way of doing this, it's not by using words, but by just showing the examples. Because he knows that I pick up on the why pretty easily. I just need to know that the difference exists, then I'll know the “why” eventually, because I'll be very interested in why that difference is. So he just kind of feeds me these little nuggets of this smarter way of doing it. I learned from that and I'm very grateful for that. Tell me how has ASF Infra changed over the years: is it proactive? Reactive? How and why did this come about? Obviously it's changed, but was it an organic thing? What's your take on that? It's changed in a lot of different ways and also it hasn't changed. And also: I don't know. It's changed in that it has become more of an obvious hierarchy now, which is not a bad thing. We have a place where the buck stops now: we have a place where decisions are being made. We have, most importantly for someone like me as a staffer, I have someone that I can defer to that I know will take care of it or will be the one with the final responsibility. That can shield us lowly peons when someone is being a bit too grumpy. That has changed which also means that we, the staffers, are not as abrasive as we used to be. I remember when I joined, the tone was a lot different. This is of course my perception as being this little timid newbie back then, but it was more, every single person had to kind of fend for themselves. Now we've got more of a cohesion. We have yearly gatherings, face-to-face gatherings. We talk about a lot of non-work related items. We have weekly calls that didn't happen before. I guess you can say it's become more of a family now than before because we interact with each other on so many different levels that are not specifically work-related now. It's also made us more friendly. The change was largely planned. Or it wasn't “planned”, but was planned as a reaction to events that happened --sometimes you come across some things when you're in any given company. We were like, “we need a change”. And this was one of changes that happened a few years back. Well quite a few years back. Actually, I think this was in Cambridge, not Cambridge, Massachusetts, but Cambridge in England. We had a meetup with our new, at that point, our new Vice President of Infrastructure, David Nalley, and the existing infrastructure team. This was the first event in my lifetime, if you will, of the team. The first face-to-face meeting we had, that was all about “what are we going to do in the future as a team”, where we worked out a lot of policies and work methods that we still use to this day. I'll not go into too much detail about why, but it was planned as a reaction to us being perceived as not the most welcoming group of people. If you go back 10 years, it was in my personal experience, a lot more daunting asking Infra for something. Do you think that's because people were just rude? Or was it a matter of them being overwhelmed? Or there was no process? What do you think was behind that? I think there was not a sense of structure in the team that we have today. People were self-led. We are, let me emphasize that, we are still very much self managing in the team, but we also have a boss and a boss's boss that let us know what they would like us to focus on for the long term processes. We didn't really have that before. It was more fend-for-yourself, figure-out-something-to-do. And if you can't, then that's just “why not”? We have a lot more structure after the Cambridge meeting. And after David started as VP Infra. Because we had gone from being --I don't know if you know this saying in the US, but there is some difference between a United States and American NATO Secretary General, and a Dutch NATO Secretary General. And that is that one is a secretary and one is a leader. One is a boss and one is a leader. We had a change in the style of management at that point. It's not that (former ASF President and VP Infrastructure) Sam (Ruby) wasn't doing his job. It's that David added something to the job that wasn't there, hadn't been there before. Sam was doing what every single VP before him had been doing. Which was fine. David came in and saw that there were things that he wanted to improve upon and he improved upon it. One of the outcomes was that, in my view, that the team also became more friendly towards people coming in with issues. But it's also a different environment that we are in now as a team. Apache in the old days, it was strictly volunteers spending their hobby time, doing what they love. It has slowly pivoted into being people that are paid. They still contribute as individuals, but they are being paid to make those contributions. They are also part of larger teams, often at big companies that have a lot of resources. The expectations and demands of the Apache infrastructure has also increased exponentially as we have become a large organization. So what we are tasked with today is also more demanding. I don't think that the infrastructure to staff 10 years ago would have the same interactions and the same good terms. You want to be on the same good terms with the contributors as we are today. So in that sense, I think David was gearing us up for what was to come. David has also a unique perspective because he had come on Board in 2012 as part of the Apache CloudStack project. So he came in as an incoming project that also needed support from infrastructure. So he has experience on both sides of the fence, so to speak. You know, Sam has a much “older” experience in terms of him being with the Foundation from a much earlier time period. So it's very interesting to see how the evolution has come about. A lot of us who've been here from the beginning see things a certain way, and don't realize that from an outsider's perspective, that experience might be completely different. It's very interesting to be able to have that balance and have someone come in and kind of make the team more cohesive based on what their perceived needs were and being able to project what projects will be needing in future. It really is. Yeah. Also, he has a very special way of --let's say he's very “godfather”-like. I don't mean that he kills people! He has a very persuasive non-intrusive way of asking you for a favor that a) I find very endearing, and b) I know why he using it: because it's very effective. That I don't think a lot of people would get away with. So what that means is we do a lot of things that David asks us because it's David, because he's built up goodwill. It's easier for him to shape the team and to what he wants it to be as to someone that was just there as a secretary and didn't really do anything. If you're not engaging with the team as a boss, and then you suddenly come in and say, do this, there will be pushback. But if you're engaging, if you're there, if you're have a presence in the daily routines and the daily water cooler chat, and then you say, "Hey, by the way, what do you think about this idea?" Then you're much more likely to get a positive response back. I think that's one of the things that David brought is a more relatable and more ... let's say he's brought in a closer bond between boss and workers. Leader and workers. And now we have Greg as well. So now we have two of them. That’s progress in the right direction. What areas are you, meaning Infra, experiencing your biggest growth? At the moment, that would be continuous integration, which is building software basically. Testing that something builds. Testing that something compiles properly, that it passes these tests. We have six or seven different platforms for that at the moment, and it is using hundreds of machines. And it's never enough! We know we have a demand and we know what the trends are, and we're also kind of blue-skying a bit on how do we solve what's ahead of us. A lot of this is throwing more money at it because that always helps. A lot of it is, again, going back to developing smarter tools that enable us to utilize the resources that we have, because we are not like a big whale. We don't have a cash whale: we don't have that much money. So we’ve got to make sure that the resources that we buy or lease or rent or whatever, are being utilized to their maximum potential. So that, again, comes back to figuring out how do we go in and monitor. Is it being utilized? What can we do if it's not? What do we do with over utilized? Can we figure out where it is bottlenecking? And a lot of other things. Builds, continuous integration, continuous delivery, I think it's called. That's the place where it's the most growth at the moment. With regard to CI, what is the most popular platform that you guys are using or what service has the biggest demand? The most used one is still Jenkins. I think we have 30-40 Travis machines building there, and that's practically nonstop. With Jenkins, we have, I think it's 150-200 machines or something that are building practically nonstop. That's by far the largest platform we have. We are using a lot of Travis and Buildbot. We can always use more of that. You’ll be talking to Gavin (ASF Infrastructure team member Gavin McDonald), who has been working a lot on splitting Jenkins into smaller components. So that major software categories, for example, get their own platform and bigger projects can get their own platform. This is because we don't want a monolith. We're splitting that up to actually save us some costs and not have so much downtime on the time. He can tell you much more about that. One of the things we did was graph out how much are we actually using and how much have we been using. Which projects are using the most of these resources? And if there's a specific project that sticks out like a sore thumb with, I don't know, 50% of all the computers or the machines are going towards that project, then we'll reach out to them and say, are you maybe doing something that's a bit too intensive? Can we scale this back a bit? Or do we need to look for a specific targeted sponsor for you or what? We're not constantly, but on more occasions than not are looking at these resource usages and seeing where can we optimize things so as to not use too much money and also not use too little money. Just the right amount. So many companies, as you know, are really struggling with their teams working from home in response to COVID lockdowns and stay at home orders. From day one, ASF has been virtual. I understand that you were stuck in another country when the pandemic lockdowns happened. How did you cope with that? Did anything change with your operations, your work? How were you impacted by that? Work-wise I was not impacted at all, which is wonderful. We are able to work from pretty much wherever we are. And this was not my first trip abroad, believe it or not. This was in Canada, by the way, I was stuck for 105 days. In the few places that I go to more than just once, I have it set up so that I can work from there in a reliable and comfortable way. By that I mean, I don't like laptops: they're a wonderful invention, but I don't like them. I don't like sitting hunched over a tiny, tiny keyboard without a mouse and looking down instead of straight ahead at the screen. Luckily I have a laptop and I travel with it all the time, but I plug it into a KVM switch which is a keyboard, video, and mouse adaptor. I have a monitor and a good old sturdy keyboard set up in the places where I frequent often. So I was able to work from there as I would with my stationary machine back home, just using my travel laptop. The only difference was the time zone difference. But we do most of our work, asynchronously. And whatever firefighting there is that always just happens at random hours. So it doesn't really matter what time zone you're in. You're going to be screwed one way or the other What do you think people would be surprised to learn about ASF infra? Surprised? I mean, probably that it's only six people. I'm sure, I remember Drew saying this and Chris and so on, but people are often surprised that it's only those five-slash-six people that are doing all the work. I know you all, and I'm astonished by it. I'm perpetually amazed by that. It is a seriously huge feat. You want to know what surprises me from the inside? That we actually manage it. It surprises us as well. It's not that “oh yeah we're just that great”. There is something about the coalition and the project that we can't really explain, and it's not explained by the individual parts. It really is the sum of the whole, that somehow… Huge, huge, huge undertaking. It's massive. And the fact that you guys do it is incredible. And yeah, you know it would take five, six times the number of people to do it elsewhere. So it's very special. I think we also have a lot more on flexibility from up above. From both our boss and his boss and his bosses. They trust that we know what we're doing in a sense that you might not find at a typical company. And I think that is part of the reason why we're able to do the things that we do so efficiently. Because we've been given this trust and we've been giving the benefit of a doubt if you will, when we choose to... They trust that we know how to manage our hours and get things done. Like it's not a strict requirement that you be here, nine to five, Monday through Friday. You can be here, I don't know, three hours one day, and then 12 hours next day. Or maybe you want to work on Sunday instead. As long as the job gets done and nothing falls through the cracks, they basically let us get our job done. And I, again, I think this is a win-win situation because it allows for us to kind of cool down when we've burned out a bit, but it also gives them the added benefit of when I feel like I will put in the extra hours because kind of as a “thank you for you let me decide my hours”. So I'm going to put in some more time here and then I'll relax when it's quiet. Because we do get quiet days. So you all have to carry the load, which is good. There's no favoritism. Everyone has the same shared responsibility --you all have to be on call, for example. Yeah. It's still quite a flat structure. I don't consider myself senior in a managerial wage to any of my coworkers. And so if I were to, or if someone else, if Gavin or Chris or Chris or Drew. If anyone were to say, "I'm not going to be on call," that would create a rift between us. I mean, there are staff members who wish they didn't have to participate in it, but we all are on call on a rotating basis. And so I think we're fortunate that we're all in a position where we're okay with it. We were able to manage it because there are legit situations where someone is not able to be on call. I think we all have them from time to time or someone else has had to cover our shift so to speak. All five of us are fortunate that we don't have things going on in our lives where you can't be on call, because those things, they can happen.

Sure, that makes sense. So what's your favorite part of the job? This is going to sound weird for a lot of people, but my favorite part is the weekly meetings. Why does it sound weird? People aren't social? I don't know. It might sound weird to normal people who don't like meetings, that I liked meetings. There's something about meetings... Even though they are very informal meetings, I like the little shred of formality that there are about them. And so that's, I think, my favorite part. And also I get to prepare for them. All right. So you must like preparing for the Board meetings too. Yes. You should read my Apache Web Server reports. What was your biggest challenge when you came into the role at Infra? There were two major challenges. The first one was learning the ropes. This is, as both I have said and a lot of people before me, it's such a complex system at ASF. There are so many things to know and it doesn't just take a year: it takes years to learn enough to get by without someone else's help. So that was, by far, the biggest ... Well, no, that was the second biggest challenge. The biggest challenge was believing in myself and not being scared of doing things unsupervised. Because again, what I can do and what my other colleagues can do with their keyboards is very, very ... We wield a lot of power over a Foundation that is responsible for so much in the world. Not being afraid of typing a command takes a long time when you know what a title can do on a machine. You know, “did you just delete this file or did you delete the entire hard drive”? And especially at the very beginning of getting into the job, I would double, triple, quadruple check everything I typed. I would wait for sometimes minutes before I hit enter just to be sure, I would look up that am I doing the right thing. Just to be sure that I'm not messing things up now. And as you to do it once, twice, three times, 10 times, a 100 times, you become more confident and you also relax more. Your first thought isn't “what if this goes wrong?” First thought is “let's see what happens next”. Or you're thinking ahead to the next debugging step or the next problem solving step instead of being stuck on what if this goes wrong? Which also means unless something goes wrong, you work a lot more efficiently. Because you're not fearing the Enter button. What are you most proud of in your Infra career to date? Let's see. I'm well, probably most proud of ... That's a very difficult question. That's why I ask it. If you don't want to answer, that’s okay. Oh, no, no, no. It's just that I've made so gosh, darn many things. What I'm most proud of is probably ... I would say that is a thing that's playing out of an Infra job I was doing that. Yeah. I'd say that's probably the thing I'm most proud of. is very powerful. We all use it every day. So that's yours? With the help of a few friends, yes. It was a brainchild of mine during some tests we had at Infra. And again, it's one of those situations where you have something that's not working and you're like, "Maybe I'll just rewrite it completely and it'll work. And then you start writing and then you find a good idea, a better way of doing some things and some things don't work. And then sometimes eventually you end up with a product that sticks. It's one of the most long lived projects I've had and that's still used today. Well, it's super useful. There's no doubting its efficacy and necessity. I mean, how many mailing lists do we have now? 1,700? It's some crazy number. I think we're nearing 2,500 if you count the private lists. And that's like 25 million emails, so ... That's insane. That's very cool. Very cool. All right, next question. This is the one that everyone kind of laughs at. How would your coworkers describe you? I'll have to think about that. They would probably describe me as the one that suddenly says something completely out of context. (Laughing) Okay. I thought I was supposed to be laughing, not you. That is funny. What happens is when I asked the question, Chris, Drew, and Greg, all laughed. Then they give me their answer and I always laugh. So it's classic. Tell me what you think are the biggest threats that infrastructure teams need to watch out for? I think the biggest threats are relying on tried and tested methods, but forgetting the change and expectations of the user in terms of user experience. I've seen a big change in what a user expects from Infrastructure in terms of user experience. I don't mean they just want it more easy, but I mean people want it more feature complete, they want it to look more intuitive, they want it to tie in together with what they are already using. They want to tie it together with whatever is the next new hot thing. If you stick with what might be good and try it and test it and it's stable, you might end up losing everyone because while it might be that, it might also not be what people are using tomorrow. If it's not what people are using today and tomorrow, then no matter how good it is, people are going to move away from it. Not necessarily outdated in the sense of technology, but more in the sense of trends. What is trendy. Yeah. I mean, it used to be Vine. Now it's TikTok and tomorrow it's going to be something else. If you don't keep up with the fashion of IT, then you're going to find yourself not wanted. That timing out happens more quickly these days, it seems. Okay, what would be advice to aspiring sysadmins or Infrastructure team members? My greatest piece of advice is basically don't be afraid because this ties back into the daunting task of having to push the Enter button after you type something in the command line. Don't be afraid because you'll lose so much time just being afraid that you could have spent fixing things or learning new things or making yourself more at ease. Just jump in with both feet and you'll be fine: you're awesome. Yeah, that's good advice. If you had a magic wand, what would you see happen with ASF Infra? Oh, interesting. I would like to see us having some magic, unified CI system that could be used across the different repository and types we had and didn't require any machines that would just build instantaneously. And yeah, be free of us needing to manage yet and pay for it. And also, if GitBox version two was suddenly a thing tomorrow and I didn't have to actually write it, which I still have to do. Okay. What else do we need to know that I have not asked yet? Gosh, I don't know. I don't know. One thing I'm really good at or one thing I'm really bad at is when you ask me an open question like that, because I don't know where to go with that. I am very good at analyzing a question and coming up with a specific response, which is why when people say, "How are you doing?" I get confused or I say, "I'm okay." And get nervous and forget to ask them how they are doing, because I don't get the dynamics that are happening there." The reason why I ask this question is sometimes people come in thinking, "Okay, this is my area of focus." They might want to talk about the “blue switch” or something specific like that, then we talk about all sorts of other things. We may derail. I may be driving the interview in a certain direction, and they have this pain in their gut because they never got to talk about the blue switch that they wanted to. The only thing I could think of would be something called pip-service, which is a new thing we're making, which is kind of like a package manager for all of our infrastructure services. Again, it's us working smarter instead of harder. And we were defining a way to easily install or run a service on any given machines and set them up without actually having to install and run then set them up. It would require a lot more time to explain in detail, but it's really nifty. Is it coming soon or is it available now? It's in production. And it's really helped us a lot. I love the efficiency of Infra, how you guys are having these new directions ... Like when you and I were dealing with the the other day for the CMS (content management system), when I was getting the Annual Report up. For 21 years, I haven't been able to deal with the ASF CMS and then you walked me through it in literally three minutes on Slack and boom: it was done. I was amazed and shocked --because I'm not a technologist. To me that was phenomenal. You guys are really helping so many different kinds of people with different profiles and different skill sets. It's very cool. I think some of that ties into, again, the CMS was cool 10 years or 15 years ago, but it's not really been able to keep up with what's going on at the moment. No one knows how to use it because it's not very intuitive… Or it's not what we do today. Right. As we’re halfway through the Infra team, who do you think I should be interviewing next? I think you should be interviewing Gavin because he knows a lot about the CI platforms that I have been on, off raving about here. Gavin's not planning to talk to me until October... Oh, well then you should talk to Chris Lambertus, because he doesn't want to talk to anyone. (laughing) Chris can talk a lot about the upgrade of our email infrastructure. We have a lot of very tough work ahead of us in that we're upgrading an infrastructure that again, it works, but it's kind of like upgrading from an IBM mainframe to a modern computer: not that much of a upgrade, but we are having to modernize heavily on our Infra email infrastructure. I understand that's a huge, huge project. It's a very big project, yeah. That's a little advice for sysadmins there. = = =

Daniel is based in Copenhagen on UTC +2 (currently on CEST). His favorite thing to drink during the workday is lukewarm, weak coffee.

Monday September 07, 2020

Inside Infra: Daniel Gruno --Part I

The fourth interview in the "Inside Infra" series with members of the ASF Infrastructure team. Meet Daniel Gruno, who shares his experience with Sally Khudairi, ASF VP Marketing & Publicity.

"...companies are not the same as ASF. They don't have 300 different departments that all have their own little tools that they want working in their specific way. And they want this to connect to that, and that's connected to some other thing. We are not afraid to create custom solutions, we're not afraid to get our hands dirty and we're not afraid to make mistakes."

What is your name and how is it pronounced? I have my official name and I have my user name and people usually ask about both of them. My name is "Dan-yell Gkhroo-no" or I will accept "Dan-yell Groo-no" which is as you read it in English. It's actually a Dutch name. So you would pronounce it "Hrooy-no" in Dutch, which I'm not even going to try to phoneticize that because, that's, well, Dutch. And my username is "Humbedooh" which is an onomatopoeia that I randomly made up in 2004 for a game called World of Warcraft, where you need a username for this character that you create. And I think I had just listened to "New York, New York", where Frank Sinatra sings "scooby doo bee doo", and I was like, "hum-be-doo-de-doo" and the name just came to me and it stuck ever since. And so for the past 15 years or 16 years, I've been primarily "Hum-beh-doo" online. By the way, Frank Sinatra sings "zoo-bee-doo-bee-doo", not "scooby-doo-bee-doo" in "Strangers in the Night", but I like your version better. Okay. Well today I learned that. When and how did you get involved with the ASF? That goes back to 2010, 2011? Again, this beautifully tied us into World of Warcraft because in that game you can make modules, add ons for the game that will do nifty things, like add ons for a Web browser. And this is written in a programming language called Lua, L-U-A, which is Portuguese for "moon". And so I started writing some programs for this game and I had great fun with it, and programing is not my official trade. I was educated in, or studied, human resource management at university actually. But it was my hobby and I had great fun doing it. And this Lua thing just got stuck in me. And then five years later or so I started writing a program for the Apache Web server called mod_pLua, the best way to describe it as if PHP and Lua had a baby. So it would be the same for people that know PHP. It would be the same structure with the less than equal sign and a question mark, and then the same thing to end it on the other end, but with the Lua language instead of the PHP language. So I wrote this program or interpreter for the Apache Web server. And I didn't really think much of it. Obviously it was mostly for my own edification if you will, and for my own use. But I had put this on a site called SourceForge, which at that time had a community manager named Rich Bowen (also Apache HTTP Server PMC Member) who took a liking to this program or this module for the Web server because the Apache Web server community, which he was a part of at that point, have been doing something similar called mod_lua or at that time mod_wombat. And that had stalled. People have interests and then the interests wane and people would move on to new jobs and the person in charge of this mod _lua had found other interests in life. And so this module was just sitting there and not really being worked on. And Rich said, "Why don't you come take a look at this program and maybe this is a place where we can collaborate." And he also got (ASF co-founder and Apache HTTP Server PMC Member) Jim Jagielski very interested in the work I was doing. And so I slowly started on my path to becoming an ASF Committer initially by fixing what's called 404s, which is basically a reference in a Webpage to a link or another page that doesn't exist. Either it never existed or it doesn't exist anymore. So I started fixing a bunch of those just to get on their good side and hopefully they would take me seriously. And I didn't have high hopes, but I think I was probably the fastest person to get committership at the Apache Web Server Project...perhaps the fastest in the 10 years preceding when I got it probably within a week. They had a vote going and I was voted in and… Within a week? Within a week. Unheard of. I was pretty much on the path to becoming a Committer. I couldn't believe it. Part of me wanted to believe it, because it was a very big validation for me. Because I had been using the Apache Web Server since 1998 and it always been a project that I looked up to and it had been this mythical "Father of the Web” program. And so to actually be a part of it and get your name on the page that says these are the Committers that actually have a say in the project and can commit code to it, that was I was quite a feat for me, especially at that time where I had stopped my studies at university and I didn't know, what am I going to do now? Because as happens with a lot of people that study something, they eventually found out that while, okay, this was interesting, but it's probably not what I want to be doing, if I'm honest. Because what I had fun with was programming. So while it was nice knowing a lot of stuff about statistics and economic models and psychology and so forth, it had started to get a little boring for me. I knew these things, what now? And so to get this validation to get an avenue of sorts where I could use my creativity in a new way that I hadn't studied for, but it naturally just came to me this programming inspiration, that was really nice, to use a very vague word. It was a tremendous opportunity for me. And then that's how I got started with Apache. Fantastic. You’re not only a Member of the ASF and an Infra team member. What other "hats" do you wear at the ASF? I have a couple of hats. I’m also the Vice President of the Apache Web Server Project, which is a great honor. And it's still to this day, three years in, fun to do. People think of it as this is a dictator role or you get to decide, but it's more of a glorified secretary really, where you keep tabs with everything or most of the things that are going on in the project. And you relay that information in a concise way to the Board of directors, whose job it is to look at these reports and say, "Are the projects doing okay? Do they need any help from us? Are they in trouble?" So basically VP is the watchdog --in these COVID days, I guess you can say it's a pulse oximeter of the project. And if you want to know if a project is still healthy and stable and progressing, the VP is the one to ask, because that's basically their job to know. As VP I don't get to decide who gets in or who gets kicked out or what direction we take in the project: I am just the person that ensures that the Board knows that the project is in good health. Do you wear any other hats or is it just the VP of Apache HTTP Server? I'm also VP of Apache STeVe. As I said, I have two VP-ships and STeVe is a whole other beast. Let's say it's very stable in that we have a code base that works and we don't really do much about it, we maintain it. In the Apache Web server, we have around 20 to 30 people actively contributing code every single quarter. And in Apache STeVe, we are basically twiddling our thumbs, waiting for something bad to happen. And it never happens. We have a program that works the way we like it. And we don't see the need for any large changes. And as long as there is sufficient oversight in a project, then the Board doesn't come in and say, "Hey, can you make this cool feature?" Because that's not the Board's job. The Board’s job is just to help us, as projects. And so if the project doesn't have anything that it feels it wants to add, but it's still there and the people are alive and well, then the Board will say, "You got it. We'll see you next quarter." And so two projects are very different and it also makes for very different reports. OK. Let’s drill into Infra, as that’s the focus of our interview series. How long have you been a member of the Infrastructure team? How did you get there? I am not sure. I think I've been a member of the Infrastructure Team since 2012. You can probably figure out when exactly I got my membership in the email archives . It started because the Apache Web server project needed a commenting system. Because we had been eyeballing the PHP project and they had a system where you could, on the documentation pages, you can enter, I have a comment about this documentation bit, or you could add some code snippets or ask a question and get an answer. And the only thing we had was send an email and get a reply and then the next person comes along and doesn't know that that email existed and sends the exact same question and gets the exact same reply and that can get tiresome in the long run. So we wanted someplace inside the documentation itself where you could go in and see, okay, I have an issue with this documentation, have other people encountered the same problem or are there some smart solutions that I can find here. And this type of software doesn't write itself, unfortunately. So I set about in writing that using the mod_lua that I had now invested a great deal of time in because A, we needed a comments system and B this was a good excuse to show off mod_lua in a production system. This could really do something, it's not only fast, but it's got a lot of features and it's got a lot of flexibility to it. And so I asked the Infra Team, which at that point was very daunting for me because they were, let's say our image has improved over time at Apache Infra, it was much more a, well basically an operator from hell vibe you got back in the early 2000s or early 2010s from the Infra Team, especially when you're someone of a more timid nature like I am. So anyway, I asked if I could get a place to set up a machine or borrow a Web server basically and put this commenting system on that I had been writing as a hobby. And they pretty much said "Sure." Which was surprising to me because normally when you go and ask for something at a company and it's very difficult, you can ask for meetings and meetings and meetings, but if you ask for actual resources, you will usually have to file a form J/99-B in triplicates and whatnot. And here they were just: “well it looks like he wants to help the project, just give him what he wants”. And so I got started on this commenting system. And other projects became aware of it and they wanted to use it as well. And then I became the comments guy, basically. And I started maintaining this system for, I think, it was seven different Apache projects at the time using it. And since you can't really maintain anything at the ASF without somehow being an infrastructure person, I was made an infrastructure person...and generally if you're a given something, you get a taste for it and you want more. And so I started volunteering for more and more infrastructure tasks. And then I became what is called infrastructure root. This was about two years later down the road. Which is a point where the Infrastructure Team says we have complete trust in what you do. Here are basically the keys to the kingdom. Do whatever you like, except don't do that. But you could do whatever you wanted to. And that was almost as awesome as becoming a member, which I had become just about a month prior. It needs to be said that at that point you could not become infrastructure root unless you were also an ASF member, because needless to say, when you have root access to an organization as wide and important as to the ASF, you get to be privy to a lot of information that you should keep to yourself. And so the logic at the time was, if you are an ASF member, you will already have access to most of this information because of your membership and so we can allow you to become an infrastructure root person. This has changed since then, we have cast a wider net when looking for new infrastructure people, this also includes a more thorough vetting process that we have now. So we feel more secure and not just requiring you got to be an Apache member before you come and help us. So we are able to look for a broader set of requirements that might not have been found within the, at that time, 400 and something members that were in the Foundation. What are you responsible for in ASF Infrastructure? Oh God. As with most infrastructure members, it's almost easier to see what are you not in charge of, which I usually say “Jenkins” with a big smile because that's things that are, I know this is going to sound silly to a lot of people reading the article, but things that are Java, I tend to shun like a vampire and sunlight. Any particular reason? Yes. I'm not accustomed to the way the output and stack traces and core dumps. And the thing about Java is it's very verbose: you can write 50 lines of code and you'll have a print Hello. And it doesn't appeal to me. So yeah, when things don't appeal to me immediately, this is one of my weaknesses, I try to not really understand it because it's easier not to. Fortunately we have some very talented people at the Infrastructure Team that knows pretty much everything there is about Jenkins and JIRA and Confluence and all the other big Java powered mod lists we have at the Foundation so I can spend my time elsewhere. What I mostly do at the Foundation day to day work aside -- because we all have basic maintenance tasks and disasters that can crop up from time to time -- is product development of the glue that binds The Apache Software Foundation together and its software infrastructure. And I'll tell you about a new thing that we've been doing, which is something called PyPubSub. I can spell it, it's P-Y-P-U-B-S-U-B, so it's a Python publisher subscriber service for the ASF. You can basically think of it as a newspaper where you have a publisher, you have an audience, you have the readers, and then you have topics of interest. Some might want the sports section or the funnies, or someone might want the financial news. And then you have, of course, the writers or journalists that make up the contents in these sections. And at ASF, these sections, they would be Subversion commit or Git commit or a new email being written, or someone got added as a Committer or someone filed a pull request, someone filed a new bug or issue, or some are discussing an issue. And the writers and journalists would be all these systems where you send an email to, or you open up a new ticket or you commit some code to it. The readers will then be either users, or there will be a lot of different software components that rely on these messages in order to operate themselves and do what they're supposed to do. So in essence, PyPubSub is, again, some glue that binds the majority of our services together. And it does so by dispatching events to basically whomever wants to read about them. We actually have something called a Pub/Sub Explorer, in real time shows every single event that happens at the ASF technology wise. So if someone sends an email to us, if someone commits something, if someone opens the poll request, if someone comments on a discussion, it all shows up in this Explorer that will update in real time. And it's very cool. (ASF Infrastructure Administrator) Greg (Stein) was saying that you do things that are uniquely different from other team members. In addition to the PyPubSub, what other things are you working on? Currently, one of the main things we manage is called technical debt, which is basically, the longer you don't maintain and upgrade a system, the more expensive it's going to become once you finally have to do it. And so I'm dealing with some technical debt that is moving the service that we have called GitBox from an old, pretty ancient set up to a brand spanking new 2020 machine and software, which also means moving from Python 2.7 to Python 3.8 for every single component that is in the service called GitBox. And that is a lot of components. GitBox is the ASF side of where a committer would commit code to if a project uses the Git version control system. The other side would then be GitHub, if a project chooses to use GitHub. And GitBox and GitHub, they kind of talk together and figure out, okay, someone pushed to me, I'm going to synchronize this with you. And I'm also going to make sure that everyone gets an email on the mailing list saying "something just happened." It's rather unique in that you can choose to either use a GitHub account, or you can choose to say, "I'm not going to use GitHub. I'll just use my Apache credentials on the Apache server instead.'' Not a lot, very, very few, in fact, organizations have this kind of interconnectivity between GitHub and a locally hosted git server. And what we have done very neatly is, we have managed to link our LDAP directory of all our committers to GitHub. Meaning that, if you go in and say, "This is me on GitHub.'' We automatically figured out, okay, that means you get wide access to this and this and this repository. And that is updated in real time. How did these out-of-the box projects come about? I remember when you first approached me about five years ago with these fantastic stats just before I was going to publish the Annual Report. I’d never seen anything like that at the ASF. It's difficult to explain. It's like asking a painter, ''Where do you get your inspiration from?'' It just happens. A lot of time --I will tell a little secret-- a lot of the time that I spend in my day-to-day work is not spent actually typing code or reading up on new fun things. A lot of it is spent what you would call idling. And by that I mean not particularly engaged in any specific task, but kind of just all over the place casually ... Like how, and I hope not to cause any offense here, but how a standard office worker would spend a lot of time on Facebook catching up on friends and family. I'll just spend mine to see whatever I'm interested in the moment that has to do with programming or mathematics or psychology. And in the back of my mind, there's always, how can I take this information that I'm reading about and apply it in a software world? My mind has a tendency to see structures that may or may not be there. And I think almost exclusively in structures. Whenever I see something I want to understand not just how does it work, but how is it basically designed? And can I replicate that? And so, a lot of my day-to-day work is, I see something cool, it might not be anything that has to do with software, or the internet, or anything. It might just be a cool gadget, or a painting, or a chart in a newspaper. And I'll be like, ''What can I use that for that would benefit the foundation? Or whatever hobby project that I'm working on?'' And then you get these aha moments where you're like, ''This I can actually use this way to fix a problem that we are having, or that problem that we could have.'' Sometimes you just make up problems that will potentially happen in the future, just so you can have an excuse to get started on something. And for some strange reason, these fictitious problems very often tend to be not so fictitious at all. And once you show three or four people, hey, I thought of this thing that's not actually a problem. And I thought of a solution. They'll be like, ''That is actually a problem for us.'' And suddenly you have a solution to a problem that you didn't think existed in real life, but it actually does. So, a lot of the things I do are “for the fun of that”. But there's always a work-related starting point in that, is this something that can be used within the software world? Or within the managerial world of software? Which is where I primarily tend to focus my energy. In terms of your day-to-day work with the Infra team, you said that you’re hands-on, not necessarily coding specific tactical solutions, but solving other problems --do you participate with the firefighting as do the other team members? You often respond to my queries about mailing lists --is that your specialty? Chris and Drew shared that everyone specializes in at least one thing. What do you specialize in? My focus is primarily, and there's this kind of a self-made problem. My focus is all the programs and services that I, unfortunately, created. You create it, you own it. Yes. There are a lot of services at Apache Infrastructure that either I made from scratch, or they have a very big thumbprint of mine on them. And so, when I started at Infrastructure, the Infrastructure team, it was expected that we do our fair bit of firefighting. We do a fair bit of the tasks that every single member of the infrastructure team knows how to complete. And I will go through tickets and I find tickets that I find manageable and complete those. I will participate in firefighting. I will do whatever I feel needs to be done right away. If there's something important, or if there's something where I feel like this should have been dealt with by now, I will do that. But it was also the expectation that I come in and help develop and maintain a lot of new features we were looking at creating for the committers and for the end users of Apache software. Simply to make for a better user experience and an easier workflow for our committers and contributors. So, a lot of what I do is maintaining and assisting with services that I have either initially authored or helped expand upon. Tell us about the structure of the Infra team --how did your work come about in a formal way? You were saying that you're creating these tools and then they just kind of got integrated. But were they looking for your sort of skill set? Or was it more of, “hey, we need another Java guy”? What clicked there? Your background is really different. Your expertise is different. Your insight is different. It's an unusual scenario to have a traditional department embrace someone like you and say, ''Hey, we're going to have a whole new type of services offered based on this one guy's vision.'' That's very unusual. Can you elaborate on that a bit? I don't think they were looking for someone like me. But I think they got someone like me and it was completely happenstance. The Infrastructure team at that point, that was early 2013. They were looking to expand with one more staffing spot. This was a part time job. And this was probably about a year and a half after I started doing things for the Infrastructure team. And they had a very narrow list of candidates at the time, because it was a very closed circle. And kind of still is because when you're a staffer, you get the keys to a very mighty kingdom. And so, they had a few people that they could consider, but I was probably by far the one putting in the most hours. And I will, gladly admit that, at that time, I did not have a job. So, I was able to put in a lot of hours. This was when Sam Ruby was VP infrastructure. When Sam initially took me aside and said, ''Hey, we are looking for this part time opening, are you interested?'' I was like, ''No, this, surely you're not, you can't be serious. There's got to be someone that's actually qualified for this job.'' I didn't consider myself qualified at all. And... But you were doing the work? I was doing the work, I just didn't have any confidence in the work I was doing. You can be creative, you can do a lot of interesting things and still have this incredible imposter syndrome going on at the back of your head saying, ''Someone else is doing this work. It's not you.'' So, I politely turned him down and said, ''Thank you, but I'm not insisted because you'll just find out I'm a fraud.'' It actually took two other Infrastructure at that point, current staffers, two other sector members to yank me aside and say, ''What are you doing? We want you for this job.'' And they had actually pretty much all internally, independently been rooting for me and trying to position me to become this new member of the team, to my great surprise. After, I think it was after a very long talk with (former Infra team member) Joe Schaeffer, I was finally convinced, maybe I should give it a shot. And I'm very glad that he convinced me. I'm very glad that the other people at that time also convinced me because it's now been, to this month, seven years since I started. And it's not been fun every day because there can be such a thing as too much firefighting going on. But it's been interesting every single day. You're never bored and you never think, ''I need to find a new job.'' Because you are respected for what you do. You are rewarded in more ways than money, honestly, and you can probably agree with this, at The Apache Software Foundation you get a very unique sense of loyalty. Not to the Board of Directors, or to the specific projects, or anything else, but to the community as a whole. To the mission that we're doing. So, I am honestly very content being where I am. I'm very happy that these people ganged up on me and, basically, forced me to get a job that was... It was kind of silly in hindsight because it's a well paying job, it's part time. So, you don't have to spend nine hours a day on it. You can work whenever you want to and... There were no setbacks except for this nagging doubt that people are going to find out the real me. Which, as I discovered myself, it turned out the real me was actually kind of awesome at this job. It's interesting because the Apache community tends to not want someone if they're not good. So, it's testament to your skill set, and who you are as a person, you're liked. You're very well liked. Thank you very much. And, you're right. The Apache community seems to be very good at finding talent, and also very good at rewarding it in ways that make that talent stick, and make them interested and continue working within the ASF community. I think that's a thing that you don't see in all software communities. We learned from (Infra team members) Chris (Thistlethwaite), Drew (Foulks), and Greg (Stein) about the scope of the work that Infra does. How is the ASF different from other Open Source foundations from an Infra perspective --are there other people doing what you do, or how our group performs, or the services that our group provides. Is this common in other Open Source foundations? It is not common in other foundations. We are different in that the breadth of the amount of services that we provide for each project. And especially at the budget that we provide it at. I think we did a count back in 2015 and it was something around 52 different distinct unique services that we had, that we were running for all projects to use. And in between these, there are possibly more than 300 machines each running, some of them running the same thing on 10 machines. And then you have another 10 machines that are running 10 different things. This is all handled by what? A team of what, seven people now? Six people actually, five of us and Greg (Stein). Greg is a bit of an übermensch, so yeah. That's amazing, in terms of the workload. It can get hectic, and I will not deny that, but we have a very, very strong cohesion. I don't want to say we finish each other's sentences, but when someone has a problem, the others know when to step in and help, when to back off, and what to do while someone else is doing their thing. We compliment each other really well. And we have a nice set of tools to help us with managing things, making sure that everything is up and running, diagnosing when something goes wrong. We have a lot of, again, by the hand of me, a lot of custom tiny services that you never even hear of or see if you're not within the infrastructure team. But that goes on automatically. Let's say you're abusing someone in a ticket multiple times, or you're spamming, whatever. We have a lot of microprocesses that go in and detect abusive behavior, both in terms of spam, but also what you would call technical hardware abuse, where someone is repeatedly using all of our bandwidth, for example, or causing the CPU to spike. We can go and detect that automatically and pull a systemwide ban on you, which it's very custom, but it saves us a lot of money. I will say that we've saved a lot of money at the Foundation by being smart about what we do and not being afraid of making a few mistakes while we make new things... Because a lot of what we do is custom-based, custom-made. Because there is not, unless you're talking about something big like Kubernetes or something at that scale, it's often very difficult to find the tools that do what we want them to do with the problem that we have. Because other companies, especially companies, are not the same as ASF. They don't have 300 different departments that all have their own little tools that they want working in their specific way. And they want this to connect to that, and that's connected to some other thing. We are not afraid to create custom solutions, we're not afraid to get our hands dirty and we're not afraid to make mistakes. That doesn't mean we make mistakes all the time, or that we're okay with all sorts of risks. How do you interact with the team? How do you stay motivated? I stay motivated by interacting with my team, I would say. Interaction is mostly on Slack, which is, for those that either don't know it or pretend they don't know it, is an instant messaging platform. We have an account for the Foundation; we have our staff channel where everything gets discussed, whether that be, the mail servers are a big backlog, or this prime rib I just sous vide-ed at 105 Fahrenheit four or five hours is awesome. I think one of the tricks or keys to success for teams like us is to really mix up the subjects and not be all business and not be all fun because you don't want it to be too boring, you don't want it to be too relaxed. I think we've somehow managed to hit a pretty good ratio of fun and serious items that we discuss on a day to day basis. So, it's fun talking to your colleagues about real-life stuff that isn't work, but it's also rewarding talking about work and learning from them and their experiences, and you being able to give them some work experiences and wisdom from your many years of being a sysadmin or infrastructure architect. I think we've hit a really good ratio there. It's an interesting perspective with that because everyone I’ve interviewed thus far has given the same answer. Can you describe your typical workday: now, I know some people don't have an exact schedule, some people do. What's a day like in Daniel Gruno's life? My typical workday is very atypical for a worker. I don't have a set schedule. I don't have a set time. I don't have a minimum amount of hours I work. I don't have, unfortunately, a maximum amount of hours I work. It all depends on the day and what happens during that day. As said earlier, a lot of what I do is developing new services for the Foundation. As such, I spend a lot of time getting inspiration, and that's done through various means of... From idling, I can be working at noon and then I'll be like, ''I should watch a movie.'' And then I'll go watch a movie. My significant other will tell you that's a lie, I don't watch movies. But that was just an example. I can't sit through two hours, I get too fidgety. And that's actually the real truth about me. I can't sit still and do something for a specific amount of hours, unless I'm in a really inspired mood. So, my typical workday is finding things to do that don't take more than half an hour to do, in between suddenly getting the greatest inspiration from up high. I'll be looking at tickets that are easy for me, not absolutely speaking, easy to fix, but tickets that I know how to fix and I'll go in and fix those. I'll catch up on every single email that I receive, which is thousands of emails every single day. I have a mania about inbox zero. If there's an email, I have to read it and sort it. Otherwise, I can't get past the inbox. I can't even close down the mail client unless I know that there is nothing in my inbox. Yeah, it's the same with Slack and IRC and all that. If there's a message pending for me, I have to check it. But that's beside the point. It gives me something to multitask between. Because there will always be a new email, there will always be someone saying something on Slack. So, a lot of my time is spent just multitasking between that, between reading up on news. And then, at some point, the inspiration that I need for that day will hit me and then comes the manic in a few hours where I just code like crazy because I have the inspiration. I tend to form fully thought out ideas which is terrible because if you have a fully formed idea in your head, you know it's going to take eight hours to complete it. But you also know that if you stop, you might forget that fully formed thought. Sometimes a work hour day can be four or five hours and sometimes it can be 10, 12, 13 hours because my muse has sung to me and the inspiration just has to be translated through the keyboard and into some sort of code or what page or documentation or just a specification for a new idea. Having said that though, don't pity me because I work 12 hours a day and don't be jealous because I work five hours a day. Because it adds up to a lot of hours on average per month. But I'm also happy to do it because it brings me joy. With this constant flow of concepts and code and inspiration, how do you keep your workload organized? You might be hammering away on a solution and imaging and envisioning something to develop --there's a lot of things happening simultaneously. A lot of people have a hard time multitasking, or focusing on one thing and managing the thousands of emails coming into the mailbox, et cetera. How do you manage that? I would say I don't manage it, but luckily I have family that helps manage it. I have a boss that helps manage it. I'm a very ... I'm on the autism spectrum and some would say that I probably have ADD as well. So I get very easily distracted and can lose focus, but I am surrounded by people that are very good at a) knowing that I lose focus very easily and b) guiding me back to the right path for that day. I think in terms of my boss, Greg's point of view, I think it's a win win because I get guided back on my path and I get to actually do something useful and not just 20 unfinished projects. And he gets some services that are working and are improving the use of experience of the people that we are there to support: the committers. So serving 350 Apache projects, initiatives and their communities, like how busy are you? How many requests do you receive a day? How do you prioritize these requests? How do you do this? Greg, Drew, and Chris talked to me about JIRA systems, et cetera. Your work, as I understand it, is not necessarily responding to user requests. How do you fit the creativity in with this process? How do you mitigate that? How do you fit everything in? I do respond to users to keep me busy because if I am, I don't want to say stalling, but if I am really idling then I lose interest so I have to always keep busy with something. So I will grab a lot of tickets just to keep busy with that. That's the thing that I had to teach myself how to do. And I don't have the recipe for it, and yet I have somehow taught myself. The thing where you have to not click on every single new ticket that opens up. And not read every single ... Well, you can read email, you just don't have to write a reply to every single email. It took a few years, I think, for me to stop doing every single ticket that came in within five minutes of it coming in. Because at that point, if you do that, plus you have 10 different projects on the side, you get burnout very quickly. And I've had a few burnouts, where I've been unproductive and doing nothing for the next week because I'd lost all hope in humanity because of the amount of tickets and angry users. So a lot of it just letting go and knowing that there are team members who know just as well as you do what this is about and how to solve it. And if they don't then they will ask you and you can help them then. So a lot of managing the workload is learning to let go of the workload. And if someone creates a ticket saying, "My forwarding address doesn't work." It's probably okay to wait more than five minutes before you fix that if you're in the middle of something. I used to be of the, not opinion, but yeah, I used to be of the opinion that this must be fixed right away. The minute I saw someone had a problem, I wanted to help them. But there comes a point where the more you try to help someone, the less you're actually helping them in the end because of the overhead of dealing with too many tasks and being burned out. I think some of it is ... Stefano… Mazzocchi? Yeah. Right. The Mazzocchi equilibrium. There is a certain point where in the effort you put in and the effort comes out of that starts to not align anymore. And so if you're not good at holding back and letting things slide just a bit, then you cross that threshold and you end up putting in maybe, I don't know, 12 hours of work. And really what you are doing is five hours of work or four hours or one hour of work because you're so not interested in what you are doing. I know that some of my colleagues use Trello or If This Then That, or other tools to organize their day, but I want to say that's not for me. I don't think it factors in the creativity that is needed in the role I have. I think without any scientific evidence whatsoever, that if your job is to think up new ideas and think of new ways to do something. These tools, they don't necessarily account for where creativity comes in because you can't put in your calendar: step one, “be creative”; or at 9:00, “be creative”. Creativity is something that just happens. I've found that it happens for me when I am idling, when I am doing a lot of non-work related things, switching between and then switching back to work. And then switching back to non work items and switching back to work. And then suddenly, a link appears between these two things and they're like, yeah, this idea could actually be used for work. But the things I am doing are not something that you can put into plan because you don't know ... I mean if I knew how to be creative, if I knew to just go to this Website, then I would be a millionaire by now. So I don't know *how* to be creative: I know that I can be creative and I know it happens when I let it happen. You have to make space for that to happen, right? You have to allow for that to happen. It's great that you have flexibility to be able to do that in your job --that part of your work is to be able to conceptualize and visualize and come up with things. It takes a while because sometimes you're not going to know the problem unless you're in the middle of it: "so, oh this is an issue … here's an opportunity for us to come up with something that'll help." It's great. And it's especially great because I think honestly if I was stuck and let's say I was doing human resource management or whatever that I studied for, if I was stuck doing Excel spreadsheets, for example, all day long ... Not that that's a bad thing, but when creativity suddenly hits me, I have to get it down on paper or it's going to haunt me to an extent where I just can't stand myself. So I'm very fortunate to have a job where I can, fire fighting aside, I can say, "Boom, I have this inspiration suddenly. I need to focus on that." And then I can go and focus on that. And I have a boss and I have a boss's boss and I have my colleagues that are understanding so that suddenly, "Oh Daniel got inspired. He's probably going to manic for the next eight hours just working on this idea he’s got." It's really wonderful being given that space to be creative because I think no matter what job I have, there would be an urge to be creative and to think up ideas. And again, when I think of an idea, it forms itself completely in my head. Some people will start with half an idea or a fingertip of an idea. For me, it's mostly been the entire idea presents itself to me right away, and I have to get as much of that as possible down on paper before it's lost. To have that opportunity is really wonderful. [END OF PART ONE]

Monday August 03, 2020

Success at Apache: I Became an Apache Solr Committer in 4,662 Days. Here’s how you can do it faster!

by Eric Pugh

On April 6th, 2020 I was invited to become a committer on the Apache Solr project.  My journey to becoming a committer started in earnest 4662 days before that!  On July 2nd, 2007, I opened SOLR-284, a ticket for adding content extraction to Solr. 

A committer on an open source project under the Apache Foundation umbrella is someone who is trusted to contribute code to the project and to help manage and drive its ongoing development. It’s an honour to have been asked and I was very proud to accept the invitation!

So, you did the math, and you realized that it took me 153 months, or 13 years (rounding up), to become a committer, and you’re wondering “What if I don’t want to wait that long?” So here’s my quick cheat sheet on ways to become a committer on an open source project, illustrated by my own journey:

  1. Start by learning the culture of the project. How are decisions made? What tools do people use? What do the various acronyms mean? Join the mailing lists and read every commit.
  2. Start small and work your way in.  Some great ways to do this are to:
    • Take existing patches and test them.  Update them to the latest code base.  Document what you’ve learned
    • Take advantage of being new to a project to bring fresh eyes to the documentation.  Every time you find yourself scratching your head on how something works, contribute a fix to the docs.   It’s a powerful way to immediately contribute.  This is the fastest way to get involved and involves the least cognitive load!  See SOLR-2232 or this email thread.
    • Answer questions on the mailing list! Being able to articulate reasonable responses to questions demonstrates how much you have learned.
    • Bug fix, bug fix, bug fix! Pick bugs that have an obvious answer so that the “correct” solution is easy to figure out. If the right approach to solving it is very ambiguous, you probably won’t get much traction. Remember to remind committers to apply your fixes when they have the time! See SOLR-13965 and SOLR-11480 and SOLR-2611 and SOLR-2263.
  3. Ready to start slinging some code?   Don’t go and refactor the core foundations of the project (at least not yet).   Instead, be like a pilot fish and latch onto one of the core committers who is being very active in the project.

Embrace their vision, and start picking up tasks related to whatever major chunk of work they are doing. Write some unit tests. See about opportunities for refactoring. Do some manual testing over multiple platforms. Once they see that you’re contributing (and accelerating what they are pushing), then work to get some of your own tickets assigned to you under that vision. I’ve seen this lead directly to committership many times, and if I had followed this route, I might have joined sooner!

Here’s to the next 4,662 days of being active in the Apache Solr project!

Eric Pugh is a member of the ASF and a committer Apache Solr. He co-authored the book Apache Solr Enterprise Search Server. Eric is co-founder and CEO of OpenSource Connections, where he helps OSC clients, especially those in the ecommerce space, build their own search teams by leading projects and by acting as a trusted advisor. He also stewards Quepid, a platform for assessing and improving your search relevance.

[this post first appeared at ]

= = =

"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works" 

Friday July 17, 2020

Inside Infra: Greg Stein --Part III

The close of the "Inside Infra" interview with ASF Infrastructure Administrator Greg Stein, who shares his experience with Sally Khudairi, ASF VP Marketing & Publicity. 

"Apache is growing: we're just seeing the demand explode and it's a hard problem for us to solve."


We were talking about ensuring that the team is up to speed with everything required of them...

So there certainly are skill gaps; this is one of the things I want to help motivate the team with, where if somebody says, "Hey, I want to go and investigate Ansible as a potential Puppet replacement," I say, "Go forward." 

This would be similar to Google having their 20% projects. I'm sure you've heard of that.

Oh, yeah.

It's almost the same where it's not 20%, maybe 5%, but it's the same as Google, no matter what they want to tell you, because everybody's got their job and you have to be really rigorous to carve out 20% of your time. And strictly speaking, it does actually make your Google manager a little upset if you carve out the entire 20%. But anyways, the concept is similar.

So for us it’s like, "Well, go in and investigate Ansible, see if it'll work for us and put your notes into the Wiki." That's how we make forward progress, up our game, and learn new skills. If someone says, "I want to go and figure this out," the response is almost always, "Okay. You go do it." There's certainly an allowance for people to learn new skills. But most of the time we simply rely on, say, Gavin (ASF Infrastructure team member Gavin McDonald), knowing more about JIRA configuration than the other guys.

That added component of sharing what you know, and adding it to the JIRA or to the Wiki actually is great because then everyone's learning. This is like the rising tide: everybody's learning about this, whether they're doing it perfectly or not. I think this is a very interesting process.

Yes, and that's also where Andrew (technical writer Andrew Wetmore) is helping us out. He’s organizing that information that we have learned, that we have documented, that we memorialized into the Wiki.

Because our (ASF’s) legacy is quite Medusa-like over all these years, it's interesting to see how everyone can get caught up and also have to go back and deal with the legacy, but you also have to be able to move forward. To be able to bring others with you is brilliant. That's really cool.

The infrastructure has grown organically over 25 years from when Brian Behlendorf first said, "Hey, I have this server called you can run a CVS repository on it for the Web server."

That computer was under his desk at the Wired offices way back when, wasn’t it...

Yes it was. And it's just grown organically over those 25 years. Then we had Minotaur and it did six different things ... now it only does half of one and we've moved the stuff out onto newer machines and newer processes and this and that. But the organic growth means that we've got some really hairy stuff. Our move to Puppet --first Puppet 3, and now to Puppet 6-- at each step we're improving it and making it less hairy and more manageable and something that somebody can come along, look at, pick up and run with it from there. That makes it a lot easier, so that we don't have to spend 100% of our time cross training.

What are your thoughts on products, the hype cycle, where everyone's demanding Kubernetes, to use that as an example. Do you decide which products to provide support for, or is that up to Apache projects in the communities? You mentioned Ansible, just not too long ago, that was your internal decision to move. But I remember not long ago, GitHub entered into the landscape. How did that happen? How did you decide to make a move like that? That's a significant thing. Can you tell me a little bit about that?

It's a lot based on community input. So if we see a lot of people asking for a particular tool, we'll like, "Oh, hey, David, can you go and take a look at that and see if that's something…” Not David (ASF VP Infrastructure David Nalley), but Chris (Infrastructure team member Chris Lambertus) or somebody else. "Can you go take a look. Is that something that we can support? Because we're getting some queries about it."

And there's a little chicken and egg problem there that if the communities don't know to ask for the egg, we don't know whether to prep the chicken. It's like, “okay, wait, they don't even know to ask for a tool because we haven't said we will make this tool available, because we're not going to make the tool available until somebody asks”. But sometimes people file tickets like, "Can I get this set up?" and we'll go, "No."

Then six months later, somebody else will file a ticket: "Can I get this set up?" and we'll go," No." But after enough of those, we're like, "Maybe that's something that we really want to do." For GitHub, specifically that’s what happened there. Well, even before that Git, where we ran our own Git server, that was a volunteer that made that happen. That was, six years ago or so.

Well...the volunteer came along and said, "Well, I'll do this. I'm not going to take any time from Infra." There's been a couple things for the past few years where I've told people, "No, Infra will not work on that. But if you want to volunteer or find a volunteer, then we'll stand it up for testing." You know what I mean? Why not? So there's a couple things where people have stood up for test examples and there hasn't really been a lot of usage.

So, we're not going to support that. But something like Ansible is our own internal workflow and the tool we’ll experiment with, then to see if it'll improve our stuff. But from the community, they pretty much have to ask and it has to be a sustained ask. That's how we ended up with Travis CI: we actually pay for capacity in Travis CI, and that's based on community input.

So many people wanted to do their continuous integration through Travis that eventually we decided to pay for it. But it's tricky because some of these systems like Travis CI and others require certain permissions that we don't want to provide to the community. So we will want to hold those only within Infra. And so it gets hard to integrate certain tools. We've had to say no, but then again, we've found other ways to improve that so that we can lock down the permissions or use a proxy or other ways that we can route around some of these issues and then integrate the requested tool.

So further to that, have you been in a situation where a project or a community has made unreasonable demands of Infra or have expectations, where it's like, so over the top or so out of scope, it totally surprised you? Have you had something like this?

Nothing surprises me.

Nothing surprises you? Okay. Have you been in this situation? Like “was never going to happen”...

Yes, yes. There's been several times where one of the guys on the team is like, "Oh man, I got this ticket. I don't know what we want to do with this. Greg, go take a look." And I go and look at it and that's where I make that call: "Okay, is the Infra team going to take this on, or do I just say ‘no’ right now?"

So, yeah, there's been a number of times where I've said no and probably two or three times where I've gotten a little bit of pushback on that no. I say, "My answer is no, but here's how you escalate." I've had escalation a few times and I'm actually, mid-process --I'm dealing with one right now. So, I've said, "no, if you don't like my no, you can go to VP Infra and VP Infra is, probably going to tell you the same thing. And then you can go to the President. Right now those are actually the same person."

The same person is a double "no".

That really is the true escalation path. I have to describe that to people and say, "I don't think you're going to get what you want." If I'm the one that says, no, you probably are not going to get it because VP Infra and President, and after that is the Board. They're probably not going to say, "Greg is wrong. Yes, we'll give that to you." But it's there. There's been a couple of times where I said "No, you have to ask the Board for the budget for those additional virtual machines." They went to the board and said, "Can we have budget for three machines?" and the Board said, "Yes."

So Infra went ahead and gave them the three VMs that they had initially requested. Strictly speaking, we would track those machines against their budget, but that detail is more than what the actual budget was. So we don't spend that time doing that, but I have had to say, no. I have had to... There was Apache Maven: they were keeping a copy of Maven Central, and Maven Central is run by Sonatype...

Which is a commercial product...

Yes. They're using the trademark “Maven”, essentially a licensing agreement from us, a MOU. So with Maven Central, you could imagine if someone decides to just turn it off one day ...we wanted a copy. Apache Maven was making a copy of it, and it just started consuming so much disk space. We were like, "We can't support that growth rate. We can't support that even for the next six months. If you want to keep doing it, go ask the Board for money to keep doing it." They never did. We turned it off.

I wouldn't call that a ridiculous request --it was something where we didn't have to just say, "No, not going to do it. Bye." A lot of the requests are mostly just, "We aren't going to run that extra software. If you want: ask for a VM and you can run it, but we're not going to take responsibility for it."

Over the years, obviously ASF Infra has changed. Was this all reactive or was it also proactive? Do you plan for those changes as you go or has it all been in response to Project X or in response to X emergency?

The growth of Infrastructure and its movement from volunteer-only to paid staff was part of just the growth of Apache. The volunteers could no longer keep up and things, like account creation, used to take sometimes four weeks to get an account. You’d put in a request for an account, four weeks later, it would finally get created.

My gosh, that queue was crazy, huh?

Well, it wasn't even a long queue, it was simply that we didn't have volunteers making sure the queue stayed empty. Today it's down to one, two, maybe three days, and the account is created, because every day a staff member goes and creates the accounts first thing in the morning.

It was how I said that my day starts with looking at messages on Slack and then reading emails to see if there's stuff to handle. Well, one of the guys on staff, first thing he does in the morning is go and look at account creation. So he's been off and on pondering on a tool to make that easier for himself; he hasn't finished the tool, so he still has to do it manually. That's his incentive.

“Work quickly”...

This is Chris Thistlethwaite. I say, "Chris, we can do something about that." And he says, "No, no, this is still my project. And every day when I run the script, it just makes me remember, I need to finish this."

So when the volunteers could not keep up with the amount of work, that's when we hired Joe Schaefer, then we hired another person, and hired another person. And so it was just trying to keep up with the rate of requests. 

That's how we ended up with hiring six people. And then I'm half a person, like I said, I'm part-time. So, it's just the growth of Apache. I think we're in much better shape than when I started. We're ahead of the curve. We can stay ahead of the curve because one of the things that I can do because I don't fight the fires every day ... that's for all the guys who know their stuff. They fight the fires and I can look at if I need to go and ask for another head count. And that's how we ended up with Andrew (technical writer Andrew Wetmore): “Well, you know, what we really need is somebody to manage all this documentation.” This was part of Sam's (former ASF President Sam Ruby), “If you had some money, what would you do with it?” That's how the technical writer/editor came around, because we've got 20 years of organic growth. We had...let's just call it “organic documentation”. That revamping project is going really well, I think.

So, in what areas are you guys experiencing your biggest growth? As I was asking Chris and Drew, is there like a geographic influence on the demand? We’ve had a huge influx of users in China. Does any of that change the way or what you guys are doing? Or is it just more of everything?

Our biggest pain point, I would say, is continuous integration/continuous development: CI/CD. Jenkins, Travis, CircleCI, and things like this, where people make a change and they want that change built and tested. The more projects we get and the larger the communities get, the more changes and the more testing and the more building and the more this, more, more, more. It's kind of one of those things where it's “expand-to-fit”. So if we gave people 100 machines, they'd use 100 machines. If we doubled it to 200, they'd use all 200. It's just this rapacious need for CI machines. It's very hard to figure out how to plan around that other than just telling the communities, “No: we just don't have that much capacity: if you want to build it, do it on your own machine. You just can't use Apache hardware to do it.”

That's an unsatisfactory answer. That's been one of our hard problems and it's also kind of a newer problem: the development workflow that uses CI probably is just maybe five years old. Before that, certainly, automated building and testing was a thing, but it's really kind of grown into community workflow much, much more over the past five years, and more and more people are wanting to do it. The communities are growing. Apache is growing: we're just seeing the demand explode and it's a hard problem for us to solve.

China is the one case where we see regional issues, and that's because of the great firewall of China. Not because we're getting more Chinese developers, but because they have problems accessing our servers because they're located outside of China, and so we're looking at CDNs, a content distribution network to essentially make our content available closer to China. We've found that even with one of those CDN drop points in Hong Kong, they still have problems just reaching it there in Hong Kong, and so ... and we don't want to buy or lease or rent a server in China because doing business in China is too high of a hurdle for the Foundation. 


You know, Microsoft and Google have to do business in China and they've got a pack of lawyers and a giant vault of money to deal with all the barriers. The Foundation does not, so it's also a hard problem to solve. We think we might be able to do it through Microsoft Azure, that they have a CDN that resides in China that Microsoft has done all that paperwork, so we're looking at that, but as far as regional things, it's not so much that we run into issues. We see Open Source communities in Europe and Brazil and Australia and Sri Lanka: none of them really have any problems because they don't have that firewall. It's not really about the Chinese people, but about the China firewall. 

That's bigger than us. And that’s not something we can fire hose.


We do see little engagement from Japan and Brazil, and that is partly for language reasons and partly because the Brazil community is more about Free Software than Open Source software. 

Yeah. They're very pro-FOSS.

Not OSS. But pro-free. And so, they're going to deal with the Free Software Foundation rather than the Apache Software Foundation.

I see. That’s an important distinction. 

And then you also have the Portuguese language barrier. People contributing from Europe and India, Sri Lanka, etc., they pretty much know English and that's fine. A lot of the Brazilian developers do not know English...this is the same with the Japanese Open Source developers. Japanese and Brazilian, they tend to not know English, and so that kind of isolates them from the larger Open Source world, or Free Software world, in the case of Brazil.

Would we consider localizing anything that we do, or are we going to continue as-is, as the ASF is all English?

The Infrastructure team will not translate our documents to serve those other languages. That's just too high of a bar.

There are a couple groups that have user mailing lists that are not English and that's totally fine, and Infrastructure will... well, you don't have to file a ticket anymore. It's, again, back to “self-serve” on Apache will create a mailing list for users communicating in Brazilian Portuguese, for example, or communicating in Japanese. But Infra doesn't do anything about that, that's just the self-serve tools. We certainly can't support non-English, and I don't think that the Foundation itself is going to make any moves towards that.

Fair enough. So a lot of companies are really struggling to accommodate their teams working from home in response to the Coronavirus and all that. These stay-at-home orders are kind of shaking companies, but from day one, the ASF has always been a virtual organization. Has anything changed with your operation on that front? Has anything impacted the ASF's day-to-day, from this pandemic?

(chuckling) Not at all. I shouldn't laugh, but no. It really hasn't changed. We've been on our team channel for all three years, three and a half years that I've been here, and the world is burning down around us, but we still sit on the team channel.

Now, that said, (Infra team member) Daniel Gruno got stranded in Canada.

Right! He’s still there?

He's still doing work from Canada. This is why when he travels to Canada for two months at a time, I don't care, you know? Because if his butt is in a chair in Denmark or in a chair in Canada, it's the same butt, so, you know...

As long as you have connectivity and a computer, you can do it. 

Right. But if he has to be offline for two months, I'd say no. Or if you want unpaid time off, well, I'm not going to pay you, of course. Certainly the discussions have changed, you know? I mean, going shopping. You know, some members are immuno-compromised and that had an effect on our team meeting that we were planning in Nashville: they were the first to say, “No way. I'm not going,” so, there’s that, but our day to day hasn't changed.

That's more of a social thing versus an operational thing. Safety first.

So the notion of, “Oh, I got to run out to the grocery store. I need to strap on a mask,” changes, but not the operation.

Right. Right. So...what do you think people would be surprised to know about ASF Infra?

I don't know if it'd be surprising, but we are global. We've got four people in the United States, one in Canada, one in Denmark, one used to be in Australia, but is now in the UK, which actually kind of hurt a little bit, because in Australia, that meant that we always had somebody in that time zone, but now we have kind of this gap of Australia/Asia time zones when...

A “Gavin” gap.

Yeah, well, I might be awake at that time, but I can't go and fix a MySQL server, so it does mean that we don't have that straight-up 24-hour coverage.

The notion that we are worldwide is kind of a neat thing about our team, and is what makes us pretty unique relative to other IT departments. I don't like being called an IT department, but that is essentially what we are. 


What's the name of that TV show? The one that's about IT...

“The IT Crowd”, is that what you’re referring to? The British show?

Yeah. So, you know, that's a funny show, but mostly when you think “IT department”, you think of some corporate people with button-up shirts, but ...most of us, we're in our pajamas.

Good one. What's your favorite part of the job?

I definitely like the team and that's why, nominally I'm part-time, but I'm pretty much constantly on the team channel and interacting, and so I think I just put that down as volunteer hours, where before I might work on Apache Subversion, but now I hang out with the team or I write some little tool or something like that. That's definitely been one of the more rewarding changes. Up until I started with this, I'd been a director for 15-and-a-half years, and that was kind of how I contributed to Apache. Now my work for Infrastructure is a new way to contribute to the Foundation. I'm also part of a new community, where before I would hang out with the httpd community, APR community, the Subversion people it's the Infra people and my hobby time is kind of blended in with my work time, and vice versa. I mean, when your work time can also be seen as a hobby time, that's pretty cool.

I do think it's the team that makes it interesting. That's what I like the most, and that I'm working with a new, interesting community to contribute to the Foundation. 

Not only did you switch roles, you switched communities. What was your biggest challenge going into this new role?

I would say probably trying to delineate what I was going to handle for the guys and that I wasn't going to tell them what to do or how to do it. It's like, “OK, I'm here to assist, to unblock things, to enable you guys, rather than to block you or micromanage you.”

To earn that trust, that I wasn't going to be some pointy-haired boss telling them how to do their work. Now, I don't know if that was ever a problem for them, but that was certainly one of my initial concerns: how to properly create my role. This was the first time Apache's even had somebody fill in this role, so I also had to find the role, which is, again, why I came up with “Infrastructure Administrator”, is because I wanted to define it as an enabler role, as an administrator, so they could get their work done but I would not be their manager. I would not be their boss: I was simply there to enable them.

So, what are you most proud of in your infra career to date?

Ooh. I don't know. I would say by being hands-on, being the “hands” of Infra, it means that VP Infra didn't run away screaming.

David said in January 2016, maybe earlier, he was like, “No way. I'm out.” And after I was on the job for about two months, he said, “Huh. All right.”

“I'm in!”

And so I get that feedback from him, “You know, you make the VP Infra hat quite easy for me.” I think that's probably what I really like about taking on the role, is that one of our volunteers got to stay rather than drop it because it was just causing so much anxiety and pain and time and frustration. Otherwise, most of the stuff I do is really boring. Not to me, but I don't have “accomplishments”. I push paperwork, basically, so the other guys can do accomplishments.

Speaking of the other guys, how would your co-workers describe you?

I have no idea. I don't know. I really don't know. (laughing)

Where I just got done talking about what I saw as an issue, trying to frame what my role would be, it might have been fine with them and I was overly worried about it, but it’s hard for me to know. We don't do 360 reviews in Infra, so I don't get any feedback, really, from the team on what they think about myself or how I'm doing my job, so you'd have to ask them. 

I have. Just kidding. So...what are the biggest “threats” that infrastructure managers or infrastructure administrators need to watch out for? What do you think is a “big thing” that people should be aware of, or is ASF so unique that you don’t feel like anyone really experiences what you experience?

There's our capacity issue with things like Travis, but I think you're asking a different question.

I am, but that's fine. What's your greatest piece of advice? What would you tell aspiring infra administrators?

Actually, one of my greatest fears is really, as a small charitable foundation, it's hard for us to compete with well-funded corporations and some well-funded start-ups.

Related to that, I touched on it earlier, is career development go into Google or Microsoft and there's a career ladder; we simply don't have a career ladder. There's salary growth. There's bonuses. If you want to have a resume or a LinkedIn profile that shows changes in growth and titles and career ladder, we can't offer that, and that's going to cut out some people. It's a very hard problem for me to solve. You know, there's things I can maybe do, but I also want to keep the team egalitarian and sort of level, rather than, “Oh, well, this guy is now the team lead.”

Given what I talked about, our social aspects, because we are all equal peers, keeping everybody with the same title, same position on the ladder means that we are peers and it's a little easier to interact that way. It's a real, real difficult problem. You ask what's scary: that's scary.

But there's a counterpoint to that. You may not have a traditional career ladder path, but to say that you've worked in Infra for Apache carries weight. That's significant. 

I believe it does, especially when you can demonstrate the hundred different types of tasks...

Well, that's exactly it. The breadth of work and the scale of what you guys do and the skill sets that you have to have and the fact that you have to play nice in the sandbox, all of it. The demand is immense, so to be able to be there and thrive and develop something from yourself in terms of a career is tremendous. Our team is exceptional. I mean, they're not expecting a linear ladder or something that others have.

You know, in other jobs, somebody might say, “I was a MySQL administrator.” Here, you're a MySQL administrator, PostgreSQL administrator… They had one role; here you've got dozens. 

If you had a magic wand, what would you see happen with ASF infra?

I'd like to solve that CI problem. The other magic wand would be upgrading our mail server from 10-year-old technology to modern technology.

Is that happening or is that literally a wish list issue?

It's happening, but it's been happening for three years. The thing is that email is so central to the Foundation that we can't really experiment with that. There are certain things we can do, but most of it, not so much, and so it means that we're being super-careful. There's about 10-12 different moving parts to it, and we're upgrading each of those a little bit by a little bit, until we can finally pull that big, scary, Young Frankenstein lever to hit the lightning bolt, you know?

Yeah: I see the visual of that.

The magic wand would be to just make that all happen and make it work. Without the wand, it's going to take another 6-12 months.

Right. What else do we need to know that I haven't asked? What should I be aware of or what should I be sharing?

Oh, I don't know. This is where my creativity ends. Ask me a coding question.

Oh no coding questions. All right. Our time has also ended. Before we go, who should I be interviewing next? 

I would say Daniel (Gruno), because his role ... he's 20-30% system administration. The rest is tool development, so that makes his role rather unique in the team.

Perfect. Thanks so much, Greg. I really appreciate it. 

= = =

Greg is based in Austin on UTC -5. His favorite thing to drink during the workday is a big 32oz cup of Diet Mountain Dew.

Monday June 29, 2020

Inside Infra: Greg Stein --Part II

The "Inside Infra" interview continues with ASF Infrastructure Administrator Greg Stein, who shares his experience with Sally Khudairi, ASF VP Marketing & Publicity.

"Who are these crazy guys spread around the world that are keeping 200 machines up and running for all these different projects and committers and contributors?"


How or what would you describe the Infra "brand" to be?

I don't really know. I've never really thought about branding or marketing ourselves, so ...

Well, you guys have a certain persona, you have those funky t-shirts you wear at ApacheCon ...there's definitely some kind of street cred that's different from everybody else. I was curious to see if that's part of your natural sense of hip, or is that something that you guys deliberately planned for.

The t-shirts and other things go back to the team bonding kind of thing. We'll give ourselves an identity, but haven't tried to create or market ourselves. I think it is something that we do need to take some control over. We hired a part-time writer in December and he's been organizing our content to provide a better and more useful front to Infrastructure.

There were a lot of pages on that have now moved over to That creates a more coherent Web space, if you will. We can really talk about those different channels. "How do you reach Infrastructure? Do I go to the Slack channel or do I file a JIRA ticket: how do I decide?" So he's helping to, while I wouldn't say "market a new face", he's certainly helping people figure out who we are, what we do, what we can help with and getting that information organized.

Which is good. That's new. Even to have you guys featured in a project like this, it's unusual and it's refreshing. I'm personally curious, and I'm sure other people are also curious about what's behind Infra.

Right, right. Who are these crazy guys spread around the world that are keeping 200 machines up and running for all these different projects and committers and contributors?

So Andrew (technical writer Andrew Wetmore) is primarily going to work on the infrastructure docs until those are whipped into shape because a lot of the material that we have, a lot of the Webpages, is really infrastructure related. He has been working with the team on those pages. What's going to be harder though is when he's kind of at a stopping point for that, what to turn his focus to, and that would be www.apache. But then it gets a lot more difficult because when he wants to update the How It Works page, who does he talk to? Who's authoritative? He can do some edits for flow and word consistency, punctuation, clarity, right, but he can't really update the process.

Right. Right. That's the Foundation thing.

Yeah. But the problem is we don't really even have a concept of who's in charge of that How It Works page, who is, you know, it's just there's nobody that the foundation is willing to say, "That person controls that process." You know what I mean?

I totally do --I come across the same pages and people go, "Are they yours?" It's hard to determine not only evolving processes, but who signs off on this or who gets it. I hear you.

I've recommended for the past year, or three, that Marketing is the owner of DubDubDub (www.), but you know, that's the "face" of Apache. You know? But the raw content, as you point out, who approves the raw content.

One thing that I asked Drew and Chris, and I'm always curious with people who are super busy and juggling 50 things, is to describe a typical workday for you.

I wake up, I look for email first, generally, sometimes I'll hop onto Slack because sometimes people ask me directly for something. Then I go look at email and sort through a number of different categories between direct team stuff, operations, the Apache Board, and then Apache in general. And then of course, if there's any vendor email to deal with. So there's a bunch of different categories in priority order. After I get through that initial work, then it's go and read all the back scroll in the team channel, which is anywhere from 200 to 400 lines of back scroll ...

Can you get any work done? Beyond just catching up on the communications?

Yes. But it does take like 30 minutes to read that back scroll. For me there's a lot in there about what the guys are doing and what they're working on, how to solve a particular problem when they're asking somebody else, "Hey, can you look at this? Can you help me with this?" But I don't, for the most part, "serve", you know ...they are the technical staff... I can do it: I have technical chops, but I let them do their jobs as they know best. I do like reading the back scroll because I'm also looking at it from the angle of "how is the team working together? Is that going well? Is there something that I need to poke and prod to improve how they're working? Are they getting jammed up on something that I can unblock for them so that they can get their work done?"

Stuff like that. That's what I look for when I go through that back scrolling, so it's important to me to read that back scroll. Most of the guys do tend to, when they first sign in in the morning, go back and scan for stuff where they might be needed. I've never really asked them how detailed they get, but I think pretty much everybody reads all of it to catch up, but they're going to be looking at it with a different lens than how I look at it. Mostly I'm looking at unblocking --are they running into problems that I can ease for them?

How do you keep your workload organized?

I don't.

Fair enough. Again, there's a lot, so it's curious to me, like everything at Apache, with the exception of a handful of things, everything could be a priority, if you're always on fire and always running around, putting out fires, you know? It's funny when I've talked to the Infra guys and you also, you all have the same reaction to that question, which is the laugh. I think that's the nature of the beast with the ASF.

Yes. That really is the nature of system administration work. My career has been product development, and you can reasonably plot that out. You can say, "We're going to develop these five new features, which is going to take us between two and four months." We'll see...we might cut a feature to try and limit our time development. The feature is going to change, unless we'll plan in time for change. But system administration is very reactive, so it's a very different beast. This is where, like I said, we were kind of treading water with four people, but we could see as Apache was growing we were not going to be able to keep up. And we certainly weren't going to be able to move ahead of the curve and do things like where, you know, before we would get a dozen tickets to create repositories and that took time. Now we don't have to do anything.

It's all, but we had to write the tool first and have enough air time to get that tool written. So I think we're ahead of the curve. We're getting some of our longer-term initiatives done, but it is still a very reactive thing. For myself, my back office work is pretty straightforward and it's a lot of email and Website work, you know, going in, paying an invoice, putting in the infrastructure credit card, sending out a purchase order, stuff like verifying and improving payroll, that doesn't require me sitting down and writing Python scripts.

The other half of my job is being present on that channel because I also help to set priorities. When something comes up, I ask, "Is this a thing that we want to do? Do we want to take on this new task? Do we want to provide this new tool to the projects?" You know, like a project is going to say, "Well, we want to integrate this thing into our GitHub repository," and we go and review it. It may require permissions that we simply don't want to allow. So there's some of those kinds of policy kind of things that I also help with. And there's always being present to help set policies and priorities.

OK... so how do you work with (VP Infrastructure) David Nalley? Are you making the decisions? Infra is an unusual type of group as opposed to other areas of activity operationally at the ASF. How do you work together?

Correct: I'm the day-to-day, so I look at it like he's the brains and I'm the hands. That said, he's like the strategic brain and I do all the tactical decisions.

I make all the tactical decisions. I am an officer of the corporation. I can make any decision that I need to, related to Infrastructure. If I feel it's a little bit weird, then I'll bounce that off David, but for most of the stuff, he doesn't feel a need to inject himself in. He feels comfortable letting me go ahead and run with the things, and rely upon me asking when it seems a little sketchy.

That's good: that process suits both of your personalities, both your sensibilities. It sounds like a good fit.

I report to the VP of Infrastructure, and that is still David, even though he became Executive VP and is now (ASF) President. He still holds that title. He's asked me, "Well, Greg, maybe you should just be VP Infra," and I said, "No way." Because we're paid people, but the Foundation is all volunteers. I told him I do not want to be a VP, because I want to report to a volunteer. I think that I (and the team) should report to a volunteer that always has a volunteer eye on the Foundation's long-term goals.

Because I manage all the day-to-day, it's a very lightweight hat for him. That VP hat is a tiny aspect compared to his President hat. One day, he'll find somebody to take over that VP Infra hat, but I've essentially mandated to him that it has to be a volunteer position.

It's not that I see we're going to go all out of control and we need a check from a volunteer; I just want a volunteer to always be able to say, "Okay, you guys are a little bit crazy, let's redirect our long-term thinking more in line with what the Foundation wants," and have a volunteer interpret what the Foundation wants.

That perfectly dovetails into what folks referred to in our ("Trillions and Trillions Served") documentary, where they were talking about Greg Stein's famous "plan for the ASF for 50 years..." This super long-term vision, which again, everyone goes back and says, "Greg Stein said..." What does that mean exactly, and how does that translate to Infra, considering that you can't really plan that far out? How does that work?

Well, actually we can plan that far out. I wrote that "50 years" in one of my Director's statements, I think it was 2014 or 2012 ...maybe earlier. Where I was going in that Director statement was the Board doesn't deal with the communities. The Board is there to support the communities. So we want the Foundation to exist for 50 years so that these communities can continue to run and see through evolution.

Some communities are going to move to the Attic, new ones are going to come along, but we want the Foundation to be viable. To say "forever" is okay. Nobody can really put that in their brain. So I just said, "OK, we can think what 50 years means." That is long enough out, but still within people's brain capacity to think, "Okay, what _does_ 50 years mean?"

And so that's where I came up with that. What does the Board need to think about to ensure that we are here 50 years from now and our projects are successful and can run through their lifetime, lifecycles. Apache HTTP and Tomcat, I don't think they are ever going to go away, but you could see maybe in 30 years they might. There might be some other mechanism in computing that would obsolete them, but the model of Apache does need to exist for at least that long.

Now, within Infra, I think we actually can plan that far out because we have growth curves. We see what kinds of computing resources people need. So we can plan for project growth, for machine growth. We can do long-term planning on how we allocate machines among our various cloud resources that we have, and start to schedule those further out. None of that really affects our day to day, but it is something that we can project out a ways and think about what kinds of resources we are going to need two, three, five years from now.

There isn't anything really that we can do for 50 years, but we can keep it in mind. Okay, that is going to be a larger team. That is going to need a larger staff, a full time manager, a full time HR person, a full time... There's different things that will change over that time, but we can actually do some of that projection, although we haven't bothered.

I do the five year plans for the Board, but mostly that is a simple cost growth as opposed to actually changing the structure of the team or the role assignments, because like I said, I think probably within 10 years, we'll probably need to add one or two more staff on top of the head count of six that we have right now. And I think supporting that would still be fine for a part-time person like myself. But once it grows to 10 or 12, then I think it's going to need a real change. Where we need to have a full-time person managing and so, we'll need to adjust the budget considerably to make that happen.

But if we ever get there, the Foundation is going to be likely in a very different position. We're talking 10 years from now. And so, who knows.

So with more than 350 projects and initiatives as we've discussed before, how do you guys stay ahead of the demand? And again, if you're trying to plan for five, ten years out, you mentioned earlier cloud computing. Not so long ago, cloud computing was a novelty. How do you plan for this?

And that is where we try and move more things to, where we look at the kinds of requests that we're getting. The kinds of tasks that we’re performing and find a way to automate that workflow and create more self-serve options for the kinds of tasks that we regularly get tickets on.

Where we used to get tickets on creating Git repositories, we get zero now and, and we can see over the past six months, we've had 20 tickets to do X, is there a way that we can automate that, so we don't have to get our hands on that ourselves and save our hands for doing things like machine upgrades, for rebalancing some of our computer resources, where things are running on an old operating system and we need to get that onto a newer version. Right now, all of our machines are managed by a system called Puppet, which does the basic configuration work for us. But today, we're on two different versions of Puppet, a really old one and a reasonably new one.

And we're trying to get everything migrated off the old stuff onto the new but once we finished that migration, we're going to have to start all over again, or maybe switch to a different tool. We're looking at a tool called Ansible to use instead of Puppet.

And so there's this never-ending ongoing set of tasks, but each time we do it, it reduces our workload by that much more. So when we upgrade from Puppet 3 to Puppet 6, we get an improvement in the maintainability of that server. And that means that we spend less time with that server going forward and have more time to do other things or to deal with project growth.

Regarding a scale of efficiency, how do you close your skills gaps? When I spoke to Chris and Drew, they both said, "We do everything." How do you do that? How do you know all of this? Do you look at this big picture and say, "Okay, we need a person to specialize in X and Y and Z," and then you send them out to learn about it? How do you cope with that?

The team definitely specializes. And the guys have specializations around different areas, but we do a little bit of cross training, but not a lot because as I mentioned, we've got like 200 machines, each individually doing their own thing. If we cross trained everybody in everything, we'd get nothing done. So, there's a little bit of cross training, but mostly some specialties. It does create a little bit of bus factor...

Which is very scary. I was just going to say, your bus factor is very scary. Talk about that.

The thing is that Puppet allows us to create configurations and that's in version control. If all of a sudden somebody leaves, another person can backfill them because if somebody leaves, it's not like they take their work with them: all the work is in version control. And so that work doesn't go with them, but we may need to backfill some education on that particular specialized area. For example, Chris (ASF Infra team member Chris Thistlethwaite) does a lot of our monitoring work. If he left, now we need somebody to get a little more familiar with NodePing and a little more familiar with Datadog, but that'll be like a week for somebody to pick that up.

It wouldn't be, "Oh my God, this is three years of expertise that we need to go backfill" ...we don't have anything that is that highly specialized.

Is that because the team is more well rounded or because you guys are more efficient or what about it? Because of technology evolution, or...

We don't deal with systems of that level of complexity. We've got 200 machines, like I said, each doing their thing, but it's not like we've got a cluster of 200 machines all trying to coordinate to create one particular outcome. It's, here's my SQL server, here's a JIRA server, here's a Puppet server. Things like that, where the amount of technology is pretty small in each little pocket ... but we just have a hundred pockets on our pants.


Tuesday June 09, 2020

Inside Infra: Greg Stein --Part I

The third "Inside Infra" interview is with ASF Infrastructure Administrator Greg Stein, who shares his experience with Sally Khudairi, ASF VP Marketing & Publicity.

"We've got about 200 different machines and each one runs something different"


What is your name --how is it pronounced?

Greg Stein. "Gregg St-eye-n"

When people need to find you, are you at gstein@? Has that always been your handle for everything?

Ever since high school, actually. I was gjs@ for a bit in college, but went back to gstein@. I started at Google early April 2004, and Gmail launched on April 1, so I was able to get my work email ID, gstein@gmail. So it’s great, but also rather annoying, because there are a lot of Gary Steins and Gertrude Steins and George Steins, and I get all of their email ... I get plane tickets, hotel reservations ... I got a proposal from the Gates Foundation once. I had some crazy bitter angry lady yelling at her husband as they were getting divorced, and she could rant. I mean, wow: that lady had a pirate's mouth.

But she didn't have his email address.

Apparently not.

When and how did you get involved with the ASF?

I left Microsoft in 1998, and the product group I was working in was building WebDAV into various Microsoft products. I thought the concept of WebDAV was very cool, and wanted the Open Source world to have it. That meant writing a module for the Apache Web Server. I think it was September 1998 when I started posting to the Apache mailing list and looking at how to plug in a WebDAV module. That was Apache 1.3 at the time. I developed a module called mod_dav for Apache 1.3, And when we started Apache 2.0 in 2000, I donated the module to Apache, and it became a standard module in Apache 2.0.

I remember that: I did the press release for that way back when. I knew you were connected with mod_dav, but didn't realize the path as to how you got there. It's very interesting.

That's what brought me to Apache, when they started putting together the foundation: it was in the Spring of '99. I remember asking Roy if I could be one of the first members of the foundation, and Roy's answer was basically like, "We already had the set of people locked in. You'll probably get nominated and voted in at our first member meeting," which occurred in September 1999. So yes, I was in that first batch of new members rather than the original membership.

You've been a member of the ASF much longer than you've been involved with ASF Infra. What were the previous hats you were wearing at the ASF? You've been here for a while, and have had a lot of different configurations.

This is true. So I'm a committer on HTTPd (Apache HTTP Server) and then a PMC Member, an ASF Member. I helped start the APR (Apache Portable Runtime) project with some of the other Web server committers, we pulled that out of HTTPd and created APR, and we used that for 2.0. We used APR, whereas Apache 1.3 was essentially the combination of the two, one big code base. Then Justin Erenkrantz and I started Apache Serf, and that was a high performance C-based client library for HTTP. But we didn't have three people in the community, so it couldn't really be an Apache project. So we took it out of Apache and started working on it on our own, and then eventually Subversion started to use Serf, and so we got more committers on Serf, and the community kind of built up around it because of Subversion. So we ran Serf externally, but just like it was an Apache community, it was Apache licensed and so on. Eventually we wanted to move it back into Apache, and I don't recall off hand, but we went straight to a TLP from our external project back to Apache Serf.

Early 2000, it was January or February, (ASF co-Founder) Brian Behlendorf approached me about helping with the network protocol for this new version control system they were starting at CollabNet, because he knew my background in HTTP and WebDAV. That “V” stands for versioning. I got involved with the Subversion project that Spring. That was also run as a very egalitarian Open Source project, very similar to how we run stuff at Apache. I was really the only Apache person, but Karl Fogel just knows how to run a great community, and so all those values that we cherish in communities at Apache were part of Subversion from day one, but was run by CollabNet. I was hired in 2001 to manage their development team. Eventually, CollabNet wanted to turn it into a vendor-neutral thing that wasn't only CollabNet, so they started a small LLC called the Subversion Corporation. Once the IP was transferred to the Subversion Corporation, people said, "Okay, let's move to Apache," because nobody wanted to deal with the overhead of the Subversion Corporation. We approached Apache at the end of 2009, and Subversion became Apache Subversion. I was the first VP for that. I think that's the only VP hat I've worn.

In 2001, I was elected to the Board at the Members meeting, and in 2002, Roy decided to step down as Chair and said, "Oh, Greg should be Chairman." He just kind of threw me under the bus that way, but I agreed, and that's when I became ASF Chairman. I was chairman until 2007, which is the longest-running chairman. I think Brett Porter did four years.

I think it was 2009 when you hosted us at the Harvard Club and Doug Cutting was appointed Chairman, but he said he didn't really want to travel, do much press stuff, or be a face of Apache. Roy came to the rescue, threw me under the bus again, and said, "Greg can be Vice Chairman, and we'll have the Vice Chairman do all that stuff”. So I held the Vice Chairman role until September 2016, when I gave up my director position, the Vice Chairman position, and VP of Subversion, because that's when I became Infrastructure Administrator.

Over the years, I did a bunch of volunteer work for ASF Infrastructure. I helped out with what we call AP mail: adjusting moderators, changing aliases, things like that. So I've had AP mail access for quite a while when I was doing that. Upayavira wrote for people to review their Member records, change their passwords, etc. I helped him with some of that stuff. That was all written in Python, so I was able to help out.

Python before Python was popular.

I've been using Python since 1995, and I've contributed to Python itself. We set up the Python Software Foundation in 2001. When I say “we”, I mean myself and Dick Hardt from ActiveState. We took the Apache bylaws, and added a different class of membership to it so that companies would become... I forget what we called them, like corporate members or something. The normal people were called nominated members, as they were nominated by somebody else and voted in. But this gave corporations a vote at the table on the board and anything else that members would get a vote on. So the core of the Python Software Foundation came from Apache.

Back to ASF Infrastructure. In 2016 we had four people on staff in Infrastructure, and our volunteer VP of Infrastructure didn't have enough volunteer time to be able to provide support and management for those four people, plus we wanted to hire two more people. With six people, he was right out. So we spent a lot of 2016 trying to figure out how to create a “manager” for the Infra team. At the time the idea of an “executive director” type position was also thrown around, but a full-time position to manage four or six staff is completely overkill, and we certainly didn't have the budget for a full-time position. Somewhere around late August, I realized that there was an email that Ross (former ASF President Ross Gardler) sent and I thought, "I can do that. That's a half-time job. I'm certainly happy to do it. I've managed engineering teams before.” Now, infra's not an engineering team, they don't really develop products, but it's pretty close to engineering management. At a minimum, it's personnel management, which I've been doing since the '90s.

So I threw my hat in the ring. Ross ran it by the Board and the team, and nobody raised a strong concern, so in his authority as President, he went ahead and hired me half-time. It was the day of our Board meeting --I resigned all my positions, and we appointed my replacement for Vice Chairman and my Director position that day, both of which I believe were Sam. He filled my role as Director, and I started as Infrastructure Administrator.

What does “Infrastructure Administrator” mean? What does it entail: are you hands-on coding solutions like the rest of the team? Are you solving problems? What do you do?

I chose the title because I didn't want to be called “manager”: I didn't want to feel like I'm the boss. I wanted to help with the administrative side, make sure the guys get paid, deal with the invoices, handle what you might call back office kind of stuff, and let the team focus on what they do best, which is the system administration. (ASF Infrastructure Team Member) Daniel Gruno does some development work in addition. I do a little bit of development work. For me, it's more like where in my hobby time I might work on Subversion, but now my hobby time is coding Infrastructure type stuff, so it's not really part of my work duties. I deal with salaries, raises, bonuses, getting the payroll done, and for our contractors, getting them paid. I also deal with third party contracts for things like Travis CI, for lists.apache,org ... that's with PonyMail. I make sure that our vendors get paid, and our contractors and employees get paid.

How was the Infra team structured, and how many are in the team?

We have five full-time people that work on Infrastructure: all five are system administrators. Daniel Gruno does maybe 30% system administration and 70% tool development. We don't develop any products, because we're not an Apache community. We write tools, but don't actually develop any products. This is why PonyMail is in the Incubator: it was originally written by one of the people on Infra, but we didn't want to run that as an Infra community. With only five people, we don't really want to be a community lead or anything like that.

The joke is if somebody wants to move into my position, they lose half their salary, because my position is part-time. It's not really a promotion: it would be a loss to do anything. So unlike a corporation with 10,000 people on staff, career development is a little more difficult. It's really a job for people that enjoy Apache and enjoy our mission, and also enjoy working with the other people on the team.

Who does ASF infra serve?

Our primary users are all the communities at Apache. We've got over 200 communities, and those are the primary users. I don't like calling them “customers”, but in a corporate world, they would probably be considered our customers, and we serve those users. There's 8,000 people with accounts that are working on different projects, but the user base is way, way larger than that, because people can file JIRA tickets and work on the wiki and do things like that without actually being an Apache Committer. So the user base is even larger. Then you start looking at all the people subscribed to all of our mailing lists, and that number goes even higher. There's probably 10% of our work which is also supporting the administrative side of the Foundation itself.

For the Board, your role in PR, and Trademarks, and Legal, and the office of the President and various other operational type stuff, we spend 5-10% of our time. A lot of what we do applies naturally across all of the user base, because the foundation uses the same tool set as our communities. Subversion, mailing lists, JIRA, Confluence, etc. We help with account creation, the LDAP management, what sort of permissions people have to access different things...

One of the neat things that we've done, and I've actually had a couple of communities ask us about it, is our GitBox setup where our projects can use GitHub. But then we also mirror all that source code back to Apache so that we have a copy of it for provenance tracking. And in case GitHub does something dumb, we have our own copy of the code. Any changes made on GitHub get sent to our mailing list or get mirrored into JIRA. Our projects can see all the activity on GitHub, and it gets mirrored into our mailing lists where we prefer that our community work is performed.

That's actually a pretty cool feature that we've done at Apache.

It's interesting to see communities outside of Apache that emulate structures and processes and solutions that the ASF has created. It's cool to see it even happening on an infrastructure level. How does ASF infra differ from other organizations or other open source foundations?

Most of them don't really have teams. Most projects out there do their work on GitHub, and don't have their own source control. They don't have account management, they don't run mailing lists. We do all this stuff that most Open Source projects just don't deal with.

They also don't have the scale that we have.

Yes. Because they're one project, and we have over 200 projects. Most projects have some repositories hanging out on GitHub or on GitLab, or wherever else that they might host: if somebody wants to run a demonstration of that project, they buy their own virtual machine and AWS, and pay that out of pocket. At Apache, all of our projects can have virtual machines hosted by Infra, where they install their software for demonstration purposes. They can point people at that VM, so they can check out the product in live motion. So that ability to run VMs is also pretty unique to the Foundation. When you look at the Linux Foundation or the Eclipse Foundation, those are a little bit different. They're not a charitable organization like us. They're a 501(c)(6), which is really like a trade association.

Like a consortium.

Yes, a consortium. I believe that they do have infra teams, but their business model is quite different from ours. If you look at Mozilla, they have the Mozilla Foundation, but that's kind of a shell; Mozilla Corporation essentially runs everything, and the foundation is like a legal shell wrapped around the corporation.

You mentioned earlier that we have 200 projects: you're referring to 200 Top-Level Projects (TLPs), but we also have sub projects and initiatives. At Apache, we have more than 350 different activities going on --you guys touch all of those. It's not like there's any aspect of ASF that you're not involved with or you're not supporting.

That's correct. And I say 200 because I'm thinking mostly from a TLP thing.

Irrespective of the existence of sub projects, you're still dealing with other communities and projects: there's more than just the 200. Hats off to you guys. It's quite a lot of work.

We've got about 200 different machines and each one runs something different. Some companies have 50 copies of a machine that they'll start up in the cloud, running some container --we never do that. Each individual machine is configured one by one and they're all different. And so 200 machines to support the 350 initiatives. It's a lot of heterogeneous work and that can be kind of distracting, but it's also very interesting because we do support such a wide variety of stuff for our projects.

There's what, five Infra team members, and we have 350 projects and initiatives going on. That's a lot of stuff happening: is it non-stop?

Yeah, it's nonstop. That's why we went from four to six people, we were sort of treading water, but we weren't really able to move forward on a number of our longer term initiatives. So when we went to six people in November 2016, that made us a lot more hands-on, if you will. That meant that we could actually make some progress on this longer term work that we wanted to accomplish. Some of that is like , where people can get things done instead of filing a JIRA ticket and having us do the work for them.

Is that popular? Do people use it?

Oh, absolutely. When somebody opens a JIRA ticket to say, "Can I have this Git repository?” or “Can you create a JIRA Space for me?" we close the ticket and say, "Go to". Before, where everybody would file a ticket for a Git repository or file a ticket for JIRA, file a ticket for Confluence, or whatever, we just close them all down now, and they use instead. We simply won't do those things anymore. So is actually quite handy. And then about four months ago we've added a feature called asf.yaml: it allows communities to control a lot of the finer grained aspects about how their repositories are used, like how do they publish Web pages from a repository, or if you make a change, where does the commit email go? Which mailing lists? Does it go to their development list? Or do they have a commit list? If somebody opens a PR on GitHub, where does notification of that go? Those used to all be tickets also, but people can deal with those just by editing a file in their repository now. So again, it reduces tickets and that's our goal where these routine tasks that all the different communities want to perform, we want to move those into a self-served mechanism so that we don't need hands-on all the time. And thus, we can support 350 different initiatives.

That's great to help empower the communities to take care of their own needs, whether they're minor or major, but that also encourages autonomy. So that's really helpful for you guys: you don't need to have a team of 40 people to support the day-to-day.

We do stay busy. You're talking about the influx and we get requests from people through email, through our Slack channel and through JIRA. Of course our monitoring system will tell us when something goes down, so our monitoring systems also give us more work to do, so it is kind of an endless string of queries. Depending on what the task is, each of those different channels is appropriate. For a quick task, hitting us up on Slack is totally fine, but if there's going to be several days of work, we like JIRA tickets so that we can track the work as it progresses.

How do you encourage the team? How do you keep them motivated? What were your challenges with such a huge load to carry: how do you keep everyone going?

One of the big benefits that we have for our team is actually that we're all remote, so we all sit on a Slack channel. We have a team-only channel that we use for communicating, "What's going on? What beer are you drinking today? What are you having for dinner?" I think about my days when I worked at Microsoft or at Google where I sat in the office by myself and it's a very individual experience that I used to have, but now, our team is there all the time on our channel. It's a very social experience: I think that makes for a much tighter team. And it provides a very different experience than what you get at a more “normal” company. That sort of team experience really helps keep people motivated.

People enjoy their jobs more. From a management standpoint, I can certainly say, "If people are sitting there talking about what they're going to make for lunch, there's a drag on the team and maybe we're not seeing the highest productivity possible," but I think that would actually run the counter. Our team is actually more productive as a result of this great team bonding. We have a conference call once a week for 30-60 minutes. And we don't really have to: the team knows what everybody's doing because we're all doing it right in front of everybody. We all get the commit messages. We have our Slack channel. We see the changes to JIRA. We know what each person is doing, but having the call actually gives us a chance to speak to another human so you're not working in your basement all day without any human contact.

We actually have that once a week, if you will, forced human voice contact.

Did that evolve organically? Or was that something planned?

The team was already doing weekly status calls. When I started, I said, "We're going to keep doing that. We're not going to switch out for just, you know, a status email or anything." Before I started, I think they were doing a group edit on a status Web page or something. I don't know if they had calls, but today I mandate the call because I want the team to get together. We've also been doing the group get-togethers at ApacheCon. We got together at ApacheCon Miami, and then the next year in Montreal. Last year we skipped the whole conference format and just got together as a team in New Orleans for four nights.

It was great because it was just us without the distractions of the conference. The conference is good because the INFRA team gets to meet the people that are their users, their customers, the people that we're actually trying to support, all those communities. And the people in the communities get to meet the team. You know, the people that asked, "Can you help me with X?" They get to put a face to those names.

There are times where one of the guys on the team will work with somebody in the community for a couple of weeks to track down some problem, get a virtual machine configured, whatever. All you see is a user ID and the kind of tone of their messages, but at the conference, you can actually put a face to that name, to that ID. That’s really good from a team standpoint. With the team bonding, we spent eight hours a day in this giant penthouse suite in New Orleans on the 30th floor looking out over the Mississippi River. It was very cool, it had space and a big dining table where we could all come in and work. And then I would go around the corner to Mothers and pick up—

Oh my gawd: the po boys ...the debris po boys.

Exactly, you know what I'm talking about.

I lived there. So, yes, I know.

It was literally a block away. So that was our lunch. Every day I was going down to the Mothers, getting a big brown shopping bag full of food and bringing it to the room. We did go there and eat once so the guys could get out of the room for lunch, and each evening we would go as a team out for dinner. After dinner, it's like, “OK, do whatever you want. It's New Orleans.” That was a really good team experience. We were set to go to Nashville this year and then, you know, pandemic ensued. So we called it off.

It's funny: I stumbled across your channel on Slack and, if I remember this correctly, someone was talking about grilling a whole steer or something along those lines. You guys deal with a lot of beef, there’s a lot of meat in this group. So ...

In the team channel, there's a lot of stories about food and beer and other forms of alcohol. We eventually created a cooking channel on Slack because there's other people like Ruth (ASF Executive Vice President Ruth Suehle) and Shane (ASF Vice Chair Shane Curcuru) and others who also like talking about making food. We still have a lot of that discussion on the team channel, but we’ve now got a dedicated channel with a larger set of people talking foodie type of stuff, so that’s very cool.

You were also talking about motivation: I work with each of the guys to find out what they're interested in exploring. Whether it's a new tool or a new product or to write a new tool to improve our workflow, it's like, "What are you interested in? Okay, take point on that, do the research, go do the experimenting." So each of the guys has gotten generally one or two long-term projects that interest them that they want to work on.


Monday May 11, 2020

Success at Apache: Remote Collaboration in the Time of Coronavirus

by Marvin Humphrey

I "arrived" at the Apache Software Foundation in 2005, unreasonably angry about a bug in Apache Lucene.  By "arrived", I mean that I sent the first few emails among several thousand I would go on to send over the next 15 years — the ASF didn't have a physical office where I could show up to buttonhole and berate some unlucky customer service representative.  An unreasonably patient Lucene contributor named Doug Cutting talked me down.

Because the ASF has always been a virtual organization, the Coronavirus pandemic has had minimal impact on its day-to-day operations.  While individual contributors may be personally affected, at the collective level there's been no mad scramble to adapt.

Others have not been so fortunate.  All around the world organizations have been struggling to revamp their processes and infrastructure to comply with "social distancing" protocols.  Sadly, many have already laid off workers, or even closed their doors for good.

And yet, there is a huge pool of work which could conceivably be performed remotely but isn't yet — or which is suddenly being performed remotely but inefficiently.  If we can accelerate and streamline the transition to remote work, many jobs and businesses could be saved.  With some creativity, our interim "new normal" could be more propsperous, and perhaps sooner than we think!

Are you an Open Source contributor?  If so, you possess expertise in remote operations which is desperately needed in today's challenging economic environment.  Let's talk about what we know and how we can help.

The Internet Turns People Into Jerks

People type things at each other over the internet that they would never say to someone's face.  In person, we calibrate our language based on feedback we receive via facial expressions, tone of voice, and body language.  But when all communication is written, the feedback loop is broken — and all too easily, vicious words fall out of our fingertips.

Suddenly-remote workers may find themselves exposed to this phenomenon as conversations that once took place in the office migrate to Slack, email, and other text-centric communication channels.  But it can be tricky learning to recognize when a conversation being conducted via a text channel has gotten overheated — it takes an intuitive leap of empathy, possibly aided by dramatic reading of intemperate material a la Celebrities Read Mean Tweets on Jimmy Kimmel.

Open Source communities have grappled with incivility for as long as the movement has existed.  Over time, "ad hominem" personal attacks have gradually become taboo because of their insidious corrosive effect; there exists broad cultural consensus that you should attack the idea rather than person behind it.

Defenses have become increasingly formalized and sophisticated as more and more communities have adopted a "code of conduct".  While the primary purpose of such documents is guard gainst harassment and other serious misconduct, they often contain aspirational recommendations about how community members should treat each other — because serious misconduct is more likely to occur in an environment of constant low-grade incivility.

Regardless of whether your organization adopts a code of conduct, it won't hurt to raise awareness among remote team members of the suceptibility of text-based communications to incivility — so that they may identify and confront it in themselves and others and shunt everyone towards more constructive patterns of communication.

Keeping Everyone "In The Loop"

Coordination is a troublesome problem even when everyone works in the same office, but the difficulties are magnified in remote environments where it takes more effort to initiate and conduct conversations.  Teams can become fragmented and individuals can become isolated unless a culture is established of keeping everyone "in the loop".

At the ASF, the problem is especially acute because its virtual communities are spread out across the globe.  Due to time zone differences, it is typically infeasible to get all stakeholders together for a meeting — even a virtual meeting held via conference call or videochat.  Additionally, many stakeholders in ASF communities do not have the availability to participate in real-time conversations regularly because they are not employed to to work on projects full-time.

"Synchronous" communication channels like face-to-face, videochat, phone, text chat, and so on are good for rapid-fire iteration and refinement of ideas, but they effectively exclude anyone who isn't following along in real-time.  Even if conversations are captured, such as with AV-recorded live meetings or logged text chats, it is inefficient and often confusing to review how things went down after the fact.

The solution that the ASF has adopted is to require that all meaningful project decisions be made in a single, asynchronous communication channel.

  • The channel must be canonical so that all participants can have confidence that if they at least skim everything that goes by in that one channel, they will not miss anything crucial.
  • The channel must be asynchronous to avoid excluding stakeholders with limited availability.

Synchronous conversations will still happen outside this canonical channel —and they should, because again, synchronous conversations are efficient for iterating on ideas!  However, the expectation is that a summary of any such offline conversation must be written up and posted to the canonical channel, allowing all stakeholders the opportunity to have their say.

At the ASF, the canonical channel must be an email list, but for other organizations different tools might be more appropriate: a non-technical task manager such as Asana, a wiki, even a spreadsheet (for a really small team). The precise technology doesn't matter: the point is that there are significant benefits which obtain if a channel exists which is 1) canonical, and 2) asynchronous.

Decision Making

In an office, decision makers can absorb a certain amount of information by osmosis — via overheard conversations, working lunches, impromptu collaborations, and so on.  That source of information goes disconcertingly dry on suddenly-remote teams, leaving only information siphoned through more deliberate action.

A canonical, asynchronous communication channel can compensate to some extent, providing transparency about what is being worked on and how well people are working together, and facilitating oversight even while most of the work gets done solo.  Because properly used asynchronous channels capture summaries rather than chaotic and verbose real-time exchanges, the information they provide is more easily consumed by observers watching from a distance.  The canonical channel also provides an arena for gauging consensus among stakeholders and for documenting signoff.

"Lazy consensus" is a particularly productive kind of signoff, where a proposal is posted to the canonical channel and if there are no objections within some time frame (72 hours at the ASF), the proposal is considered implicitly approved.  Provided that the channel is monitored actively enough that flawed proposals get flagged reliably, "lazy consensus" is a powerful tool for encouraging initiative — a precious quality in contributors collaborating remotely.


Organizations are adapting in myriad ways to the economic crisis brought on by the Coronavirus pandemic.  In the world of Open Source Software where countless projects have run over the internet for decades, we've accumulated a lot of hard-learned lessons about the possibilities and pitfalls of remote collaboration.  Perhaps our experiences can inform some of the suddenly-remote teams out there straining to find their way in these difficult times.  Let's help them to do their best!

Marvin Humphrey is a Member Emeritus of the ASF and a past VP Incubator, VP Legal Affairs, and member of the Board of Directors.  These days, he is focusing on family concerns and consulting part-time.

= = =

"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works"  

Monday May 04, 2020

Success at Apache: bringing the Apache Beam firefly to life

by Julián Bruno

Creating the Apache Beam firefly was the first opportunity I had to contribute my skills as a designer and illustration artist to an open source project. I didn’t know anybody working in open source until I moved to San Francisco from Buenos Aires, Argentina. I knew about open source software for video games, like Unity or Unreal Engine... This allowed gamers to make modifications, like adding new levels or creating new character models, and upload them to the same engine that hosted the original game for other gamers to use. This practice enabled a sense of community, where users can share ideas, passions, and express creativity. There are so many things you can do when you work in collaboration with others. This spirit of community is one of the things that made me excited about contributing to Apache Beam. 

Living in an area where technology is everywhere really piqued my interest and drove my curiosity to understand how technology evolves. When the opportunity came to contribute to Apache Beam, I was interested right away. I didn’t know about the project before I got involved, and I certainly didn’t know there was a community behind it, working together to build this amazing solution. Building a mascot for a group of people is different from working for a brand because this firefly represents a group of people and what they find valuable. There is an extra layer that makes it more human. For this type of work, designing a mascot is usually a decision reserved for a small group, and the larger community is not involved. It is refreshing and very meaningful that the community had a chance to step into the process. I saw it as an opportunity for self-expression,participation, and one more exercise in community building. 

In order for this process to be inclusive, I built a group-wide communication system for the community to input during the process. I think that having open and frequent communication was key because, ideally, I wanted everyone to feel that the mascot represents them. I created questions that would help Apache Beam contributors understand what I needed as an illustrator. The questions helped me understand what they liked. This ensured that the mascot was aligned with the community’s taste. Some questions were about colors and visual styles they preferred, if the eyes are too big or small, and preferred line art style. There were 4 rounds of feedback, plus a final vote, where 18 people participated. Engagement increased with every new round. The Apache Way for communities to operate reminded me of a lot of animation forums I participated in the early 2000s. I’m glad to see that some of these practices are still around, because they help make processes more inclusive and build a sense of community.

This communication with the Apache Beam community helped me to create a mascot with features that are unique to the project. When I started, I was given a few concepts that I needed to work with, such as: cute, innovative, fast, data processing, and futuristic. The first few decisions, like making the mascot look as aerodynamic as possible were easy to make. Conveying "data processing" was a bit harder to figure out, butI eventually chose to communicate this concept by changing the mascot's color. What really gave the mascot its unique identity came from using Pokémon-like character style. I built the rhetoric for Apache Beam's logo by combining two concepts that have nothing to do with each other, Pokémon and data streaming, and created something new. 

In the end, I created the Apache Beam mascot and its model sheet, so that anyone can reproduce it, a version of the mascot learning (a key focus for the project at the moment), and a version of the firefly doing what it does best… stream data! I really enjoyed working for Apache Beam and contributing my skills as an illustration artist to open source. I think the most interesting part is the community: creating something in collaboration with others adds a lot of value to what you are making for the world.

Julián is a digital artist based in San Francisco, California. He has spent over 10 years in the animation industry and has developed his skills in art direction, 2D animation, illustration, and visual art development. My passions include art and cartoon animation, as well as connecting with people and creating new projects. He was born and raised in Buenos Aires, Argentina, where he studied Graphic Design at University of Buenos Aires (UBA). Find Julián's work on Artstation and Instagram.

= = =

"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works" 

Monday April 27, 2020

Inside Infra: Drew Foulks

The second in the "Inside Infra" interview series with members of the ASF Infrastructure team features Drew Foulks, who shares his experience with Sally Khudairi, ASF VP Marketing & Publicity.


"I am in the business of making life easy for people who do phenomenal stuff."

What is your name --how is it pronounced?

My name is Drew Foulks. “Droo Follx”.

If folks were to find you at the ASF, like on Slack or elsewhere, what's your handle? How do they find you?

They'll find me at Warwalrux, spelled with an X, so W-A-R W-A-L-R-U-X.

So, “War Walrus”, but with an X at the end. Where did that come from?

Kind of embarrassing story actually. I got picked on a lot in middle school because I was always really good with computers, but as bad as it sounds, I never really wanted to be. I always wanted to be one of the one of the cool kids and the cool kids were not going to computers. One day I got into a fight at school and one of my friends just absolutely made me lose it afterwards. I was sitting there on the ground crying and he said, "Man, you were a fighting walrus, like the walrus of war or something. It was awesome." I lost it. But ever since then, I’ve just been like, "You know what? I'm not even going to be ashamed about that anymore." I've been that since I started doing tech, which was actually not that long ago compared to the other guys on the team.

How long have you been in tech?

I'm 29, and have been in tech since I was 16, so 13 years.

When did you get involved with the ASF? How did you get here?

I was working at NASA for four and some change years, and I decided that I wanted to pursue some other opportunities because they really were not supportive of that work from home culture. And at the time I had a lot of stuff going on. My wife was sick, my daughter, my youngest, has special needs and stepson actually also has special needs, so being at home was something I had to do. A buddy of mine tipped me off on a Website called We Work Remotely. I ran across your ad there and thought, "There is no way that is who I think that is and I'm going to apply for the hell of it." Surprisingly, two months later, I got a call back.

You do understand how many interview candidates we had, right? A lot of people were competing against you.

It blows my mind. I heard the stories after I got hired and I was just like, "Man, that's nuts." And then when I got hired, I was actually told, jokingly, of course, the ASF was looking to launch its own brand of internet satellite. So that’s why we hired people from SpaceX and NASA.

The Infra guys have such a dry sense of humor! How long have you been a member of the team?

One year and one month, 13 months.

For some reason it feels like you’ve been part of the Apache family for years. What’s your role in ASF Infrastructure? What are you responsible for?

My latest contributions have been the Website builders, so I'm working on helping people migrate off of CMS. Some of the ways that I've chosen to do that are by working with Humbedooh (the handle for ASF Infrastructure team member Daniel Gruno) on his ASF.YAML project, that so many projects seemed to be really enjoying.

YAML? Yet Another Markup Language?

That's it. Yet Another Markup Language.

So basically, I built the system that lets you build Websites from ASF.YAML and you just specify your Website builder, whether it be Pelican or Jekyll --those are the two that we support right now. And you give it a source branch and a target branch and every time you check in, boom. It builds your website.

Who is this aimed at?

This is for Apache Projects building their TLP Websites. When you commit your Website to the repo, say any project, they've all got Websites, but some of them are generated via Jekyll. Some of them are generated with Pelican, some are generated in a custom way with a Jenkins job. It's just how each project is determined to generate their website, but we're trying to make it easy and provide lots of options for projects to migrate off of the old CMS. But still projects are allowed to be able to choose their own method of publishing or their method of creating a site, but you have to be able to enable all of that to happen.

Did you have to learn this or was this knowledge something that you came into the position with?

I learned it.

Was it difficult? How long did it take you to get this project up?

The Pelican one was a lot harder than the Jekyll one. So, Pelican took a couple of months. Really, Greg had a prototype when I came in that apparently had been kicking around for a little bit, so I tightened it up and pelicanized it. I think it works pretty well. I've not heard any complaints about it.

That took a while before I wasn't doing primarily Python programming, I was doing lots of different ops things just in a completely different way than what I do now. To be honest, I still haven't wrapped my head around exactly what it is I do here.

Do you mind sharing a little bit about that?

I came here from the government world, which is very silent. I worked for the OCIO, Office of Chief Intelligence Officer Data Center for NASA Langley, which is a very old NASA center. Older than NASA itself actually. Their infrastructure, as you can probably guess, is not the newest: It's 100 years old. They have wind tunnels from the 1920s. There are parts of the infrastructure that are 100 years old and it's insane. Everybody has a specialty, everybody's a subject matter expert in something, and there's nothing more permanent than a temporary government program, so if you take something on, expect to be doing that for the rest of your life. It's very regimented. If you’ve ever seen Hidden Figures, the computational research facility where they’ve opened the Katherine Johnson Research Center, was my data center.

And then to come to the ASF, it's like, "Okay, so we've got like 11 different Cloud providers and these are all the projects that we're supporting. Do you know this, this, this, this or this?” Jenkins, Buildbot, VMware, any of the Docker, Puppet and all that stuff. Do I know any of these myriad Open Source technologies that one doesn't really get to use a lot of in the government sphere. I mean, I've been doing Ansible there for three years.

It was very monolithic. We had VMware. I ran a data center. I had hardware. I had to track all of that. Coming here, everything is completely different. It's like, "We're juggling all these different Cloud providers, and oh, wait: we’ve got to migrate out of this one today, so let's do that. Okay. All right. Where are we going with this?" It's just like there's no end in sight. As technology progresses, so do we. It's just that we do it so much faster than anywhere else I've ever been.

Is that exciting or scary?

Oh, gosh. I've never stopped long enough to think about it. It is a bit of both. It is intimidating for sure, because before it was very silent. Like I said, I did my thing and I had my interests, my extracurricular interests, running home network setups and private media servers and whatnot. Then I come here and those hobbies go away, now I’m doing that for the Foundation instead.

Yeah, that's cool, though.

It is. I'm a professional hobbyist.

To get paid for doing your hobby is pretty rewarding.

It is. Yeah.

This has become your hobby in a different way, of course, because I'm sure you weren't planning on dealing with ~11 different Cloud providers.

No, I was not.

In our chat with Chris Thistlethwaite last month, we learned more about who ASF Infra serves and the scope of the work that you provide. Can you tell me more about the who and how it works exactly? So, who Infra serves and to what capacity or what is it that you guys do? Because I get every person's perspective is slightly different because I get the same, we do it all answer, and is that true? I mean, you're saying that so far, it sounds like it's true. I guess no one has a reason to expand upon it in terms of embellishment, but tell me more.

We serve Apache project developers and development teams. It’s not just the people who sit down and write the code, the people who orchestrate these very complex processes of building testing, checking, doing the sanity work behind the scenes, the people coordinating releases, PMCs planning out the future of these projects, we serve them, too, and we have to serve them in a capacity beyond, "Hey, here's a build platform," it's: "We support your email communications, we’re there to facilitate the goings on of the Project." Infra's domain is almost everything but the coordinating and writing of code.

Taking care of their code management systems, providing them with the means to do build testing and having it not kill us in the process. That's a big, big addendum to that requirement. Like I mentioned, email, I call them the central services, things like LDAP, authentication, your virtualization services, file sharing, all of those things that make the business of a TLP easy(ish). I am in the business of making life easy for people who do phenomenal stuff. That's honestly how I view my job and it's very, very different than my old one.

In my old job, I had one customer who I bent over backwards for; here, it's very much, "Listen, my job is to provide these services and to facilitate what you guys do, not do it for you." Drawing that line sometimes becomes difficult for me personally because I don't have as much experience in the ASF, I think. But that seems to be a skill that the other guys have is when to bounce back and say, "No, this is definitely a PMC or a PMC issue that you guys should be dealing with because it sets a bad precedent if I make this decision. I'm not going to do this work for you." It wouldn't be a right to pollute a project like that.

What you're saying doesn't come across as odd. One thing that I always want to know is how ASF compares with other infrastructure operations in general. Chris had said this also, here you have 300+ projects and all sorts of different groups that you're interfacing with, so it's a completely different type of interaction. Your response is totally legitimate: it takes a certain type of personality to be able to handle that because most people would likely be overwhelmed and run away. The fact that you're here and thriving and our projects are expanding is awesome.

Thank you. You can thank my wife for not letting me run away.

Based on my understanding, as a team you're autonomous yet coordinated. Is that the right way to describe how you work together?

Yes. That is a good way to describe how we work together.

Do you feel like that model works or do you think something else should be happening or how does that work for you?

That's a tough question because I'm not sure that the answer would make any sense, but I'll give it a go anyway. By constantly talking with each other, the team gets a sense for the direction that we need to be heading. Leadership is very organic and not spontaneous, but they're like a current guiding us towards the goal, really, whatever that is, so all of the decisions that we make on the daily really kind of help us towards that goal, because fighting the current is difficult.

In a lot of ways that long-term coordination is really facilitated by this, I'm going to call it “on a current of progress”. It's not forceful. That's kind of what it feels like. The team is driving towards something, it's not random, to be honest with you. It's typically a goal that we have in mind, but all of the work that we do is just like, "There's a cool idea that I had related to this, so let's just work on that." And we end up getting there. It's crazy.

Describe your typical workday. Are you on a rolling schedule? Do you guys work on a shift? How do you get it all done --and you're down one person now-- how do you get it done?

I have no idea. So really, personally, I have a nine-hour a day week schedule that I follow every day. So basically I start work and I break it up into two or two-and a half hour chunks and I do four of those, take little breaks in between, try to keep myself sane, try to throw in a dog walk. Really, I just approach it like I approach any other job, one ticket at a time.

Do you work in shifts? How do you cover those 24/7? How do you balance the load?

So there's a one week on-call rotation. So right now there are the... gosh, how many of us are there? Five? Anyway, so there's one week on-call rotation and that person is on 24/7 for the week, Monday to Monday. And then after that, it's pretty much just you cover your time zone. Yeah. So the scheduling, it's so loose that I mean really as long as you're putting in your eight hours a day, nobody really cares when you do that. I choose to have that nine-hour work day because kids really. It's fantastic for having a family, but whether you want to jump on at 1:00 in the morning and work for six hours, that's fine.

OK, so as long as someone's there, and it doesn't have to be you, you can work on your own timeframe. Are you guys usually slammed? Is it low-level? Is there a busy time for Infra on the whole? Is it like tax season if you're an accountant, or is it constantly just 24/7/365?

It's pretty much 24/7/365, but we do definitely have “seasons” as well. We do a one week on-call rotation, so somebody's always on, but the scheduling is very relaxed. So, it's optional, the hours you'd like to keep. I choose to work a work day because of the family and that just kind of fits in nicely actually. Some people may decide that, "I'm awake It's 1:00. I can't sleep. I might as well get some work done and I do that." And I've certainly done that before. So, yeah, it's pretty whatever and we're all kind of, I don't want to call us workaholics because I think that's a bad word, but we're all …

“Work enthusiasts.”

I don't know that I've called them busy seasons as much as busy cycles.

What are they? What triggers them?

Typically? Releases. The most tickets coming in is when some project is putting out a build or is putting out a release. For a large project release, we'll have a lot of tickets sent in because they're utilizing a bunch of resources and stuff gets backed up. That's typically it.

So whoever is on call during that time period, it's really their responsibility to handle: it's not like when Apache Wombat or whatever Project has an issue, it becomes “Drew's issue”. You're not assigned to a project to facilitate that, it's whomever is there will help them however possible, correct?

Yeah. And I think that you said it earlier: everybody that you've talked to says that we do it all. I'm going to tell you that we do it all. It's every project from Apache Zeppelin to Airflow, whatever the first one is. That's not our work.

I don't know if this is actually the case, but I'm curious: is it possible for an ASF Infra team member to be an introvert or do you all have to be “client-facing”? I know that we don't have an office, and you see people from time to time at ApacheCon, but do you have a wall that you can hide behind or do you have to interface with people all the time?

Did you go to the end for Lightning Talks?

I was not at Lightning Talks at ApacheCon/Vegas, but I heard it had quite an activity that happened there, Chris told me about it during his interview, let's put it that way. No one said anything to me up until that interview, so I was surprised. Fill me in with some more. What do I need to know?

[laughing] So, an introvert and two extroverts that are way too drunk, get up on a stage in front of people and proceed to just make fools of themselves for a minute. That's pretty much it.

I guess I know who the introvert was.

Yeah. So the original plan was to go up there and make thunder noises because that is the sound of lightning talking. That was a fun experience. Not one that I would do again, I think but it was fun.

Let's go back to the daily schedule for a minute. This is always a curiosity for me for anyone who's super busy, which is pretty much everyone at Apache: how do you keep your workload organized? Your structure for your day is very impressive, I have to say, this two-and-a-half hours times four. I think it's fascinating. But your actual workload, for example, you get one of these huge releases, how do you manage all that?

Okay, so the first part of my day is typically spent organizing my day as awful as that sounds. We get so much email that I think that it's literally impossible to read it all. I'm pretty sure it's literally impossible to read it all and so much email, so the first order of the day is sift through that while you drink your coffee because there's no way I can get through that. I catch up on the stuff that the team has been talking about, catch up on all the slack channels, look at my tickets, prioritize my workload, and that usually takes about an hour. So right at 8:30, I'm ready to actually start doing stuff. Then it's usually tickets and then a break. And then I don't like to check my email too terribly often. I wish I could three, four times a day, because I think it gets me off task, but that's not really something I have the luxury of being able to do all the time, so I do have to monitor my Ubuntu alerts as emails come in, scanning for anything important. But yeah, it's ticket work for the first half of the day, a project work for the back half of the day. And then right after lunch, I'll sit down and I'll figure out where I am on my project, and then try to move forward from there. Typically, that involves research, but yeah, I like to spend the last couple of hours of my day trying to do something. So, typically project work, because I don't like doing ticket changes at the end of the day.

Why is that?

Well, if you're going to nail your foot to the floor, don't be surprised when you can only run in circles.

I presume when you do ticket work, more things come out of it, too, so it never ends.

Yes. Typically, ticket work involves making a change of some sort, to something that's actually being used, whereas project work is kind of this nebulous, unused, non-production thing.

I'm hearing that you need to know a little bit about everything in addition to your own areas of expertise. How do you stay ahead of the curve? How do you learn about everything that you need to know especially if you don't know what you need to know? How do you do that?

I don't think that you do stay ahead of the curve. I really don't. I think that we do our best to ride it. Getting ahead is so immensely difficult. This technology essentially fractalizes into these many different various facets of high computing.

From virtualizing, networking, programming, you have all of these facets. Nobody can really, truly stay ahead of the curve. I mean, holy cow, the guys in the Infra team, they are all 12-pound brain-type dudes. They'll go from talking about hardware specs to talking about virtualization. They'll bounce around all these different facets of technology, and obviously you have strengths and weaknesses, I don't think anybody can really stay ahead of the curve at this point, and I feel like it's been a long time since anybody has. Technology has just gotten so complicated. We've really tried to, without specializing too much ... kind of pick out some of the non-essential fluff, the stuff that we don't use. I mean, hypervisors aren't really like super in these days. It's all about the Cloud, which is really just an abstract hypervisor, but whatever.

So, we don't really have any “machines” anymore, spec-ing out a physical machine is not something many of us do very often. It's not part of our job anymore, but that's definitely one area of technology that continues to advance as they put out better processors and whatnot. Mostly we try to stay ahead on the DevOps side of things without focusing too much on this operational infrastructure portion. And that's where I came from, this operational infrastructure, the data centers, the servers, the hypervisors, making VMs for people. That's what I used to do and now it's a lot less of that and a lot more fine-tuning this nebulous system of intermeshed tools that I don't fully understand yet.

Seeing that you and others can't stay ahead of the curve, can ASF Infrastructure actually stay ahead of the demand? I mean, is there any way you aren’t constantly in a reactive mode of “this new thing we're responding to, or here's a new part.” Can you get your house in order, or is the house in order?

At the ASF, especially Infra, we do a very good job of listening to our projects because we as individuals cannot stay ahead of the curve *and* have every good new idea that there ever was to be had. Our community is large, and our community is very smart as people and as a group. We have a lot of really excellent ideas that come in from tickets and you say, "You know? I think I'm going to look into that today." And you look into it. You realize that it has all this potential and suddenly, that's the service that we're now using, some things like Travis, which is a third party build validator, came to us in that way.

Since I've been here, some of them have come to us via tickets, where it's been, "Hey, I saw that GitHub has this new thing, you should check it out." So one of us will check it out and we’re like, "Dude, that's awesome. We should use that." I think that we're constantly being batted in front of the curve by a community, by a boots-on-the-ground community that knows what's up. We obviously have our own interests and our own passions, but I don't think if left to our own devices, it would look quite the same as if Apache TLPs couldn't put in tickets.

So it's been one year and one month, but how has Infra changed for you since you've come on board or has it changed?

Nope, still terrified. [chuckles]

How is the team coping with the ASF's unstoppable growth? We have 45 projects in the incubator and there's more than 300 projects out there … there's a geographic influence now on demand, fan increase in users and committers and projects from China, for example. Are there any issues that the team feels like, "Oh boy, we got to deal with this?" Is computing an international language, where it doesn't matter where you're from or what's happening? Are any shifts going on from the ASF’s growth impacting you guys beyond more of what you're already doing?

So, typically, all of my jobs really have been this kind of larger, national or international affairs so basically, since I was 20. I worked for a really large mortgage company, and then I left there and I went to a massive health insurance company. Lots of international folks and so, aside from the language barriers, yeah, I would say that computing is kind of an international thing. As far as the unlimited growth, I don't really know. I'm not sure. That sounds like a question that I would definitely advise you to go ask one of the board members about.


Right: “Management”.

You had mentioned that you were working on the no-longer-CMS project. Is there another project that you're doing? Are you a go-to guy for something?

I don't think I'm the go-to guy for anything really. I just try to pick up whatever is there to be picked up. One of the things that I'm working on right now in the “demise of CMS” project is this custom builder. I'm still working on it, so it’s still a work in progress, but the idea is that you'll be able to have a custom build environment that would allow you to, from the ASF.YAML file, write a script, do a “thing” to create your own custom build environment so that we can really, really make a hardcore concerted effort to get off CMS.

Why? What was the issue with CMS? Why do we have to migrate from it? What was the problem?

To be honest with you, I've never actually used CMS. Fortunately, I have never been asked, too. John (former Infra team member John Andrunas) was, but I was not. I was spared, by the CMS gods, they shone their countenance upon me. It was pretty awesome. From what I understand, it's very cumbersome to use and not very friendly and also very old. My understanding is that although it works, there are changes we wish we could make to it that we cannot, so it might be time to just move on to something newer that maybe works a little bit better for us because our use case has changed.

You're still rather new to the role: when you first came on board, what was the biggest challenge or surprise? What really opened your eyes?

So, what really opened my eyes was how much of a learning curve there is. Man, that was rough.

Is that still the case?

Yes, that's still the case. It's just not as bad as it was. Where I was before, I was using all of the stuff that we're not using here, all the Enterprise Edition stuff. So I came in with a completely different toolbox than what I was handed, so the learning curve was massive. I had to relearn how to use the automation software and we were all Splunk, so I had to learn the ELK stack stuff and we were Ansible or they were Ansible, the Foundation is using Puppet. Just all of it down to the monitoring. We didn't have any third party monitoring because, “government”: we had this really unfathomably convoluted Xymon setup, which was interesting but  we were using RCS for everything. So instead of git or subversion or even CVS.

Yeah, they're stuck with their legacy, that's for sure.

Yeah. You got text files in there that have got 10,000 versions in RCS. It was like, "Oh, my God. What am I going to do with this?"

So, I tried to implement some of the new hotness there. The git workflow, gitflow, actually, the exact same kind of thing that we do here.

I had a good understanding of how ASF did business from an operational standpoint. I understood it, because I've helped implement it elsewhere, but this is the first time I've ever been fully immersed in the river of PRs and tickets and all that other stuff, so it's been a hell of a learning curve, like it has really, really kicked my butt.

But you're kicking it back. I mean, you're here. You're making it work.

Oh, yeah, hustle, man. That's really all you’ve got to have is hustle.

As you're describing the way the ASF is and you were talking about some of the tools and the orchestration requirements, is this a common thing that Infrastructure today in general is heading in that direction, or is it an anomaly not only from your personal experience, obviously, but that is an anomaly but from the way you see the industry? Does “infrastructure” in general seem to be headed in this direction, or is ASF really a unique animal in that way? Do people really have to be more jack-of-all-trades?

So the ASF is a unique animal. It is. Typically, people don't have 11 Cloud providers and if they do, they've usually got some sort of system underpinning all of that whereas ours is tribal knowledge and text documents and we're really trying to get this knowledge codified and our technical writer Andrew Wetmore was really doing a kick ass job with that. But, yeah, typically an infrastructure team of this sophistication would probably have a different set of tools.

It's surprising that we're not using, like Vagrant and Packer and Teraforms which abstract the way Cloud providers make VMs. We still make them by hand. It's work, and really the only way to be good at that is to know what you're doing and to be confident in that particular UI, which is always its own special kind of awkward, trying to get used to a new UI, finding out where all the options are, and we're doing all these things by hand … everybody just picks up this knowledge through osmosis, just by stumbling through these tickets from time to time and it's really crazy to see sometime how much process there is and how little documentation there is. So I'm really happy to have our documentation writer on board.

That's Andrew, right? Andrew Wetmore is working on the documentation?

Oh, yeah. Yep, and he's doing a really good job, helping us sort it out.

And he hasn't left screaming and running either, so that's a good sign. It's a lot of work.

That's true. Yeah. It is. It is a lot of work and he has not left running, but he is a really chill dude.

Our infrastructure is unique in that we do all of the things that are kind of necessary. There really isn't too much of a go-to guy for any of this stuff. If there's a problem in the build system, you take care of it. If there's a problem with a Web server, you take care of it. That's where the autonomous nature of Infra comes in. If there's a problem, you just take care of it. You have these tools, you know how to do it, you just do it.

How do you know that someone's not fixing it on their own at the same time? If something's broken, you're like, "Hey, this is broken. I'm dealing with it" or something else?

Just slack, typically. I always check.

Yeah. Okay, what's your favorite part of the job?

Oh, gosh. My favorite part of the job is not feeling icky at the end of the day. I've worked for some companies that kind of made me feel a little ick in their mission. So one of the stories that my wife likes to tell is that I quit [MEDICAL INSURANCE COMPANY] because I disagreed with them as a company and I paid $5,000 to do so. But yeah, so I worked in the mortgage industry a little while shortly after the housing collapsed and I just thought about it. It was like, "Man, I really don't feel good about this job anymore." And then I moved to [REDACTED], which was arguably a bad move.

Big Health.

I was there for like 11 months. I signed a contract, I got a sign-on bonus, I moved to get there, so the stipulation was I stayed a year. I stayed 11 months and three weeks and I quit. I couldn't take it anymore. I'm just like, "I'm not doing this. I'm not doing this."

I was walking on an image parser for the Affordable Care Act pipeline, which was awful. They were still implementing it. This was 2012, 2013.

It was really bad. So after that, I went to NASA and I finally felt good about what I was doing and to have made a move where, again, I agree ethically and morally with what we're doing. I mean, it really is noble work, not specifically the work that I do, but the work that the people that I support do, and so, by proxy, my work is also.

At Apache, we have volunteers that dedicate hours of their life to these projects that we distribute freely because it really does make the world a better place. I mean, where would the world be without HTTPd?

What you just said right now has totally touched me. I feel like I’m ready to burst into tears, that's amazing. Really: I mean, wow. That's from the heart. I totally get you about doing things for people you don't believe in. That's so hard.

That sucks so much.

I totally get it and you're right. This is such a crazy group. It should not work and they do and it's incredible: 21 years of this. It's amazing.

Yeah, it's like trying to watch an eight-legged horse run.

[laughing] A what?!

An eight-legged horse. Somehow twice as fast, but you have no idea how it's working. Or which direction it's going to go.

I can’t stop laughing over the visual of that.

It's actually really funny because I'm a huge classics and mythology nerd. Technology was not my first choice in careers. I wanted to be a Latin teacher.

I love this. These are the backstories that everyone wants to know. You want to be a Latin teacher?!

I wanted to be a Latin teacher, yeah. I did Latin from freshman year in high school until I decided that college wasn't for me. So sophomore year, I took six years of Latin and it is really awesome what learning Latin does for your programming ability because it’s surprisingly similar to learning to code. But yeah, I make a lot of really, really stupid classics and mythlogy puns. So my daughter, her nickname is actually Livy, in reference to the famous historian, which is not something a lot of people get, but that's okay, it makes me chuckle. And Odin had an eight-legged horse that was twice as fast as the other horses, supposedly really fast because it had twice as many legs.

It's interesting with your career, you've worked at places that are big names and people would be very impressed with that, but you're stressing that just because it's a big name or big group, it's not what it's all cracked up to be. What are you most proud of with your career, your Infra career, with Infra as a whole? What makes you say “yay”?

To be honest, becoming an Apache Member was pretty freaking awesome. When I got here, when I start a new job, I always try to set a goal for that job. Sometimes I get it and sometimes I don't, and sometimes I don't realize how hard it is to actually do what I'm setting out to do when I start. My goal at NASA was to win a silver Snoopy, but that was never going to happen.

Silver Snoopy? What’s that?

That's an award given by astronauts to engineers. They don't typically give that to IT folks, but I didn't know at that time.

But here, it was to kind of become a Member and really to be accepted. I feel like I'm doing okay on that. That's pretty cool. That's going along really well.

You fast tracked. I mean, if you've been here for 13 months and you're in as a Member, that's pretty cool. That's good timing, good performance on you.

Well, thank you. I have no idea of how well or badly I am doing. I'm just doing things in the hope that they affect the universe in a positive way.

You're there, we couldn't do it without you.

That's excellent. Thank you.

You got to pat yourself on the back for the work that you're doing, because with our community, you know if you weren't doing it, you'd hear it. People would grump about it.

That's true. That's very true. But again, this is a mindset that's really prevalent in IT is the Tetris mindset where when you're playing Tetris, you fill up a row and it disappears. As such, those are your successes.

The Tetris mindset really is being bogged down by the monument to failure that you've built because really, when you're playing Tetris, that's what you're looking at is the monument of your failure, places you haven't quite gotten the row completed yet and shifted out of your bucket. And it's really easy to succumb to that mindset, especially in a place like this.

And I really, really enjoy the fact that the Apache Community is they seem eager to call out wins for other people and that is an awesome attitude for a community. It's something I've not experienced a whole lot of being called out for successes. I think that on the whole, the community and being embraced by the community has really kind of helped me not fall into that funk, that Tetris mindset just doesn't seem to be prevalent in this community, which is nice.

Do you think that puts people in a kind of "I'm not good enough" mindset because there's not a reward? You're young enough to be part of that community that likes or is accustomed to getting trophies for showing up. Apache doesn't allow that. It's nice for you to show up, but you're not going to be rewarded. Do you think there's an impact with that?

I was on a soccer team once and I did get a participation trophy. You know what? I couldn't even tell you what the name of that soccer team was because I didn't want to play soccer. So, really, I think that if you're coming to The Apache Software Foundation, you're not doing it for the participation trophy, you're doing it because you want to, so the reward doesn't matter. You're doing it because you want to. It's really weird to be surrounded by people who are motivated by nothing other than the fact that they want to be here doing this.

And it's refreshing and I love it. I do.

I love hearing that, that's great. Here come the somewhat personal questions: there's just a few of them. Chris was laughing hard when I was asking them; I don't know if you read the full Chris interview, but it's always interesting to hear what they have to say. So ... how would your co-workers describe you?

Less cool than my wife.

What is your greatest piece of advice... what would you tell aspiring infra people, sysadmins, people like yourself, what would you give them for work advice or career advice or life advice: what would you say?

Oof, that's tough. I guess I would have to say that if at the end of the day you don't feel like your job is worth it, it's probably not.

So, if you're going to do something, make it worth it. That's my advice.

If you had a magic wand, what would you see happen with ASF Infra?

What would I see happen? Well, obviously bonuses and pay raises, but I have no idea. If I had a magic wand, I'd probably turn it over to someone who I thought could make the wish better than I could, but yeah, I have no idea.

What else do we need to know that I haven't asked?

Oh, gosh. So many things, but none of them would make sense out of the context of this particular conversation. To be honest, I'm still under the impression that everybody knows more about this than I do still, so I don't know.

Drew is based in Tennessee on UTC -5. His favorite thing to drink during the workday is a black coffee prepared using a French press or the pour-over method.

# # #

Monday April 06, 2020

Success at Apache: Welcoming Communities Strengthens the Apache Way

by Jarek Potiuk

During my career, I have been a software engineer, Tech Lead Manager at Google, a robotics engineer at an AI and robotics startup, and am currently the Principal Software Engineer of a software house, Polidea, which I helped grow from 6 to 60 people within 6 years as CTO. Over the past year and a half I was a user, then contributor, then committer, and now a Project Management Committee (PMC) member of Apache Airflow.

Although I took on many roles through the years, including being the main organizer of the international tech conference (MCE), deep in my heart I was always a software engineer. It took me many years to find a place where I could explore my true potential. Then I became part of the Apache community. I first learned about the Apache Software Foundation (ASF) 20 years ago when I used the Apache HTTP server at the beginning of my career. I had only made small contributions to OSS projects up to that point, and becoming involved with Apache Airflow was the first time I contributed seriously to one. As a Principal Software Engineer at Polidea, several of our customers were using Apache Airflow and wanted to contribute back to the project to help other users. Better integrations of services with Airflow would significantly improve future releases of the software. 

The needs of our customers made me and my team go from users of the project to contributors and more. We have people in our software house who understand open source, know how to follow the OSS rules, and contribute changes from customers to help other people in the community. We know how to communicate well, we can also represent a vendor-neutral point of view because we represent the view of several customers and collaborate with all the stakeholders in the project. People in our company also contribute to other OSS projects, such as Apache Beam and Flink. 

We also discovered a great model where our customers wanted us to contribute to an open-source project to make it better because they were using it and wanted to improve it for future users. This allowed us to do it full time (or even 150% of the time if you add all the out-of-hours contributions). I invite you to read about it in my blog post The evolution of Open Source - standing on the shoulders of giants.

Committing to Apache 

I found exactly what I was looking for in the Apache Software Foundation. It’s a great organization for people like me: individual contributors who are also good at working with others, the ones who don’t shy away from organizing and making things happen, who thrive when they can do meaningful work with others. 

This made me think: since the ASF is so great, how come for 20 years I was not contributing to OSS projects more? And since so many software engineers use Apache technology, why is participation not more common? I got lucky because I was in a position that allowed and supported my contributions to an open-source project. For me, it’s a dream-come-true. But what about others? There must be more people willing to contribute and get involved in the OSS community, they probably just don’t know how to go about it yet or did not like the experience.

So here I am, sharing my thoughts on what can be done to help others to get to know ASF sooner and get involved.

Apache Airflow and the initial experience 

Apache Airflow is an exciting project. It is a platform created by the community to programmatically author, schedule and monitor workflows. It started in AirBnB in 2014, was submitted to the ASF incubator in March 2016 and it graduated to a top-level project in January 2019.

When I started working with the Apache Airflow project, I quickly realized that it was hard for me to contribute to. It was not clear how to develop and debug Airflow, how to start, and how to communicate. The project had a number of channels for communication including a developer list, a Slack channel, issues and pull requests alongside the code. As a newcomer, it was not easy to understand which channel is used for what and whether it’s OK to raise certain issues using those channels. It was not clear what were the common protocols: for example how to see that one thread is a discussion and one is voting on an already discussed topic. 

Is our community welcoming enough?

At a party after a conference where I spoke about Apache Airflow, I had a long discussion with a young engineer who was new to the field, Fabian. Fabian told me that often OSS projects create some invisible barriers around communication and onboarding. I explained to him the “Apache Way” and how transparency and openness help with those barriers, fiercely protecting the fact that “we are open”. We carried on with our friendly discussion and it was really eye-opening.

That conversation stayed with me for a while. After some time, I realized that maybe our project was not as welcoming and accessible as we thought: there should be an easier way for people to contribute and join the community. I recalled my case—when I joined the community, I made a mistake by writing that something “will happen” before discussing it with the community. A long-time community member reminded me that this is not the way we should communicate at Apache Airflow. Just to note - each project is autonomous within Apache and it defines its own communication rules. It was done in a very good and friendly tone and I took it as a lesson, but some people might be put off by such a response. Not everyone has the determination, experience, thick skin and willpower to overcome all the obstacles and some people might be put off by such responses - even if they are nice and friendly. 

Could we do better to communicate the ideals of our community more straightforwardly? Without coming across as harsh? Maybe we could find a way of explaining to the future contributors how they should communicate rather than do it by trial-and-error?

Becoming a more welcoming community

A few days and emails after the discussion with Fabian I started a thread at the developer’s list of Apache Airflow “[NON-TECHNICAL] [DISCUSS] Being an even more welcoming community?” that kicked off a conversation that included people who rarely had spoken before. As a result, we managed to introduce many changes to the processes for new members. Thanks to the input of people such as Karolina Rosół (Project Manager at Polidea), we came to the conclusion that the way seasoned community members communicate at Apache Airflow is not obvious at all to newcomers. We added missing chapters to our CONTRIBUTING documentation regarding communication channels, expected response times, and more. 

What helped a lot was that we were able to improve our documentation for the development process during last year’s Google Season of Docs program. Apache Airflow was one of the first projects at the ASF to participate in the program. I was one of the mentors to Elena Fedotova, a technical writer assigned to our project. She improved and restructured the documentation, and made it more readable and easier to understand. Many people took part in reviewing and correcting the docs. Also, we took on the task of creating a new website for Apache Airflow with modern, clean design, and well thought UX addressing different personas of visitors in mind (including new contributors, users and potential partners of Apache Airflow). One of my colleagues and fellow Apache Airflow committer, Kamil Breguła, put enormous effort into both building the website and also restructuring the documentation. 

As a result of the discussion at the developer’s list, we also introduced the mentorship options and even handled (via mentorship) a few difficult cases that could have lost us valuable contributions. A great example of the improvements we’ve done as a community might be this tweet from Vanessa, a research software engineer who had no experience with the community.  Vanessa had tried to contribute support for Singularity—a popular container technology for high-performance computing - a year earlier, and came back for a second try after much of this work was done:

Are we there yet?

Looking back, it’s been a long (yet satisfying) journey trying to make the Apache Airflow community more welcoming. But how do we know it works?

At the beginning of this year, we started to participate in the Outreachy and Google Summer of Code programs, where people from around the world with different backgrounds can be paid for contributing to open source projects. Together with my friend and PMC member Kaxil Naik, we became program mentors and started to receive a flood of requests from the Outreachy members. Initially, we were overwhelmed but soon realized that we have everything we need to answer the questions of the candidates (and future committers) to let them teach their lessons and easily follow the “contributing” documentation. The contribution environment was available for them to get started, and the documentation detailed how they could learn how to prepare contributions and communicate via various channels.

Just two days later, we approved a few pull requests from those people! That’s quite a difference from 1.5 years ago when it took days, if not weeks, to understand the environment and how to work with Apache Airflow. It was truly a team effort; many community members participated in the process and made the Airflow project much more welcoming to newcomers.

Despite having challenges during our experience getting started, I was never going to quit. I loved the project and people almost from day one. Realizing how hard it was initially to start contributing (other people had told me so as well), I decided that I would put a lot of effort (both professionally and also personally) into making the project easier and more open and accessible for people with different backgrounds and experiences. My experience starting as a contributor, then becoming a committer, and now a PMC member proves that this is possible.

To me, Success At Apache means making the community and the spirit of Apache Way more accessible to people around the world. With the difficult times that we are going through now with COVID-19, it’s more important than ever to build and strengthen various communities. And to strengthen the community means to be open to others and be welcoming, We hope that our experience will encourage you to take a look at your project and see if you can make your community more welcoming.

# # #

Jarek Potiuk started to work on the Apache Airflow project in September 2018. He became an Apache Airflow committer in April 2019 and a member of the Apache Airflow Project Management Committee (PMC) in October 2019. He is an Apache project mentor in Outreachy and Google Summer of Code and was a mentor in Google Season of Docs. Jarek is a Principal Software Engineer at Polidea and always keen on making it easier for people with different backgrounds to join OSS projects.

= = =

"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works" 

Tuesday March 31, 2020

Inside Infra: Chris Thistlethwaite

"Inside Infra" is a new interview series with members of the ASF Infrastructure team. The series opens with an interview with Chris Thistlethwaite, who shares his experience with Sally Khudairi, ASF VP Marketing & Publicity.

"I get very attached to the technology that I'm working with and the communities that I'm working with, so if a server goes down or a site's acting wonky, I take that very personally. That reflects on how I do my job.

Let’s start with you telling us your name --how is it pronounced?

It’s “Chris Thistle-wait” --I don’t correct people who say “thistle-th-wait”-- that’s also correct, but our branch of the family doesn’t pronounce the second “th”.

What’s your handle if people are trying to find you? I know you’re "christ" (pronounced "Chris T") on the internal ASF Slack channel.

Yeah --anything ASF-related is all under "christ".

Do people call you "Christ"?

They do! I first started in IT around Christmastime and was doing desktop support and office-type IT. When people started putting in tickets, and my username  was "christ" there, they were asking "why was Christ logging into my computer right now?" and it became a thing. When I was hired at the ASF I told Greg (Stein; ASF Infrastructure Administrator) about that story, he said "you gotta go with that for your Apache username."

When and how did you get involved with the ASF?

A long time ago I started getting into Linux and Open Source, and naturally progressed to httpd (Apache HTTP Server). Truth be told, that’s where it started and stopped, but I’ve always been interested in Open Source and working with projects and within communities. Three years ago I was looking for a new job and stumbled across the infra blog post for a job opening. I fired up an email, sent it off to VP infra and that’s how everything started. The ramp up of the job was diving deep into everything there is with the ASF and Open Source --which I am still diving. I don't think I found the bottom yet with the ASF.

How long have you been a member of the Infrastructure team?

This November will be my fourth year.

What are you responsible for in ASF Infrastructure?

Infrastructure has a whole bunch of different services that are used by both Apache projects as well as the Foundation itself: the Infrastructure team builds, monitors, supports, and keeps all those things running. Anything from Jenkins to mailing lists to Git, SVN repositories and on the back end of things we keep everything working for the Foundation itself within, say, SVN or mailing lists, keeping archives of those things, keeping your standard security and permissions set up and split out. Anyone you ask on the Infra team will say: "I do everything!" It's too hard to explain --it's quite possibly a little bit of everything that has anything to do with technology --as broad as it can possibly be.

So you really have to be a jack-of-all-trades. Do you have a specialty, or does everybody literally do everything?

Everyone on the team generally does everything --for the most part any one of us can jump into the role of anyone else on the team. Everyone has a deep knowledge of a particular or a handful of services that they’ll take care of --like, Gavin (McDonald; ASF Infrastructure team member) knows more about Jenkins and the buildbot and build services than most people on the team. At any one given point we’re on call and need to be able to fix something or take a look at something, so everyone needs to be versed enough in how to troubleshoot Jenkins. That can also be said for not just services that we offer, but also parts of technology, like MySQL or Postgres or our mail system or DNS: we do have actual physical hardware in some places, and we have VMs everywhere too, so sometimes we’re troubleshooting a bad backplane on a server or why a VM is acting the way it is. There's a very broad knowledge base that all of us have but there are specifics that some people know more about than others.

How does ASF Infrastructure differ from other organizations?

There are a lot of similarities but a ton of differences. A big part of how Infra is different is, to use a "Sally-ism": if you look at it on paper, it wouldn't work --I've heard you describe the ASF that way. If you explained the way things work at the Foundation to somebody, they would literally think that you're making it up and there's no way that it would possibly be working the way that it does. There's a lot of that with the Infrastructure team too: many people that I keep in contact with that I've worked with over the years, from my first job where we would buy servers, unbox them, rack them, wire them up, set them up, and run them from the office next door to us --I'd be impressed whenever I had 25 servers running in our little "data center" at that job, and now I talk to these guys about what we do at the ASF: we have 200 servers in 10+ different data centers that are vendor-agnostic and we make it all work. They ask: "how the heck do you do that?!" We just do --it's an interesting thing as to how it all works together because we solve problems that others have as well, but their problems are often centralized to one thing, or a data center that they control and own, or one cloud provider that they control and own, where they deal with a single vendor and possibly at most have to talk with the same vendor in two different geographical areas. We're having to deal with stuff with one cloud vendor that's a VM and other stuff on the other side of the world that's actual hardware in a co-location or data center running and the only thing that makes them the same is that they're on the Internet.

It's a good summation of the team too due to the fact that we’re all based out of worldwide locations, we’re not all in one spot doing something.

Describe your typical workday. Since you're all working on different things on such a huge scale, what's it like to be you?

"It's amazing" [laughs]. Everyone on the team generally has some project or projects that they are working on --long-running things for Infra. 

I'm currently working on rewriting a script for Apache ID creations. The process of putting your ICLA in, sending off to the Secretary, the Secretary says, "OK good," puts in all your data, and that gets put into a file in SVN ...currently, we have a script that we manually run that does a bunch of checks on the account and whatnot, and then creates it, sends off a welcome email, whatever. I'm rewriting that because it's an old script, it's in several different languages. It's actually six scripts that all run off of one script. Consolidating that into one, massive script, that's in a supported language for us, and then moving forward with it into something that we could potentially automate, versus me having to run a script manually a couple of times a day.

Fluxo (the ID/handle for Apache Infra team member Chris Lambertus) was working on some mail archive stuff in our mail servers. Gavin (Apache Infra team member Gavin McDonald) is working on some actual build stuff. Everyone has kind of "one-two punch" tasks that they work on during the day, and then the rest of the time is (Jira) tickets or staying on top of Slack, if people are asking questions in the Infra channel or in our team channel or something like that. The rest of it is bouncing around inside the ASF and checking things out, or finding out new projects to work on, or ways to improve such-and-such process. 

How many requests does Infra usually receive a day, in general?

Over the past three years, we've resolved an average of 6 Jira tickets a day, year-round. We've had 213 commits to puppet repositories in the last 30 days. We handle thousands of messages on our #asfinfra Slack channel, and have had 659 different email topics in the last year.

Dovetailing that, how do you keep your workload organized?

Everyone on the team does it their own personal way. I have a whiteboard and a Todoist list. We also have Jira to keep our actual tickets prioritized and running. We have a weekly team meeting/call and talk about things that are going on, and is the more social aspect of what we do week-to-week.

How do you get things done? You're juggling a lot of requests --what's the structure of the team? How do you prioritize when things are coming in? Is there a go-to person for certain things? If you're sharing everything, how do you balance it and who structures it? How does that work? 

To one end, the funnel to us starts with Greg and David (ASF Infrastructure Administrator Greg Stein and VP Infrastructure David Nalley). It's different from other places that I've worked, where I'm on a team of other systems administrator/engineering people, and we have a singular, customer-facing site. Someone says, "Hey, this should be blue instead of red," there's a ticket and we make the change and then it goes to the production.

There're many different ways to get a hold of the Infrastructure team. Everyone gets emails about Jira tickets and gets updated as soon as one of those comes in. If it's something that you know about --say, the Windows nodes that we handle-- those all fall into my wheelhouse because I'm the last one to work with Windows extensively. Everyone else knows how to work with them, but it makes more sense for me to pick it up in some cases. 

Most of the stuff in Jira are very "break-fix" kinds of things. A lot of the requests on Slack are too, for example: "DNS is busted," and we fix DNS. It's a very quick, conversational, "Let me go change that," or, "I'm going to go fix that real quick." Of course, some of the Jira tickets are very long-running, but the end result is they're fixing something that used to work. 

We were originally running, and Git WIP, so we hosted our own internal Git servers and we would read-only mirror those out to GitHub. Somewhere along the line, Humbedooh (the ID/handle for Apache Infrastructure team member Daniel Gruno) started writing out Gitbox or building Gitbox based on the need to have writable GitHub repositories. He built Gitbox and set up with the help of some other people on the team, got it going, and that became our replacement for While we still host our own Git repositories, people are free to either write to ours or write to GitHub, and the changes are instantaneously mirrored between the two.

We had Git hosted at the ASF, and had GitHub as a read-only resource. The need arose to have rewrite on both sides: Humbedooh went and built out MATT (Merge All The Things), which does all of the sync between GitHub and our Git instance.

MATT started a while ago, and Humbedooh added on to that to do the rewrite to GitHub. Basically what all that does is once your Apache ID is created, or if you have one already, you go on, you add your GitHub username in there and then MATT --there's another part of that called Grouper-- MATT and/or Grouper will run periodically, pull data from our LDAP system and say, "Oh, ChrisT at has ChrisT as his GitHub ID. I'll pull those down." It says, "ChrisT is in the Infrastructure group. Hey look, there's an Infrastructure group in GitHub. I'll give ChrisT write access to the GitHub project." In a nutshell, that's what that does.

There's a ton of other house cleaning things, if you get removed from the LDAP group ... we run LDAP and keep all this stuff straight. If you get removed from the Infrastructure group at LDAP then MATT/Grouper will go and say, "Oh, this person's not in this LDAP group but they do have access in GitHub. Let me pull that so that they don't have access to that any more." It does housekeeping of everything as well as additions to groups and that kind of thing. There's a ton of technical backend to that, and that's what Humbedooh's doing. 

At first when Git and GitHub were set up, it was fine: the ASF has to keep canonical everything about what goes into each project. You could only write to our Git repos. Then it was conveniently mirrored out to GitHub because there's a lot of tools that GitHub has that we didn't have or weren't prepared to set up. GitHub has a very familiar way of doing things for a lot of developers. Once GitHub Writable came along with Gitbox and the changes to MATT, that opened up a whole other world of tools for people on projects to use. If they wanted to use pull requests on GitHub, they could start using pull requests on GitHub to manage code. They could wire up their build systems to GitHub with Jenkins so that whenever a PR was submitted and got approved, it would kick off a build in Jenkins and go through unit tests and do all the lovely things that Jenkins does.

It was really an evolution of, "Here's the service that we have. Someone, somewhere, be it infrastructure or otherwise, once they have writable GitHub access, here we go." And here's the swath of things that that now opens up to projects inside the ASF that if they could come and set up a project with us, and then never, ever actually commit code to the ASF, it would always go to GitHub but still be safe and saved on our GitHub servers for ASF project reasons.

At the same point, we saw a need and said, "Let's build this out and go." Another funnel that comes into us is when we're on-call, something breaks and we ask, "Why do we do it this way? We should be doing it a different way." We then come up with a project to fix that or build it. It's a very interesting process of how work gets into the Infrastructure team.

It's been an interesting ride with that one.

There's always stuff that we're working and fixing and making better. For the most part, Gitbox as it is now is kind of in a state of "It's getting worked on". If there are bugs that need fixed, it gets fixed, but I don't know what the next feature request is on Gitbox. There's talk of other services GitLab. If someone wanted to write code and put it in GitLab as opposed to  GitHub, then someone would need to come in and write the connector from Gitbox to GitLab. So it's possible. I don't know if that's necessarily an Infrastructure need as much as it is a volunteer need for infra. But it's a system that can be set up to any other Git service as long as someone goes in and writes that.

You brought up an interesting point here, which is volunteers. Do volunteers contribute to Infra also? 

We sometimes have volunteers, yes. We have a lot of people on the infra mailing lists that will bounce ideas back to us or they'll work on a ticket or put in a pull request.

Well, the need is not as critical because you have a paid team, versus Apache projects. 

Right. That's exactly true. There's a bit of a wall that we have to have because we work with Foundation data, which not everyone has access to. Granted, we're a non-profit, Open Source company and everything's out there to begin with, but usernames and passwords of databases and things that we have encrypted that the team has access to isn't necessarily something that you would want any volunteer to have access to.

How do you stay ahead of demand? This is a really interesting thing because part of it is you're saying, "Necessity is the mother of invention." You guys are doing stuff because you've got those binary, "break-fix" types of scenarios. In an ideal situation, do you even have enough runway to be able to optimize your processes? How do you have the opportunity to fix things and improve things as you're going along if you're firefighting pretty much all day long?

That's a really good question about just how our workflow is. In other companies that I've been in, there's the operations people that are doing the "break-fix", and then there's the development people that are doing "the next big thing". The break-fix folks are spinning the plates and keeping them spun without breaking, and that's a lot of firefighting. That's literally all that job is. Even when you're not firefighting, you're sitting around thinking about firefighting in a sense of, “when is this going to fall over again? If it does fall over, what can we do to fix it so it doesn't do that anymore?" And in the past, the break-fix guys, the firefighters, would end up saying, "Hey, there's this thing that needs fixed." And it would fall over the wall to the developers. They would develop the fix for it, and then it would go back into production and then the cycle continues. 

To some extent, that's kind of where DevOps came from: if you merge the two of those together, then while you're firefighting you can also write the fix for the problem, and then you don't have to wait for the lag between the two. We don't have that split here. Everyone on the team is firefighting with one hand and typing out the solution with another. And a lot of the times our project work, like getting a new mail server spun up or my task to rewrite the workflow for new Apache ID creations, I've been working on that for a very long time because it will keep falling off ... it gets put on the backburner while we're like, "Hey, we found out that our TLP servers are getting hammered with downloads from apps and people trying to use them instead of the mirror servers." So, let's set up and we can funnel stuff over to that so that that server can get hammered and do whatever it needs to do so that our www. site and all the Apache Project websites stay up and running in a more reliable way.

What's the size of the teams that you were dealing with before that had a firefighting team and a dev team versus ASF infra?

The last "big" corporate job I had was ...six ops people that kept the site going, four database people, another eight technical operations-type people… all-told it was about thirty.

There were technically thirty firefighting people and we had a NOC (network operations center) that was literally people that only watched dashboards and watched for alerts. Whenever those go off, they’d call the firefighting people. The NOC was another 20 people. And then the development teams were ... twenty to fifty people.

What kind of consumer base were they accommodating? Does it match the volume that ASF has? Was it more of a direct, enterprise type of, "We have a customer that's paying, we have to respond to them" situation? Or is it different?

This was at a financial services company that transacted on their Website: completely different from the type of stuff we're dealing with here at the ASF. Volume-wise, they were much smaller, but it was much more ...visible, as their big times were at the start of market and end of market. After end-of-market came all the processing for the day to get done before markets started the next day. The site had to be up 100% of the time. We had SLAs of five minutes. If you got paged or something broke, you had to get the page and respond to it in a way of, "Hey, this is what's going on and these are the people that I need involved with it," all within five minutes of it going off. That was the way the management structure was. It was intense.

In scale, Apache probably does way more: they do way more traffic across all of our services in any given day. If someone doesn't get mail for a little bit, then they come and tell us or we get alerted of it by our systems, and we go and we fix it and we take care of it. But with the financial services group, people were losing money: dealing with people and money is just a very stressful situation for anyone working in technology because you have to get it right and it has to be done as fast as possible before someone's kids can’t go to college anymore. It was a completely different minefield to navigate.

The type of stress that's involved or the type of demand or the pressure is different, but you also have the responsibility with ASF that systems have to be up and running. I understand it's not mission critical if something goes down for more than five minutes, which is different in the financial sector, but do you feel that same type of pressure? Is it there or is it completely different for you? 

No. I think I do because we also have SLAs here: they're just not five minutes. We have structure around that and the way that we handle uptime and that kind of thing. I get very attached to the technology that I'm working with and the communities that I'm working with, so if a server goes down or a site's acting wonky, I take that very personally. That reflects on how I do my job. If a server's not working or if something's broken either because of me or something externally that's going on, I want to get that up and running as fast as possible because that's how I would expect anyone to work in a field that has ...any technology field, for that matter. And generally, that's the same attitude the rest of the team has as well.

How has ASF Infra changed over the years?

It's matured quite a bit. When I first started, it was Gavin, Fluxo, Humbedooh, Pono (former ASF Infrastructure team member Daniel Takamori), and me. There were five of us. The amount of stuff that we got done, I'm like, "Man, there's no way that five people can do this."

That's kind of what I'm pointing at. If you're a team of eight or five or twelve or whatever, compared to the other thing that you did with the other job that had maybe a core team of twenty, thirty --that in itself is insane.

We were five people, everything was very, "Here's the shiny thing we're working on," and then something else would come up and we'd have to jump on that. Then something else would come up and we'd have to jump on that. We were very ...I don't want to say we were stretched thin, but there wasn't necessarily ...time for improvement.

There was a lot of stuff we had still in physical hardware, and a couple of vendors that we no longer use. But things were moving more towards a configuration-based infrastructure with Puppet instead of one person building a machine, setting up all the configs themselves, installing everything and then letting it go off into the ether to run and do its job. We were moving everything towards Puppet to where you configure Puppet to configure the server. So then if the server breaks, or goes down or goes away or we need to move vendors or whatever, all you need to do is spin up a new server somewhere else, give it the Puppet config, it configures itself and then goes off into the ether to run and do whatever it needs to do.

That's great. More automation.

Right. We were automating a lot more stuff right when I first started. Over the course of the next year, the team kind of ebbed and flowed a little bit until we were eight in the last year. We started to get to the point of "where can we point the gun to next? What can we target next to get it taken care of and done?" That's where we started taking on more specific infra projects, for instance, mail. Our mail server has been around since the dawn of time, and it's virtualized so it moves servers every now and then, but the same base of it is quite old in technology standards.

Fluxo started moving this on to newer stuff and he got that going. We started taking care of projects that were not broken, but needed to be worked on. Instead of waiting for it to break, we're fixing and upgrading and moving down that path versus firefighting, break-fix, that kind of thing. We were moving more towards, "Hey, I see a problem. I have time. I'm going to take care of that and make that into a more serviceable system." 

Automation has helped quite a bit with that. I also think that just as the team grew, it just got to a point where I think tickets were getting responded to quicker, emails, chat was responded to quicker. And then we also could focus more on the tools that we use for the foundation. Like, HipChat was going away. We needed a new chat platform, so we chose Slack. And then we updated and moved everything over to Slack, and that's where we are with that. It started following into its own with workflows of like, "Oh, okay. How do we get this done? Let's go do that."

What areas are you experiencing your biggest growth? Is it a technical area? Like, "Hey, all of a sudden mail's out of control"? Or, "Hey, we need to satiate the demand for more virtual machines," or is it a geographic influence that's coming in in terms of draw? Where are you guys pointing all your guns to?

Currently we're trying to get more out to the projects and talk to people more often. Not that we didn't do that before, because ApacheCons and any Meetups that we had, Infra would always have a table. We were always accessible, but we were always passively accessible. We weren't really going out and talking to projects proactively to say, "Hey. What do you guys need from us? What are we doing with this?" So I think that's one part of it, that I think that we're moving towards a little bit. It's not at all technical, but more of a foundation broadening, community broadening thing that we're doing.

That's one part of it. The other thing that we're doing too is from a more technical or infrastructure standpoint, is we're really trying to get our arms around all of the services we provide, and then really take a look at those and say, how is this used inside the ASF? How is it used in the industry as a whole? Do we need to put more time and energy towards those things in order to make the offerings of the infrastructure team a little bit a more solid platform, kind of thing? Generally, that ... and on top of any other automation and that kind of stuff, I think that's really the two spots that I see infra growing in a lot in the next year-ish of just really boiling down our services to, "Hey, we've seen a lot of people using this. And a lot more projects are using this. It's not just a flash in the pan. We need to build out more infra around blah service, so let's really do that and make that a solid platform to use."

What do you think people would be surprised to know about ASF Infra? When you tell someone something about your job and they go, "Whoa, I had no idea" or, "That's crazy." What would people be surprised to know?

That Apache has an infrastructure team. [laughs]

Why are you saying that?

Because honestly, I don't think a lot of people know about the Infrastructure team. Those that do, have used us for something, not used us for something, have talked to us about something, and worked with us on something. Those that don't are like, "Oh, I didn't know the ASF paid people to be here," --that kind of thing. That's kind of the two reactions I've got from people. It's like, "Oh, that's cool. You work for the infrastructure team." Shrug. And then the other people are like, "Oh, sweet. Yeah, that's great. I know Gav. I've worked with him on blah, blah, blah." But that's not necessarily surprising. I mean, it is in a sort of way. 

When people ask, "What are you doing for work?" and you say you work for ASF, do people even know what that is? Do they know what you're doing? Do they care? Are they like, "Oh, okay. Whatever"?

There's literally three types of people that I've run into that ask, "Oh, what are you doing for work?" One person is the person that has no idea what the ASF is, not even the vaguest hint of Apache, and they're like, "Oh, okay. That's cool." There's that next person that does, and may or may not know about the ASF but knows of Apache, the Web server, or some other lineage of that.  They're like, "Oh, whoa. That's super cool. It's impressive.” That's wild. Then the third people ask "Why are ‘Indians’/Native Americans running software? That doesn't make any sense to me" and "Are you on a reserve?" I swear to God I've gotten that question before. I don't even know how to answer that. I'm like, "No, buddy."

Are these technologists or are these just guys off the street? Are they in the industry?

Guys off the street. I say Apache Software Foundation, and they're like "Apache" and "software" doesn’t make sense. Actually I've gotten mean tweets too whenever I've been tweeting about being at ApacheCon. Things like I'm "taking away" from Native Americans and whatever...

We also get that on Twitter, on the Foundation side: we get included in tweets about some kind of violation along the lines of, "Stand up for the ..." I get it. From time to time we also get sent these "How dare you?" letters, that sort of kind of thing. It's an interesting challenge, the whole issue of "why do Native Americans run this thing?" misinterpretation.

Let’s move on. What's your favorite part of your job?

The whole job is my favorite part of the job.

That's funny because everyone at Infra ... You know how people have bad days or may be grumpy or whatever, in general you guys seem to all like each other. You all have a great camaraderie. You all get along. You work really closely together. It's a very interesting thing to see from the outside. Is that true? Or are you just playing it up? Does it really work that way?

That's absolutely true. I've found that generally speaking, when you get a bunch of nerds together, they either really like each other and everything works or they really don't like each other and nothing gets done. The team is great, and it's like no other team I've ever worked with before. But it's very odd because you go through the interview process, and the interviews are interviews. I mean, you get to know people in interviews, but not really. Then you start working with people, and at some point you start getting below the surface. And at some point you get deep enough to where you find out whether or not you gel with all these people. 

It's very odd that all of us have the same general sense of humor. We'll talk about food non-stop in the channel, and recipes and cooking, and different beers or different whatevers. It's nice to get to that point with a team that you're comfortable enough with everybody to ... like I said, I've been here three years and there is still so much that I don't know, both technical and non-technical, about the ASF. I ask very dumb questions in channel and say, "I have no idea why this is doing this this way," or, "Can someone else take a look?" or, "I don't know what I'm doing here." And never in the entire time I've been here, from the day one until now, has anyone ever chastised me for not knowing something or said anything about the way that I work or something like that. Well, at least not in channel. At least not publicly. 

Everyone's very supportive. It doesn't matter if you know everything there possibly is to know about one singular product or thing you're working on, or don't know anything about it. You can ask questions and really learn about why it was done the way it was done, or figure out how to fix a problem. No problem on the team. It's just like, "Okay, yeah. This is what you have to do." Or, "Here's a document. Read up on it." Or, "I don't know either." And then out of that comes an hour of conversation and then a document pops out, and then the next person that asks, we can say, "Here, go read the doc." Yeah. I mean, we're all very happy. Very happy.

Which is really good. Looking back when you first started, what was your biggest challenge when you came onto the team?

Oh man. I look back at that and I feel like the learning curve was ... It wasn't a curve. It was a wall. I've used Linux, I've used Ubuntu for a while and various other flavors of Debian and whatnot, so getting spun up on all of ...expanding my Linux knowledge was a big deal, expanding everything about the ASF and how it works. Which I'm still trying to figure out. If you know, send me something to read to figure out how that all works. I mean, I don't want to sound like I was completely out of my depth and I have no idea what I'm doing, but I feel like I was completely out of my depth and I had no idea what I was doing. 

There's a lot about the ASF that is just tribal knowledge, and there's a lot about Infra that's tribal knowledge. It's just no one has anything written down --"the server's been running under Jim's desk for the last 15 years in a basement that has battery backups and redundant Internet, so it's never gone down. But don't ever touch that server, because if it goes down, then all of our mail goes down" or whatever. There was a lot of figuring all that out for myself and digging around. Which is, frankly, one of the parts that I really enjoy, is just, "Hey, this thing broke. I've no idea what that thing is. I've no idea where it lives," and just diving in and trying to figure out what's going on with it and how it's built, and then the hair trigger that sets it off to crash and never work again. Yeah. That's an interesting question too.

What are you most proud of in your Infra career to date? You're talking about overcoming these challenges, I'm always curious just to see what people are like, "Yeah, I'm patting myself on the back for that one" or, "Ta-da. That's my ta-da moment."

I did lightning talks at ApacheCon Las Vegas and didn't get a phone call from you when I was done. [laughs]

I wasn't at lightning talks --what did you say? What would make me call you?

I didn't say it. We were on stage, and it's John (former ASF Infrastructure team member John Andrunas), Drew (ASF Infrastructure team member Drew Foulks), and I, and we figured we'd do lightning talks: "Hey, we're the new guys: ask us infrastructure questions." A week or two before ApacheCon, there was a massive outage at a particular vendor. It wasn't: "Oh, our server's down for a while," the server went down and then it was *gone*. It got erased from the vendor side. I can't remember what service it was. There was something that disappeared two weeks before Vegas and never came back. 

It wasn't just us, though: tons of companies had this issue. So we're on stage answering questions, and someone asks where this service went: "What happened to XYZ?" And John has the mic and he goes, "You should probably go ask [vendor name]." At that point it was very widely published that the vendor"s response was like, "Whoops, someone tripped over the cord that powered the data center. And when it came back up, then deleted all of your VMs.” They totally acknowledged it and they didn't give refunds for it, so it was a little bit of a PR kerfuffle for them. The vendor is in the other room handing out buttons and stickers, and John was like, "Oh yeah, go ask the [vendor] guys what happened to your server. That's their fault," he said it jokingly but my jaw dropped. 

[laughs] No one told me this story. No one said anything. Someone's trying to protect you. I had no idea this happened ...oh my gosh.

Well, David Nalley was in the back of the room, and he's screaming with his hands cupped around his mouth, "Don't badmouth the vendor and the sponsors." I deflected and quickly moved onto something else. [laughs]

But yes, that's another good question that I haven't actually reflected on. Looking back and seeing where Infra was when I first started and where it is now, it was a very runnable and very good team then, and it's a very runnable and it's a very good team now. I feel like a lot of the work that I've done and a lot of the work that the team has done over the last three years has been getting from a spot of "everything's on fire, who's holding up what this weekend?" to things being stable and us nitpicking on whether or not something needs to be updated or not. That's huge. That's a big step from like starting a company and treading water to being profitable and having resources to do other things versus just keeping your employees paid. I mean, it's a big step for a company and it's a big step for Infrastructure.

I love your talking about how you guys are tightly-knit and all that. How would your co-workers describe you?

The other odd part about that too is being completely remote and not having day-to-day, face-to-face interactions with people. You get a very odd sense of people through text for a 24-hour period that you're online reading stuff. It's a different perspective than if I was in the office every day, working on something and interacting with people. Even though every day, except for the weekends, I'm online talking to these guys and doing stuff. How would they describe me? Dashingly good looking and ... I don't know. [laughs]

I know that Infra's "just Infra," right --you guys are all under the Infra umbrella. Do you have a title? When you got hired, what do they call you?

We're all systems administrators. The only person that actually has a title is Greg, and he's Infrastructure Administrator.

What are the biggest threats you face? For infra folks or systems administrators or infrastructure administrators even, what do you need to watch out for these days? What's big in the industry? Is everyone saying, "Oh, XYZ's coming"? In terms of your role in the job: is there something that you need to keep your eye on? Is there something that you would advise other people, "If you're in this job keep an eye out for blah, this is a new threat" or anything along those lines?

General scope stuff. 16 years ago, everything was hardware: you bought hardware and you had to physically put it somewhere. And virtual machines came along about the same time. People were starting to do virtual stuff to where you could have a physical machine and then multiple machines running on that, sharing resources. Then cloud and infrastructure as a service, and everything's been moving more and more towards that over the years.

Of course, there's still people that work in office IT, doing desk support stuff or office infrastructure type things.Those are still a majority of how things run at companies. As everything is moved more towards the cloud or hosted services, more systems administrators are becoming more like software engineers. And software engineers are becoming more like systems administrators. They're kind of melding into one, big group of people. Now of course, there are still people that only write software. But gone are the days where it used to be someone would write some code and say, "I need to deploy it and get it out to all these computers." They would write the code, they'd hand it off to a systems person. Systems would go and configure on whatever server to get it out to however many machines and hit the button and go. The software developer never really needed to know hardware specifics of the systems that it was going to run on. And the systems people never really needed to know what software packages this was getting put together. There's exceptions to that, but for the most part ... 

Over the years, it's fallen into a thing now where the software developer knows exactly what systems this is going to run on and how it's going to run there, so it's more efficient and things work better and they're releasing less buggy code based on the fact that they know they're closer to the hardware. And the systems people, they want to troubleshoot it more and work with it and fix problems because they're closer to the software and know more about its internal workings and how it's going to run on systems. Everything is getting more and more chunked down into, first it was VMs, then it's cloud, then it's containers with Docker and things like that, and it's going to get more virtualized down into that. Knowing about Docker orchestration and things like Kubernetes and Apache Mesos. The reality is other people run Kubernetes, people run Docker, people run everything. That's the interesting thing in terms of how they do it at ASF. We don't require folks to do just one thing.

In terms of where the industry's going ... everything's getting pushed down to "a developer can work in a container on a set of systems, write software for that and then deploy that to a machine themselves, never involving a systems engineer at all, and build a product using that." It's getting stuff out the door faster, and it's also keeping the unicorn of the industry a while to go ... even today, I developed this thing, it works on my machine. If I move it over to another computer, it stops working. Why? What's the problem with that? Containering or containers fix that problem. The container you run on my system runs the same way as it does on every system everywhere. It takes the "runs on my machine" thing out of the equation. 

What's your greatest piece of advice? What would you tell aspiring sysadmins?

Part of the ASF is the community behind it, and a giant part of that is what makes it work. I mean, you could say all of it. That's what makes everything work with this. Right when I first started the sysadmin kind of thing, I didn't get into Meetups and Linux Users Groups and any of that stuff. I didn't get into the network. I didn't go into the community that I had around me. And honestly, I don't know if that's because it didn't exist or because I didn't know about it or what, but now that I'm older and wiser, the community part of it is really ...there's a massive benefit to that. Aside from socialization, or networking and how to get a better job through networking, getting together with like-minded people and talking through your problems is an amazing tool to use. And I didn't do that enough when I was a sysadmin starting out, and looking back it's something that I sort of regret not doing, was really sharing knowledge with other people in the community and building a group of people that I could ping ideas off of, or help with other ideas, or share in the knowledge of, "Hey, this is what's going on in the industry" or, "Hey, I saw this at work the other day. How do we work around that?" or that kind of thing. It's much easier these days with social media: the never-ending amounts of social media. But it's a big, important part of my day-to-day now, that I wish I had 16 years ago.

That's powerful. OK, If you had a magic wand, what would you see happen with ASF infra?

If I had a magic wand, I'd update our mail server instantly or maybe magic wand a few other projects.

Wait. I know you're joking, but what is the problem with the mail server?

It's running on an older version FreeBSD that doesn't play well with our current tools. Some form of that server has been upgraded, patched, moved, migrated, etc for the last 20 years. We want to bring it up to more modern standards. Mail runs fine for the most part, but it's probably the most critical service we have at the ASF and we want to make sure everything continues to hum along. Because of that, it's a huge project that touches a ton of different parts of our infrastructure.

How big is it?

It's all of our email. Every email that goes through an address.

This is a huge project and Chris (Lambertus) has been working on it for a while --it's not a simple thing to fix. It's very, very complicated. We couldn’t do it without him.

Back to the magic wand thing: I'd wish for more wands. 

Chris is based in Pennsylvania on UTC -4. His favorite thing to eat during the workday is chicken ramen.

# # #

Wednesday March 04, 2020

Success at Apache: Google Summer of Code Mentorship --inside the GSoC 2019 Mentor Summit

by Sanyam Goel & Kevin A. McGrail

Sanyam first came to the ASF as a Google Summer of Code (GSoC) student in 2017; since then he has become a committer and contributor to Apache Fineract and active participant with Apache community initiatives. Sanyam, along with Kevin (a.k.a. “KAM”), a long-time  ASF Member involved with the Apache Incubator and SpamAssassin projects, were selected to represent the Apache Software Foundation at GSoC’s 2019 Mentor Summit.

Google Summer of Code is a global program focused on introducing students to open source software development. Students work on a 3 month programming project with an open source organization during their break from university.

Since its inception in 2005, the program has brought together 15,000+ student participants and 25,000+ mentors from over 118 countries worldwide. Google Summer of Code has produced 36,000,000+ lines of code for 686 open source organizations.

As a part of Google Summer of Code, student participants are paired with a mentor from the participating organizations, gaining exposure to real-world software development and techniques. Students have the opportunity to spend the break between their school semesters earning a stipend while working in areas related to their interests.

About the ASF and GSOoC: “The Apache Software Foundation has been a GSoC mentoring organization every year since the program’s inception. As a mentoring organization, the ASF is able to draw attention and new talent to many of its projects; Apache projects benefit from contributions and galvanize new community members by mentoring students; and students have an invaluable opportunity to gain experience by working directly with the individuals behind Apache projects. This, in turn, enriches the Apache community as a whole, and furthers the ASF’s mission of providing software for the public good.”

At the ASF, GSoC is overseen by Apache Community Development (“ComDev”), the committee that welcomes new participants to the Apache community and mentors them in “The Apache Way”. Former ComDev VP and Google Summer of Code administrator Ulrich Stärk, along with Apache OpenMeetings VP and GSoC mentor, Maxim Solodovnik, helped lead the ASF’s participation in GSoC this year, with the support of numerous Apache community members.

The ASF provides an established framework for intellectual property and financial contributions that simultaneously limits contributors potential legal exposure. Through a collaborative and meritocratic development process known as “The Apache Way”, Apache projects deliver enterprise-grade, freely available software products that attract large communities of users. The pragmatic Apache License makes it easy for all users, commercial and individual, to deploy Apache products.

As we gear up for Google Summer of Code 2020, we wanted to take a moment and share some of the experiences from last year’s GSOC!

In Google Summer of Code 2019, 23 students were selected by a careful analysis and ranking.  17 students successfully completed their Google Summer of Code projects with the support of 45 mentors spread across dozens of Apache projects that include Allura, AsterixDB, Beam, Camel, Fineract, Gora, Kudu, Mnemonic, Nemo (Incubating), OODT, SpamAssassin, and more.

Quick Report on the GSoC 2019 Numbers for

Accepted projects: 23

1st evaluation: 22 passed, 1 failed

2nd evaluation: 17 passed, 5 failed

3rd evaluation: all passed

Total Apache Mentors: 45

Sanyam and KAM were lucky enough to be selected as the delegates of the Apache Software Foundation for the GSoC Mentor Summit & the 15th GSoC anniversary.

On 10th March 2019 we got our invitations from Google: “You have been invited to be a Mentor for The Apache Software Foundation in Google Summer of Code 2019”.

With this invitation, there comes a huge pool of responsibilities to mentor students.  For Sanyam, it was his first time to provide mentorship at such a great level and to drive the complete project with the college student.

Sanyam: “By providing the complete guidance throughout the GSoC Period at the same time, though I had provided mentorship to at the university level to juniors in college. I also learned to manage the project and how to play the role of project lead to fulfill the project with the timelines with the student.

I was really excited to meet Google Open Source team in person and Kevin A. Mc Grail (KAM) along with 332 mentors from 162 organizations and 42 countries to share their ideas about open source and to discuss their experience of GSoC 2019. I would like to thank Ulrich Stärk and Maxim Solodovnik for serving as an organization admin for the ASF community.”


Day 1: Thursday | Munich, Germany - Marriott München

Day 1 of the summit is started by checkin into the Marriott Hotel, where we met the Google OPSO team just near the entrance and reception of the hotel.

Google OPSO team was very welcoming and welcomed every mentor by providing a Goodie bag along with a mouth watering sweet.

At the reception, we met Mario Behling from FOSSASIA community along with mentors from various organisations like Mifos Initiative, SCoRE Labs and DBpedia where we talked about the pocket science project. 

Then we all headed to lunch, where we met dove into the discussions about the OSS and how umbrella organisation manages the student applications to select the students for Google Summer of Code.

GSoC Mentor Summit started with the opening reception dinner along with opening notes from the Google OPSO team which lead to a small game named as person scavenger hunt which had a sole purpose to connect and meet the mentors from different organisations and to interact with them to discuss more about open source with some drinks and food.


Day 2: Friday | Munich, Germany - Fun Day (City Scavenger hunt / Castle Tour)

On the celebration of the 15th anniversary of GSoC, Google allocated an extra day this year at the mentor summit for fun activities like Castle tour and City Scavenger hunt.

Sanyam participated in the Scavenger hunt where some group of mentors had to explore the city on their own to find the clues and the top 2 teams got the prize. Sanyam was lucky enough to be with the winners team. And some mentors like KAM went for a really nice castle tour thanks to our host, Google.

The day ended up with informal conversations among the mentors over dinner and games in the ballroom of the Marriott.


Day 3: Saturday | Munich, Germany - Unconferences (Yay!!)

Day 3 was one of the most exciting days at the event. We had a lot of sessions organized by different organisations in the form of an unconference, which is “a loosely structured conference emphasizing the informal exchange of information and ideas between participants, rather than following a conventionally structured programme of events.”

Mentors organized the unconference sessions on Saturday and Sunday. The unconference slots were planned with two rounds of lightning talks but ended with three rounds of lightning talks :-). A lightning talk is a platform for organisations to present on the work of their GSoC 2019 and GCI 2018 for 3 minutes. KAM also presented a lightning talk for ASF and Apache SpamAssassin on Saturday morning.

After lunch, all the mentors and the Google OPSO team gathered in a lawn just outside the Marriott for a group photograph.

[“GSoC 2019 Mentors Photo”]

We were involved in various unconferences sessions like:

How to get more Women interested in FOSS

The Fundraising Session (Presented by Kevin A. McGrail)

Source code preservation

Google Season of Docs (GSoD)

Intro to licenses and why we need them

After attending all the talks, we also discussed how to retain students after the completion of the GSoC period.

After the last lightning talk we all managed to spend some more time together to enjoy dinner, playing foosball, making funny poses on the photo booth along with enjoying the famous chocolate room (Oh, did we forget to mention about the famous chocolate table? This year, Google managed to have a complete room of chocolates!) where mentors across the globe shared the local country chocolates with each other!

Day 4: Sunday | Munich, Germany - Final day  :( 

Unfortunately, it was the last day of the mentor summit. The day started with continuation of lightning talks where Sanyam and KAM almost managed to attend all the lightning talks and got to know more about the other GSoC organisations and their amazing projects from GSoC 2019.

We attended some more unconference sessions on the following topics

GCI Info & Feedback with Google

GSoC Feedback session

Breaking the barrier for the newcomers

Interviews at Silicon Valley

Then we all headed for the final lunch of the summit.  By this point, most of us knew each other and some are planning to extend the trip by visiting some other cities, or some are planning to return back to their home countries. We all gathered for the closing session and all mentors had made a great network of cool people in the open source community!

We have also met a lot of mentors who were previously GSoC students. We had a lot of discussions about the experiences of being a student as well as a mentor, what motivated them to become a mentor and how they're contributing to their community.

Left to Right: Joey Schlichting, Sanyam Goel & Kevin A. McGrail

Overall, it was one of the lifetime experiences for every representative. The trip was full of memories and we got to learn so much, we also made new and special friends throughout the summit.

The GSoC Mentor Summit-2019 was a wonderful experience and we would like to thank the Google, The Apache Software Foundation, and once again, the ASF GSoC Organisation Admins, Ulrich Stärk and Maxim Solodovnik and the event hosts from the Google Open Source Team.

GSoC 2020 is underway now and we are just gathering project ideas and mentors.  Students looking to get involved, please see

Sanyam Goel started his journey with ASF by participating in GSoC 2017 as a student and continued contributing actively to OSS, currently serving as a committer of Apache Fineract. He also participated as a mentor in Google Code In and Outreachy programs for Mifos Initiative and DIAL community and always keen to spread the word about OSS to create an impact around the globe and focus on reducing the barriers for newcomers into OSS.

Kevin A. McGrail, better known as KAM, is a VP emeritus of the Apache SpamAssassin project where he has battled spammers for years.  In addition to helping the SpamAssassin project, he has served as in the office of treasurer and fundraising for the Apache Software Foundation.  He is also a member of the Apache Incubator project where he mentors new projects at the ASF including echarts, IoTDB & brpc. In his $dayjob, he works at doing cybersecurity for critical infrastructure.

= = =

"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works" 

Monday February 03, 2020

Success at Apache: Literally

by Chris Thistlethwaite

I became part of the Apache community as a member of the ASF Infrastructure team in 2016, and was elected an ASF Member in 2019.

Browsing through the other "Success at Apache" posts made me reflect on the word "success". Years ago, I was asked in a job interview, "How do you define success?". After a pause, I asked back, "In what?", which threw the interviewer off a bit. That's just too broad of a question for me to define one answer: success in a career, success as a human, success as a team member, success at a software release, the list goes on and on. 

Every day there's a giant list of possible successes and failures, and that’s even before you get to work keep that in mind as you continue reading.

In August of 2016 I came across a blog post that would change my life forever. 

At the time, I was looking for a new job that was taking longer than I expected. Taking a long shot, I sent off a very sparse email replying to the post. Two days later David Nalley (VP Infrastructure) replied, introducing me to Daniel Gruno who'd be doing the first round of interviewing. Fast forward a few months, and, spoiler alert: I got the job.

My first day "in the office" was in Seville, Spain, on November 14th during ApacheCon EU. Let me jump back a bit: most of the "Success at Apache" posts talk about the extensive background the authors have, both in the Open Source community and the ASF. While I use httpd, LAMP, etc. all the time, I never really found out how the "sausage was made". Apache has well-made products and the philosophy of how they were built intrigued me. My career until that point has mostly been inside Microsoft shops, usually with me suggesting FOSS solutions in meetings and only getting to use them in small-ish batches. A few MySQL boxes here, a few other Linux machines there, but not "full stack" kinda stuff: I ran it where I could but I was very happy with Microsoft products. "Best tool for the job", right? 

Anyway, back to Spain. I don't travel as much as I should, my Spanish is terrible (or enough to get me into a bar fight), and I'm traveling to a country I've never been to.

Friday November 11th was the last day at my previous job. Saturday afternoon, I left my wife and kid to jump on a plane for Seville, Sunday-ish I landed, and on Monday I started work in another country, at a job that was 98% Linux-based (Windows Jenkins build nodes), with people whom I’ve never seen before because no one used video chat during the interviews --at a conference held by the foundation I now work for. 

You may ask yourself, "How did I get here?", as I sure did: queue "Once in a Lifetime" by the Talking Heads...

My time at the ASF has been very interesting to say the least. With such a huge range of users of Apache software, some days I'm helping a large global company trying to get a product out the door, other days I'm troubleshooting a broken commit for someone working in their basement between dinner and baths for the kids. That's what makes this place special: those contributions help the community and help the common good of the project. The unique perspective I have is from within Infra. We don't just support the ASF, we support all projects in one way or another. One project might just be getting started with automated builds in Jenkins while another has been using CI/CD for years. That's a true strength of the ASF: disparate parts come together as a whole in a way that wouldn't work otherwise. Some days my job has nothing to do with technology, it's just getting the right people together on an email to figure out how to solve a problem, leveraging the different parts.

As mentioned earlier, "success" is a moving target, and at Apache, it's no different. Though in my case, any success at my job means I'm helping the ASF become successful, which in turn helps the projects and communities it supports. Behind every commit is a person, just working towards their own success.

I'm glad that I took the chance to respond to the job opening. Every job, company, and environment have a fair share of unpredictably and diversity. At the ASF, those traits are celebrated, leveraged, leaned on, and held up by the great people I get to work with and the community that I'm proud to be a part of.

Chris Thistlethwaite has been fixing problems and herding cats since before he can remember. He likes digging through log files to find solutions to complex problems and then turning his findings into pretty charts and graphs. After working at Avenue A | Razorfish, Sharebuilder, and some small startups, he brought his unique perspective on DevOps/Systems Engineering to the ASF Infrastructure team, where he specializes in monitoring systems. In his spare time, he enjoys homelabbing and spending time with his family.

= = =

"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works"

Monday September 23, 2019

Success At Apache: "Mentor Your Mentor"

By Patricia Shanahan

After retiring, I wanted to continue programming but without the pressure and constraints of a job, so I started contributing to Apache. Open software development the Apache way is a great retirement hobby, offering social contacts, intellectual challenge, continuous learning, and the pleasant feeling of making a contribution.

I just got back from having a wonderful time at ApacheCon NA 2019 in Las Vegas. While there, I met relatively young people, and older people who had been involved in Apache for up to 20 years, but joining as a retiree seemed to be unusual. 

Encouraging retirees could benefit Apache in many ways. 

Often, a retiree has a range of experience and skills that take time to accumulate. I have worked, for several years each, on applications, operating systems, compilers, system performance, and architecture of servers with dozens of processors. People like me who were programming in the 1970's have experience surmounting memory limitations, a skill that may be useful again for Internet of Things projects. I can imagine several reasons for a lack of retiree recruits. The most basic is that the computing profession was relatively small when a 2019 retiree would have started their career. That is a good reason to develop ways of helping retirees join Apache, so we will benefit from increasing numbers over the next few decades.

Some retirees already have plans that will take all their time and energy, and have zero interest in another hobby. Among those who might choose Apache as a hobby, there are several possible blocks, such as just not thinking of it, lack of confidence in returning to doing after a period of managing, outdated skills, and skills that may have atrophied through disuse.

The concept behind "Mentor your Mentor" is that someone who is active in Apache should watch for opportunities to bring the idea of open source as a retirement hobby to the attention of a retiring colleague, even if the retiree has been their mentor, and no matter how senior the retiree.

If the retiree is interested, the Apache contributor can offer various forms of help and support such as:

• Introduction to how Apache operates
• Encouragement
• Help selecting a project
• Help identifying resources for technical learning and relearning

In summary, the Apache contributor would do for the retiree the things a good mentor would do for someone new to IT. 

If you are an Apache contributor reading this blog, ask yourself: who in your network has retired from the computing profession? Reach out to them! Apache projects are a great opportunity for retirees to reconnect with innovation in computing. If you are a retiree and do not have an Apache mentor, don't let that stop you. Begin at

Patricia Shanahan worked from 1970 to 2002 in various programming and computer architecture roles for NCR, Celerity Computing, FPS, Cray Research, and Sun Microsystems. She then went to UCSD as a graduate student, receiving a PhD in computer science in 2009, after which she retired.

= = =

"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works"

Saturday September 21, 2019

Success at Apache: Why you'd want to become an Apache Committer

by Dmitriy Pavlov

All newbies in Open Source communities may sometimes think that they’ll never be able to become Committers. Many treat this role as a prestigious one, granted only for special feats, and after having written a ton of code. But not all things are so simple, and I hope my story will help you. 

Election as Apache Committer

My journey with The Apache Software Foundation began relatively recently, in 2015. I was working for Openway Group, and was enthusiastic about in-memory computing. I got to know about Apache Ignite at a local developers conference. I implemented the POC of a backend system based on Apache Ignite. I was so impressed with the clear API and documentation, and it was also very convenient that I could start prototyping without passing through the approval process. I suggested using the Apache product instead of a source-available solution. I met Konstantin Boudnik (cos), who helped me to understand the difference between Apache projects and source-available/closely-governed products.

Luckily for me, GridGain, the company that initially donated Ignite to the Apache Incubator, has a development center in my city (Saint Petersburg, Russia). I joined the GridGain team in 2017.

As part of my day job, I provide patches to the product. I actively joined the dev.list discussions (some fellows sometimes say “too actively”). I’ve created a number of wiki pages - ‘Apache Ignite-Under the hood’ to help developers understand product internals. Also, I developed ‘Direct IO’ plugin. I was elected as a Committer.

In 2018 I was concerned about reviews of patches from members of the community not affiliated with GridGain. Since I had a commit-bit now, I’ve started to review patches and ask others to review them, too. I don’t know for sure, but I suppose - these social achievements in the community development were a basis for me being elected as Project Management Committee (PMC) member. 

I asked several questions about The Apache Way on the Community Development (“ComDev”) dev list. I was very impressed by how friendly and welcoming they are. I very much like such a positive atmosphere, and feel it influences the success of Apache projects. Now I’ve also joined Apache Training (incubating) community as Committer and PPMC (Podling Project Management Committee) member.

Quite funny for a software developer with 17 years of experience… being elected as a Committer, that is to say, because of the social aspects and documentation. 

Who is a typical Committer and where does his or her strength lie?

When creating an Open Source product, we always let the users explore it in action -- as well as allow them to modify it and distribute modified copies. But when such modified copies are replicated in an unsupervised manner, we don’t get contributions into the main codebase and the project stalls. It’s here where we need exactly such a person – the Committer – someone who is authorized to merge user contributions into the project.

Why should you become a Committer?

First of all, being assigned to a Committer role is extremely motivating. The professional community acknowledges you and your work, and you clearly see the results of your work in action.

How different is that from some enterprise project -- where you have no idea why you must continually keep shuffling various XML fields?

The second pure advantage of being a Committer is an opportunity to connect with top professionals and also pull some cool ideas from Open Source into your own project. But, if you aren’t one of the top professionals, certainly don’t be afraid to join -- the community has various tasks for different folks.

Besides, being a Committer is a jewel in your CV --and even a greater plus for junior programmers, because at interviews you are often asked to show code samples. If you know an Open Source project well, a company supporting or using it will be happy to hire you. There are some people who will tell you that great positions are unreachable without first committing in Open Source.

There are some bonus goodies, too! For example, Apache Committers get an IntelliJ Idea Ultimate license for free (albeit with some limitations).

How do you become a Committer?

You should be committed to the project --it’s just that simple. Development, writing tests and documentation, and simply answering questions on lists are also good ways to start working towards committership.

Yeah, the contributions of QA engineers and technical writers in the community are valued no less than the developers’ contributions.

If you think there are no tasks for you on some project, you are wrong. Just join the community you are interested in and start working on its tasks. 

The Apache Software Foundation has this dedicated page that lists what contributions are needed.  

Committer — to be or not to be?

Committer activity is a good and useful endeavor, but one shouldn’t strive to become a committer per se. This status can be granted not only for code and it doesn’t justify your proficiency. 

Find a project you may be interested in: it will probably be a project whose software you already use. Dive into its code and say hi to the community; offer help, improve docs, complete a newbie ticket or answer to a user. You may just be surprised how welcoming and open folks are there.

Strive to gain the expertise (knowledge and experience) while researching the project, tweaking it and helping others to solve their problems, and, hopefully, enjoying collaborative development in an Open Source project.

Getting started at will help you on your way.

Dmitriy Pavlov is a Java developer enthusiastic about Open Source and in-memory computing. He is interested in system performance, information security, and cryptography. He created and donated utility for monitoring tests for Apache Ignite, and is a former Community Manager for Apache Ignite at GridGain. Dmitriy represents the Apache Ignite Project Management Committee (PMC) at local meetups in Russia. He runs workshops and training for Apache Ignite developers and users, and is a frequent speaker at meetups and conferences.

= = =

"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works"



Hot Blogs (today's hits)

Tag Cloud