The Apache Software Foundation Blog

Monday November 30, 2020

The Apache Software Foundation Announces Apache® TVM™ as a Top-Level Project

Open Source End-to-End Deep Learning Hardware Compiler Stack in use at Alibaba Cloud, AMD, ARM, AWS, Facebook, Huawei, Intel, Microsoft, NVIDIA, and Xilinx, among others.

Wilmington, DE —30 November 2020— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today Apache® TVM™ as a Top-Level Project (TLP).

The ASF's first  full stack software and hardware co-optimization project, Apache TVM is an end-to-end open deep learning compiler stack for CPUs, GPUs, and specialized accelerators. TVM enables machine learning developers to optimize and run computations efficiently on any hardware backend. The project originated in 2017 as a research project at Washington University and entered the Apache Incubator in March 2019.

"It is amazing to see how the Apache TVM community members come together and collaborate under The Apache Way," said Tianqi Chen, Vice President of Apache TVM. "Together, we are building a solution that allows machine learning engineers to optimize and run computations efficiently on any hardware backend."

Apache TVM’s extensible full-stack framework enables deep learning applications to efficiently deploy across an array of hardware modules, platforms, and systems, including mobile phones, wearables, specialized chips, and embedded devices. Features include:

  • High Performance: compilation and minimal runtimes commonly unlock ML workloads on existing hardware.

  • Runs Everywhere: automatically generates and optimizes tensor operators on backends, CPUs, GPUs, browsers, microcontrollers, FPGAs, ASICs, and more.

  • Flexible: deep learning compilation models in Keras, Apache MXNet (incubating), PyTorch, Tensorflow, CoreML, and DarkNet, among other libraries. Supports block sparsity, quantization, random forests/classical ML, memory planning, MISRA-C compatibility, Python prototyping, and more.

  • Easy to Use: easily build out production stacks using C++, Rust, Java, or Python. Deploy deep learning workloads across diverse hardware devices.


Apache TVM is in use at dozens of organizations and institutions that include Alibaba Cloud, AMD, ARM, AWS, Carnegie Mellon University, Cornell University, Edge Cortix, Facebook, Huawei, Intel, ITRI, Microsoft, NVIDIA, Oasis Labs, OctoML, Qualcomm, University of California/Berkeley, UCLA, University of Washington, Xilinx, and more.

"ML compilers and runtimes thrive on diversity of models supported and HW targets, which is a perfect way to show the power of Open Source communities," said Luis Ceze, CEO of OctoML and Professor at the University of Washington. "It has been fantastic to see Apache TVM's fast adoption among hardware vendors and ML end-users, being well on its way to becoming a de-facto industry standard."

"Apache TVM brings unique value to deep learning researchers and developers. It closes the gap between model development and the demand to efficiently deploy it on various hardware targets," said Yizhi Liu, Senior Software Development Engineer at AWS and member of the Apache TVM Project Management Committee. "I'm thrilled to see Apache TVM now becomes the Top-Level Project and looking forward to further collaboration with the community."

"Congratulations to the Apache TVM community for graduating to be one of the Top Level Projects of The Apache Software Foundation," said Henry Saputra, ASF Member and Apache TVM Incubating Mentor. "The Apache TVM ecosystem has a healthy mix of representation and contribution from the industries and academia that provides a good balance of innovations and production readiness for wider and faster adoption. As one of the mentors of the podling, I am grateful and glad to be part of the journey."

"The key to Apache TVM's success is its open community," added Chen. "We welcome everyone interested in the field to join us and shape the future of ML compilation together under The Apache Way."

Catch Apache TVM in action at the annual TVM Conference being held 2-4 December 2020. The online event is free of charge to participate: for more information and to register, visit https://tvmconf.org/  

Availability and Oversight
Apache TVM software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache TVM, visit http://tvm.apache.org/ and https://twitter.com/ApacheTVM 

About the Apache Incubator
The Apache Incubator is the primary entry path for projects and codebases wishing to become part of the efforts at The Apache Software Foundation. All code donations from external organizations and existing external projects enter the ASF through the Incubator to: 1) ensure all donations are in accordance with the ASF legal standards; and 2) develop new communities that adhere to our guiding principles. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. For more information, visit http://incubator.apache.org/ 

About The Apache Software Foundation (ASF)
Established in 1999, The Apache Software Foundation (ASF) is the world’s largest Open Source foundation, stewarding 227M+ lines of code and providing more than $20B+ worth of software to the public at 100% no cost. The ASF’s all-volunteer community grew from 21 original founders overseeing the Apache HTTP Server to 813 individual Members and 206 Project Management Committees who successfully lead 350+ Apache projects and initiatives in collaboration with nearly 8,000 Committers through the ASF’s meritocratic process known as "The Apache Way". Apache software is integral to nearly every end user computing device, from laptops to tablets to mobile devices across enterprises and mission-critical applications. Apache projects power most of the Internet, manage exabytes of data, execute teraflops of operations, and store billions of objects in virtually every industry. The commercially-friendly and permissive Apache License v2 is an Open Source industry standard, helping launch billion dollar corporations and benefiting countless users worldwide. The ASF is a US 501(c)(3) not-for-profit charitable organization funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, Amazon Web Services, Anonymous, Baidu, Bloomberg, Budget Direct, Capital One, Cerner, Cloudera, Comcast, Facebook, Google, Handshake, Huawei, IBM, Inspur, Pineapple Fund, Red Hat, Target, Tencent, Union Investment, Verizon Media, and Workday. For more information, visit http://apache.org/ and https://twitter.com/TheASF 

© The Apache Software Foundation. "Apache", "TVM", "Apache TVM", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Sunday November 29, 2020

Inside Infra: Andrew Wetmore --Part I

The "Inside Infra" series with members of the ASF Infrastructure team continues with Part I of the interview with Andrew Wetmore, who shares his experience with Sally Khudairi, ASF VP Marketing & Publicity.



"...I really had a distant but benevolent appreciation of Apache until I started to get more and more involved with Royale and began to understand from that angle all the things that the Foundation does to support these little projects that could not survive without it. Of course, now that I've become part of the Infrastructure team, I'm awestruck by the amount of work that the team does to support all these little projects, so they can do their thing."


What is your name and how is it pronounced?


I'm An-drew Wet-more. The "Wetmore" is like a rainy day, very easy to pronounce.


When and how did you get involved with the ASF?


I was a Flex and ColdFusion developer. When Flex came to end-of-support with Adobe and they passed it over to Apache, I followed along. I wasn't an active committer: I was a participant in the Apache Flex project and contributing in my little ways here and there. Then when the Apache Royale project split off Apache Flex, I went there, but I was not an active, not a heavily significant contributor. That is, I was helping with documentation, a bit of testing, a bit of organizing and helping. I was truly surprised when I was invited to become a Committer. Then at some point, somebody noted on the Apache Royale list that the Apache Software Foundation Infrastructure Team was looking for a documentation person. I thought, "Well, that's interesting. Maybe I would be able to contribute to that."


I followed that up. I wrote to (ASF Infrastructure Administrator) Greg Stein and introduced myself and said, "Oh, I'd be interested if this is something that's happening." Then for one reason and another, nothing happened for quite a long time. That was fine. He told me nothing was going to happen for a while. He was migrating some monstrous mountain of something. Then when that long time was up, I pinged him again and said, "I'm still around if that's an interesting possibility," and we got talking. He did that wonderful interviewer thing of saying, "Well, if you were going to hire someone for this sort of a job, that has this heading, what sort of job description would you write?"


He made me write the job description. I thought, "this is cute: I'm happy to help. I don't know what person not me is going to get this job, but I'm happy to write what I think is a good job description for this thing." Truly, I really expected this to go out and a whole bunch of people to apply for it and that I would get a participation trophy. I was very pleased when I was invited to join the team.


… You got the real trophy.


Yes, I did.


You got involved when Flex came to Apache, so that goes back to 2011, you've been with the Foundation for nine years or so?


I was aware and downloading builds as soon as there were builds to download and participating. I was still building my own Flex stuff, but I don't think I was really contributing significantly until around maybe 2015. Then I didn't become a Committer until 2018.


The other things you were doing prior to Infra were limited to Apache Flex and then onto Royale?


Yeah, I had a glancing awareness of Apache. Without even thinking about it, of course, I was using Apache tools like Apache Tomcat packages, but I really had a distant but benevolent appreciation of Apache until I started to get more and more involved with Royale and began to understand from that angle all the things that the Foundation does to support these little projects that could not survive without it. Of course, now that I've become part of the Infrastructure team, I'm awestruck by the amount of work that the team does to support all these little projects, so they can do their thing.


It's interesting with Apache projects because they're mostly ingredient brands versus a customer-facing final product. Of course we do have those too, but the majority of them power something else. A lot of times people aren't aware until they're in it: then they’re like, "Oh, wow, Apache is everywhere."


Well, I keep trying to improve myself and I go and choose a product project at random and read its homepage and its "about our product" thing and see how far I can get before I've hit five things that I don't understand at all. I don't even understand what I would do with the thing that I do understand which is not a knock on those projects. It's what you just said, they're not end-user-facing. I, as a Flex developer, was a Flex developer. I was using Flex and now Royale to build other things, not getting in with the toolkit and adjusting and tweaking Flex or Royale.


Right, like a commercial product. Explain your role within the Infra team. What is it exactly? What's your title? When did you get there?


Let's see. It was at the very end of last year. I'm coming up on 11 months on the team. I started two weeks before the end of 2019. My title is either editor/writer or writer/editor. My job is ... well, the situation of the Infrastructure team from a point of view of a written material, of documentation, is that for the past 20 years people have been doing their very best they can to document what they do and how people should do things. They've been adding to that material and adding to that material. We have a built-up pile of stuff, some of which is no longer relevant, some of which contradicts other bits, some of which is written very provisionally for something that we now take for granted that we're always going to use.


My job is to go around with my little broom and dustpan and clean up things. I go and find a page in the Infrastructure documentation and do what I would do when I visit project pages. I start to read it. When I get lost, I start to edit it or I start to ask questions to the team and say, "Is this even a thing anymore?" The team sometimes says, "Oh, we were doing that in 2011, was it? I forget." We know that that page can go into history. There are other times when I run into things, there are many pages, far too many pages where there's a sentence like, "At this point, such and such is true." It doesn't say anywhere what "this point" is.


Then when I dig in a little bit, I find out it was again back several years, where either the conversation came to a stop or the conversation continued and whatever that uncertainty was on the page has been resolved. Now I'm able to update that page and say something that's more useful to the visitor like me who's coming now and doesn't want to read a historical document, so much as get help about doing the thing they want to do.


Right: "what does that mean for me?" This is the legacy dilemma.


Sure. Well, it's also a factor of people trying very hard to do too many things all at the same time. Let's just write enough explanation so people can get going, because surely they'll understand what we mean. That's often true for probably the vast majority of people who are active in an Apache project. The pages that stump me don't stump them.


We have this issue of tribal knowledge not scaling. Part of it is that people are moving on. Part of it is that processes are changing, technology is changing, people forgetting about pages. You're in a very interesting position. If I'm understanding it correctly it’s trying to rectify whatever the intention of what's historically been in place versus what people are doing now, even establishing relevance and trying to find out who's the de-facto source to go for this, considering the amount of contributors, is incredible.


There's another piece in this. I'm trying to come at my review of pages from the point of view of a person of good intention for whom English is not the first language. I'm trying to think, "What on Earth would one of my colleagues from Laos understand from this sentence that has a jokey little acronym or aphorism in it?" Then I tried to figure out how I can say that more clearly and not in a boring way, but in a more accessible way. The three flaws that all our documentation tends to have, one is to write in an academic way. One is to write with a heavy use of acronyms. One is to use a heavy serving of passive voice in the material.


Let me explain what I mean by those three. The academic style: so I have a page here that's going to tell you about X. If I'm an academic, I might write, "The intention of this page is to explain the modalities by which a user might accomplish X." As the reader, especially the reader for whom English is not the first language, I'd like to see, "Here's how to do X."


The second one is the acronyms. Even acronyms that should be, you would think would be obvious to everyone. "The PMC of a TLP." It's like reading Hebrew where there are no vowels, but a simple rule is to always spell out what the thing is and put the acronym in parentheses the first time you use it in a document. It's easy to forget if we know the acronym so very well ourselves. 


The third thing is passive voice. Passive voice correlates strongly to the academic approach. It has a way of hiding who does what. "After the file is uploaded, it is passed to the server and verified." I don't know who does any of those things, if I'm the owner of that file, I don't know if I have to do any of those things or if I just sit back and watch them happen.


Sometimes when I ask—I actually enjoy doing this because it is my naïve or newbie character—I go to the team and I say, "Here's this sentence. Who is doing this?" In the conversation about that sentence, sometimes the team discovers that there's a difference of opinion about what part of the Infrastructure does it or whether someone has to do it. We might end up not only fixing the text but improving the process.


What I'm hearing, it's beyond just writer/editor. It's writer/interpreter.


I came into software through the QA door. Over 15 years, I ran teams that were generally documentation and quality assurance, documentation and testing. I think the two are very tightly connected. When I go to edit a document, to the extent that I'm competent to do so, I test it. If it says, "You go here and do that," I go over there and look, is the thing that I can do available there?


It does what it says on the tin. It doesn't correct that.


Well, we have innocent blind alleys that are built into our documentation out of good intentions, pages written about our software repositories, back just a few years, presumed that everything happens in a Subversion repository for projects.


For projects, almost everything happens in Git repositories now and the instructions of how to do something in the two repositories may be different. We need to go back and find those pages and make the path comfortable for someone who couldn't care less about Subversion, but they have to do something in Git.


Let's talk about the scale of what you're working on. How many pages are you handling?


Oh, good Lord, I don't know. I started out within Infrastructure itself, looking at several packages of pages. There was a set of pages on the Apache Website under the subhead dev, a set of pages on the Apache Website under infra.apache.org. There had been a set of pages under something called reference. There's a set of documents in a Subversion directory. Then there's a very large Infrastructure section of the Confluence wiki. One of the first things I had to work through is how did these relate to each other? How does anyone find their way? I proposed and the team seems to have accepted that the pages at the apache.org/dev area are the introduction. Here's what someone trying to figure out what goes on with dev and possibly the Infra team would need to look at.


The next step in is the infra.apache.org area. If you can't find what you need there, you follow the links through into the wiki. God help you if you have to go to the Subversion repository. Really, what should be in the Subversion repository is only, it seems to me, the instructions for restarting or rebuilding the Confluence wiki. I am gradually moving other stuff out of the Subversion repository. 


This feels like a byproduct of individuals or projects or committees or communities actually having this "scratch your own itch" issue, right? "Hey, we need to document that somewhere on our thing and here's our particular experience." Apache is open enough. You can do that. The whole directive of integration or coordination or making sure that one hand is speaking to the other has never really been a mantra of ours. It's interesting to see as we're scaling, it reflects directly on what you’re experiencing.


There's the thing. If it were a team of six people in a room or at a project level, it wouldn't be a problem figuring out where to put what or how to fix a documentation collision. When you're talking about hundreds and hundreds and hundreds of people, some with decades of experience, literally two decades of experience in the Foundation, someone bopping in brand new can bring things to a standstill. People can get lost or disheartened.


The disorientation is very common. I hear people going, "How do I find …?" "Where do you start? Where do you go?" It's great that there's assistance.


In early January, I started with some thoughts. "I'll be a new person." "I'm going to read the apache.org site and go where it tells me." I got so depressed. I could not understand how the information on one page related to the information on this other page that I was looking at. There seemed to be two sets of instructions for doing many things. Part of my fun is trying to make it easier for people who come in: you have to help them find their way without giving up the whole thing and throwing the computer out the window.


Are you still at the audit and discovery stage or are you actually at the rewriting stage also? Are you in course-correction or new content?


I've moved almost everything out of the apache.org/dev area that shouldn't be there. What is left is introductory material. I've edited every page on the infra.apache.org area. I've edited almost every page, I think, on the wiki. I'm still digging around in the Subversion thing. I am doing a long march through the top level apache.org pages. I've been in most places, but I'm not done. Beyond this, I guess there are some areas I haven't touched or I need to ask permission. For instance, the Incubator has a set of interesting pages in which the material is clear, in large part. There are some suggestions I'd like to make for those pages, but I don't have any mandate to go in there and start changing things.


What I would do in that situation is write up a little sample report, "Here's how I would suggest changing this page," share it with the Incubator team and say, "This is yours to do as you'd like. Would you like some more?"


… Chances are they'd say yes.


Well, indeed, but I don't want to assume that. I know how I would feel if I turned around and found out someone was changing all my sentences.


I might very well feel that not having passed through the Incubator wouldn't have a good grip on what needs to be said.


It's really important for us to have these fresh eyes with respect to what the outside world is seeing: people who are new to it, how is their interpretation or misinterpretation? Again historically, there's been this issue of, "well, it's obvious". Not just has Apache evolved and the communities have evolved, but Open Source has evolved. The expectation is very different. Similar to what you were saying before with Subversion and Git, it's a completely different space now. We have to grow with that. I think again because we're not a corporation, we don't have these marching orders of "go, bring it up to speed or bring it in alignment." It's great that you were there to audit, align, and course-correct.


There are pluses for this that I had suspected but hadn't been sure I would find ... I'll give you an example. One of my projects is about Apache's content management system that's nearing end-of-life. We have to migrate all the projects that use that content management system to generate their Project websites to some other technology. We've been working on this for over a year, I think, but there were 40 or 50 projects that hadn't gotten started on that migration. I started conversations with them, saying, "Hey, are you going to move? What do you need help moving? What do you need to know?"


As I was getting feedback, I was able to improve the documentation we provide on the wiki on how to migrate your project off the CMS. Along the way, I've met some really interesting people and am having fascinating conversations with people deeply engaged in Projects; the output of which I know nothing. It's a lot of fun because these are very, very smart people. They're doing really significant stuff. I want to make it as easy as possible for them to turn from that highly significant stuff to this rather mundane thing of moving the way they built the Website from the current way, which is creaky but they know it, to a new way.


We long have had this "do your own thing" culture --no one's telling anyone that there's one, official, way to develop your Website. Hundreds of Apache Projects are developing their own Websites their own way. No doubt some that are using the current CMS, there is an opportunity to offer them a different development direction versus a giant arm sweep stating, "We're going to pull down 300 project sites and rebuild them all at once," as would be done in other organizations when they choose to rebrand or upgrade their CMS or backend. It's like little mushrooms popping up where everyone is producing their own site at their own pace, using their preferred tools. It's very, very interesting.


It's educational to me also because the Infra team has a series of recommendations, "We really recommend you go to this technology to build your Website." The subtext is because that gives you the most options, the most flexibility, and means Infra has to do less to hold things together, but then we have other Projects that say, "Oh, no, we don't like that. We really would like to use this." The Infrastructure response is, "Show us how you can possibly use that in a way that matches these requirements we have." For instance, that the landing page for the project Website has to be a thing that can be branded as [projectname].apache.org and hosted on our servers.


People, as a project, demonstrate, "Oh, they can use this technology" that we had not thought of, then we have the documentation for that and that might encourage some other project that doesn't like the vanilla package we're providing to migrate using this new thing. We're down to about 20 projects I think that haven't really gotten very far on their migration.


… Is there a deadline for that?


At some point, the content management system is just going to fall over. We'd like to get everyone out before that happens. We set the end of the year as, "Let's do this before the end of this year, but there's not a switch." There's not the end of a license or something like that that's going to happen. We have a little bit of wiggle room.


… We're not pulling the plug, so to speak.


No. We're not pulling the plug, especially since treasurer.a.o. hasn't moved and we wouldn't want to annoy them.


Right. Are there additional responsibilities that you take care of?


Well, that what I just described about helping or encouraging teams to migrate was not part of my job description. I just saw something that I could do that involved being engaged with projects to get them on the path, leaving the other team members available to do things that I can't do. I'm the least technologically savvy person on the team. I might as well do the stuff that involves words and interaction.


What's the process of sorting through 21 years of ASF history on apache.org? How far along are you? Is this a never-ending project or is there a specific milestone that you want to hit to say, "Hey, okay, we've done"? Is there an end in sight to this or …?


I think we'll get to a point where we'd say, "We're pretty well caught up. Now what?" That could happen within the next couple of months, but then remember, at that point, the process of doing doesn't end. Where new material is being created, technology changes. We're migrating server things from one kind of server to another kind of server. We have to document what that new server does. Git for instance, or GitHub, I guess, has provided a couple of new options for things projects can do. The Infra team has to learn how to support those, then we have to document them and help teams understand how they can use them for their benefit.


As long as the Foundation keeps doing stuff, the same problem of uncurated information silting up will recur. Hmm. That's my lifetime employment plan.


[laughs] Going to apache.org/dev, how did you decide where to start? Were there any active fires that you were told you had to put out or it was more a bunch of low simmer, "We'll get to that someday," types of sections of the site? Again, it just seems so like a Medusa situation. How did you decide to divide, conquer, and get started?


For apache.org/dev, I just started at the top file or the top link that said anything about dev and went into it. "Why is this out here? This looks very much like the same thing we say over here in infra.a.o. Why are we saying it twice in two different places in two different ways?" I started pretty much by grabbing anything. It's like the way you might go into your grandparents' attic when they're downsizing to a smaller house and you're going to help them move. All you can do is pick up the first box and see what's in it and give the best guess about where that should go.


… Then there's those people who just grab it and just donate everything.


That's true.


… Not even looking through it, they're just purging and starting afresh.


Fairly early. I tried to elaborate that tiered idea of Infra information, so that if you land on dev, you're getting high-level stuff. If you go to info.a.o, you're getting more thorough stuff. If you need to, you can go over to the wiki to get code snippets or very detailed instructions. If you're an Infra team member, you go over there and get stuff. Only in the direst need you go down into the Subversion repository.


Before, everything was mixed around. Where the most essential and the least essential stuff was, was not consistent or logical.


Have you had to learn about the Apache Way of community-led development or other processes in order to get the job done? Even if they're talking about a technical thing, you're testing it out. Are you kicking the tires along the way saying, "Okay, this doesn't make sense," or are you not at that stage yet in terms of content?


I'm doing a fair bit of tire kicking. Of course, as a participant in the Flex and in the Royale projects, I've engaged myself to understand the Apache Way from them. The PMCs I work with modeled the management and development style of Apache. I learned it organically. I'm not seeing a conflict between what I learned on the Flex and Royale teams and the larger Apache Way of doing things. I think that's really good. You stumble into a small project with a very minor, very focused goal to do this thing, this bit of technology. You take in through your skin how to make decisions and how to share information and how to support each other.


… Continuity for the win: that's good to hear. What kind of influence do you have on content development? You said you're adjusting a page if it's not saying what it's supposed to do, but beyond that, are you saying, "Look, this really needs to take a different approach"? Are you deciding on your own? Is there a review committee that has to oversee every edit or is the process completely autonomous? How do you know what you're writing is factually correct? Who signs off on that?


The Infra team is in constant contact, 24-hour chatter every day on the Slack channel. There's an asynchronous conversation going on. When I run into something I don't understand ... Well, there are two things that happen. I can suggest things there that might be useful, but also when I notice people discussing something that's new or something went wrong and what they have to do to make it right, I often say, "Is that something we should write down, do you think? Where should we write it down?" That begins the conversation about documenting whatever the thing is.


One of the first things I created on the wiki page, the Infra wiki site, is a page for me called the job jar. Each time I come up with something that has to be written, I start a new item on a checklist and write in what that thing is to the best of my knowledge. Then, if I can't see any way to write because I don't have a clue what that thing is about, I go to the Slack channel or I go to the team meeting, which we have every Thursday, and say, "Who can help me write this? I just want you to blurt out the facts and then I'll turn it into pretty language." I can't direct that we have to write anything, but we work interactively.


If I write a new thing, I post it on the Slack channel. Someone will come back and say, "Well, you totally missed this thing. Here, let me fix it for you." We go back and forth like that until it's ready to make available to the larger public. Greg, of course, keeps a close eye on me, so I don't accidentally delete everything.


We review regularly what needs to be added or what can be sliced away because often, if you say less, you can communicate more.


Going back to "delete everything", when I first joined W3C 25 years ago, I remember making copies of everything because I was terrified that I was going to delete the Web's original history, there were thousands of legacy pages. Do you do the same thing? Do you make copies of things and edit that then just do merges? How do you actually do that?


I have a strong reliance in the team's guarantee that everything is version controlled. Actually, I'm more shy about changing things than they are to encourage me to do it. They just said, "Go ahead and do that. We'll fix it later." In that sense, I'm truly not afraid of deleting everything. I am afraid of inadvertently causing annoyance. I have an example: when I first started to move pages from a.o/dev area to the infra.a.o area, some of the pages I wanted to move that had titles that didn't really match what was in the document. God, I've got to improve this. I changed the name of the file at the new location. Then that was a pain because how do you redirect from the old location to the new direction?


I learned very quickly that I was causing trouble for my colleagues, but beyond that, I was causing trouble for people on projects who might have a link on their page to an Infra page. I really don't want to cause an information barrier because, in my mind, I'm making things more efficient. On the a.o/dev area, there are all sorts of pages sitting there now that are just stubs or just shells of their former selves. If you click on the link to go to that page, there's a little gearcranking and all of a sudden you're over at the same page at infra.a.o … most of the time. Sometimes it just does not work and then I have some sad people.


It's interesting you were saying about not wanting to upset people, but I think this is actually a parallel with good documentation and good data management. It becomes un-intrusive and a natural byproduct of your experience online. The whole point is you don't want to say, "Hey, there's some underhanded entity there that's controlling it." It's natural in terms of what you're seeing, what you're reading. In terms of comprehension, it's great UX. It's a very interesting comment that you made about you not wanting to ... this, "do no harm" approach --the outcome is very positive. 


If you want to make a really highfalutin image, we're surgeons working on something together. There's a thing going on the table there that's going to ... Things are going to go bad if we don't do our job well. If I go moving around where the implements are that we're going to reach for on the tray from where they normally are just because I think they should be alphabetical or something, things are not going to go well for the patient.


… Someone might even die, right?


Fortunately, there's a limited amount of trouble I can cause because I'm not turning the nuts and bolts on the servers. Not yet.


But I am ... You asked earlier what sort of, I don't know, influence I have to bear, I'm in there asking questions whenever I can understand a question to ask about, "Shouldn't we update this list here of the servers? This doesn't look like it's been updated since 2014. Shouldn't we make this list more accessible to the people who have to look at it?" That makes it sound like my colleagues are bumbling along and inattentive. They're very attentive, they need to document what they're doing and they're very patient with me when I get fixated about a semicolon while they've done everything else right on that page, except that damn semicolon.


It's important. Both parties: that's a good dovetail of talent, right? You're talking about a page that hasn't been touched since 2014. We have pages that have been untouched since 2001. I'm sure you're coming across them.


Here's a situation that probably is of low impact, except when it has high impact. I've been reading the memorial pages for past committers. I got to a page that said many kind things about the person, "who died in a car accident this last week". This "last week".


… *When* was that, right?


Yeah. I'm making little reports on those pages and the people who have ownership of the pages have to decide what to do with those reports. I'm not going in and changing those pages, but I suggested, "Let's figure out the year at least, maybe the month and make that more accurate, so someone like me now visiting this memorial page about a person who died before I joined Apache can understand what happened." In some ways, that's important for remembering and honoring the people who have been with us and are gone. There is more painful stuff when we haven't updated something or we've left a sentence that says, "As of this writing, so and so is the case, but I don't know if it's going to be that way for long." Again, there's no date. I think it's scary.


… There's no frame of reference at all.


Exactly. It makes the whole thing provisional. We have, under this COVID-19 crisis right now in the province of Ontario in Canada, a very complex Website that purports to tell you if you're in the city of Toronto what you can do in different parts of Toronto, what the lockdown level is. At the very top of the pages, it says, "Latest information."


If you go in there, it's three months old. The latest information is elsewhere in the page. To me, it throws the whole thing. If I'm someone who's trying to find something out from that site, I tend not to believe any of it.


[END OF PART ONE]

Friday November 27, 2020

The Apache News Round-up: week ending 27 November 2020

Farewell, November --we're wrapping up the month with another great week. Here are the latest updates on the Apache community's activities:

ASF Board – management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 16 December 2020. Board calendar and minutes https://apache.org/foundation/board/calendar.html

ApacheCon™ – the ASF's official global conference series, bringing Tomorrow's Technology Today since 1998.
 - all videos from ApacheCon@Home now available at https://www.youtube.com/c/TheApacheFoundation/  

ASF Infrastructure – our distributed team on three continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield uptime at 100%. Performance checks across 50 different service components spread over more than 250 machines in data centers around the world. http://www.apache.org/uptime/

Apache Code Snapshot – Over the past week, 291 Apache Committers changed 503,622 lines of code over 2,498 commits. Top 5 contributors, in order, are: Gary Gregory, Oleg Kalnichevski, Pavel Tupitsyn, Till Rohrmann, and Andrea Cosentino.                

Apache Project Announcements – the latest updates by category.

API --
 - Apache APISIX Dashboard 2.0 and Dashboard 2.1 released https://apisix.apache.org/

Application Performance Monitoring --
 - Apache SkyWalking Python 0.4.0 released http://skywalking.apache.org/

Big Data --
 - Apache ShardingSphere UI 5.0.0-alpha released http://shardingsphere.apache.org/

Blockchain --
 - Apache Tuweni (Incubating) 1.3.0 released https://tuweni.apache.org/

Data Management Platform --
 - Apache Unomi CVE-2020-13942 Remote Code Execution in Unomi https://s.apache.org/u0gg5

Libraries --
 - Apache Daffodil (Incubating) 3.0.0 https://daffodil.apache.org/

Messaging --
 - Apache Qpid Proton 0.33.0 and JMS 0.55.0 released http://qpid.apache.org/

Servers --
 - Apache Tomcat 7.0.107 available http://tomcat.apache.org/

Web Frameworks --
 - Apache Wicket 8.11.0 and 9.2.0 released https://wicket.apache.org/

 
Did You Know?

- Did you know that you can help secure the future of Apache? Your tax-deductible contribution helps the ASF steward, develop, incubate, and make $20B+ worth of Apache software free to all at 100% no cost. Thank you for making a one-time or monthly donation https://donate.apache.org

- Did you know that Pulsar Summit Asia is taking place this weekend? Join the Apache Pulsar community 28-29 November online: sign up today --free-of-charge https://pulsar-summit.org/

- Did you know that the Apache TVM (incubating) community is holding its annual TVM Conference online 2-4 December 2020? Register for this free-to-participate event https://tvmconf.org/ 

Apache Community Notices

- Apache Month In Review: October 2020 https://s.apache.org/Oct2020

- ASF FY2020 Annual Report https://s.apache.org/FY2020AnnualReport 

- "Trillions and Trillions Served" documentary on the ASF: 1) full feature https://s.apache.org/Trillions-Feature 2) "Apache Everywhere" https://s.apache.org/ApacheEverywhere 3) "Why Apache" https://s.apache.org/ASF-Trillions 4) “Apache Innovation” https://s.apache.org/ApacheInnovation 

 - The Apache Software Foundation Statement on the COVID-19 Coronavirus Outbreak https://s.apache.org/COVID-19  

 - The Apache Software Foundation Celebrates 21 Years of Open Source Leadership https://s.apache.org/21stAnniversary

 - Apache in 2019 - By The Digits https://s.apache.org/Apache2019Digits

 - The Apache Way to Sustainable Open Source Success https://s.apache.org/GhnI

 - Foundation Reports and Statements http://www.apache.org/foundation/reports.html

 - "Success at Apache" focuses on the people and processes behind why the ASF "just works". https://blogs.apache.org/foundation/category/SuccessAtApache

 - Inside Infra: the new interview series with members of the ASF infrastructure team --meet 
    Chris Thistlethwaite https://s.apache.org/InsideInfra-Chris
    Drew Foulks https://s.apache.org/InsideInfra-Drew
    Greg Stein Part I https://s.apache.org/InsideInfra-Greg
      ...Part II https://s.apache.org/InsideInfra-Greg2 and Part III https://s.apache.org/InsideInfra-Greg3
    Daniel Gruno Part I https://s.apache.org/InsideInfra-Daniel1 and Part II https://s.apache.org/InsideInfra-Daniel2
    Gavin McDonald Part I https://s.apache.org/InsideInfra-Gavin and Part II https://s.apache.org/InsideInfra-Gavin2

 - Please follow/like/re-tweet the ASF on social media: @TheASF on Twitter (https://twitter.com/TheASF) and on LinkedIn at https://www.linkedin.com/company/the-apache-software-foundation

 - Do friend and follow us on the Apache Community Facebook page https://www.facebook.com/ApacheSoftwareFoundation/ and Twitter account https://twitter.com/ApacheCommunity

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

= = =

For real-time updates, sign up for Apache-related news by sending mail to announce-subscribe@apache.org and follow @TheASF on Twitter. For a broader spectrum from the Apache community, https://twitter.com/PlanetApache provides an aggregate of Project activities as well as the personal blogs and tweets of select ASF Committers.

Friday November 20, 2020

The Apache News Round-up: week ending 20 November 2020

Happy Friday! Here's what the Apache community has been up to this week:

Inside Infra – the interview series featuring members of the ASF Infrastructure team.
 - Gavin McDonald --Part II https://s.apache.org/InsideInfra-Gavin2 

ASF Board – management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 16 December 2020. Board calendar and minutes https://apache.org/foundation/board/calendar.html

ApacheCon™ – the ASF's official global conference series, bringing Tomorrow's Technology Today since 1998.
 - ApacheCon@Home sessions now online at https://www.youtube.com/c/TheApacheFoundation/  

ASF Infrastructure – our distributed team on three continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield uptime at 99.74%. Performance checks across 50 different service components spread over more than 250 machines in data centers around the world. http://www.apache.org/uptime/

Apache Code Snapshot – Over the past week, 310 Apache Committers changed 806,646 lines of code over 3,127 commits. Top 5 contributors, in order, are: Jarek Potiuk, Kaxil Naik, Andrea Cosentino, Mark Miller, and Maruan Sahyoun.            

Apache Project Announcements – the latest updates by category.

Application Performance Monitoring --
 - Apache SkyWalking Could on Kubernetes 0.1.0 and SkyWalking Client JS version 0.2.0 released http://skywalking.apache.org/ 

Big Data --
 - Apache BookKeeper 4.11.1 released http://bookkeeper.apache.org/ 
 - Apache ShardingSphere 5.0.0-alpha released http://shardingsphere.apache.org/ 

Content --
 - Apache Jackrabbit Oak 1.36 released http://jackrabbit.apache.org/ 

Libraries --
 - Apache Log4j 2.14.0 released http://logging.apache.org/
 - Apache CXF CVE-2020-13954 Reflected XSS in the services listing page via the styleSheetPath https://s.apache.org/tsb9p

Messaging --
 - Apache Qpid Proton-J 0.33.8 released http://qpid.apache.org/

Servers --
 - Apache Tomcat 8.5.60, 9.0.40, and 10.0.0-M10 available http://tomcat.apache.org/

 
Did You Know?

- Did you know that the following Apache projects are celebrating their anniversaries this month? Three cheers to Apache Ant (18 years); HttpComponents (13 years); Attic, Buildr, CouchDB, and Qpid (12 years); Community Development ("ComDev", 11 years); OODT and ZooKeeper (10 years); Kafka and Syncope (8 years); Ambari (7 years); BookKeeper, Drill, and MetaModel (6 years); Brooklyn, Groovy, Kylin, and REEF (5 years); Geode (4 years); Guacamole, Impala, and Mnemonic (3 years); Griffin (2 years); and Petri (1 year). Many happy returns! https://projects.apache.org/committees.html?date 

- Did you know that you can make a tax-deductible individual or corporate contribution to the ASF to help us continue to provide more than $20B worth of software to the public at 100% no cost? Donate today at https://donate.apache.org/ 

- Did you know that Alibaba’s real-time computing platform processed data streams totaling 4 billion items per second during Singles Day, the largest global online shopping event, using Apache Flink? https://flink.apache.org/ 

- Did you know that more than 300 videos from ApacheCon@Home are now available, including keynotes, plenaries, and sessions on Big Data, Clinical Data, Community, Content Delivery, Fintech, Incubator, Integration, IoT, Machine Learning, Mobile, Observability, Search, Servers, Streaming, and more are now available? https://www.youtube.com/c/TheApacheFoundation/ 

Apache Community Notices

- Apache Month In Review: October 2020 https://s.apache.org/Oct2020

- ASF FY2020 Annual Report https://s.apache.org/FY2020AnnualReport 

- "Trillions and Trillions Served" documentary on the ASF: 1) full feature https://s.apache.org/Trillions-Feature 2) "Apache Everywhere" https://s.apache.org/ApacheEverywhere 3) "Why Apache" https://s.apache.org/ASF-Trillions 4) “Apache Innovation” https://s.apache.org/ApacheInnovation 

 - The Apache Software Foundation Statement on the COVID-19 Coronavirus Outbreak https://s.apache.org/COVID-19  

 - The Apache Software Foundation Celebrates 21 Years of Open Source Leadership https://s.apache.org/21stAnniversary

 - Apache in 2019 - By The Digits https://s.apache.org/Apache2019Digits

 - The Apache Way to Sustainable Open Source Success https://s.apache.org/GhnI

 - Foundation Reports and Statements http://www.apache.org/foundation/reports.html

 - "Success at Apache" focuses on the people and processes behind why the ASF "just works". https://blogs.apache.org/foundation/category/SuccessAtApache

 - Inside Infra: the new interview series with members of the ASF infrastructure team --meet 
    Chris Thistlethwaite https://s.apache.org/InsideInfra-Chris
    Drew Foulks https://s.apache.org/InsideInfra-Drew
    Greg Stein Part I https://s.apache.org/InsideInfra-Greg
      ...Part II https://s.apache.org/InsideInfra-Greg2 and Part III https://s.apache.org/InsideInfra-Greg3
    Daniel Gruno Part I https://s.apache.org/InsideInfra-Daniel1 and Part II https://s.apache.org/InsideInfra-Daniel2
    Gavin McDonald Part I https://s.apache.org/InsideInfra-Gavin

 - Please follow/like/re-tweet the ASF on social media: @TheASF on Twitter (https://twitter.com/TheASF) and on LinkedIn at https://www.linkedin.com/company/the-apache-software-foundation

 - Do friend and follow us on the Apache Community Facebook page https://www.facebook.com/ApacheSoftwareFoundation/ and Twitter account https://twitter.com/ApacheCommunity

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

= = =

For real-time updates, sign up for Apache-related news by sending mail to announce-subscribe@apache.org and follow @TheASF on Twitter. For a broader spectrum from the Apache community, https://twitter.com/PlanetApache provides an aggregate of Project activities as well as the personal blogs and tweets of select ASF Committers.

Tuesday November 17, 2020

Inside Infra: Gavin McDonald --Part II

The "Inside Infra" series with members of the ASF Infrastructure team continues with Part II of the interview with Gavin McDonald, who shares his experience with Sally Khudairi, ASF VP Marketing & Publicity.



"...you don't know you need somebody until somebody like that arrives."

Earlier you mentioned growth: preparing for growth and being able to accommodate that growth. What areas are you guys experiencing the biggest growth? Is there a specific type of request that's coming in more than others? I'm hearing all the time when I'm dealing with our  Targeted Sponsors, for example, I'm hearing "We need more CI. We want more services in this area. We want more credits." What is it that you guys are feeling, or what are you dealing with in terms of big picture? What's the biggest demand, where are things coming from?

Yeah, as far as I'm concerned you're spot on with CI.

… That's you.

I mean it's not me totally. I have been concentrating on it more as others have been concentrating on other things, yes. Jenkins for example, we had this one --I call it a mega monolith of a service that had all the project services that was on one server, one Jenkins instance. And it was the same instance being upgraded for the last 10 years. So it was time to migrate it, so it's been migrated to five smaller Jenkins services.

BuildBot is also being upgraded and moved to a bigger server. You've got actually Travis is being used to its full capacity.

What does that mean, "to its full capacity"?

When Travis came out, projects started using it, and we're now at the stage where it's at the full capacity ASF is provided.

Oh okay. So if they're giving us 20 whatever, we're at 20.

Right. It's not unlimited. This is why a lot of projects have decided to start moving to GitHub actions, which is also not unlimited. So the more that's provided, the more is needed. I don't think we'll ever keep up with the pace.

Greg says that often. When I talk to him about donations and things like that coming from different companies, he just says "more" and "more" and "more", so he's not exaggerating, right?

No, no, he's not exaggerating.

... If we offer, they'll take it?

Yeah. There's ways in which projects can use CI in the same way they can use other things. And they will use everything that's given to them.

… Insatiable need.

Yeah. Not understanding that there's 300 other Projects that could be using those same services. But there's a few beginning to realize, and there's talks on certain mailing lists that how can we make this more efficient, how can we projects help each other in managing the best usage of these services that we've got. Because they don't want to have it all. They've just been creating whatever they feel is needed for their project. Then sometime later they realize, "Oh, I'm using 80% of what everyone's been given."

… So it's not malicious, just a lack of awareness. When you guys get a new service do you go, "Hi PMCs, we have this thing, there's 20 units available, use with your discretion," or does someone say, "Hey, Infra has this now. We're going to run and take it all," without realizing they're taking it all. How do you introduce new services to the projects? I'm curious because I never see that side of the activity.

Sometimes it's via the mailing list, users@infra. Projects can come to us and ask questions on that mailing list if it's not appropriate for a Jira ticket. People join that mailing list because they're interested in what Infra is up to. So we use that mailing list as a heads up for whatever it could be: "Jira is getting upgraded this weekend, or there's going to be some downtime on this", or it could be things like "okay we've now enabled GitHub actions across the board" or whatever. There were some new features added to one of our self serve things is asf.yaml, which I know you've spoken to Daniel Gruno about.

… You guys actually published a blog post on that. I saw that on Twitter and  just added it to today's weekly news roundup.

Right.

… I was really surprised to see that. It was exciting. Is that new for you to be announcing publicly like that, sharing outside of the ASF's mailing lists?

It is and it isn't. We used to do it all the time years ago. Then as we've become bigger, it's paid staff not volunteers, we're busy all the time. Blog posts got put out of the picture I guess for a while. So this really new cool feature that was provided, the code was provided by a volunteer via a GitHub pull request. And we looked at it, Daniel made some comments, the changes came backwards and forwards until it was ready to be committed the other day. And it's fantastic new features that projects keep asking us for all the time via Jira tickets. "Can you enable this? Can you check this checkbox?" It's work that we shouldn't have to be doing. So now we don't have to do it. They can just edit their own yaml file in their own GitHub repository and those GitHub features are enabled. It was worthy of a blog post.

… I was pleasantly surprised to see that. That is good. It's interesting because we often don't have enough time to do the work, put out the fires, deal with normal life. You have a family by the way, oh yeah let’s not forget that, but you also have to find time to share this sort of information with the outside world. For me, working on the Inside Infra project is amazing because not only do I get to learn about what's been going on, all these years with the ACF for me it's been kind of a black box in many instances with Infra, but I'm also hearing from outside folks who are saying, "This is great to learn," because how would we know otherwise? It's cool that you are able to share more and more however you can. I was really excited to see that blog post.

Hopefully there'll be more coming. Infra does have a blog and it's been not used as much as it should be I think.

When we publish, we'll be sure to get that out (https://blogs.apache.org/infra/). Let's talk about project requirements. Of course Apache projects have been setting the standard across countless "usual suspect" technologies: servers and Big Data and build tools and libraries and so much more. We now have incoming projects in Edge computing and IOT and blockchain and even hardware acceleration, all of which are coming in through the Incubator. Does Infra need to know anything technical about these topics in order to get the job done? Or do you just handle the back-end support and it doesn't matter what's coming, who's the project or the category they’re in? Are there instances that cause you to say, "We have to learn/do something completely different for this project"? Is there anything that's coming in that changes the way you're getting your work done?

No.

… Great; so it doesn't matter.

No, it doesn't matter. Obviously the Foundation welcomes all types of technologies coming in, and Infra provides what it needs to provide. We don't need to know the ins and outs of 300 projects' code to get our own work done. That would be impossible anyway.

But when you came in, you came in through Apache Forrest: you came in with a project. Do you look at stuff that's coming in through the Incubator, out of curiosity, are you keeping an eye on that?

I do, yes, just on a personal level I do just because it's something I like.

… Okay, so it's a curiosity thing, rather than your job depends on it.

Right.

With so much evolving in this space, if you need to learn something new, is it top-down -- someone saying, "Hey Gavin, go to Jenkins University"? How do you figure out what needs to get done, and how do you learn how to do the job? How do you close your skills gaps?

I think each team member has their own way of learning more. Obviously more needs to be learned all the time, and that could be going to the particular pieces of software's Website, having a look through their documentation. I mean when we're implementing things, we're at a vendor's Website every day of the week, you know? We use Puppet a lot: it's core to what we do. We farm out to 30 Jenkins nodes or whatever, it would all be done through Puppet. So we need to keep up with what's happening with that project. So I'm on the Puppet Website looking through the documentation all the time. New versions are coming out, new features are being enabled, and that could be something that makes things easier for us, so we can implement that from our side. I do a lot of that reading. You talk about Jenkins University. I actually have--

… That exists? I was just making that up.

Well, there is a Cloudbees university. So in my own time I've actually done half a dozen courses myself on Jenkins through Cloudbees University. I'm keeping myself up-to-date and ahead of the curve on that. It's a deep interest of mine, so in my own time I take those courses.

Do you bring it back to the team and share it with them? Or is every man for himself? Do you guys have a lunch-and-learn --"Hey on Thursdays during our weekly call we're going to discuss topic X"? Is there anything like that that happens on a bigger scale, or is it on an individual basis?

Mostly individual basis, but obviously anything that people learn is going to be shared amongst the team at some point, it's going to be "oh I learned this today". Someone could be learning from someone through a PR (pull request), you know? A new piece of code comes into our infrastructure that someone committed, then the others ... everybody looks at everybody's code. So they would look at it and go, "Oh that's neat. Didn't know that." And that's because someone has gone out and learned it from some Website or some course they've done.

Right. Plus your team is super, super close-knit. I'm sure you are like, "Hey, I found this cool thing, you got to check it out" --you're sharing things together, right?

Yes, yes, we have our own Slack channel where we talk all sorts of things. That could be new software coming out, yeah, or new ways of doing things. Or new cooking methods.

That's the thing I've been hearing: the most common thing that everyone tells me is they talk about food and drinking --not code. That's good too, because that's the fabric that connects everyone together.

You can't talk about work all the time. It doesn't matter what industry you're in. If you're in an office somewhere or you're working in a restaurant. It doesn't matter what you're doing, you're not standing there talking work all day long. You're talking about your kids or what place is a good place to go to or have you seen this, have you seen this movie. You know? We do the same talking, we just do it online.

To that end, ASF since day one has been a virtual organization: anyone can work from anywhere, there's always these great stories about people meeting each other for the first time at ApacheCon after collaborating online for 10 years. It's a really cool thing to see. And for years you've been our man in Australia. Has anything changed with that as we're growing? Obviously you're no longer in Australia, but do we need to have an Infra presence on every continent? Where are things going with this? Has anything changed, or we're still business as usual since day one, because it doesn't really matter?

There's no specific place you need to live to do this job. Obviously global coverage is a good thing to have. When I was first employed, I'm pretty sure one of the bonus points was that I was in Australia. There was that one was US-based, then there's Australia-based, so you've got a fair bit of coverage there. So somebody can sleep while the other ones can then fight those fires. With there being five or six people now, there is still a bit of a gap, but not too much. People do all sorts of hours. And I plan on going back there sometime.

… I was going to say that we're going to dispatch someone to Siberia or someplace in order to cover the timezone. But you are going to go back you think?

Yeah, at some point.

… You miss it?

Yeah. The weather is nice, and we have family there.

This is a question I've been asking everyone. What are the biggest threats that infrastructure managers need to watch out for? Not necessarily a doom-and-gloom threat kind of lightning bolt coming out of the sky, unless there is something like that to be aware of. With the pandemic there's been all these security crises and all sorts of weird stuff happening. Is there anything in general you think people in this role need to know about or watch out for? Is there any advice that you think folks should be aware of?

Oh I have no idea. The pandemic hasn't really affected me in terms of the role. I mean being an all virtual organization, I'm not sure it's affected the Infrastructure team at all on a work level.

… Remarkably with that, we remained business as usual while everybody else was scrambling trying to figure out how to cope. So I agree with you on that. Another question I ask is what would you think people would be surprised to know about ASF Infra?

I'm sure the same answer has come out of everybody's mouth. That there's only five, six of us looking after this many projects. I don't really know, yeah. There's just so few of us I guess doing this.

As there are so few of you doing it, and you think that would be surprising for people to know, how many people do you think normally should be doing this work? I'm curious about that, because I don't understand how you guys are able to do this. For me there's always this awe of how you guys are making this work as I can't figure out how.

From my perspective I don't know any different. I've not worked at Google or Microsoft or any other tech company down the road. I've not worked in offices, so I've got no comparison. I wouldn't know whether it takes 20, 30, 40 people. All I know is that we could do with a few more people to take the strain off, because we're all working hard. There's not a day we can say, "Oh there's not much today."

… Are you on a schedule? Do you have days off, or are you on some level seven days a week?

It's different for everybody. Working from home which I've done for these last 10, 11 years is I guess you get used to it. You get up in the morning and you have your coffee, then you're straight into the work. That could be 7:00, 8:00 in the morning. Then you're doing bits and pieces. You're getting some work done. Then "oh hang on a minute, I need to go to the shops," so you can take an hour or two off and go to the shops. Then you come back, then you find yourself closing your laptop at 10:00, 11:00 at night.

… From bed.

Yeah. I like to work the weekends. If I wanted to take a Tuesday, Wednesday off or something like that, I'll just take Tuesday and Wednesday off. During these pandemic times, I haven't had to do much of that.

But the flexibility is crucial and it's super helpful, because you need to keep your mindset. I mean it is very easy to go mad. You could very easily get overwhelmed with this type of workload.

You need to try your best to balance it out. I know other people that work remotely say you must have your own office at home, your own space. And the rest of the family needs to know when you're in that office you're at work.

… Daddy is not accessible. That's not realistic either, right?

No, I mean 10 minutes ago I don't know if you heard my three year old coming up to me.

… I did, and I was going to ask you if you needed to stop because I heard the little one.

My office at the moment for this interview is the dining room table. The kids are upstairs.

You partially answered the next question that I ask everyone. Which is, if you had a magic wand, what would you see happen with ASF Infra?

More staff.

… That's what you mentioned earlier without me asking, so more staff, more people. What kind of roles? More of a generalist thing, or do you need a database guy or a specific type of person?

Either of those would be nice. I mean we've got a dozen services that use databases, so we all know databases. I don't know how many of us are database admins. None of us are, I don't think. But we know enough that we need to know for our services that we run. So do we need a DB admin? I don't know. It could be that if a DB admin came along he would say or she would say, "Look at this, I can improve all this. You've been doing it wrong all these years." And maybe we have, I don't know. So you don't know you need somebody until somebody like that arrives.

Do I think we need somebody specific? Maybe another Python individual. Because we are focused on any new code that we write internally for our own benefit would be in Python. But Daniel is pretty good at that, and so is Greg.

Do we desperately need somebody like that, or do we need a generalist? Probably a generalist at this point.

… To handle the volume.

Yeah.

What's your favorite part of the job?

Interesting ... Let me think about that for a second. I think when you do something and somebody from one of the projects comes back and says, "Thanks very much. You've been a great help." I enjoy helping the projects.

I mean obviously I enjoy working on CI stuff. I enjoy maybe providing a new self-serve tool to help projects and help the team out. Various things. I enjoy the job. I've been here that long, obviously I enjoy the job. But yeah. It's nice when somebody from a community says good job. 

… Is that infrequent? I know that we tend to be very high standards and very "that's expected" mentality. Do folks scrimp with the appreciation?

Yeah. Yeah pretty much. But it does happen. I've had a few "thanks very much, you've been great". Sometimes you need to work with a project not for an hour or a day, but sometimes you might need to work with them for a few weeks on something. It might be a migration or something like that. By the end of it you say, "Okay that's done, I'm going to leave your mailing list now or leave your Slack channel," then you get a message saying, "Thanks very much, we really appreciate your work."

That's great. When you first came into the role, when you first came Infra, what was your biggest challenge?

Hard question. I don't know to be honest. I mean the people that I was working with, whether it was paid staff or volunteers, I knew them anyway because I'd been volunteering previously. I don't know of any challenges to be honest. It was moved from volunteer role to paid role. Gradually over time I got to know the people, got to know the Infrastructure layout, where things were at a technical level.

... Do you think there was any sort of attitude shift towards you when you moved towards paid? It was new then: you were one of the earlier team members. Being paid anything kind of raised an eyebrow at the time. I remember that really clearly. Did you face anything like that? Were you challenged by that?

It wasn't an issue for me. If there were any attitude changes, I didn't notice. I don't think the volunteers leaving over the following couple of years was related.

… Well people burn out. And it's a lot of work. That's the other issue: "Hey I'm here to help out, I know Apache needs help, I'll lift or I'll help raise the building or whatever," --that's one thing, but the fact that the demand is constant, you can't expect people to be spending their free time to do all this work. We do need people to be dedicated and paid for. I'm all for that. 

As you've been with us for such a long time, I'm sure there've been highlights for you. What's your proudest contribution or role or moment in this position?

That's a particularly hard question for me. I think I don't get proud over anything much to be honest. I enjoy doing what I do, but I don't have anything specific to call out. Getting my 10 year t-shirt was a proud moment, I guess.

… There you go. Are you saying that because you're shy or you're super humble?

I don't know.

… Or you're super hard on yourself? I mean there's that too: "That's it, that's all I've done." 

No, I'm not one for calling out the self. That's probably the English in me, I guess.

I appreciate you as one of the gold star participants in my Media & Analyst Training, because you've done it so many times. But I noticed that in you, and that was part of the thing I was always trying to tweeze out of you. "Oh we need to talk about ourselves," and you're like, "No." A lot of people have that tendency to downplay or just not go around "rah rahing". I have to mention that's a very American thing to do, but it happens across the industry. Whether you're interviewing for a job or talking to someone at a cocktail party or whatever. Now even more so with social media, so it's the challenge of balancing that. I want you guys to get the recognition. You are doing massive work. You guys are heroes. I know it might sound weird just to hear that, but it's true.

Yeah. Yeah. I guess. Yeah, I don't have anything to call out. I mean if somebody said to me, if I was in ... not that I've had a job interview in over a decade, but I guess that's one of those questions, isn't it? "What are you most proud of in your last role? What did you accomplish? What was your best thing that you did?" And that's why I've not been good at interviews, because I wouldn't know what to say.

… That's where that t-shirt comes into play. I survived and I thrived over 10 years.

Yeah.

That's great. What about your co-workers, how would they describe you?

You see, we just talked about how quiet I am. But on the Thursday team meetings, I probably talk the most.

… I do not believe that. How do you mean?

It's a standing joke, "Oh this week's team meeting was only 20 minutes because Gavin didn't attend," you know? But I don't waffle on, I do talk about work. It's just I've got a lot to say.

That's good. You stick to the point, you're not waffling on. All right. What else do I need to know that I haven't asked? Is there anything specific that you want to highlight, a project you're working on, an achievement, something? Anything that I'm not touching upon that's like "I need to make sure that Sally knows X"? Is there anything you haven't touched upon that we haven't discussed?

No, I don't think so. I was thinking before the call, "What are we going to talk about?" 

… Media training. There you go, you're preparing your speaking points.

Yeah. 

… Ta-da, so that's my proud achievement. I got Gavin to think about speaking points. 

Yeah. I mean yeah, as far as how the Infrastructure team are doing at the moment, are there any big things coming up? Not from me, I don't think. No.

I started thinking about this series when we were in Vegas (ApacheCon) last year. We had so many people show up at media training, I was thinking, "This is such an interesting group that no one ..." --you're not faceless, you're there, but it's so hard to discern the individual behind the group, right? And the group, like you said, is so small, so I think there is such an amazing story with you guys. And you're so integral to the Foundations actual operation. Because you're so ubiquitous, people don't think about it. And I think that's ... Maybe because I'm non-technical, for me it's this massive lift. It's like shooting a rocket up. It's incredible. So I certainly appreciate you guys. I think it's just amazing what you do, and you deserve a pat on the back and an attaboy and all sorts. I'm here to cheerlead.

Thank you.

You bet. Thank you so much. I really appreciate your time today. And thanks for doing this, and thanks for passing on the word with the team that it's not a painful experience. I'm also trying to encourage other members of the community to consider doing something like this. We do the "Success at Apache" posts, with individuals writing about themselves and their experiences. Apache wouldn't work if it weren't for the people. Right? It's all about the people. This is not from a PR perspective, but after 21 years I am always ... I keep wanting to know the same things about everyone. And I know it's not limited to me. If I'm curious, I'm sure other people are curious.

= = =

Gavin is based in the UK on UTC +0 (currently on CET). His favorite thing to drink during the workday is coffee with milk, no sugar, and consumes about 10 cups per day.

Friday November 13, 2020

The Apache News Round-up: week ending 13 November 2020

It's Friday already --the week has zipped by. Here's what happened:

ASF Board – management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 18 November 2020. Board calendar and minutes https://apache.org/foundation/board/calendar.html

ApacheCon™ – the ASF's official global conference series, bringing Tomorrow's Technology Today since 1998.
 - ApacheCon@Home sessions now online at https://www.youtube.com/c/TheApacheFoundation/ 

ASF Infrastructure – our distributed team on three continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield uptime at 99.92%. Performance checks across 50 different service components spread over more than 250 machines in data centers around the world. http://www.apache.org/uptime/

Apache Code Snapshot – Over the past week, 407 Apache Committers changed 2,336,295 lines of code over 3,425 commits. Top 5 contributors, in order, are: Jarek Potiuk, Mark Miller, Andrea Cosentino, Mark Thomas, and Kaxil Naik.            

Apache Project Announcements – the latest updates by category.

Libraries --
 - Apache Log4j 2.14.0 released http://logging.apache.org/
 - Apache CXF CVE-2020-13954 Reflected XSS in the services listing page via the styleSheetPath https://s.apache.org/tsb9p


Did You Know?

- Did you know that more than 300 videos from ApacheCon@Home are now available, including keynotes, plenaries, and sessions on Big Data, Clinical Data, Community, Content Delivery, Fintech, Incubator, Integration, IoT, Machine Learning, Mobile, Observability, Search, Servers, Streaming, and more are now available? https://www.youtube.com/c/TheApacheFoundation/ 

- Did you know that Tencent uses Apache Pulsar to power federated learning with trillions of concurrent training? http://pulsar.apache.org/ 

- Did you know that the Norwegian Agency for International Cooperation and Quality Enhancement in Higher Education is powered by Apache Wicket? http://wicket.apache.org/


Apache Community Notices

- Apache Month In Review: October 2020 https://s.apache.org/Oct2020

- ASF FY2020 Annual Report https://s.apache.org/FY2020AnnualReport 

- "Trillions and Trillions Served" documentary on the ASF: 1) full feature https://s.apache.org/Trillions-Feature 2) "Apache Everywhere" https://s.apache.org/ApacheEverywhere 3) "Why Apache" https://s.apache.org/ASF-Trillions 4) “Apache Innovation” https://s.apache.org/ApacheInnovation 

 - The Apache Software Foundation Statement on the COVID-19 Coronavirus Outbreak https://s.apache.org/COVID-19  

 - The Apache Software Foundation Celebrates 21 Years of Open Source Leadership https://s.apache.org/21stAnniversary

 - Apache in 2019 - By The Digits https://s.apache.org/Apache2019Digits

 - The Apache Way to Sustainable Open Source Success https://s.apache.org/GhnI

 - Foundation Reports and Statements http://www.apache.org/foundation/reports.html

 - "Success at Apache" focuses on the people and processes behind why the ASF "just works". https://blogs.apache.org/foundation/category/SuccessAtApache

 - Inside Infra: the new interview series with members of the ASF infrastructure team --meet
    Chris Thistlethwaite https://s.apache.org/InsideInfra-Chris
    Drew Foulks https://s.apache.org/InsideInfra-Drew
    Greg Stein Part I https://s.apache.org/InsideInfra-Greg
      ...Part II https://s.apache.org/InsideInfra-Greg2 and Part III https://s.apache.org/InsideInfra-Greg3
    Daniel Gruno Part I https://s.apache.org/InsideInfra-Daniel1 and Part II https://s.apache.org/InsideInfra-Daniel2
    Gavin McDonald Part I https://s.apache.org/InsideInfra-Gavin

 - Please follow/like/re-tweet the ASF on social media: @TheASF on Twitter (https://twitter.com/TheASF) and on LinkedIn at https://www.linkedin.com/company/the-apache-software-foundation

 - Do friend and follow us on the Apache Community Facebook page https://www.facebook.com/ApacheSoftwareFoundation/ and Twitter account https://twitter.com/ApacheCommunity

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

= = =

For real-time updates, sign up for Apache-related news by sending mail to announce-subscribe@apache.org and follow @TheASF on Twitter. For a broader spectrum from the Apache community, https://twitter.com/PlanetApache provides an aggregate of Project activities as well as the personal blogs and tweets of select ASF Committers.


Friday November 06, 2020

The Apache News Round-up: week ending 6 November 2020

Welcome, November --we're opening the month with another great week. Here's what the Apache community has been up to:

The Apache Month in Review – highlights of what we've accomplished over the past month.  
 - 
October 2020 https://s.apache.org/Oct2020 

Inside Infra – the interview series featuring members of the ASF Infrastructure team.
 - Meet Gavin McDonald --Part I https://s.apache.org/InsideInfra-Gavin

ASF Board – management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 18 November 2020. Board calendar and minutes https://apache.org/foundation/board/calendar.html

ApacheCon™ – the ASF's official global conference series, bringing Tomorrow's Technology Today since 1998.
 - ApacheCon@Home was a success https://s.apache.org/d3stb All sessions now online at https://www.youtube.com/c/TheApacheFoundation/ 

ASF Infrastructure – our distributed team on three continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield uptime at 99.77%. Performance checks across 50 different service components spread over more than 250 machines in data centers around the world. http://www.apache.org/uptime/

Apache Code Snapshot – Over the past week, 401 Apache Committers changed 4,517,272 lines of code over 3,562 commits. Top 5 contributors, in order, are: Jean-Baptiste Onofré, Claus Ibsen, Gary Gregory, Andrea Cosentino, and Adrian Cole.      

Apache Project Announcements – the latest updates by category.

Application Performance Monitor --
 - Apache SkyWalking 8.2.0, Client JS 0.1.0, and Kubernetes Helm Chart 4.0.0 released https://skywalking.apache.org/

Big Data --
 - Apache HBase 2.3.3 released https://hbase.apache.org/
 - Apache ShardingSphere ElasticJob 3.0.0-beta released http://shardingsphere.apache.org/elasticjob/

Content --
  - Apache Jackrabbit 2.20.2 released https://jackrabbit.apache.org/

Libraries --
 - Apache Commons VFS 2.7.0 released http://commons.apache.org/proper/commons-vfs/

Natural Language Processing --
 - Apache NLPCraft 0.7.1 (incubating) released https://nlpcraft.apache.org/

Search --
 - Apache Lucene 8.7.0 and Solr 8.7.0 released http://lucene.apache.org/

Security Framework --
 - Apache Shiro 1.7.0 released https://shiro.apache.org/
 - Apache Shiro CVE-2020-17510 Shiro with Spring, a specially crafted HTTP request may cause an authentication bypass https://s.apache.org/lxsuw

Did You Know?

- Did you know that you can support the ASF's day-to-day operations with a one-time or recurring donation? http://donate.apache.org/ 

- Did you know that Omnichannel Order Management Systems handle real-time inventory visibility and synchronization using Apache OFBiz? https://ofbiz.apache.org/

- Did you know that the Call for Presentations is open for Apache MXNet Day? Submissions are due by 16 November https://s.apache.org/4jbhx 

- Did you know that Virtual Druid Summit will be held on 18 November? See "Upcoming Events" at http://druid.apache.org/


Apache Community Notices

- ASF FY2020 Annual Report https://s.apache.org/FY2020AnnualReport 

- "Trillions and Trillions Served" documentary on the ASF: 1) full feature https://s.apache.org/Trillions-Feature 2) "Apache Everywhere" https://s.apache.org/ApacheEverywhere 3) "Why Apache" https://s.apache.org/ASF-Trillions 4) “Apache Innovation” https://s.apache.org/ApacheInnovation 

 - The Apache Software Foundation Statement on the COVID-19 Coronavirus Outbreak https://s.apache.org/COVID-19  

 - The Apache Software Foundation Celebrates 21 Years of Open Source Leadership https://s.apache.org/21stAnniversary

 - Apache in 2019 - By The Digits https://s.apache.org/Apache2019Digits

 - The Apache Way to Sustainable Open Source Success https://s.apache.org/GhnI

 - Foundation Reports and Statements http://www.apache.org/foundation/reports.html

 - "Success at Apache" focuses on the people and processes behind why the ASF "just works". https://blogs.apache.org/foundation/category/SuccessAtApache

 - Inside Infra: the new interview series with members of the ASF infrastructure team --meet
    Chris Thistlethwaite https://s.apache.org/InsideInfra-Chris
    Drew Foulks https://s.apache.org/InsideInfra-Drew
    Greg Stein Part I https://s.apache.org/InsideInfra-Greg
      ...Part II https://s.apache.org/InsideInfra-Greg2 and Part III https://s.apache.org/InsideInfra-Greg3
    Daniel Gruno Part I https://s.apache.org/InsideInfra-Daniel1 and Part II https://s.apache.org/InsideInfra-Daniel2

 - Please follow/like/re-tweet the ASF on social media: @TheASF on Twitter (https://twitter.com/TheASF) and on LinkedIn at https://www.linkedin.com/company/the-apache-software-foundation

 - Do friend and follow us on the Apache Community Facebook page https://www.facebook.com/ApacheSoftwareFoundation/ and Twitter account https://twitter.com/ApacheCommunity

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

= = =

For real-time updates, sign up for Apache-related news by sending mail to announce-subscribe@apache.org and follow @TheASF on Twitter. For a broader spectrum from the Apache community, https://twitter.com/PlanetApache provides an aggregate of Project activities as well as the personal blogs and tweets of select ASF Committers.

Monday November 02, 2020

Inside Infra: Gavin McDonald --Part I

The "Inside Infra" series with members of the ASF Infrastructure team continues with Gavin McDonald, who shares his experience with Sally Khudairi, ASF VP Marketing & Publicity.



"...The Foundation itself has a responsibility to the Projects to ensure that there is solid infrastructure there. So there's got to be a requirement that there's people there all the time to maintain this infrastructure. The Infrastructure team has become more professional over the years. The Projects have become customers, I guess. Volunteers are always welcome; at Infra we still have plenty of areas in which volunteers can help out."

All right, let's get started. What is your name and how is it pronounced?

Nice and easy one. Gavin McDonald. Just McDonald as in Big Mac and fries McDonald's. It's M and C, no Mac.

When and how did you get involved with the ASF?

That was back around about 2005. I was looking for something different to do than what I was doing. And I came across the Apache Forrest Project. I knew a little bit about XML and websites and stuff like that. So I started contributing to the Apache Forrest Project. And some months later they made me a Committer.

So you first got involved with the Forrest project, then at some point you became part of infra. How did that evolution happen?

That's me looking around for more things to do. I've always been involved in and interested in system administration work. My first real communications with the Infra team was whilst working on a Forrest Solaris Zone and needed some help with it. Shortly after that I started volunteering there. 

First of all, I saw a huge number of tickets regarding mirrors, you know for our software downloads. I'd say it was probably around 150 tickets outstanding for mirrors wanting to join.

... What?!

Yeah.

... One Hundred and Fifty...

Something like that; some of them had been outstanding for quite a while. At the time there was only one person being paid. There were volunteers obviously looking after the machines and stuff like that. Mirrors were sort of lagging behind as they were less important. So that was my in. I started off with getting karma to add all the mirrors.

There was a certain standard that mirrors have to have, certain configurations. So I was going backwards and forwards with the mirror providers and making sure they were up to scratch, then adding them into our configuration.

From then, I introduced BuildBot to Infrastructure. And I think maybe a year after that, this is now talking 2009, a position opened. I think more or less the rest of the Infrastructure volunteers said, "Gavin is doing the job anyway. Let's give it to him."

That was my interview.

Around October, November 2009 I became paid staff.

Are you the longest serving member of the current Infra team?

Yes. Last year at ApacheCon I got presented with a 10 year t-shirt. Next time there's a physical conference I'll be bringing it along.

10 years thumbs up: that's good! Explain the structure of the Infra team and your role in it.

There are six of us, plus Greg (Stein), our Infra Admin, and David (Nalley), VP Infra. One of them is a documentation guy, that's Andrew (Wetmore). The rest of us all various system administration devops work. We look through tickets, what's needed to be done, and obviously we're looking to improve our infrastructure uptime and software and updates. So we all do what's needed, basically. Everyone has various roles.

What's your role?

Well it's a bit of everything, I think. I have been concentrating quite a lot on the CI/CD side of things. That was written into my original contract, which is now not part of the contract. Basically that means the whole entire time I've been here, I've been involved in BuildBot and Jenkins and other CI/CD stuff, and I've been doing a lot of that lately as well. Migrating Jenkins over to new Cloudbees software, and on a whole load of VMs, mainly in AWS.

You mention that CI/CD is a key part of your role. Is that what you're specifically responsible for within Infra? Are you "the CI guy"? Are there other things you do? Everyone says to me, "Hey we do everything." That sounds amazing, but how is that possible? Do you do everything else in addition to the CI work?

Yeah pretty much. Yeah. Everyone can do pretty much everything that we touch on. Some just choose to do certain things that they're more capable of or more used to working with or they like it better. Nobody is told, "You're working on this."

That's interesting. Fill that part in: if there's six things that need to get done, but five of you are actually hands-on sysadmins, so you guys do what you like to do or what you prefer to do? No one says, "Okay you go handle that mail server"? How does it work?

Obviously there's 24 hours in a day and there's people all around the world. If there's an emergency going on or a mail server breaks down or something needs doing, then whoever's around at the time would step up and say, "Okay I'll take a look at that."

So everyone's pitching in --it's not, "Hey I'm not going to do it. Wait for ‘the mail server guy’."

No, no. Obviously I'm sort of known for the Jenkins and BuildBot stuff, but if I'm not around, everyone else can just jump in and get on with it.

So how did you become the Jenkins guy? How did you get to be the BuildBot guy? You were saying earlier that you kind of evolved into it because it was needed, but is this something that you've personally had interest in to start? Or is it just, "Hey there's a fire here, I need to put it out," and it became this cellular memory, a habit, it's now your thing.

I think a little bit of both. I started off introducing BuildBot not long after I started. Jenkins had already been going a little bit at that time, but I've been involved in that also since the start. And it's over the years become more important to projects. Back when it started, it was a nice-to-have sort of thing. There was none of this pipelines, and CI wasn't an integral part of releases, whereas these days it's more and more a requirement. Jenkins and BuildBot have gone from second-class citizens, if you like, to one of our core services that needs to be kept on top of all the time. It's one of the most important aspects of our infrastructure for projects. There's a great demand for it. And it's increasing all the time.

That's interesting to see it go from a supporting role to the lead demand. That's been what, over 11 years now?

Yeah, 11 years.

In earlier interviews, I spoke to Chris and Drew and Greg and Daniel (Infra team members Chris Thistlethwaite, Drew Foulks, Infra Administrator Greg Stein, and Daniel Gruno) and they've all given me their perspectives on the many areas that infra is involved with. Tell me about the scope of the work that you guys do, and how is it different from other Open Source foundations?

Not sure how I can answer that. I'm not involved in any other Open Source foundations.

Okay, well tell me how does Infra operate at the ASF? You support the Foundation, you support projects. How do you help?

We have, as you know, over 300 projects. Each of those requires a Website, each of those requires an area for their code, whether that be Subversion or Git. We obviously over the last couple of years have been more involved in supporting GitHub for Projects. And we have the Confluence wiki and Jira for the issue trackers. So all of the services that they need to operate as an Apache Project is what we offer them.

So every project needs a Website, as you said. Each Apache Project is responsible for their code, their communications, and their community. So they run their own Website, but Infra handles the backend? What is it that you do for them?

Yes, we handle the backend. We've got Web servers that all the Websites get published on, but they write their Website content, and that could be written in many different languages. So we support them being able to provide their Website content in whatever manner they want. This could be just plain HTML, it could be compiled in Maven or in Pelican, there's a million different things. GitHub pages. So we provide the publishing methods for them to be able to go from there ... most projects these days just want to be able to commit a change and leave it at that. Then that change automatically gets published to the Web via automated mechanisms at the backend, you know? We watch for a commit. That could be via a gitpubsub, could be via Jenkins, via BuildBot, GitHub Actions, all of these methods. We'll see a commit, and it'll publish it and build it if necessary before publishing. So they just commit a change and leave it at that.

So the magic that's associated with that automation, is that something you guys are building to support them? Or is it something that pre-exists? How are you integrating all these different languages or platforms? How is this happening?

Well, software like Jenkins and BuildBot ... those mechanisms we can provide pre-built code to watch their repositories for commits to their Website repository. It'll automatically build it, and then it'll push it to the websites. There's also recently GitHub actions will also, instead of being on Jenkins or BuildBot or Travis or any of those, GitHub actions will take a commit straight out of the GitHub repository. It'll do the building of the Website, then it'll push it to usually what's called an asf-site branch. And then we pick it up from there and publish it. The actual GitHub actions code themselves is written by the projects. So that's self-serve.

If there is a fail for that commit, who fixes it? Is it the Project’s responsibility or is that your responsibility? Who's under the hood dealing with that?

It depends. If it's a coding error, then it's theirs, the Projects. If there's some kind of hardware failure, or if there's a piece of software gone down, communications error, yeah, it's up to us to track that down and find out what happened.

I'm understanding a trend here. If you go to other foundation sites, they seem more “corporate” in the sense that everyone's site looks, feels, and performs the same way, they operate the same way and they tend to be under the same infrastructure altogether, right? They're not using 50 different CMS's.

Right.

... That in itself is highly unusual.

Oh okay, yeah. We don't mandate how projects make their Website look, or we don't mandate how they must build it.

That in itself, the autonomy to do what works best for the Project, I think is highly unusual.

Okay. That's good to know.

In terms of ASF Infra and other foundations, you guys don't sit together and compare notes or talk to each other or anything. A lot of groups copy us, so I presume there's little interaction other than socially, right? I didn't know if there was, "Hey, Linux Foundation does that. We should do the same thing," kind of thing. The ASF does its thing and so be it. 

As far as I know, we have no interest in what other people are doing in terms of how they do things. We do things how we think it's best to do them for us and our Projects, how it works best for us. Whether other team members go off and have a look at how other foundations are doing things, I don't know. But I don't.

... Uniquely Apache.

Yeah.

In terms of services, what's the difference between what you offer for individual Apache Projects and their communities versus Foundation-level initiatives? I presume there's a difference --is the majority of your work serving the Projects? What's the percentage of work that you do that's for the Foundation versus Projects? Is it all for Projects? Or is it all considered one thing.

I don't really see a difference. All the work that needs doing is for the Project or Foundations as a whole. It's all the same to me.

What about incubating projects? Do they have special needs or requirements? How do you support them?

Not really unless they're coming into the Incubator with something they've always used that we don't do. Then we would look at that and decide whether it's something we can do for them or not. There's been a few projects that come in like maybe OpenOffice in 2011.

That was exactly in my head in terms of pre-existing groups that have pre-existing infrastructure. OpenOffice was a whole community altogether in a completely different way. How did Infra support them? What did you do? I knew that there were some issues with the codebase and licensing. What else did you do to support that project?

Oh that was a while ago.

... That's fine, I was just curious as to what you guys did. I just remember it was a huge lift from everybody, from all sides. Licensing and code and every aspect of that project coming in seemed to me to be very, very, very challenging, but we got through it. So that's great.

I know there was a lot of work bringing the code in, and not just from the licensing perspective, but also it was an enormous amount of code that needed to come in. I don't know whether they were in Subversion beforehand, but we provided them their space in Subversion and their Website space. I think a lot of the work was done by the project themselves.

Wow, wow. That was a lot of work. How do you handle Projects or communities that make unreasonable demands from the team? How do you guys deal with that?

There are some projects ask more of Infra than others. Some we never hear from at all. There's kind of a fine balance. Projects that are fairly new, we probably spend a bit more time with them helping them out, making sure they get all set up. They may ask new things, there may be some initial push backs, then all of a sudden there's another two or three projects interested in the same thing. So then we have to take a serious look and decide whether that's something we need to support ongoing.

We do get each of the team members I'm sure gets private PMs on Slack and emails and stuff like, "Hey, can you help me out with this?" Or whatever. Sometimes you just do it. But we're sort of encouraged to ask them to go through the proper channels via a Jira ticket or email to the appropriate list.

Not to name names, but have any Project's expectations been so unusual or so out of scope that it shocked you guys? Have you had situations where it's just been absolute, where you guys have been floored by it?

There's been maybe one or two projects that have just been incessant in their demands on Infra, as if we were their personal team. But we deal with it as in, "okay, slow down, what do you need? File a ticket." If they keep going on and on and on, then obviously we've got escalation levels. We can say, "Hang on," and we can pass that onto our boss and say these are being a bit unreasonable.

For those "colicky baby" types of projects, I've been hearing more and more about additional services being offered through Self-Serve. Are these guys able to take on Self-Serve and go, "Yeah that works for us and we'll do it." Have they been able to kind of self-satiate their needs, or has it always been "Infra do it for us"? How successful has Self-Serve been in terms of wicking away demand?

It's been hugely successful. You're referring to selfserve.apache.org: we introduced that three-four years ago maybe. It was a way to ... help the projects help themselves so they don't have to wait for Infra, because they know Infra is busy. Sometimes waiting two or three days for something is ... from their side of things they're like, "It's been two or three days. Still hasn't been done." But from our side of things, "it's only been open two or three days, what are they worried about?"

... "You're in the queue, wait."

Yeah: self-serve was introduced as a way for them to help them, and also it helps us, there's an awful lot of tickets now that don't get filed because of that. They can create their own Jira Project. They can create their own Confluence wiki. They can create their own Git repositories. 

... On their own completely? Without intervention, without "mother may I?", anything? They just go do it?

Yep. There's an awful lot that they can do on their own. And we introduce more self-serve things all the time that otherwise we'd have hundreds more tickets if they weren't able to do that on their own. They can create their own mailing list now: they don't need us.

Do you have to be a PMC member to do that? Can any Committer can do that? Who gets to administer these types of services for projects?

I believe some of the self serve options are PMC chair, and others are PMC members.

… So not just some person who's like, "Hey I'm committing code, I'm going to go and futz around with the site and break something."

Yeah, no.

That's good. Controls obviously are necessary. This is terrific: what a huge difference.

Yeah definitely.

We've got hundreds of projects that have successfully incubated and graduated under the Apache banner. How do you guys develop new products and services to help support that innovation? We get all sorts of projects coming into the Foundation. Going back to OpenOffice as an example, we've never had a project like this of that scale, and consumer-facing. There were so many different things about that that was so unique, and yet we said, "Yeah you're part of the Foundation, you're coming in, you're part of the family."

Yeah.

We’ve had to adapt as we grow. Is there a way for you guys in anticipation ... feel like you need to have a different type of runway in order to accommodate new projects coming to the ASF? Or do you deal with it as it comes along?

Infra is not in control of what projects come to the Foundation. We don't have a say in that. When a project comes to the foundation and they have different requirements, then that's when we get to know about it. And we would deal with it appropriately then.

Obviously there's growth and we know that there's going to be more and more projects coming to the ASF all the time. So we anticipate growth as such.

… So you are setting yourselves up to accommodate more growth, not specifically a matter of "we need more Jenkins" or whatever.

Right. I mean whatever it is that we are looking after, we need to know that that particular service is going to be able to connect with growth.

Got it. How many requests do you receive a day? In general in terms of what constitutes "hey we're slammed" versus a regular day of "we've got 40 things in the hopper", that's normal? What's the volume that you are dealing with?

I want to give you a figure as far as Jira is concerned, which is only one aspect of the things that we handle. Not everything is done by Jira tickets. But I'd say on an average month, we probably get between 150 to 200 tickets a month.

I've been on the Infra channel on Slack, and it's constant. It's nonstop.

Yeah.

Explain to me a typical workday. How do you manage between "hey I'm focused on a long-term project, this new request is coming in, Sally's hair is on fire because she needs help with a mailing list" and whatever else is going on? There's just constant demand on you guys. How do you not go crazy? How do you manage this?

We just get used to it, I guess. Obviously each individual handles their own time in their own way. At any one time there could be one, two, or all of us could be on Slack. So as requests come in on Slack, if it's a two minute, five minute job, we might just say, "Okay, all right, I'll sort that out for you now." Or if we feel it's going to be a little bit more in depth then we say, "Okay file a Jira ticket." Then one of us can pick that ticket up and take a look at it.

We do get people pinging individuals on Slack saying, "Can you help me with this?" Or whatever. Which is often negative to them in a way that they're narrowing their scope of help they can receive by targeting a specific individual. That person might be extremely busy for the next four or five hours, day and a half, whatever it is. And there's another four or five people that could help them with that question.

Typical day, obviously you get up, you check your emails, you see what's urgent, are there any fires to fight straightaway. You go on Slack, that stays open all day. As requests come in, you check Slack all day long. That's just one of those things. You check your tickets, your Jira tickets, what needs doing today, what can wait, or if you've got plenty of time then even the ones that can wait get done.

Whatever order you feel is most important. Then yes, everyone's got longer-term projects on. So myself personally, if I can spend a day or two on a long-term project, then get back to doing tickets, it's the way it is. If there's a lot going on in ticket land, then your project gets put on hold. If something breaks down ... The other week we had to move our Jira server because the hardware broke, so on a Sunday things broke down. Quickly fire up a new server and move everything across. Not sure anybody noticed, which is a good thing.

That's always a good thing. Business as usual, no one knows. With all this stuff coming at you and servers breaking down on the weekends, et cetera, how do you keep everything organized?

It depends on the day, I guess. Some days are good, some days are ... some days you can't see your hand in front of your face for things going on. Each day as it comes. There's no plan. I don't plan what I'm doing tomorrow. If there's a long-term project and I think things have slowed down, projects aren't asking for things, tickets are coming in slowly, I think I'll get on with my project tomorrow. Then you wake up tomorrow and something different happens. There's no real plan.

You don't use any special tools to keep your work checklist in order or anything like that other than the Jira? 

I tried to use various products over the years. You've got Trello and these other kanban board type things. You actually got to open it up and fill it out, haven't you?

It's so interesting you say that because I think some people find that structured way of working extremely efficient, then it's exactly that solution for them. Spending the time to actually do it is taking away from doing other things ... so I don't know if that works for everyone.

It doesn't work for me. I did start one of these boards, but it doesn't fit in with the job. You've got ... "okay, this has got to be finished in three weeks, this has got to be finished in two days." And it sends you reminders and emails and this and that. I mean there is no time limits on things. We're not a software project. We don't have to release something next week.

… True, you don't have hard delivery dates.

Like you say, time is taken away by filling out these things that are supposed to help you organize. So I just don't do it anymore.

Do you have other challenges with that? Balancing everything and getting everything done?

No, feeling okay. I mean I'm still here.

That t-shirt is evidence, that's true. Since Day One, the ASF has been known for creating their own rules for success. They're like, "We're going to do it our way," right? And Infra --even before there was an official infra-- played an important role. You can't exist without that kind of support. How has --and you've been with the Foundation long enough to see patterns and changes --how has infra changed over the years?

Good question. When I officially came onboard as a contractor, I was the second contractor at the time. And everybody else was a volunteer. There were quite a few volunteers. And they were there a lot. At least a dozen people that were active as infrastructure volunteers, even though they knew that there were two people getting paid to do the same thing, they were still there. Still volunteering.

Over the years, things have gotten a bit more professional, I guess. The service requirements have become more of a professional level. Down time is ... years ago if something was down for a couple of hours, it was like "there were just volunteers that are handling it. They'll get to it when they can". But as more and more paid staff had come onboard, to a grand total of six, a reverse happened with volunteers. They've mostly gone. You've got now maybe two or three volunteers that have stuck around and been around for a while. Because there's paid staff doing it. It's changed as in "who wants to volunteer for something when there's people being paid to do it?"

Was this shift proactive or reactive? Was it a matter of the demand coming from a Project and for us to go, "Well we better change this," or was it a matter of we're feeling like we're having volunteer burnout or whatever and we need to make this a more professionally oriented organization? Do you recall how this shift happened?

It happened gradually over the years. As the Foundation grew, more projects came in, more hardware was required, more services are required, more hands-on time is required. So you increase the staff one by one to handle this. Then I think over time as volunteers start dwindling away, due to the fact that there's people getting paid to do it.

That's one aspect. The Foundation itself has a responsibility to the Projects to ensure that there is solid infrastructure there. So there's got to be a requirement that there's people there all the time to maintain this infrastructure. The Infrastructure team has become more professional over the years. The Projects have become customers, I guess. Volunteers are always welcome; at Infra we still have plenty of areas in which volunteers can help out. And, we don't bite!

Obviously the SLA is related to that shift too. They're becoming customers versus "we're all in it together and everybody figure out how to make it work". I'm sure the expectations also were higher, right? Because now you have a team, what's your excuse for not getting it done?

Right.

[END OF PART ONE]

Sunday November 01, 2020

Apache Month in Review: October 2020

Welcome to the latest monthly overview of events from the Apache community. Here's a summary of what happened in October:

New this month --

 - ApacheCon™ – the ASF's official global conference series, bringing Tomorrow's Technology Today since 1998.
   -- ApacheCon@Home 2020 was a huge success https://blogs.apache.org/conferences/entry/apachecon-home-2020-was-a
   -- All sessions, including Plenaries and Keynotes from ApacheCon@Home are available https://www.youtube.com/c/TheApacheFoundation/

 - Apache Software Foundation Operations Summary: Q1 FY2021 (May - July 2020) https://s.apache.org/2mefr

 - The Apache Software Foundation Celebrates 20 Years of OpenOffice https://s.apache.org/86lex

 - "Inside Infra" – the interview series featuring members of the ASF Infrastructure team
   -- Daniel Gruno --Part II https://s.apache.org/InsideInfra-Daniel2

 - Apache Month in Review: September 2020 https://s.apache.org/Sep2020


Important Dates --

  - Next Board Meeting: 18 November 2020. Board calendar and minutes http://apache.org/foundation/board/calendar.html


Infrastructure --

Our seven-member Infrastructure team on three continents oversees our highly-reliable, distributed network under the leadership of VP Infrastructure David Nalley and Infrastructure Administrator Greg Stein. ASF Infrastructure supports 300+ Apache projects and their communities across ~200 individual machines, 1,400+ repositories, 5-6PB in traffic annually, ~75M downloads per month, and 2-3M daily emails on 2,000+ lists. ASF Infra performs 7M+ weekly checks to ensure services are available around the clock. The average uptime in October was 100%. http://www.apache.org/uptime/

Committer Activity --

In October, 789 Apache Committers changed 12,272,284 lines of code over 15,524 commits. The Committers with the top 5 highest contributions, in order, were: Andrea Cosentino, Claus Ibsen, Mark Miller, Mark Thomas, and Andi Huber.

Project Releases and Updates --

New releases from Apache Ant (Build Management); Any23 (Content); Arrow (Big Data); APISIX (API); BookKeeper (Big Data); Calcite (Big Data); Camel (Integration); Commons DBCP (Libraries); Commons NET (Libraries); Commons Pool (Libraries); Druid (Big Data); Flink (Big Data); Groovy (Programming Languages); Hadoop (Big Data); HBase (Big Data); HttpComponents Client (Servers); Ignite (Data Management Platform); Jackrabbit (Content); Kylin (Big Data); Lucene (Search); NiFi (Big Data); Oak (Content); OpenMeetings (Web Conferencing); Pulsar (Messaging); Qpid Broker (Messaging); Skywalking (Application Performance Management); Solr (Search); Struts (Web Frameworks); Tomcat (Servers); Tuweni (Blockchain); Wicket (Web Frameworks); XMLBeans (Library).

The Apache Incubator is the primary entry path for projects we invite you to review the many projects currently in development in the Apache Incubator http://incubator.apache.org/ . New releases from incubating podlings include: Apache Teaclave (Incubating; Computing); StreamPipes (Incubating; IoT); TVM (Incubating; Machine Learning).

# # #

To see our Weekly News Round-ups, visit https://blogs.apache.org/foundation/ and click on the calendar in the upper-right side (published every Friday) or hop directly to https://blogs.apache.org/foundation/category/Newsletter . For real-time updates, sign up for Apache-related news by sending mail to announce-subscribe@apache.org and follow @TheASF on Twitter. We appreciate your support!

Calendar

Search

Hot Blogs (today's hits)

Tag Cloud

Categories

Feeds

Links

Navigation