What's your name and how is it pronounced?

My name is Chris Lambertus (“Kris Lamb bert uhss”): it's pronounced exactly how it's spelled.

When and how did you get involved with the ASF?

I've been aware of the ASF probably at least since the inception of the ASF. I've been working in IT for quite a long while, and I've been very familiar with the ASF projects, because I use them daily in my career. I didn't actually get involved with the ASF until a buddy of mine who was working on the CloudStack project mentioned to me that (ASF VP Infrastructure) David Nalley was looking for somebody to do some contract Infra work. Long story short, I talked to David and I tossed in an application. I was eventually hired as a part time contractor.

...So CloudStack, we're talking about 2012, when CloudStack first came into the Apache Incubator, or was it after that?

I've been aware of the ASF probably since HTTPd, since the original Web server came out. I joined the team late 2014.

Explain your role within the Infra team —how did you get here? Were they looking for someone who specializes in something particular?

My understanding was David was really looking for somebody that had a background in production systems engineering and had been doing it for a long time in a production environment. That's something that I had been doing since 1992: I've been essentially a professional production systems administrator. I knew that skill set was definitely in line with what David was looking for. I think I brought that to the table pretty well. That's basically what I've been doing ever since as a contractor.

What are you responsible for specifically?

That's a complex question, because the ASF Infra sysadmins are essentially responsible for everything. All of us are all responsible for all of the things. We do tend to specialize a little bit. My current project is probably reengineering the mail system: it's the largest one I'm working on right now. I do tend to focus a lot of my efforts on backups. Beyond that, I do a lot of JIRA and Confluence work with Gavin (ASF Infra team member Gavin McDonald). But Puppet, configuration management, again, all these things are things that all the Infra guys support.

In past interviews everyone has basically said, “we do everything”. How does it work? There's no hierarchy. Everyone does everything. Do queries come in and everyone jumps on them? Do you have a round-robin way of getting stuff done? How do you manage with so much going on with Infra? How do you cope with that?

“Cope with it” is an accurate term. Each of us has... I don't really want to call it a specialty, but definitely a focus. If some question comes in about the new mail routing or something that I've been specifically working on, that would go into my bucket as a priority. Certain people have history with certain types of projects. Gavin (Infra team member Gavin McDonald), for example, has been heavily involved with the Continuous Integration infrastructure for many, many, many, many years. So, he tends to be the font of knowledge for all things CI-related.

We tend to break things up that way. Some of the team members definitely have skillsets above and beyond general system administration work. Humbedooh (Infra team member Daniel Gruno’s username) is a very skilled programmer. He then ends up owning a lot of the software that Infra has developed and he has developed. So, questions regarding that, and specialty configurations related to software that he's written tend to go into his bucket. Because of the nature of the team, and because of the nature of the time zones that we're all in, the responsibility of dealing with issues follows on whoever is on call, first of all, and then whoever is awake and available, to handle any situation that comes up regardless of who "owns" the technology.

Describe a typical workday for you.

Apache work for me is basically: I wake up in the morning, hopefully not at 3:00 in the morning, but get out of bed and plop down in front of the computer. Essentially, my lifestyle is I've always been a computer guy. I've always been really focused on computer system administration, not only as my work but also as a hobby. So, I spend the vast majority of my day behind the computer, whether I'm working on Apache stuff or working on other projects, things like that. That'll go on until 11:00 at night. So, my "workday" is essentially me living my life and doing tasks as they arrive and doing projects as necessary and getting things done that need to get done.

...All the things.

Regardless of the time of day, yeah.

How do you keep your workload organized? Folks have all sorts of different systems. Are you an Evernote type of person, or do you keep your own journal? Do you have a certain system to help manage your workload?

Jira is the primary basis for managing my workload with the ASF. We've done a lot of work in terms of building technologies around Jira. Our service level agreement reporting tools I find extremely useful for seeing what's in the queue, what needs to be done, what hasn't been touched in a while, things like that. That really drives a lot of my day-to-day efforts in terms of replying to tickets and servicing customers.

In addition to that, I also use Jira to track my projects. So, if I have a project going on, that's usually a Jira ticket. And then I can go back and refer to those and see where things are, what needs to be done. I've never been a big one for lists of notes. I do have notes that I keep, but by and large, the things that are on the top of my stack maintain on the top of my brain at the same time. So, I don't feel like I forget a lot of things, but I don't take a lot of notes, which is what it is.

…And then there's people who have everything except the monitors covered in Post-it's.

I don't do a Post-it thing, but I have little text files everywhere with notes and things in them.

So, you all have day-to-day tasks that you manage, as well as things that require your immediate attention, as well as long term projects. In my earlier interviews with other Infra team members, everyone's been saying that I have to talk to you, because you're handling “The Email Project”. For those who aren't aware, standard operating procedure at the ASF is “if it didn't happen on-list, it didn't happen”. So, you have, if I'm understanding this correctly, 21 years’ worth of email archives that you're working on. What's going on with this project? What are you handling? Why is it so important?

Well, as you know, email is the lifeblood of the Foundation. Everything that happens here happens on a list. Because of that, the Foundation has amassed a very large quantity of email archives. Those archives are fundamental to the provenance of the Foundation. So, maintaining those and keeping those safe and available is really a top goal of the Infra team.

The mail project, such as it is, is essentially to upgrade and migrate our existing legacy email system to a modern, more supported system. The current email system as it stands was engineered by folks, volunteers, some staffers, I would guess, over 10 years ago, maybe 15 years ago, running on FreeBSD, which we don't really use too much anymore. Actually, we don't really use it at all. They used technologies that were interesting at the time, but are perhaps not so well supported today. So, a lot of it is modernization.

A lot of it is taking a lot of that old tribal knowledge that really doesn't exist anymore and bringing it into the modern era, documenting all the weird little settings that we have and all the edge cases that we manage in email, management of the list systems, mailing lists and their configuration, and making sure that gets upgraded, migrated, modernized. Doing that all in such a way that we don't a) lose anything, or b) suffer any downtime. So, it's a large project. That's really what I've been working on probably for the better part of the last two years, bringing that up to the present era.

You’re like the Titan Atlas: carrying the heavens on your shoulders. That's a massive, massive undertaking. Is there like a deadline for this —where's the end for this project? Is it never ending?

I feel more like Sisyphus than Atlas, but the deadline is as soon as possible. The thing that we're fighting against is the safety and longevity of the old technology. For quite some time, our primary concern was that the hardware that was running the old email system, which was 15 years old, was going to fail. In fact, it did. But fortunately, I basically copied the whole thing off to a separate colocation facility. So, we had an archive of it when it went down, and I was able to bring it all back up.

So, that wasn't a problem. I mean, it was a problem, but it wasn't a disaster as it could have been. So, the deadline is as soon as possible. But in reality, it's going to work until it stops working. I'm not sure how to better state that, because the technology is so old and we really need to get off of it and onto new technology. But there's no hard and fast timeline. Nobody's really cattle prodding me to get it done, but it's the absolute top priority that I have.

...That was actually my follow-up question. Is the “as soon as possible” official, or is this something you're setting for yourself because you just want to get it done?

Oh, that's definitely an official timeline. Yeah.

...I remember our first email servers were a machine under Brian Behlendorf’s desk at the Wired offices. So, we've come a long way since then.

We have, yes.

...You're handling this behemoth. Are you also dealing with the day-to-day putting out the fires, as well as everybody else?

Absolutely, yes.

The volume and scale of this project seems so huge. Again, the word 'cope' keeps coming to mind, because knowing what I know —and I don't even know— it's just scratching the tip of the iceberg: it seems astronomical in terms of scale and scope. Are you building everything from scratch for this project? Are you using any kind of commercial packages? This is a huge overhaul. Tell us more about it.

Multitasking has been in my blood for my entire life. I don't typically have a problem of splitting my time and my attention and my energies between multiple projects. You are absolutely right: this is a titanic project. It's one of the reasons why it's taken so long. Like I said, we've been working on this for several years at this point. The reason it's taken so long is twofold: One is I can't spend 100% of my attention on it or else I would go absolutely crazy. So I partition that. I partition my mind and my time, if you will. Just a little bit of time here working on this, working on this particular aspect of it, then I'll go work on some tickets. So, I'll go work on something else. If I was only working on the mail, then other things wouldn't get done, right?

I have to partition it that way. I think the main way I've tackled this type of project... Again, my experience in system administration going back so far, I've worked on a lot of very large scale projects. So, this is in the middle in terms of the scale. But the biggest thing is to break it down into multiple components as small a component as you really can. The first thing to do is to analyze the existing system. "What is it? How is it running? How is it tied together? How are these things all related? Where are the pieces? Where are the tendrils? How far do they go?" “Write that down.”

I started developing documentation that explained a lot of stuff. There was some documentation that existed. I take that and I carry it forward then into the new system. Okay, "what things do I want to keep? What things do I HAVE to keep? What things are legacy? What things don't we use anymore?" That process of discovery, of understanding how it was built, why it was built and what we're still using, and what we don't need to use anymore, is probably the vast majority of the work--just to understand it. Once that's done, we say, "Can we use the old technology, or do we need to use a different technology?"

In the case of the Foundation, we're extremely tied to the way that ezmlm, our mailing list system, works. ezmlm is extremely tied to Qmail. So, converting those into other tools, basically, I'll say, it's too complicated. With the amount of data that we have and the amount of dependence that we have on those configurations, migrating it to a different system would be incredibly difficult. So, what we've done is there are modern versions (and updates for) these pieces of software, ezmlm and Qmail.

What we've done is I've taken those packages and I built them for modern operating systems. I've patched them with current technology, TLS and various modern email stuff, and put that into configuration management and built a system that deploys all those packages in a reproducible fashion. So, at any time, I can just turn on a new machine. I could type in, "This machine is the new mail router," and run Puppet, our configuration management software, on it. It'll deploy all that software automatically.

That's probably the second part of this huge phase of developing this. The phase that we're in right now is testing it to make sure that it works the same way as the old one works. Once that's verified, then we can actually look at migrating the old data onto the new system and deploying it into production. I think that answered your question.

I think so, but it made me think of another question: How did this wind up being "your" project? Was this assigned to you? Did you jump on it going, "Yeah, I'm taking it"? How did you wind up with this?

That is a very good question. I don't really know. I think probably just because I had been working with... Back in, 2015 maybe, we were actually having this exact same discussion: "what do we need to do to migrate this EZMLM, all these mail archives, all this stuff to a new modern system?"

One of the things that we looked at was, "Can we transfer this? Can we translate this to something like Mailman or some newer type of mailing list management system?" We looked at a couple of options. The biggest problem we had was that the archivers were terrible. So, Humbedooh basically ended up writing this thing that became Pony Mail as the answer to that system. Ultimately, that turned out to be a great effort. I think it's going to take us a long way. But in the end, I was the one to continue to work on the email system. For whatever reason, I guess it just became my thing. Maybe because I was the only one willing to do it. I don't know.

...Is the legacy system going to be powered by Apache Pony Mail (incubating) at some point, or is it already in the process?

So yeah, lists.apache.org is our primary advertised archive system. That is what we're telling people to use. In terms of what happens to the old system, that remains a little bit under discussion. I don't know the ultimate disposition of that, but the current plan is lists.apache.org will be the primary access to the mail archives.

I noticed that Pony Mail goes back quite a bit, but it didn't originally go back as far as it does now in terms of the archives. I’m curious to see if everything eventually is going to be migrated to it.

Yes, yes, we actually have a plan to load the previous archives in there. We loaded a subset when we first started it up. I believe they go back to 2012 right now. So yes, we do have a plan to load the previous archives.

Great. I understand some Apache projects and their communities are always asking for new services. How does Infra decide which products you support? Who gets assigned to take the lead on introducing new services or new products? I understand that you develop your own custom solutions as well. How do these get divvied up? Is everything in queue? How does it get done?

When you're talking about a project requesting a service, I think the first thing we look at is, "Is this service extremely specific to this one project, or is it something that has broad appeal to the Foundation?" If it has broad appeal to the Foundation, we've got multiple requests for it, it's a service that we feel we can provide, given the amount of time that we have available, then it's something that we would consider doing.

Obviously, there's a lot of other thought that goes into that in terms of what it is, what it does, what it needs to do, who needs access to it, that we have to evaluate. But generally, if it has broad appeal to the Foundation, it would be something we would look into. If it doesn't, if it's something that's very specific to a certain project, what we typically recommend is that a project request their own VM. They can run the service themselves. That's typically how we’ve approached that in the past.

Has the team been in a situation where you're like, "Hey, this is a really cool thing, let's bring it in," and then throw it on projects or see if anybody wants to do it? Does the converse happen also, where you guys have insight as to something that's hot and new and you think that would be a great fit for Infra, but you have to find a "problem" to connect it to; or is that not something that you deal with? Is your work all reactive, or do you ever come into a situation where you say, "Long-term planning: we want to introduce something brand new"?

I think probably up until maybe five years ago, the work was almost entirely reactive. But the team and the processes that David (Nalley) now put together have really pushed us more in a direction of future planning, of taking the time and taking the mindset of, "What can we do long term to better support projects?" I think selfserve.apache.org is a great example of that. That's something that grew out of a small subset of tools. We got very positive feedback from Committers and Projects about using selfserve.apache.org.

That tool has grown extensively since it was developed. I think one of the best things that we provided recently is the .asf.yaml system, which allows projects to essentially set up 90% of their project metadata in GitHub. It lets them set labels. It lets them set notifications. It lets them set all kinds of things, all self-service. So, it's taken a huge load off of Infra in terms of responding to tickets, and also put a lot of that control in the hands of the projects. That's been incredibly well received. It's definitely, I think, one of the best things that we've done for projects in a while. I think it's a fantastic tool.

That's great. Now, it's also a new way of institutionalizing, so to speak, of "scratch your own itch", but in a way that that's a common deployment. You can do your own thing, but there's a common mechanism or method of doing it, because before —it was like the Wild West, back in the '90s— everyone's just doing their own thing. It didn't really matter, but it wouldn't scale properly: you guys can’t really support them because everyone's doing something and it was a one-off. It's interesting to see that selfserve.apache.org has standardized or unified that process.

Yeah, and one of the things that I really like about it too is because we have so many different projects --they're so varied-- the people that work on them are so varied in their skill sets and their desires and their interest level and their skill level and all this. What we want to be able to do is empower projects to use the tooling and take advantage of the skill sets that they have available. So, we don't want to arbitrarily enforce, "Oh, you must use this particular technology," but we also don't want random technologies to proliferate, like you alluded to, the Wild West. So, it's a very refined balance between, "How do you allow projects to do their own thing in a way that's scalable and supportable?" That's a complex task. It's difficult to manage. I think Self-Serve (selfserve.apache.org) goes a long way to support that.

Speaking of Self-Serve and other solutions the team is providing, the strategic process of figuring out where to go —direction— I know you have David (Nalley), I know you have Greg (ASF Infrastructure Administrator Greg Stein). Does the entire team participate in this? How does this work: is it top-down, or is it bottom-up? Are you guys saying, "Hey, there's a new thing that we should do"? I presume you don't have an annual strategy, but rather an ongoing rolling process; how do strategic decisions get made?

It's a collaborative effort, for sure. I think we do have an annual —when we get together at ApacheCon, we do tend to have a lot of discussions about strategy and about future direction. That's one of the things that we try to do as Infra with our team meetups, and with ApacheCon as well, to get together in person in a room and talk about where we're going to go, what we want to do. I say the process is collaborative, because sometimes it comes from the direction of Greg, or the Board, or David, or whoever. Sometimes it comes from a staffer saying, "Hey, it'd be cool if we could do this."

Sometimes it comes from Projects, or Committers, and they say, "Hey, can we go in this direction? I think it would be useful for X reasons." It just depends. By and large, the decisions for a future strategy are brought up by whoever thinks of it and are discussed within the team at a peer-to-peer level, right? We have very few situations where Greg or David or somebody will come down and say, "Thou shalt do it this way." Yeah, very uncommon to have that happen. It's a very collaborative environment, which I appreciate and works well for me.

So, in light of the pandemic, you guys didn't have your face-to-face. Did you do a virtual annual meeting? Or did it just not happen?

Well, we have a weekly team meeting. Yeah, we didn't do any virtual thing beyond that.

With so many projects at the ASF now, with 350 projects and initiatives and growing and so few of you in Infra, you must be constantly learning new things. How do you keep abreast of what's new? How do you close your skills gap? How do you stay ahead of everything?

I follow a few mailing lists, discussion boards, Reddit, and other similar sources. I typically learn new things when I need to implement new technology to solve a problem. "How do we provide 'X'?" I’ll go research it and learn that way. I also find out about new things from my hobby projects or other work.  

...It's not like "I want to take Blah University to become certified in X" or anything like that, right? I mean, you’d do that from your own interest, but it's not something that's required of the job unless it comes up, right?

No, that's never been required of the job. Personally, I'm very much a self-directed learner. If I'm interested in something, I will absolutely seek out the resources to do so. I will say that there's not a lot of time for that stuff, at least not for me. I got a lot going on, right? So, having the time to sit down and take a class or go through that process, I find very difficult. I don't really learn that way very well either. So, class-based learning has never been for me.

...Not linear. Yeah.

Yeah. So, typically, if I want to learn something new —I've been trying to learn Python, because it's definitely a gap for me— I find it incredibly difficult, because it's very hard for me to sit down and watch a video on programming, right? I got to have a reason. I got to have a thing to do. I need to have a project that requires it. And then I go and I figure it out.

...Got it. So, it's purpose-driven education. You need an end result.

Yeah, exactly. That's how I've always operated.

[END OF PART I]

Let's talk about your background and your road to the ASF. How did you become a technical writer and editor? What sorts of projects were you working on?

Well, let's see. I spent 20 years as an ordained minister and I was working for the Episcopal Church in the US and the Anglican Church in Canada. I got to the point where I preached my many thousands of sermons and it was time to stop. It was about then I moved over into QA and documentation with a company building healthcare software in DOS. That tells you how far back we are. One of my first great excitements was helping that team through the Y2K tensions. I got myself a bit smarter and took courses and had a lot of hands-on experience. I became proficient as a tester as well as a documenter. I worked over the next 15 years for a number of companies, large corporations, startups, and nonprofits, and leading teams or participating in teams, both for documentation and testing, but also at one point, I was the director of user experience. It was designing the front-end for a big complex project.

I've built applications from end to end, usually using Flex which compiled to Flash in the days when we could trust it, when we hadFlash to play with, ColdFusion for munging things around and communicating with the database and a MySQL database.

… That's a blast from the past with ColdFusion.

It's still around. There's a new ColdFusion. It's even brighter and shinier, I'm sure. I was just looking at it and thinking, "Gosh, I really should take a look at the tutorial and see if I still recognize anything."

I'm curious, when you tell people what you do, how would you describe the ASF to the uninitiated?

I would say it is a benevolent community home for a whole bunch of highly focused teams who are trying to do good stuff. The benevolent community home provides the support features that let those teams do their things without crashing into each other. 

How do you explain what you do?

Do you know the movie Fifth Element?

… Yes. That's a cult film in the tech community. I've only seen parts of it superficially, I don't know it years after so I might have to watch it again.

You were very young. Your parents probably had to give you permission to go to it.

No. I'm older than you think.

In that movie, there's a sequence when a bad guy is explaining economics by knocking a drinking glass off his table and it breaks and there's a mess. Out from the baseboards of the wall come all these little robots, one robot with a broom, one robot with a dustpan and one robot with a vacuum cleaner and a duster and they go. They run around and they clean up the mess and they disappear back in the baseboard. That's me on the Infra team.

I'm the little guy with the dustpan.

… I love it. I understand that you are also very active with ApacheCon --were you involved in this past ApacheCon that we had in September?

I was. I thought Royale had some things to say that could be said and I looked around and nobody seemed to have the time or have paid attention to the fact that there should be a Royale track. I said, "Oh, there's going to be a Royale track and I guess I'll coordinate it." 

… You volunteered to do that, you decided to do it, you just rolled up sleeves and dove it?

Well, I said to the team, "If nobody else will do it, I'm going to do it."

I was sure, I was so hoping, that someone else would say, "Oh, no, I'll do that," and then I'd be in a support role, but that didn't happen. I also engaged with the team that Rich had to put ApacheCon together, but in a very minor way. I didn't help as much as I felt I should have helped just from a lack of time.

… You were on the Planners list; you were involved with that as well?

Yeah, in the regular meetings and so on and testing out things like the --

… Hopin platform.

I have my own Hopin account now because I found it quite useful.

Was that your first ApacheCon or have you gone to a face-to-face event before?

I've never been to an ApacheCon until this one. Obviously I've never been in a face-to-face one because there hasn't been one since. In fact, the Infra team was going to have one of its annual face-to-face meetings. We were all geared up to do that just when the lockdown happened this year. I haven't even met my colleagues.

… That was actually one of my questions, so we'll get to that later; that's interesting too. Just before we leave ApacheCon, Apache Royale, you mentioned, for them, "it's code once and run anywhere".

That's the theory.

The Project is popular with folks who are programming for mobile devices and other applications. Is this something you're considering with your work that you're doing on apache.org? Is there any cross-pollination or is it completely on a content basis only?

I've not got to the point of suggesting that Apache as a group or that the Infrastructure team use Royale. One reason is that Royale is at the 0.98 stage release. It's really darn good, but it has some weaknesses still. I'm thinking that the suggestion, "Hey, why don't we use the thing which is after all an Apache project to power our front-ends?" should better happen once we're at the 1.0 version.

… I'm all about eating your own dog food, but when you're ready, right? Early on I'd always wondered why don't we require our projects to use Apache projects for everything and was constantly told, "That's not how we work. We don’t dictate what projects use --they’re free to use what they want." Very interesting position, compared to other groups. I was just curious, are you coming across things on the site and saying, "Oh, Royale will be a good fit for this."

One of the ways that Royale could be useful would be as the front-end for the Apache project pages. However, the Apache project landing pages are static. That's their primary thing. They're static HTML pages. Royale really shines when you're doing data-driven pages, when--

… You’re developing across platforms and devices

That's right. Sally logs in and Sally sees this, that and the other thing because of her role on the site. The Royale-built site dynamically knows what to show her. Andrew logs in and doesn't necessarily see what Sally sees.

… That's amazing.

It's cool. I've built apps like that using Flex that were serving people in many different roles, doing many different things on a common project without this huge massive duplicative pile of code to do it with. I could do my elevator pitch for Royale anytime, but that's not what this talk is about.

… No, but it's interesting because it's about site development. I was curious if it impacted what you're working on.

The main difference is that Royale has more power than a project site needs. Pushing them to use Royale because it's an Apache tool might be requiring people to use a shotgun to kill a fly.

With the expansion of the Web and how everyone's becoming a "publishing expert", many people want to break into technical writing and editing. They want to get their hands on site content and content development and don't know where to start. What do you think are some good entry points? Are there special considerations for Open Source, specifically Apache? Is there anything that people need to consider doing? Are you doing something that's very unique to you because you're doing that for the Foundation or is this a more common type of job that you could take anywhere?

I'm not a super expert on anything. I think that would be probably the theme of my life, but what I bring is curiosity and a willingness to try to see things, not just from my stance, but how would this look like to someone who doesn't have the same preexisting knowledge that I have. Any person who would like to become a documentation person for something really just needs to find something and say, "Hey, can I write about it?" Almost any project in the Apache galaxy would jump on a person who is willing to help with the documentation. 

There's not a project here … well, there are a few projects that are very, very thorough with their documentation, but there are a lot of them where this is the thing that you're asking a technical developer to turn over and use another part of their brain and become a writer about the result of the technical development. That's not the easiest thing for most developers. If somebody with writing skills shows up and says, "I really like this cool thing you're doing. Could I write about it, so I could understand it better and maybe others could understand it better?" I think any team in Apache would embrace that person.

Since Day One for us, it's always been "documentation, documentation, documentation", but that’s often lacking. It's a challenge because you want people to do what they're best at and most comfortable with and happiest doing … which … the majority of folks want to develop code and yet we have an uptick with contributors that are non-code contributors. It's an interesting thing to see, "Where can they find the talents?"

In the "real world", the world of a corporation trying to make a buck off their code, they'd have X number of testers anda build and release person and X number of documenters and a middle manager that would help to make all this stuff happen. We don't have that structure. Indeed, the poor developers are called upon to try to put into words what they're doing. It's just tremendous if we have more --I want to say code sympathetic writers, not people who don't have any clue what's going on, but people who have some idea. At least they can ask the right question and say, "How come I don't understand what you're saying here? I think I tried to do it and I can't do it." Then the developer can say, "Oh, that's because you need to be logged in here. We forgot to write that down."

… Right. The obvious missing element to it.

Well, it's so clear in front of you that you can't see it. I flew in a plane once, years ago. The guy sitting beside me had been on the quality assurance team for one of the Gemini missions, the space missions, and this was the early days of manned spaceflight. He told me about how they were testing the escape procedure if something went wrong on the launch pad and they had this 14-step procedure that was attached in front of the astronaut. They thought, "This looks pretty good. We'll go through it." Sat someone down inside the spacesuit in the chair and said, "There's your procedure. Do it. This is step one. Do this. Two, three, four, five." Step six is blow the explosive bolts to release the door. Boom, bang, bang, bang, bang, the bolts go. The door flies away. With it goes the list.

… Right. Forgot about that. Vacuum. There goes the astronaut too with it.

That's right.

… Wow: in the midst of it, you don't think about it?

Well, exactly. What I feel sad about is people who become excited in a software project and try to figure out how they can use it to solve their own problems or answer their own needs. They can't make it run or they can't make it do what they want and they can't figure out from the help docs how to even ask the question they want to ask and they give up and go away. That's a silent vote that we don't really hear.

… That is unfortunate. Unlike other Infra team members who have said to me; and I'm sure you read it --"everyone does everything".

I do nothing. I do nothing.

You're uniquely responsible for optimizing site content. Do you collaborate with any specific Infra team members? You said you talk to Greg. Is there someone you have to go to every time? Do you work with anyone else out of Infra?

I don't go to any one person because I really don't want to make that person roll his eyes when I contact them. It's a small team. First off, I usually lob out a question when I have an issue. I say, "I'm over here on page so-and-so. I haven't a clue what this thing relates to because it looks like it hasn't been touched in a few years. Who knows?" Sometimes, someone right away will say, "Oh, you do this and do that," or they'll say, "Oh, no. Drew knows about that or Gavin knows about that. Go ask him." Then I do that. And sometimes there's silence. Then I won't ask again on the list. I'll wait until the team meeting.

When we've got everyone on the call and it's my turn and say, "I'm stuck on this thing. Who can help me?" someone always steps up and says, "Yeah, put me down for that. I'll help you in a couple of days." All of these people can do everything. The codicil to that is that all day long they are doing everything. I don't want to be hauling on the same person's elbow all day long saying, "Help me with my little thing." I want to spread out my requests, so I don't pull away any one person too often from the essential tasks, the core tasks of keeping the Infrastructure running.

… Do you work with anyone outside of Infra or you only work within the Infra team to get your work done?

There are a number of people I consult, especially specifically with this migration off the CMS. I'm dealing with people on various projects. I probably have 25 conversations going on with people specific to their projects about what are the various pluses and minuses of the alternative technologies that are available or how do we even do step whatever in the list that's on the wiki page. Actually, I'm so proud of myself when I can actually answer one of those questions.

… How do you collaborate with everyone? Do you use certain tools or is it just email or Slack? How do you work with those folks?

Primarily Slack. Well, there are two things. Slack conversation is going on all the time. I also keep my eye out for whenever a page is updated on the confluence wiki Infra area. As soon as it is, like that little robot with the dustpan and the broom, I go and look at the page. I just scroll down it and maybe fix a little punctuation, change this word and that word to make it a little bit more cleare. Usually, I save it in such a way that they know I've been there. Sometimes if I'm just being really, really finicky, I turn off the thing that says, "Tell the team that you've done it." I'll just stealth in and change that run-on sentence into something more legible, but I don't want to draw attention to it.

… Is that more common than not, or …?

I'm not going to say.

… The "non-intrusive" thing. Describe your typical workday.

I get up around 5:00 or 5:30 and get on the Slack channel and see what's happening. The nice thing about my part of this work is that almost never do I check in and they say, "Andrew, you got to fix this." There's usually not a hanging message about an urgent issue. Then if there is no such hanging message, if I know that I have an ongoing project, like the top-level apache.org pages, I go and tackle the next one. That will keep me busy for many days to come. I keep the Slack channel going partly because, as I said before, I like to monitor what's on people's minds because I may say before they think of it, "Oh, do we need a page about that?" 

I'm only working part time, so I only have basically a half of every day available for Apache. I tend to do a couple hours in the morning and another in the afternoon and another at the end of the day. I have another job with a small publishing house. As I'm working on that, I have one eye on the Apache Slack channel to make sure there isn't anything that requires me to jump over there really quick.

How do you keep your workload organized? It sounds like you're super immersed in everything.

Well, I've got, as I mentioned, a job jar page. Basically, it's a long, long list, a checklist of things that need to be done. It's divided into sections. There's a section of things that need to be done for which I need input from other team members. Then there's basically another section where it's just stuff I need to get around to doing. Then if the team comes up with something that they think I should be doing, they're not above adding an item to my list.

We are familiar with that. It's like, "Oh, I left many, many, many more items." How do you stay motivated? What are your challenges?

This is fun. It's partly fun because the ... Let me turn this around: I worked for a corporation at one point. This is fairly early in my career in software and I realized I didn't like what they were selling. I didn't like the way they were selling it. I didn't like what they hoped would happen with the stuff they were selling. It made it harder and harder to go into work. I was very happy when that relationship ended. Here, I am playing around at the edges of a very exciting machine shop or toy workshop with complicated gears and sliders and rheostats and bubbling things that I barely understand, but that I can be helpful with.

Really, I can't see any trouble with motivation. I don't have to say, "Oh, really, you have to go put in your hours." It's more like, "I know, I also have to do this thing outside, my editing job, but I really wanted to do this thing for Apache."

… That's awesome.

The nice thing is that the Infrastructure team does so much so well and almost making it look easy that any project in Apache that's really got itself organized to do its work is going to find success, because there's going to be no roadblock or brick wall or power failure that will keep them from it. That makes me feel like I'm engaged in a very small way in a very large good thing.

With regard to the Infra team, you're looking at it as an "inside outsider", right? You're someone who's working with Infrastructure, but you're not a sysadmin or not a DevOps person. Is this the first time ...

My camouflage is that we all have beards. I fit in that way.

[laughing] … is this the first time you've been part of an Infra team, because usually folks with your profile are usually part of a content team or marketing or PR or sales engineering or some other division that's more again outwardly facing from an acting standpoint. Is this the first time you're dealing with the underbelly?

Not purely. The very first software company I worked for, within a few weeks, I was in charge of build and release. That was as far down into the bowels of the code as I was. I wanted to go with that point, but it was certainly not outward facing or documentation related. At another company, I was in charge of localization across 17 languages. Although of course, there's words involved, it was very much words in terms of, "Will the German for this thing fit on the button we have for it?" I've been the inside-outside guy in other situations before.

Cool. Website administration, as we've been talking about, running Websites in general changed so much over the years. What has been the biggest change or hurdle that you've experienced?

Purely in Website administration or in dev?

... Anything for documentation, what you're doing now, and what it relates to administrating site content.

Gosh, I think the disappearance of paper for me as a writer is one of the big changes. Almost everything I do now I can do digitally. One of the companies I worked with, I helped supervise the transition from printed user guides. There was a great big room full of boxes of spiral-bound user manuals that we stopped doing. We moved over to a product which we could dynamically create, so we could create a manual for Sally and her role and a manual for Andrew and his role that would come off the same text source but would have different chapters with the stuff relevant to what they were doing. Getting away from the physical artifacts to me has been the biggest change in the writing world.

Remember, I work in a publishing company, when I'm not at Apache and what we turn out as books are very much in the physical world. I was just working with an author this past week where I'd send her a PDF of the very final, final, final, "This is the final version of the book before we go to press." She said, "I need a physical copy. Send me one."

We printed off this endless book and drove it to her house. Then she found two tiny little things that way and we fixed those and everybody was happy.

… That's good. When I teach media training, I require everyone to put their laptops away and write, even if it's on the tiny little hotel notepad, but write it down because your brain processes things differently when writing.

Absolutely.

… Some people say, "I feel like I don't know how to write anymore." That's such a sad situation, but it’s our reality.

When I'm doing my own writing, first drafts are always physical. Pretty much always. Then the nice thing then is when you move it over to digital form, even if you try not to, you see ways you can improve it. You're already into version two with a better document because you wrote the thing first by hand and then you write it again on the keyboard.

You've seen a lot of transition in this space. How do you close your skills gaps in order to stay ahead of everything? How do you do that?

Boy, my skills gaps are larger than my skills. I'm wonderfully good at Apache Flex but it's not a skill much in demand now ... If I had known three years ago that I would be on the Infra team now, I would have learned Python, become very comfortable with it, because it's a Python shop. I'm learning Python on the side, not in order to become a coder, but just so I know what the others are talking about.

Got it. But that's not required for you to actually do your editing work?

No. Nowadays, the writing tools are good enough and versatile enough that they're almost transparent. If you learn how to write in Word or on Apple pages and it turns out you have to write using Markdown language in Git pages, that'll take you 90 seconds to figure it out and then you can do it.

Great, so that's not a problem. Earlier we're talking about you not having actually met the team: the offsite was cancelled, and you haven’t been to a previous ApacheCon. A huge advantage for the ASF is, as you know, especially during the pandemic, is that we've been virtual since Day One. We literally didn't have to change anything in order to maintain our day-to-day operations: it's just business as usual. Just keep going. For you, have there been any advantages or disadvantages of working remotely from the team? Do you think your work could be improved in any way or is it no big deal?

Because that's how the team works and because the expectation is asynchronous, that is you ask a question and you may not get your answer until the person four time zones away wakes up, it's not been a big impediment to me. If I need to have a private conversation with someone, I know how to ping them on Slack and we can open a private conversation. I will say I was working remotely from 2013, I guess. I moved from Boston up here to Nova Scotia holding on to the same job I had with the company I was working for. I was into working remotely from then and have continued pretty much without too much disruption.

When that job came to an end, I went primarily freelance. I was working with clients in Japan and Laos, Germany, all over the United States, South America and so on. Of course, I never met any of them face-to-face. This Apache experience ac,tually, I feel closer to this team because we're on Slack all the time than I had felt with many of the other teams I've worked with over the past decade.

… I love that. That's great.

We share cooking recipes. That's really important.

… I hear a lot about that: Infra’s cooking, drinking, and eating. I ask everyone this question and it tends to make folks pause, but what do you think people would be surprised to learn about ASF Infra?

I think ASF Infra is like the person who's not the main act in Cirque du Soleil. This is a big thing happening on Cirque du Soleil. Over in the corner is a person who's throwing two chainsaws and an apple, an egg and a baby, juggling those all at the same time and just doing that. It just happens and somehow it's essential to make it possible for everything else to happen, but that juggler is good enough to just make it happen and make it look easy. I watch what my team colleagues are doing. I'm just in awe of all the things they managed to do. The Apache teams are doing their things and something goes crazy on a server somewhere. We get an alert about it. Whoever is awake at that hour from the team jumps on it and maybe they call in another person. Then they realize it's because of this third thing over on this other server that's gone wrong and they fix that. All the individual project team might notice is that their email is delayed for a few minutes.

… Right. Everything else is being juggled in the background. They're not aware.

Juggling without dropping an egg or a baby.

… Incredible. If you had a magic wand, what would you see happen with what you're handling within Infra, with your role specifically because you're the only one doing that? What would you like to see change?

I don't know that I have a wish at this point. It's pretty straightforward. I guess I wish I could time travel back and learn Python and come back here and be at this point knowing Python, rather than trying to pick it up on the side.

What's your favorite part of the job?

My favorite part is when I hear back from people, "Oh, now I get it. I read that page again now that you've edited it and now I get the thing."

When something we've changed, something we've provided makes it possible for people to do what they need to do.

What was your biggest challenge when you came into the role, when you started? Was it a wall of, "Oh, my God"? What was it like?

I guess it was grandma's attic, trying to figure out what box to pick up first and feeling in the first weeks that I was working, I was afraid. I was very, very preoccupied with not offending anybody, with not implying that they were ... Not correcting their writing in such a way that I insulted them. It took me a little while to realize of course, they are perfectly happy to have their typo corrected, but it was a matter of ... In the first few weeks, I was still feeling out my colleagues and understanding how much they were going to appreciate some documentation help, how much they would find it as an intrusion or a waste of time.

What is your greatest piece of advice for aspiring technical writers and editors coming into this type of role? What advice would you give them?

I would say ask more questions. When you're stuck, don't presume it's your fault. Ask a question. Someone may say, "Oh, that's because X and we never said X."

What are you most proud of in your Infra career to date?

I haven't broken a single damn thing, except one thing and I'm not telling you what it is.

All right. How would your coworkers describe you?

I think the robot with the broom and dustpan. I think they bought that one. I suggested it.

What else do we need to know that I haven't asked?

I'm a playwright. I play the banjo. For $10, I won't play the banjo. At this point in my career, f you hand me a banjo and a cup of coffee, I'll be happy.

= = =

Andrew is based in Nova Scotia on UTC -4. His favorite thing to drink during the workday is countless cups of black tea, accompanied by homemade pumpkin-seed-flour bread served hot with butter.