Apache Infrastructure Team

Tuesday April 30, 2019

Apache and GitHub - a friendly PSA about awesomeness

With the news of the Apache Software Foundation teaming up more closely with GitHub, we feel it natural to elaborate a bit on what has been going on and what this means for you as a committer and/or user of Apache software.

A little bit of history

The Apache Software Foundation started experimenting with git as a source code repository system in 2008, and ventured into GitHub in 2010, where we were graciously offered whatever resources we needed.

At first, this was merely a mirror of our existing git and subversion repositories, but as time went on, and projects expressed an interest in utilizing the many user-friendly features of GitHub, we started work on enabling projects to make proper use of GitHub some three years ago in the middle of 2016. This project, aptly named `gitbox`, ensured that committers could make full use of the GitHub features, while we kept a place within our own infrastructure for people inclined to continue using our infrastructure for their work. As git is decentralized by its very nature, we were able to use GitHub to augment rather than replace our git workflow, bringing our software development to the millions of users on GitHub in addition to the existing Apache community and committers, on a case-by-case basis.

In 2018, we made the decision to combine the two different git service offerings we had into one service, allowing all Apache projects to use GitHub if they so desired. Before then, we had two distinct git services: gitbox and git-wip-us, the initial git service that had been available since 2010. We coordinated the move from git-wip to gitbox with the various Apache projects, and in early 2019 we had migrated all projects to the new service, enabling GitHub features for all git-based Apache projects.

With Microsoft's acquisition of GitHub in 2018, and their commitment to help strengthen open source development, we have received additional resources to help lower the bar for contributions, and we'd like to thank GitHub for their support of the Apache Software Foundation through all nine years of using their platform.

What this means for you as a committer


As stated above, our GitHub integration is an augmentation of our existing service. It is available to all committers on git-based projects to make use of, should they so wish. All new git repositories will automatically be available on both GitHub and Gitbox.

For those wishing to take full advantage of GitHub's features, one can link their GitHub and Apache accounts through https://gitbox.apache.org/setup/ which will grant their GitHub account write access to the repositories you'd traditionally have access to at Apache.

People that wish to continue using their Apache committer accounts to commit code may continue doing so on gitbox.apache.org with their Apache credentials. Nothing has changed in that respect.

As Apache is a very email-centered organization, all GitHub activity is naturally linked to our mailing lists to ensure the same level of openness in the development of our software.

What this means for you as a user of Apache software


For many projects, the move to GitHub means a lower bar to both contributing as well as troubleshooting and submitting issues to the projects, through the GitHub issue and pull request features.

Our commitment to provenance, quality and open governance remains the same, and with our tight integration with GitHub through our linked account service, we are able to bring what made Apache a mark of quality to the many users and contributors on GitHub.


As always, if you have any questions, comments, remarks or feedback about this, we welcome you to reach out to the Apache Infrastructure Team at: users@infra.apache.org

Sunday January 27, 2019

Rate-limiting on Apache services

Over the past few days we have implemented rate limiting on selected services across the ASF.

As our foundation grows, so do the number of users and robots utilizing our services. In order to accommodate as many as possible with what resources we have, we have opted to implement rate-limiting to ensure that everyone can get their fair share of use of our services across the globe. The first services to have rate-limiting implemented are:

  • JIRA (issues.apache.org)
  • MoinMoin Wiki (wiki.apache.org)
  • BugZilla (bz.apache.org)

If you are a normal user of our services:

This very likely will never affect you, and you can go about your business just like normal :) If you DO experience errors or 429 (rate limited) response codes, please do let us know.

If you are a robot or otherwise automated tool:

There are now limits in place for how much CPU time you can use, varying from service to service. If you get limited, you will receive a HTTP 429 response instead of the normal 200, and a short text blob will explain that you have crossed our resource limits and have been rate-limited. It will also explain why, and when you can expect to be unblocked again (generally within two minutes time). Scrapers, bots etc using our services should check for a 429 response code and act accordingly (or just slow down the discovery pace in general, as that benefits all of us).

A general note about the rate limiting system, now and in the future:

Rate limits are applied across IP blocks to discourage distributed abuse, thus if you have 1.2.3.4 abusing a service, 1.2.3.5 would potentially also be affected by the rate limits till they expire.

Later this year, we will be rolling out rate limits on more services, and we encourage people automating tasks to honor the 429 responses across all ASF services.

We would also like to point out that there are, as before, additional global limits in place regarding the use of our services, which can be found at: https://www.apache.org/dev/infra-ban.html

Thursday January 10, 2019

Roller updated to 5.2.2

We've updated blogs.a.o to the latest version of Roller, 5.2.2!!


Friday December 07, 2018

Relocation of Apache git repositories on git-wip-us.apache.org to gitbox.apache.org

[IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE DISREGARD THIS POST]

Hello Apache projects,

I am writing to you because you may have git repositories on the git-wip-us server, which is slated to be decommissioned in the coming months. All repositories will be moved to the new gitbox service which includes direct write access on github as well as the standard ASF commit access via gitbox.apache.org.

Why this move?
The move comes as a result of retiring the git-wip service, as the hardware it runs on is longing for retirement. In lieu of this, we have decided to consolidate the two services (git-wip and gitbox), to ease the management of our repository systems and future-proof the underlying hardware. The move is fully automated, and ideally, nothing will change in your workflow other than added features and access to GitHub.

Timeframe for relocation
Initially, we are asking that projects voluntarily request to move their repositories to gitbox. The voluntary time frame is between now and January 9th 2019, during which projects are free to either move over to gitbox or stay put on git-wip. After this phase, we will be requiring the remaining projects to move within one month, after which we will move the remaining projects over.

To have your project moved in this initial phase, you will need:

  • Consensus in the project (documented via the mailing list)
  • File a JIRA ticket with INFRA to voluntarily move your project repos over to gitbox (as stated, this is highly automated and will take between a minute and an hour, depending on the size and number of your repositories)

To sum up the preliminary timeline;

  • December 9th 2018 -> January 9th 2019: Voluntary (coordinated) relocation
  • January 9th -> February 6th: Mandated (coordinated) relocation
  • February 7th: All remaining repositories are mass migrated


This timeline may change to accommodate various scenarios.

Using GitHub with ASF repositories
When your project has moved, you are free to use either the ASF repository system (gitbox.apache.org) OR GitHub for your development and code pushes. To be able to use GitHub, please follow the primer at: https://reference.apache.org/committer/github We appreciate your understanding of this issue, and hope that your project can coordinate voluntarily moving your repositories in a timely manner.

All settings, such as commit mail targets, issue linking, PR notification schemes etc will automatically be migrated to gitbox as well.

Monday September 17, 2018

Position Available: Infrastructure Systems Administrator

UPDATE:  We have received enough applicants at this time. Thank you all for your interest. 

The Apache Software Foundation (ASF) seeks to fill an Infrastructure Systems Administrator position. You will be responsible for working with the existing technical infrastructure team.  The ASF manages a world-wide network of open source software which includes more than 1600 software code repositories, a worldwide distribution and mirroring system for software, change management, issue tracking, and software management for 300+ open source initiatives and over 10,000 contributors around the world.

Applicants should have a strong background in Computer and Information Science, and should be familiar with modern DevOps environments. Applicants must demonstrate the ability to work in a remote team environment alongside others working in diverse locations around the world and in different timezones. The successful applicant will work with the existing Infrastructure team to manage the ASF's critical infrastructure and resources. Infrastructure team members are expected to work a weekly on-call rotation with the rest of the team.

Our infrastructure team also supports our broader community by enabling the creation of self-service tooling. The successful candidate will be able to balance the needs of our critical infrastructure and the needs of our community to self-serve. These two demands can often be in conflict and thus an ability to navigate such complex environments is a distinct advantage.

Familiarity with Puppet (or a similar configuration management tool) Linux (Ubuntu-based), Virtual Machines, Subversion/Git, Python, and full development environment stacks are a requirement. Further, the candidate should possess great documentation skills and should be well versed in not only developing and assisting in technical solutions, but in documenting them.

Daily tasks will include handling of alarms, outages, and security concerns on a timely basis; working with our many communities on their needs and issues; managing tickets on a timely basis; rolling out and upgrading services; reducing our large technical debt; and maintaining a professional and collegial atmosphere. The team coordinates primarily through daily chat usage, weekly meetings, and email. Social skills and ability to integrate closely with the team are expected.

Preferred qualifications include a Bachelor's Degree in Computer Science or similar background from an accredited university, though demonstrable and appropriate on-the-job experience is an acceptable substitute for formal qualifications. Familiarity with how open source communities work is a definite positive.

English as a spoken and written language is required in order to facilitate team collaboration.

This is a remote work position, the ASF does not require nor provide office locations. Travel once per year is required, for a team meetup and will typically be coincident with an Apache conference.

Sunday March 26, 2017

Bringing GitPubSub to the Apache Jenkins build server

When it comes to [Jenkins], it has long been known that [polling must die].

While we could go and create post commit hooks in all the ASF hosted Git repositories, that is something that realistically is just creating an added maintenance burden. In any case, we have [GitPubSub].

The question then becomes, how do we integrate [GitPubSub] with [Jenkins]? Thankfully, ASF committer stephenc is also an active committer to the [Jenkins] project and created a [plugin] that connects to [GitPubSub] parses the events and passes them through to the Jenkins [SCM API].

What does this mean?

* You can turn your Git polling down - way way down - to something like once per day. This should significantly reduce the load on both the ASF git servers and builds.apache.org
* Your builds will be triggered in seconds rather than having to wait for the next polling run.
* You can try out using Multi-branch projects much like the [Maven] project has been doing for [Maven core] and [Maven Surefire]

If the reaction to this change proves positive, the next step will be to integrate SvnPubSub with Jenkins and bring the benefits to the Subversion based projects too

See also this blog post by Stephen Connolly:

https://www.cloudbees.com/blog/using-multi-branch-pipelines-apache-maven-project

[polling must die]: http://kohsuke.org/2011/12/01/polling-must-die-triggering-jenkins-builds-from-a-git-hook/
[GitPubSub]: https://www.apache.org/dev/gitpubsub.html
[Jenkins]: https://jenkins.io/
[plugin]: https://github.com/stephenc/asf-gitpubsub-jenkins-plugin
[SCM API]: https://plugins.jenkins.io/scm-api
[Maven]: https://maven.apache.org
[Maven core]: https://builds.apache.org/job/maven-3.x-jenkinsfile/
[Maven Surefire]: https://builds.apache.org/job/maven-surefire-pipeline/

Posted on behalf of Committer Stephen Connolly (stephenc)

Sunday January 01, 2017

blogs.a.o moved, upgraded and improved

Hi All,

blogs.apache.org   - the site you are reading now! has had a bit of an update.

1. We moved it from an aged VM Host to the Cloud (thanks LeaseWeb!)

2. We puppetised the entire service, from install to deploy (see our Github Mirror )

3. We upgraded the Apache Roller software from 5.0.3 to the latest 5.1.2

4. We enabled LDAP for logins. That's right! Every single ASF Committer can now just login! No more creating an INFRA Jira ticket just to get a Roller account on blogs.apache.org

Other stuff remains the same - meaning if you are a Blog Administrator you still need to invite committers into your blog, you still need to choose to make them an Author or Admin etc - Roller doesnt support anything more than login auth for LDAP currently - but I bet the project would love to see the LDAP integration extended and improved if you feel the need!.

Anyhow, our first new year present to our ASF Committers, a shiny updated blog instance,

 Enjoy, and have a great 2017!!


Monday July 25, 2016

Position Available: Infrastructure Systems Administrator/Architect

UPDATE:  We have received enough applicants at this time. Thank you all for your interest. 

The Apache Software Foundation (ASF) seeks to fill an Infrastructure Systems Administrator/Architect position. You will be responsible for working with the existing technical infrastructure team, and VP of Infrastructure at the Apache Software Foundation. The ASF manages a world-wide network of open source software which includes more than 750 software code repositories, a worldwide distribution and mirroring system for software; change management, issue tracking, and software management for 300+ Open Source initiatives and more than 11,000 contributors around the world.


Applicants should have a strong background in Computer and Information Science, and should be familiar with modern Dev/Ops environments. Applicants must demonstrate the ability to work in a remote team environment alongside others working in diverse locations around the world and in different timezones. The successful applicant will work with the existing Infrastructure team and VP, Infrastructure to manage the ASF's critical infrastructure and resources. Infrastructure team members are expected to work an on-call rotation with the rest of the team.

Our infrastructure team also supports our broader community by enabling the creation of self-service tooling. The successful candidate will be able to balance the needs of our critical infrastructure and the needs of our community to self-serve. These two demands can often be in conflict and thus an ability to navigate such complex environments is a distinct advantage.

Familiarity with Puppet (or a similar configuration management tool) Linux (Debian-based), Virtual Machines, Subversion/Git and full development environment stacks are a requirement. Further, the candidate should possess great documentation skills and should be well versed in not only developing and assisting in technical solutions, but in documenting them.

Preferred qualifications include a Bachelor's Degree in Computer Science or similar background from an accredited university, though demonstrable and appropriate on-the-job experience is an acceptable substitute for formal qualifications. Familiarity with how open source communities work is a plus.

English as a spoken and written language is required in order to facilitate team collaboration.

This is a remote work position, the ASF does not require nor provide office locations. Travel required for conferences and general team meetups.

Contact vp-infra@apache.org with your CV.

Thursday June 30, 2016

ASF JIRA Outages and Troubleshooting

As people have noticed, our JIRA instance (arguably the largest public instance in the world) has been suffering from a yet unknown issue as of late. We are reasonably sure that this is related to specific queries being made against the instance (possibly automated queries from scrapers), but have yet to identify the exact cause of the problem.

The failure condition arises when the database connection pool is exhausted, despite being configured and sized appropriately. These connections all appear idle, but when the pool is full, no new connections can be established, and the instance falls over, requiring a restart. 

We are working closely with Atlassian, the creator of JIRA, to remedy the situation. Unfortunately, this requires running diagnostics on the production JIRA instance, which in and of itself causes performance degradation and downtime. Over the past several days, we've identified and implemented some changes to the pool parameters which we hope will help stabilize the instance while we continue our diagnostic work.

We expect that there may still be some moments of downtime and occasional restarts. Any longer duration outages will be announced via Twitter/infrabot and status.apache.org.

Friday February 12, 2016

AppVeyor CI now available for GitHub Mirrors

The ASF Infrastructure team is happy to announce that projects can how have AppVeyor CI setup on their GitHub mirrors.

 The only thing you need to do is create an INFRA ticket at issues.apache.org with the following information:

  • Repo Name
  • Mailing list to send build notifications to (optional)

There are already a few projects using AppVeyor on their GitHub mirror, and we now have an Organization role account for central management (and I have gone through an updated previous tickets with new links to badges).

If you have any questions, you can ask us in Hipchat or you can email infrastructure@apache.org

Monday October 19, 2015

Dear Apache

My name is Daniel Takamori and I'm so happy to be joining the Infra team here at Apache.  I'm from Oregon in the United States and really enjoy the rain.  While at Oregon State University I studied mathematics and physics with a lean towards error correcting codes and mathematical modelling.  Some of my hobbies are playing Go in which I'm ranked 6.9 kyu by the AGA, cooking with eggs and green things, and old school platforming video games.  In a former life I worked on underwater remotely operated vehicles and automated gardening systems.  Traveling is something I liked to do once; living in Hungary was awesome and I hope to visit again. Oregon is a great place to live, with all the trees, rain and burritos but maybe things will change in the future.  My handle Pono is my Hawaiian name, and I'm really proud to use it.

Previously I was at the Oregon State University Open Source Lab and really enjoyed my time there; getting to know the Open Source communities and even work with Apache!  It was a real eye opening experience to the world of what software and DevOps (lol who knows what that even means).  I'm very excited to continue working with the community and even more excited to start this next chapter with such an amazing group.

See you around internets!

Wednesday August 19, 2015

Planned downtime for ReviewBoard

The ReviewBoard vm ran out of space and despite our best efforts to fix the space issue without restarting the service, that is the only option left.

The plan is to restart the vm on Thursday August 20th at 21:00 UTC (14:00 PDT), but if it fills up again before then, the resize will take place earlier.

A tweet via @infrabot will be tweeted 1 hour before the scheduled downtime and a planned maintenance notice will be posted to status.apache.org.

The actual downtime should take no more than 30 minutes.

The next email about this will be after the service has resumed from the planned downtime.

Thanks!

Geoff Corey

Monday August 03, 2015

Planned downtime for Jira

There will be a planned reboot of Jira on Friday 7th August at 00:00 UTC.

This is 72 hours notice as recommended in our Core Services planned downtime SLA.

Currently, Jira requires a reboot when adding new projects to it. There is an outstanding
ticket with Atlassian about this. They require logs and so these will be gathered at the
time of the planned reboot.

Projects being added to Jira at this time will include:-

INFRA-9713 - Whimsy

and any more that get requested between now and downtime.

Any projects requiring issues to be imported from other issue trackers will NOT be done at
this time.

A tweet via @infrabot will be tweeted 24 hrs and 1 hr before.
A planned maintenance notice will be posted on status.apache.org.

Actual downtime should be no more than 10 minutes all being well.

The next email about this will be after the service has resumed from the planned downtime.

Thanks!

Geoff Corey

Tuesday July 14, 2015

Mirroring to GitHub issues

As some of you are aware, there have been some issues syncing changes from repositories on https://git-wip-us.apache.org to the mirrors on GitHub.

The issues we are seeing:

  • Pull requests not being closed when they should be
  • Changes not being synced to the GitHub mirrors
  • Bots other than asfgit closing PRs on Apache GitHub mirrors.

We are looking into why changes are not being synced, as well as why PRs are not getting closed and why some PRs are being closed by other bots such as hubot.

We will update this blog post as we get more information about the sync issues.

Monday June 29, 2015

Buildbot master currently off-line

Update (2015-06-30 ~12.00 UTC):

The replacement buildbot master is now live. The CMS service and the ci.apache.org  website have been restored. The project CI builds are mostly working but builds that upload docs, snapshots etc. to the buildmaster for publishing are likely to fail at the upload stage while we ensure all the necessary directory structures are in place to receive the uploads. Work to resolve these final few issues is ongoing.

We continue to try and contact the owner of the account where the IRC proxy was running. In case their account has been compromised, it remains locked. In addition, all their commits have been reviewed by other project committers and that review has comfirmed that no malicious commits have been made by the account in question.

The review of aegis.apache.org  is ongoing. No evidence of compromise beyond the possible compromise of the single, non-privileged user account has been found.

Original post (2015-06-29 ~21.00 UTC):

As per the e-mails to committers@ earlier today, aegis.apache.org is currently offline after a report was received that suspicious network traffic had been observed from that host. This blog post will be updated as more information becomes known.

What we know:

  • At ~16.00 UTC 28 June 2015 a report of suspicious network activity from a buildbot host was reported to the Apache security team.
  • Further information was requested and at ~18.00 UTC 28 June 2015 the Apache Infrastructure team received a copy of network logs that showed a number of suspicious IRC connections originating from aegis.apache.org
  • These IRC connections were traced to a non-privileged user account on aegis.apache.org  running an open IRC proxy
  • At ~20.00 UTC 28 June 2015 the user account concerned was locked for all ASF services and the proxy process terminated.
  • At ~10.00 UTC 29 June 2015, after further discussion within the infrastructure team, aegis.apache.org was taken off-line as a precaution.

It remains unclear whether the open IRC proxy was installed by the user that owned the account or whether their account was compromised and the IRC proxy was installed by an unauthorized user.

It is worth stressing that no further information came to light between 20.00 UTC 28 June 2015 and 10.00 UTC 29 June 2015 that triggered the decision to take the host off-line. The host was taken off-line purely as a precaution while we reviewed the available information. That process is ongoing. So far we have found no evidence to even suggest anything more than a user account being used to run an IRC proxy and plenty of evidence that suggests that this was the only activity this account was used for.

Risks:

There is no risk to released source or binaries for any ASF project. There are multiple reasons for this:

  • buildbot is a CI system used to build snapshots, not releases
  • no builds are performed on aegis.apache.org

Buildbot is used to build some project web sites and / or project documentation. The risk of compromise here is viewed as very low for the following reasons:

  • the builds do not take place on aegis.apache.org
  • diffs of every change are sent to the relevant project team's mailing list for review and an unexpected / malicious change would be spotted

Project impact:

The following services are currently off-line and will remain so until the buildbot master is restored

  • All buildbot builds
  • Projects that use the CMS will be unable to update their web sites (the CMS uses buildbot to build web site updates)
  • the ci.apache.org  website

Work in progress:

Analyzing aegis.apache.org  is going to take time and, while we view the chances of a wider compromise of this host as very, very small, we are not willing to bring the host back on line at this point. This host was due for replacement so the decision has been taken to pull this work forward and rebuild the buildbot master on a new host now. We have taken this decision not because we believe aegis.apache.org  to be compromised, but because it is possible to complete this work far more quickly than it is possible to confirm our view that aegis.apche.org is not compromised.  We currently estimate that the rebuild of the new buildbot master host will be completed by 1 July 2015.

We continue to analyze the information we have obtained from aegis.apache.org  and from other sources and will update this blog post as more information becomes available.

Questions:

Questions, concerns, comments etc. should be directed to infrastructure@apache.org

Calendar

Search

Hot Blogs (today's hits)

Tag Cloud

Categories

Feeds

Links

Navigation