Entries tagged [apache]

Tuesday April 30, 2019

Apache and GitHub - a friendly PSA about awesomeness

With the news of the Apache Software Foundation teaming up more closely with GitHub, we feel it natural to elaborate a bit on what has been going on and what this means for you as a committer and/or user of Apache software.

A little bit of history

The Apache Software Foundation started experimenting with git as a source code repository system in 2008, and ventured into GitHub in 2010, where we were graciously offered whatever resources we needed.

At first, this was merely a mirror of our existing git and subversion repositories, but as time went on, and projects expressed an interest in utilizing the many user-friendly features of GitHub, we started work on enabling projects to make proper use of GitHub some three years ago in the middle of 2016. This project, aptly named `gitbox`, ensured that committers could make full use of the GitHub features, while we kept a place within our own infrastructure for people inclined to continue using our infrastructure for their work. As git is decentralized by its very nature, we were able to use GitHub to augment rather than replace our git workflow, bringing our software development to the millions of users on GitHub in addition to the existing Apache community and committers, on a case-by-case basis.

In 2018, we made the decision to combine the two different git service offerings we had into one service, allowing all Apache projects to use GitHub if they so desired. Before then, we had two distinct git services: gitbox and git-wip-us, the initial git service that had been available since 2010. We coordinated the move from git-wip to gitbox with the various Apache projects, and in early 2019 we had migrated all projects to the new service, enabling GitHub features for all git-based Apache projects.

With Microsoft's acquisition of GitHub in 2018, and their commitment to help strengthen open source development, we have received additional resources to help lower the bar for contributions, and we'd like to thank GitHub for their support of the Apache Software Foundation through all nine years of using their platform.

What this means for you as a committer

As stated above, our GitHub integration is an augmentation of our existing service. It is available to all committers on git-based projects to make use of, should they so wish. All new git repositories will automatically be available on both GitHub and Gitbox.

For those wishing to take full advantage of GitHub's features, one can link their GitHub and Apache accounts through https://gitbox.apache.org/setup/ which will grant their GitHub account write access to the repositories you'd traditionally have access to at Apache.

People that wish to continue using their Apache committer accounts to commit code may continue doing so on gitbox.apache.org with their Apache credentials. Nothing has changed in that respect.

As Apache is a very email-centered organization, all GitHub activity is naturally linked to our mailing lists to ensure the same level of openness in the development of our software.

What this means for you as a user of Apache software

For many projects, the move to GitHub means a lower bar to both contributing as well as troubleshooting and submitting issues to the projects, through the GitHub issue and pull request features.

Our commitment to provenance, quality and open governance remains the same, and with our tight integration with GitHub through our linked account service, we are able to bring what made Apache a mark of quality to the many users and contributors on GitHub.

As always, if you have any questions, comments, remarks or feedback about this, we welcome you to reach out to the Apache Infrastructure Team at: users@infra.apache.org

Wednesday February 12, 2014

Improved integration between Apache and GitHub

After a few weeks of hard work and mind-boggling debugging, we are pleased to announce tighter and smarter integration between GitHub and the Apache Software Foundation's infrastructure.

These new features mean a much higher level of replication and retention of what goes on on GitHub, which in turns both help projects maintain control over what goes on within their project, as well as keeping a record of everything that's happening in the development of a project, whether it be on ASF hardware or off-site on GitHub.

To be more precise, these new features allows for the following:

  • Any Pull Request that gets opened, closed, reopened or commented on now gets recorded on the project's mailing list
  • If a project has a JIRA instance, any PRs or comments on PRs that include a JIRA ticket ID will trigger an update on that specific ticket
  • Replying to a GitHub comment on the dev@ mailing list will trigger a comment being placed on GitHub (yes, it works both ways!)
  • GitHub activity can now be relayed to IRC channels on the Freenode network.

As with most of our things, this is an opt-in feature. If you are in a project that would like to take advantage of these new features, please contact infrastructure, preferably by filing a JIRA ticket with the component set to Git, and specifying which of the new features you would like to see enabled for your project.

On behalf of the Infrastructure Team, I hope you will find these new features useful and be mindful in your use of them.

Wednesday March 06, 2013

paste.apache.org sees the light of day

Today, the Apache Infrastructure team launched http://paste.apache.org, a new ASF-driven site for posting snippets, scripts, logging output, configurations and much more and sharing them with the world.

 Why yet another paste bin, you ask?

Well, for starters, this site is different in that is it run by the ASF, for the ASF, in that we fully control what happens to your data when you post it, or perhaps more important, what does NOT happen to it. The site enforces a "from committers to everyone" policy, meaning only committers may post new data on the site, but everyone is invited to watch the result. While this is not a blanket guarantee that the data is accurate or true, it is nonetheless a guarantee that what you see is data posted by an Apache committer.

Secondly, committers have the option to post something as being "committers only", meaning only committers within the ASF can see the paste. This is much like the "private" pastes offered by many other sites, but with the added benefit that it prevents anyone snooping around from watching whatever you paste, unless they are actually a committer.

 Great, so how does it work?

It works like most other paste sites, in that you go to http://paste.apache.org,  paste your data, select which type of highlighting to use, and you get an URL with your paste. For text-only clients, raw data will be displayed, while regular browsers will enjoy a full web page with the ability to download or edit a paste. Currently we have support for httpd configurations, C/C++, Java, Lua, Erlang, XML/HTML, PHP, Shell scripts, Diff/Patch, Python and Perl syntax highlighting. If you want to have any other type of highlighting added, don't hesitate to ask!

Since this site enforces the "from committers to everyone" policy, you are required to use your LDAP credentials when making a paste. To allow for the use of the service within console applications (shells etc) that might not (or should not) provide authentication credentials (on public machines you'd want to avoid storing your committer credentials for instance!), we have equipped the site with a token generator, that both allows you to pipe any output you may have directly to the site as well as gives you some hints on how you may achieve this.

Imagine you have a directory listing that you'd only want your fellow committers to see. Publishing this, using the token system, is as easy as doing:
$> ls -la | privpaste               

And there you have it, the command returns a URL ready for sharing with your fellow committers. Had you wanted for eveyone to be able to see it, you could have used the pubpaste alias instead (click on "generate token" on the site to get more information about tokens and the useful aliases).

We hope you'll enjoy this new service, and use it wisely as well as often. Should you have any questions or suggestions, we'd be most happy to receive them through any infra channel you want to use.

Monday October 12, 2009

DDOS mystery involving Linux and mod_ssl

In the first week of October we started getting reports of performance issues, mainly connection timeouts, on all of our services hosted at https://issues.apache.org/.  On further inspection we noticed a huge amount of "Browser disconnect" errors in the error log right at the beginning of the ssl transaction, on the order of 50 connections / second.  This was grinding the machine to a standstill, so we wrote a quick and dirty perl script to investigate the matter.  Initial reports indicated a ddos attack from nearly 100K machines targetting Apache + mod_ssl's accept loop, and the script was tweaked to filter out that traffic before proxying the connections to httpd.

As we started getting a picture of the IP space conducting the attack, the prognosis looked rather bleak: more and more IP's were getting involved and the ddos traffic continued to increase, getting to the point where Linux was shutting down the ethernet interface.  So we then rerouted the traffic to an available FreeBSD machine, which did a stellar job of filtering out the traffic at the kernel level.  We unfortunately didn't quite realize how good a job FreeBSD was doing, and for a time we were operating under the impression that the ddos was ending.  So we eventually moved the traffic back to brutus, the original Linux host, and patched httpd using code developed by Ruediger Pluem.

And back came the ddos traffic.  In a few days the rate of closed connections had nearly doubled, so we had little choice but to start dumping the most frequent IP addresses into iptables DROP rules.  5000 rules cut the traffic by 2/3 in an instant.  But the problem was growing- our logs indicated there were now over 300K addresses participating in the attack.

We started looking closer at the IP's in an attempt to correlate them with regular http requests.   The only pattern that seemed to emerge was that many of the IP's in question we're also generating spartan  "GET / HTTP/1.1" requests with a single Host: header to port 443.   Backtracking through a year of logs revealed that these spartan requests had been going on since August 6, 2008.  The IP's originating these requests were as varied as, and more often that not matched up with, the rapid closed connection traffic we started seeing in October.

So what exactly is going on here?  The closed connection traffic continues to rise, and the origin of the associated spartan requests is currently unknown.

Monday March 23, 2009

Roller installed for use by Apache Projects

blogs.apache.org open for ASF Projects[Read More]



Hot Blogs (today's hits)

Tag Cloud