Entries tagged [apache]
Improved integration between Apache and GitHub
After a few weeks of hard work and mind-boggling debugging, we are pleased to announce tighter and smarter integration between GitHub and the Apache Software Foundation's infrastructure.
These new features mean a much higher level of replication and retention of what goes on on GitHub, which in turns both help projects maintain control over what goes on within their project, as well as keeping a record of everything that's happening in the development of a project, whether it be on ASF hardware or off-site on GitHub.
To be more precise, these new features allows for the following:
- Any Pull Request that gets opened, closed, reopened or commented on now gets recorded on the project's mailing list
- If a project has a JIRA instance, any PRs or comments on PRs that include a JIRA ticket ID will trigger an update on that specific ticket
- Replying to a GitHub comment on the dev@ mailing list will trigger a comment being placed on GitHub (yes, it works both ways!)
- GitHub activity can now be relayed to IRC channels on the Freenode network.
As with most of our things, this is an opt-in feature. If you are in a project that would like to take advantage of these new features, please contact infrastructure, preferably by filing a JIRA ticket with the component set to Git, and specifying which of the new features you would like to see enabled for your project.
On behalf of the Infrastructure Team, I hope you will find these new features useful and be mindful in your use of them.
Posted at 01:16AM Feb 12, 2014 by humbedooh in Development | |
paste.apache.org sees the light of day
Today, the Apache Infrastructure team launched http://paste.apache.org, a new ASF-driven site for posting snippets, scripts, logging output, configurations and much more and sharing them with the world.
Why yet another paste bin, you ask?
Well, for starters, this site is different in that is it run by the ASF, for the ASF, in that we fully control what happens to your data when you post it, or perhaps more important, what does NOT happen to it. The site enforces a "from committers to everyone" policy, meaning only committers may post new data on the site, but everyone is invited to watch the result. While this is not a blanket guarantee that the data is accurate or true, it is nonetheless a guarantee that what you see is data posted by an Apache committer.
Secondly, committers have the option to post something as being "committers only", meaning only committers within the ASF can see the paste. This is much like the "private" pastes offered by many other sites, but with the added benefit that it prevents anyone snooping around from watching whatever you paste, unless they are actually a committer.
Great, so how does it work?
It works like most other paste sites, in that you go to http://paste.apache.org, paste your data, select which type of highlighting to use, and you get an URL with your paste. For text-only clients, raw data will be displayed, while regular browsers will enjoy a full web page with the ability to download or edit a paste. Currently we have support for httpd configurations, C/C++, Java, Lua, Erlang, XML/HTML, PHP, Shell scripts, Diff/Patch, Python and Perl syntax highlighting. If you want to have any other type of highlighting added, don't hesitate to ask!
Since this site enforces the "from committers to everyone" policy, you are required to use your LDAP credentials when making a paste. To allow for the use of the service within console applications (shells etc) that might not (or should not) provide authentication credentials (on public machines you'd want to avoid storing your committer credentials for instance!), we have equipped the site with a token generator, that both allows you to pipe any output you may have directly to the site as well as gives you some hints on how you may achieve this.
Imagine you have a directory listing that you'd only want your fellow committers to see. Publishing this, using the token system, is as easy as doing:
$> ls -la | privpaste
And there you have it, the command returns a URL ready for sharing with your fellow committers. Had you wanted for eveyone to be able to see it, you could have used the pubpaste alias instead (click on "generate token" on the site to get more information about tokens and the useful aliases).
We hope you'll enjoy this new service, and use it wisely as well as often. Should you have any questions or suggestions, we'd be most happy to receive them through any infra channel you want to use.
Posted at 06:37PM Mar 06, 2013 by humbedooh in General | |
DDOS mystery involving Linux and mod_ssl
In the first week of October we started getting reports of performance issues, mainly connection timeouts, on all of our services hosted at https://issues.apache.org/. On further inspection we noticed a huge amount of "Browser disconnect" errors in the error log right at the beginning of the ssl transaction, on the order of 50 connections / second. This was grinding the machine to a standstill, so we wrote a quick and dirty perl script to investigate the matter. Initial reports indicated a ddos attack from nearly 100K machines targetting Apache + mod_ssl's accept loop, and the script was tweaked to filter out that traffic before proxying the connections to httpd.
As we started getting a picture of the IP space conducting the attack, the prognosis looked rather bleak: more and more IP's were getting involved and the ddos traffic continued to increase, getting to the point where Linux was shutting down the ethernet interface. So we then rerouted the traffic to an available FreeBSD machine, which did a stellar job of filtering out the traffic at the kernel level. We unfortunately didn't quite realize how good a job FreeBSD was doing, and for a time we were operating under the impression that the ddos was ending. So we eventually moved the traffic back to brutus, the original Linux host, and patched httpd using code developed by Ruediger Pluem.
And back came the ddos traffic. In a few days the rate of closed connections had nearly doubled, so we had little choice but to start dumping the most frequent IP addresses into iptables DROP rules. 5000 rules cut the traffic by 2/3 in an instant. But the problem was growing- our logs indicated there were now over 300K addresses participating in the attack.
We started looking closer at the IP's in an attempt to correlate them with regular http requests. The only pattern that seemed to emerge was that many of the IP's in question we're also generating spartan "GET / HTTP/1.1" requests with a single Host: 188.8.131.52 header to port 443. Backtracking through a year of logs revealed that these spartan requests had been going on since August 6, 2008. The IP's originating these requests were as varied as, and more often that not matched up with, the rapid closed connection traffic we started seeing in October.
So what exactly is going on here? The closed connection traffic continues to rise, and the origin of the associated spartan requests is currently unknown.