Apache Software Foundation Gold Sponsor Profile: Bloomberg

Posted at 02:25AM Jan 15, 2019 by Sally in SuccessAtApache  |   | 

The Apache News Round-up: week ending 11 January 2019

Happy Friday --with the year off to a hot start, we're picking up the pace with a new Top-Level Project, more events, and 80% more commits over the past 7 days. Let's take a look at what's happening with the Apache community:

Do you have a few minutes to help the ASF? In advance of our 20th Anniversary, we'd love your insight on The Apache Way. Our community is integral to our success, and we'd appreciate your feedback on this quick survey https://s.apache.org/oxTr

Success at Apache –a monthly blog series that focuses on the processes behind why the ASF "just works".
 - Success at Apache: Accidentally Finding Awesome by Daniel Ruggeri https://s.apache.org/e3is

ASF Board –management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 16 January. Board calendar and minutes http://apache.org/foundation/board/calendar.html

# # #

Posted at 02:57PM Jan 11, 2019 by Sally in General  |   | 

The Apache Software Foundation Announces Apache® Airflow™ as a Top-Level Project

Open Source Big Data workflow management system in use at Adobe, Airbnb, Etsy, Google, ING, Lyft, PayPal, Reddit, Square, Twitter, and United Airlines, among others.

Wakefield, MA —8 January 2019— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today Apache® Airflow™ as a Top-Level Project (TLP).

Apache Airflow is a flexible, scalable workflow automation and scheduling system for authoring and managing Big Data processing pipelines of hundreds of petabytes. Graduation from the Apache Incubator as a Top-Level Project signifies that the Apache Airflow community and products have been well-governed under the ASF's meritocratic process and principles.

"Since its inception, Apache Airflow has quickly become the de-facto standard for workflow orchestration," said Bolke de Bruin, Vice President of Apache Airflow. "Airflow has gained adoption among developers and data scientists alike thanks to its focus on configuration-as-code. That has gained us a community during incubation at the ASF that not only uses Apache Airflow but also contributes back. This reflects Airflow’s ease of use, scalability, and power of our diverse community; that it is embraced by enterprises and start-ups alike, allows us to now graduate to a Top-Level Project."

Apache Airflow is used to easily orchestrate complex computational workflows. Through smart scheduling, database and dependency management, error handling and logging, Airflow automates resource management, from single servers to large-scale clusters. Written in Python, the project is highly extensible and able to run tasks written in other languages, allowing integration with commonly used architectures and projects such as AWS S3, Docker, Apache Hadoop HDFS, Apache Hive, Kubernetes, MySQL, Postgres, Apache Zeppelin, and more. Airflow originated at Airbnb in 2014 and was submitted to the Apache Incubator March 2016.

Apache Airflow is in use at more than 200 organizations, including Adobe, Airbnb, Astronomer, Etsy, Google, ING, Lyft, NYC City Planning, Paypal, Polidea, Qubole, Quizlet, Reddit, Reply, Solita, Square, Twitter, and United Airlines, among others. A list of known users can be found at https://github.com/apache/incubator-airflow#who-uses-apache-airflow

"Adobe Experience Platform is built on cloud infrastructure leveraging open source technologies such as Apache Spark, Kafka, Hadoop, Storm, and more," said Hitesh Shah, Principal Architect of Adobe Experience Platform. "Apache Airflow is a great new addition to the ecosystem of orchestration engines for Big Data processing pipelines. We have been leveraging Airflow for various use cases in Adobe Experience Cloud and will soon be looking to share the results of our experiments of running Airflow on Kubernetes." 

"Our clients just love Apache Airflow. Airflow has been a part of all our Data pipelines created in past 2 years acting as the ring-master and taming our Machine Learning and ETL Pipelines," said Kaxil Naik, Data Engineer at Data Reply. "It has helped us create a Single View for our client's entire data ecosystem. Airflow's Data-aware scheduling and error-handling helped automate entire report generation process reliably without any human-intervention. It easily integrates with Google Cloud (and other major cloud providers) as well and allows non-technical personnel to use it without a steep learning curve because of Airflow’s configuration-as-a-code paradigm."

"With over 250 PB of data under management, PayPal relies on workflow schedulers such as Apache Airflow to manage its data movement needs reliably," said Sid Anand, Chief Data Engineer at PayPal. "Additionally, Airflow is used for a range of system orchestration needs across many of our distributed systems: needs include self-healing, autoscaling, and reliable [re-]provisioning."

"Since our offering of Apache Airflow as a service in Sept 2016, a lot of big and small enterprises have successfully shifted all of their workflow needs to Airflow," said Sumit Maheshwari, Engineering Manager at Qubole. "At Qubole, not only are we a provider, but also a big consumer of Airflow as well. For example, our whole Insight and Recommendations platform is built around Airflow only, where we process billions of events every month from hundreds of enterprises and generate insights for them on big data solutions like Apache Hadoop, Apache Spark, and Presto. We are very impressed by the simplicity of Airflow and ease at which it can be integrated with other solutions like clouds, monitoring systems or various data sources."

"At ING, we use Apache Airflow to orchestrate our core processes, transforming billions of records from across the globe each day," said Rob Keevil, Data Analytics Platform Lead at ING WB Advanced Analytics. "Its feature set, Open Source heritage and extensibility make it well suited to coordinate the wide variety of batch processes we operate, including ETL workflows, model training, integration scripting, data integrity testing, and alerting. We have played an active role in Airflow development from the onset, having submitted hundreds of pull requests to ensure that the community benefits from the Airflow improvements created at ING.  We are delighted to see Airflow graduate from the Apache Incubator, and look forward to see where this exciting project will be taken in future!"

"We saw immediately the value of Apache Airflow as an orchestrator when we started contributing and using it," said Jarek Potiuk, Principal Software Engineer at Polidea. "Being able to develop and maintain the whole workflow by engineers is usually a challenge when you have a huge configuration to maintain. Airflow allows your DevOps to have a lot of fun and still use the standard coding tools to evolve your infrastructure. This is 'infrastructure as a code' at its best."

"Workflow orchestration is essential to the (big) data era that we live in," added de Bruin. "The field is evolving quite fast and the new data thinking is just starting to make an impact. Apache Airflow is a child of the data era and therefore very well positioned, and is also young so a lot of development can still happen. Airflow can use bright minds from scientific computing, enterprises, and start-ups to further improve it. Join the community, it is easy to hop on!"

Availability and Oversight
Apache Airflow software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache Airflow, visit http://airflow.apache.org/ and https://twitter.com/ApacheAirflow

About The Apache Software Foundation (ASF)
Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server --the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 730 individual Members and 7,000 Committers across six continents successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official user conference, trainings, and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, Anonymous, ARM, Baidu, Bloomberg, Budget Direct, Capital One, Cerner, Cloudera, Comcast, Facebook, Google, Handshake, Hortonworks, Huawei, IBM, Indeed, Inspur, LeaseWeb, Microsoft, Oath, ODPi, Pineapple Fund, Pivotal, Private Internet Access, Red Hat, Target, Tencent, and Union Investment. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Airflow", "Apache Airflow", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 11:00AM Jan 08, 2019 by Sally in General  |   | 

Success at Apache: Accidentally Finding Awesome

Posted at 01:16PM Jan 07, 2019 by Sally in SuccessAtApache  |   | 

The Apache News Round-up: week ending 4 January 2019

Welcome, 2019! We're testing a slightly adjusted format --do let us know what you think:

Apache in 2018 - By The Digits
 - Stewarding 300+ projects "The Apache Way" https://s.apache.org/Apache2018Digits

ASF Board –management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 16 January. Board calendar and minutes http://apache.org/foundation/board/calendar.html

# # #

Posted at 03:53PM Jan 04, 2019 by Sally in General  |   | 

Apache in 2018 - By The Digits

Together, our Members, Committers, contributors, users, supporters, and sponsors continue to build on our mission of providing Open Source software for the public good and are helping keep Apache software accessible to everyone.

Wishing you the best in 2019!

Posted at 07:30AM Jan 01, 2019 by Sally in General  |   | 

The Apache News Round-up: week ending 28 December 2018

It's the last Round-up of the year --the past 12 months have been tremendous. Thank you to all who have contributed to our collective success. Here's to a great 2019!

Let's review what took place over the past week:

Support Apache –help offset our day-to-day operating expenses.
 - We appreciate everyone who supported The ASF with a tax-deductible donation. Please make a year-end gift today: show your ongoing support and help keep Apache software for everyone. Every dollar counts! http://donate.apache.org

ASF Board –management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 16 January. Board calendar and minutes http://apache.org/foundation/board/calendar.html

# # #

Posted at 02:55PM Dec 28, 2018 by Sally in General  |   | 

The Apache News Round-up: week ending 21 December 2018

As we wrap up the week, we wish everyone a festive holiday season for those who celebrate. Like Santa's elves, the Apache community is always working! Here's what happened over the past week:

Support Apache –help keep Apache software for everyone.
 - Just a few remain to make a tax-deductible donation to the ASF in 2018. Every dollar counts! http://donate.apache.org

ASF Board –management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 16 January. Board calendar and minutes http://apache.org/foundation/board/calendar.html

# # #

Posted at 03:08PM Dec 21, 2018 by Sally in General  |   | 

The Apache News Round-up: week ending 14 December 2018

Happy Friday! Let's review what the Apache community has been up to over the past week:

Support Apache –help keep Apache software for everyone.
 - Two weeks remain to make a tax-deductible donation to the ASF in 2018! 750+ generous contributors donated $80K+ as part of our Individual Giving campaigns. Every dollar counts! http://donate.apache.org

ASF Board –management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 19 December. Board calendar and minutes http://apache.org/foundation/board/calendar.html

# # #

Posted at 03:55PM Dec 14, 2018 by Sally in General  |   | 

The Apache Software Foundation Announces Apache® Griffin™ as a Top-Level Project

"We appreciate the Griffin project which really helps so much in our daily data jobs.After years of struggling with the complexity of data quality issues, we turned to Apache Griffin for a new platform that would simplify our data quality pipeline," said Jianfeng Liu, Director of Real-time Data Department at PPDAI. "Because of Apache Griffin's unified model for both batch and stream processing, we've been able to replace legacy systems with one solution that works seamlessly in our production environment. Griffin DSLs have allowed us to dramatically simplify our pipeline and to reduce our efforts a lot. I'm very proud and excited to see that the project is graduating."

Posted at 11:00AM Dec 12, 2018 by Sally in General  |   | 

The Apache News Round-up: week ending 7 December 2018

Greetings, December: counting down to the end of the calendar year. All the while, the brilliant Apache community remains productive:

Support Apache –help keep Apache software for everyone.
 - Join 730+ generous contributors who have donated nearly $80,000 to the ASF as part of our Individual Giving campaigns. Giving to the ASF feels great and is so easy --every dollar counts! http://donate.apache.org

Success at Apache –a monthly blog series that focuses on the processes behind why the ASF "just works".
 - Success at Apache: Cookie Monster by Isabel Drost-Fromm https://s.apache.org/cnSe

ASF Board –management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 19 December. Board calendar and minutes http://apache.org/foundation/board/calendar.html

# # #

Posted at 08:14PM Dec 07, 2018 by Sally in General  |   | 

Success at Apache: Cookie Monster

by Isabel Drost-Fromm

As a researcher interested in machine learning, Web- and social graphs I joined the Nutch mailing lists back in 2005 when the project was still on SourceForge. I started tinkering with Nutch Writeables to store the data I needed for my analysis – something that today some may know as Hadoop Writeables – the Nutch wiki still has a link to the material that I could get published out of those experiments: https://wiki.apache.org/nutch/AcademicArticles

After leaving academia I remained on the Nutch and Lucene mailing lists - until one day I saw the idea of an "Apache Text" project mentioned: https://lists.apache.org/thread.html/ac22faddbef946b66d544e590fe1b2a54b60215c98cc38a2f995ee06@1176254016@%3Cdev.lucene.apache.org%3E ... I got in touch with Grant Ingersoll, over the course of half a year that vague idea was turned into a plan to have a scalable machine learning project at Apache: Scalable in terms of community, dataset size but also commercially friendly when it comes to licensing – Apache Mahout was created.

Some ideas turn into something with a life on its own. The story I'm going to tell has little to do with great technical or economic achievements that were made with software developed at The Apache Software Foundation. However it has a lot to do with the kind of cross community links that exist between projects at Apache. It also has a lot to do with the fact that there are people active in Apache projects for whom the project is more than merely a day job.

But let's start at the beginning: Little over a year ago, in April or May 2017 Stefan Rudnitzki, one of my then-new colleagues at Europace AG was showing me around the office – mentioning in particular that there's space for meetups of 100 up to 200 people. It was the year when it was unclear whether or not there would be an ApacheCon EU. The combination of those two pieces of information put  an interesting idea in our heads: Why not pull ASF interested people to Berlin and have them discuss cross-community, behind-the-scenes, OSS economics, decentralized project management, coordination of work without discretionary power topics?

In a first step we ran a rough version of the idea past a handful of friends at Apache – and received encouragement. The idea got bigger, new aspects were added and we thought "Let's get more specific!".

In a working backwards model the next thing that was written was a press release (in big, bold, red letters marked as "draft, imaginary, DO NOT PUBLISH!!!!!!!") describing a conference on all things open source behind the scenes. The format helped identify important open question marks – like: 

• "We don't have a name for the event yet!"
• "We need to decide on a date."
• "We need to come up with a clearer list of topics to cover."
• "What's our target audience?"
• "If this is a full day event – what will we do about catering?"
  
  

What helped me personally was having learnt from Sally in her ASF media training what a real press release actually should look like. 

As for the name that was found missing in the initial press release draft:  After weeks of trying several approaches to come up with a catchy name, I went to pick up my child from kindergarten. What caught my eye was a poster announcing a beneficial concert to collect donations for better equipment and toys – an *a capella* concert: .oO(FOSS A Capella?) .oO(FOSS Backstage?)

The press release formatted version of the vision was first run by Europace – though people here are fairly regularly running after hours meetups, hosting an entire full-day conference is a slightly different scale. After the idea had been met with approval here, it was run by the Apache Community Development mailing list – which we used to keep current planning status transparent and public. 

With the idea out in the open it grew beyond something that can easily be run as a small side project. Years ago to create Berlin Buzzwords I had been working together with an event agency called newthinking communications GmbH. They were founded in 2003 by Andreas Gebhard and Markus Beckedahl in the spirit to create a network on the interface between digital technologies and society. Today, the focus lies in the organisation of events such as Berlin Buzzwords and FOSS Backstage as well as content management services (based on Drupal) for NGOs and political parties as well the conferences named above. So I got in touch with newthinking – and was delighted to receive "Sure, we are going to help out" as an answer. 

So, what about the cookies? One of the first offers I received after publishing that we were to run a FOSS Backstage full day Micro Summit in November 2017 was: "If you need support with providing cookies for the coffee break – I'm happy  to bake some, if there's no more than 40 attendees." Half jokingly I responded that I would add another 40 cookies, lest someone sends me a 3D model of an ASF feather cookie cutter. Lo and behold – the next thing I know is that someone sends me a model file for an ASF cookie cutter (which by now even made it to the then VP trademarks – who was interested in putting it to good use himself). Just a few weeks later I attended Open Source Summit in Prague. Guess what happened? Someone who knew I'd be there brought some printed cookie cutters with him from Australia.

In the meantime we had a one day / two tracks FOSS Backstage Micro Summit in November 2017 kindly hosted by Europace AG. I was able to talk several people into baking ASF cookies (including sugar coatings in the appropriate colours). In addition with the support of both, Newthinking communications GmbH, the ASF planners, and the ASF community development PMC an Apache Roadshow was co-located with the actual FOSS Backstage in June this year – a two day, multiple tracks event featuring Danese Cooper and Shane Coughlan for keynotes, a host of speakers with all sorts of relevant and inspiring stories to share, as well as fishbowl discussions on topics like Open Source monetization. One of the loveliest feedback we received: "This doesn't feel like an inaugural conference, given the professional organisation. You surely did manage to successfully invite people from a great variety of FOSS projects and foundations."

Having a press release draft ready was helpful when starting to drum up interest for the real event: With all details filled in, the "Draft/ Do not share"-warning removed it ended up getting sent to the press and published for real.

We started with a scope of all things FOSS economics, decentralised organisation, cross-cultural team-building, volunteer motivation, licensing and legal. In 2019 we want to align these aspects towards InnerSource, work collaboration principles and modern work models so that teams, companies and organisations can learn from the experiences we all make while working on Open Source projects. We are glad to have the event backed by newthinking GmbH next year again.

"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works" https://blogs.apache.org/foundation/category/SuccessAtApache

Posted at 11:41AM Dec 03, 2018 by Sally in SuccessAtApache  |   | 

The Apache News Round-up: week ending 30 November 2018

Farewell, November: you certainly were a fast-moving month. This past week was fast-moving as well, with the following activities from the Apache community:

Support Apache –help keep Apache software for everyone.
 - From Giving Tuesday to every day, more than 700 generous contributors have donated nearly $80,000 to the ASF as part of our Individual Giving campaigns. Giving to the ASF feels great and is so easy --every dollar counts! https://s.apache.org/9cCy

ASF Board –management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - The Apache Software Foundation Operations Summary: August - October 2018 https://s.apache.org/d2Fq
 - Next Board Meeting: 19 December. Board calendar and minutes http://apache.org/foundation/board/calendar.html

# # #

Posted at 02:45PM Nov 30, 2018 by Sally in General  |   | 

The Apache Software Foundation Operations Summary: August - October 2018

Second Quarter, Fiscal Year 2019 (August - October 2018)

(c) The Apache Software Foundation 2018.

Posted at 08:06PM Nov 29, 2018 by Sally in General  |   | 

Support Apache: Individual Giving and Corporate Gifts Campaigns

Posted at 04:35AM Nov 27, 2018 by Sally in General  |   |