Newest addition to Apache Big Data ecosystem used for continual, incremental processing of data at petabyte scale

Forest Hill, MD –26 July 2017– The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today that Apache^® Fluo™ has graduated from the Apache Incubator to become a Top-Level Project (TLP), signifying that the project's community and products have been well-governed under the ASF's meritocratic process and principles.

Apache Fluo is a distributed system for incrementally processing large data sets stored in Apache Accumulo (the sorted, distributed key/value store based on Google's Bigtable, built on top of Apache Hadoop, Apache Zookeeper, and Apache Thrift). With Fluo, users can continuously join new data into large existing data sets without reprocessing all data. Unlike batch and streaming frameworks, Fluo offers much lower latency and can operate on extremely large data sets.

"I am very excited to see Apache Fluo graduate and I would like to thank our mentors for all their help, the Apache Incubator Project Management Committee for its advice and guidance, everyone in the Fluo community, and Google for publishing the research upon which Fluo is based." said Keith Turner, Vice President of Apache Fluo. "As a result of collaboration within the community, we are graduating with a beautifully designed piece of software."

Based on Percolator (built on top of Bigtable to support incremental updates to the search index at Google), Fluo makes it possible to continually-update the results of a large-scale computation, index, or analytic as new data is discovered.

"Apache Fluo is a very clever piece of software, elegantly supplementing Apache Accumulo's ability to store and maintain very large indexes," said Christopher Tubbs, ASF Member and Committer on Apache Accumulo and Apache Fluo. "Its support of transactions enables Accumulo to solve a whole new set of big data problems, and its observer framework makes designing ingest workflows fun."

An example of how Fluo works is a use case of counting phrases in unique documents. This could be accomplished by two MapReduce jobs: one job to get a unique set of documents and a following job to count phrases. Where petabytes of documents are concerned, running both jobs for a small amount of new data is inefficient. Apache Fluo enables continuous, quick computations of these two joins as new data arrives, constantly emitting deltas of phrase counts. Anything could consume the emitted deltas. For example, a query system could be continuously updated using them.

"We are excited that Fluo is becoming a Top-Level Project at the Apache Software Foundation," said Dr. Adina Crainiceanu, Apache Rya (incubating) Committer and Associate Professor, Computer Science Department, United States Naval Academy. "Heartfelt congratulations to the Fluo community for achieving this important milestone. The Apache Rya project uses the observer framework in Fluo to cache and maintain answers to complex SPARQL queries for large RDF datasets. Using cached answers greatly improves Rya's performance for complex queries. Fluo complements Rya by allowing the incremental and continuous update of the cached answers. Fluo is particularly useful because it allows updates to happen as new data is ingested, reduces updates latency, avoids stale results, and circumvents the periodical reprocessing of the entire dataset. We are confident that Apache Fluo will become one of the important frameworks for updating indexing results in a dynamic data-acquiring context."

"Fluo fulfills an important role in the Apache Hadoop ecosystem, significantly expanding existing capabilities for working with large data sets," said Billie Rinaldi, ASF Member and former Vice President of Apache Accumulo. "I was excited to see this project come to the Apache Incubator, and am even more pleased to see it graduate to a top-level Apache project."

"We welcome new users and contributors to Apache Fluo," added Turner. "If you are interested in trying Fluo, check out the Fluo Tour on the project Website. Join our mailing lists to discuss how Fluo may be a good solution for your problem, as well as for help with debugging and finding starter issues."

Catch Apache Fluo in action and meet members of the Fluo community at Accumulo Summit, 16 October 2017 in Columbia, MD. http://accumulosummit.com/

Availability and Oversight
Apache Fluo software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache Fluo, visit http://fluo.apache.org/ and https://twitter.com/ApacheFluo

About the Apache Incubator
The Apache Incubator is the entry path for projects and codebases wishing to become part of the efforts at The Apache Software Foundation. All code donations from external organizations and existing external projects wishing to join the ASF enter through the Incubator to: 1) ensure all donations are in accordance with the ASF legal standards; and 2) develop new communities that adhere to our guiding principles. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. For more information, visit http://incubator.apache.org/

About The Apache Software Foundation (ASF)
Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server --the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 620 individual Members and 6,000 Committers successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official user conference, trainings, and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Alibaba Cloud Computing, ARM, Bloomberg, Budget Direct, Capital One, Cash Store, Cerner, Cloudera, Comcast, Confluent, Facebook, Google, Hortonworks, HP, Huawei, IBM, InMotion Hosting, iSigma, LeaseWeb, Microsoft, ODPi, PhoenixNAP, Pivotal, Private Internet Access, Produban, Red Hat, Serenata Flowers, Target, WANdisco, and Yahoo. For more information, visit https://www.apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Fluo", "Apache Fluo", "Accumulo", "Apache Accumulo", "Rya", "Apache Rya", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 16 August. Board calendar and minutes http://apache.org/foundation/board/calendar.html
 - ASF Annual Report for 2017 Fiscal Year https://s.apache.org/IDn5

ASF Infrastructure –our distributed team on three continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield shipshape performance at 94.89% uptime http://status.apache.org/

Apache Geode™ –Big Data management platform that provides a database-like consistency model, reliable transaction processing and a shared-nothing architecture to maintain very low latency performance with high concurrency processing.
 - Apache Geode 1.2.0 released http://geode.apache.org/

Apache Hive™ –Big Data warehouse software that facilitates querying and managing large datasets residing in distributed storage.
 - Apache Hive 2.3.0 released https://hive.apache.org/

Apache Jackrabbit™ Oak –a scalable, high-performance hierarchical content repository designed for use as the foundation of modern world-class Web sites and other demanding content applications.
 - Apache Jackrabbit 2.4.8 and Jackrabbit Oak 1.6.3 and 1.4.17 released https://jackrabbit.apache.org/

Apache Libcloud™ –a Python library that abstracts away the differences among multiple cloud provider APIs.
 - Apache Libcloud 2.1.0 released http://libcloud.apache.org/

Apache PDFBox™ –an Open Source Java tool for working with PDF documents.
 - Apache PDFBox 2.0.7 released http://pdfbox.apache.org/

Did You Know?

 - Did you know that 20% of all Apache lines of code are comments (nearly 3x the entire Linux codebase)? https://projects.apache.org/statistics.html

 - Did you know that the Apache Directory Fortress access control system was designed for Java runtime environments with millions of users and thousands of requests per second? http://directory.apache.org/

 - Did you know that the Apache OpenWebBeans Meecrowave microprofile server contains all you need to run a JavaEE-based microservice from the command line, in only 9 MB? http://openwebbeans.apache.org/

Apache Community Notices:

 - "Success at Apache" focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be

 - If you're looking for the ASF on social media, we're @TheASF on Twitter and on LinkedIn at https://www.linkedin.com/company/the-apache-software-foundation (follows/re-tweets/shares/likes appreciated!)

 - Presentations from ApacheCon https://s.apache.org/Hli7 and Apache: Big Data https://s.apache.org/tefE are available; as well as videos https://s.apache.org/AE3m and audio recordings https://feathercast.apache.org/

 - Check out the Apache Community Development blog https://blogs.apache.org/comdev/

 - Do friend and follow us on the Apache Community Facebook page https://www.facebook.com/ApacheSoftwareFoundation/ and Twitter account https://twitter.com/ApacheCommunity

 - The list of Apache project-related MeetUps can be found at http://apache.org/events/meetups.html

 - The CloudStack European User Group will be held 17 August in London https://www.eventbrite.co.uk/e/cloudstack-european-user-group-tickets-35565783215

 - Catch the Apache Ignite and Spark communities at the In-Memory Computing Summit 24-25 October in San Francisco https://imcsummit.org/

 - ASF Annual Report https://s.apache.org/FY2017AnnualReport

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 19 July 2017. Board calendar and minutes http://apache.org/foundation/board/calendar.html
 - ASF Annual Report for 2017 Fiscal Year https://s.apache.org/IDn5

ASF Infrastructure –our distributed team on four continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield great performance at 97.03% uptime http://status.apache.org/

Apache Jackrabbit™ –a fully conforming implementation of the Content Repository for Java Technology API (JCR).
 - Apache Jackrabbit 2.14.2 released https://jackrabbit.apache.org/

Apache HAWQ (incubating) –an advanced enterprise SQL-on-Hadoop analytic engine.
 - Apache HAWQ 2.2.0.0-incubating released http://hawq.apache.org

Apache HTTP Server™ –the world's most popular Web server.
 - Apache HTTP Server 2.2.34 and 2.4.27 2.2.34 released http://httpd.apache.org/
 - CVE-2017-9789: Apache httpd 2.4 Read after free in mod_http2 http://mail-archives.apache.org/mod_mbox/www-announce/201707.mbox/%3CCACsi2528RzhCmN78sbpUYNYhLvsMkJmaB1hGV%2BzxaU64fjBL2g%40mail.gmail.com%3E
 - CVE-2017-9788: Uninitialized memory reflection in mod_auth_digest http://mail-archives.apache.org/mod_mbox/www-announce/201707.mbox/%3CCACsi252%3DWGN3zy9qsErsuvy5p-%2BL9p%2BDMUoZkFvyg4zH25N%2BkA%40mail.gmail.com%3E

Apache Impala (Incubating) –a high-performance C++ and Java SQL query engine for data stored in Apache Hadoop-based clusters.
 - CVE-2017-5652 Apache Impala (incubating) Information Disclosure http://mail-archives.apache.org/mod_mbox/www-announce/201707.mbox/%3CCA%2BLM4Mt5Nk_qJom7YhKaZrLQArqWWY%3DofUDK7h%2BMM7mmOVhp_w%40mail.gmail.com%3E
 - CVE-2017-5640 Apache Impala (incubating) Information Disclosure http://mail-archives.apache.org/mod_mbox/www-announce/201707.mbox/%3CCA%2BLM4Mur2%2Bmdv_YukGG0s5OWrj6xdZyLZbPRaB0g6Mm_WH%2BN9g%40mail.gmail.com%3E

Apache Lucene™ Solr –an Open Source enterprise search server based on the Lucene Java search library.
 - CVE-2017-7660: Security Vulnerability in secure inter-node communication in Apache Solr http://mail-archives.apache.org/mod_mbox/www-announce/201707.mbox/%3CCAOOKt53EgrybaD%2BiSn-nBbvFdse-szhg%3DhMoDZuvUvyMme-Z%3Dg%40mail.gmail.com%3E

Apache ODE™ –a WS-BPEL 1.1/2.0 compliant Web services orchestration engine.
 - Apache ODE 1.3.7 released http://ode.apache.org/

Apache Olingo™ –a Java library that enables developers to implement OData producers and OData consumers.
 - Apache Olingo 2.0.9 released http://olingo.apache.org/

Apache OpenMeetings™ –provides video conferencing, instant messaging, white board, collaborative document editing and other groupware tools.
 - Apache OpenMeetings 3.3.0 released http://openmeetings.apache.org

Apache OpenNLP™ –a machine learning based toolkit for the processing of natural language text.
 - Apache OpenNLP 1.8.1 released http://opennlp.apache.org/

Apache OpenWebBeans™ –a CDI container (Contexts and Dependency Injection for Java) and targets the CDI-1.2 specification (JavaEE 7).
 - Apache OpenWebBeans-1.7.4 released http://openwebbeans.apache.org

Apache Phoenix™ –enables SQL-based OLTP and operational analytics for Apache Hadoop.
 - Apache Phoenix 4.11 released http://phoenix.apache.org/

Apache Struts™ –a comprehensive and modular tooling stack for creating Web-based Java applications.
 - Apache Struts 2: possible RCE in the Struts Showcase app in the Struts 1 plugin example in the Struts 2.3.x series http://mail-archives.apache.org/mod_mbox/www-announce/201707.mbox/%3CCAMopvkPHCrVRX7TuyyzdHp2wv6Pc8-f4%3DsqmSxPiE04md7SMug%40mail.gmail.com%3E
 - Apache Struts 2.5.12 GA with Security Fixes released http://struts.apache.org/

Apache Subversion™ –an Open Source, centralized version control system.
 - Apache Subversion 1.8.18 released http://subversion.apache.org/

Apache Tika™ –a toolkit for detecting and extracting metadata and structured text content from various documents using existing parser libraries.
 - Apache Tika 1.16 released http://tika.apache.org/

Apache Wicket™ –an Open Source Java component oriented Web application framework.
 - Apache Wicket 7.8.0 released http://wicket.apache.org/

Apache Yetus™ –a collection of libraries and tools that enable contribution and release processes for software projects. - Apache Yetus 0.5.0 released https://yetus.apache.org/

Did You Know?

 - Did you know that if your employer has a matching gift program, your contribution to the ASF can be generously increased and will help even more to support its mission? http://apache.org/foundation/contributing.html

 - Did you know that the Apache Incubator site had a redesign/facelift? http://incubator.apache.org/

 - Did you know that the value to users from Apache OpenOffice's 200+M downloads (over the past year) exceeds $25M per day? https://openoffice.apache.org/

Apache Community Notices:

 - "Success at Apache" focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be

 - Presentations from ApacheCon https://s.apache.org/Hli7 and Apache: Big Data https://s.apache.org/tefE are available; as well as videos https://s.apache.org/AE3m and audio recordings https://feathercast.apache.org/

 - Check out the Apache Community Development blog https://blogs.apache.org/comdev/

 - Do friend and follow us on the Apache Community Facebook page https://www.facebook.com/ApacheSoftwareFoundation/ and Twitter account https://twitter.com/ApacheCommunity

 - Apache ActiveMQ Call For Logo https://blogs.apache.org/activemq/entry/apache-activemq-call-for-logo

 - The list of Apache project-related MeetUps can be found at http://apache.org/events/meetups.html

 - The CloudStack European User Group will be held 17 August in London https://www.eventbrite.co.uk/e/cloudstack-european-user-group-tickets-35565783215

 - Catch the Apache Ignite and Spark communities at the In-Memory Computing Summit 24-25 October in San Francisco https://imcsummit.org/

 - ASF Annual Report https://s.apache.org/FY2017AnnualReport

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 19 July 2017. Board calendar and minutes http://apache.org/foundation/board/calendar.html
 - ASF Annual Report for 2017 Fiscal Year https://s.apache.org/IDn5

ASF Infrastructure –our distributed team on four continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield stonking performance at 96.68% uptime http://status.apache.org/

Apache Jackrabbit™ –a fully conforming implementation of the Content Repository for Java Technology API (JCR).
 - Apache Jackrabbit 2.10.6 and Jackrabbit Oak 1.7.3 released https://jackrabbit.apache.org/

Apache jclouds™ –Open Source multi-Cloud toolkit.
 - Apache jclouds 2.0.2 released https://jclouds.apache.org/

Apache Juneau (incubating) –a toolkit for marshalling POJOs to a wide variety of content types using a common framework, and for creating sophisticated self-documenting REST interfaces and microservices using very little code.
 - Apache Juneau 6.3.0 (incubating) released http://juneau.incubator.apache.org/

Apache OFBiz™ –an Open Source product for the automation of enterprise processes that includes framework components and business applications.
 - Apache OFBiz 16.11.03 released http://ofbiz.apache.org/

Apache POI™ –library for reading and writing Microsoft Office file formats, such as Excel, PowerPoint, Word, Visio, Publisher and Outlook.
 - Apache POI 3.17-beta1 released https://poi.apache.org/

Apache Syncope™ –an Open Source system for managing digital identities in enterprise environments.
 - Apache Syncope 2.0.4 released http://syncope.apache.org/

Apache Subversion™ –a simple, reliable Open Source centralized version control system.
 - Apache Subversion 1.9.6 released http://subversion.apache.org/

Apache Tomcat™ –an Open Source software implementation of the Java Servlet, JavaServer Pages, Java Unified Expression Language, Java WebSocket and Java Authentication Service Provider Interface for Containers technologies.
 - Apache Tomcat 7.0.79 and 8.0.45 released http://tomcat.apache.org/
 - End of life for Apache Tomcat 8.0.x announced http://mail-archives.apache.org/mod_mbox/www-announce/201706.mbox/%3C19f2ddf7-aa64-5e56-7b97-a464aa319162%40apache.org%3E

Apache Wicket™ –an Open Source Java component oriented Web application framework.
 - Apache Wicket 6.27.0 released http://wicket.apache.org/

Did You Know?

 - Did you know that recordings from ApacheCon are available on Feathercast? https://feathercast.apache.org/

 - Did you know that Apache Syncope (enterprise identity management) has been downloaded 4K/month in 2017 thus far? http://syncope.apache.org/

 - Did you know that Apache Drill (Big Data SQL query engine) is well suited for data exploration? http://drill.apache.org/

Apache Community Notices:

 - "Success at Apache" focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be

 - Presentations from ApacheCon https://s.apache.org/Hli7 and Apache: Big Data https://s.apache.org/tefE are available; as well as videos https://s.apache.org/AE3m and audio recordings https://feathercast.apache.org/

 - Check out the Apache Community Development blog https://blogs.apache.org/comdev/

 - Do friend and follow us on the Apache Community Facebook page https://www.facebook.com/ApacheSoftwareFoundation/ and Twitter account https://twitter.com/ApacheCommunity

 - Apache ActiveMQ Call For Logo https://blogs.apache.org/activemq/entry/apache-activemq-call-for-logo

 - The list of Apache project-related MeetUps can be found at http://apache.org/events/meetups.html

 - The CloudStack European User Group will be held 17 August in London https://www.eventbrite.co.uk/e/cloudstack-european-user-group-tickets-35565783215

 - Catch the Apache Ignite and Spark communities at the In-Memory Computing Summit 24-25 October in San Francisco https://imcsummit.org/

 - ASF Annual Report https://s.apache.org/FY2017AnnualReport

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 19 July 2017. Board calendar and minutes http://apache.org/foundation/board/calendar.html
 - The Apache® Software Foundation Announces Annual Report for 2017 Fiscal Year https://s.apache.org/IDn5

ASF Infrastructure –our distributed team on four continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield steady performance at 96.13% uptime http://status.apache.org/

Apache Allura™ –an Open Source implementation of a software forge, a Web site that manages source code repositories, bug reports, discussions, wiki pages, blogs, and more for any number of individual projects.
 - Apache Allura 1.7.0 released https://allura.apache.org/

Apache Calcite™ –a dynamic data management framework.
 - Apache Calcite 1.13.0 released http://calcite.apache.org/

Apache Directory™ Fortress –provides Role-Based Access Control components that are fully ANSI INCITS 359 compliant and production ready.
 - Apache Fortress 2.0.0 released http://directory.apache.org/fortress/

Apache Fluo (incubating) – a distributed system for incrementally processing large data sets stored in Apache Accumulo.
 - Apache Fluo Recipes 1.1.0-incubating released https://fluo.apache.org/

Apache Groovy™ –a multi-faceted programming language for the JVM.
 - Apache Groovy 2.4.12 released https://groovy.apache.org/

Apache Ignite™ –an In-Memory Data Fabric providing in-memory data caching, partitioning, processing, and querying components.
 - [CVE-2017-7686] Apache Ignite Information Disclosure http://mail-archives.apache.org/mod_mbox/www-announce/201706.mbox/%3CC6768663-E7C6-48CC-A480-400218F23486%40apache.org%3E

Apache Jackrabbit™ –a fully conforming implementation of the Content Repository for Java Technology API (JCR).
 - Apache Jackrabbit 2.15.4 released https://jackrabbit.apache.org/

Apache Kafka™ –a distributed, fault tolerant, publish-subscribe messaging.
 - Apache Kafka 0.11.0.0 released http://kafka.apache.org/

Apache MyFaces™ –a JavaServer(tm) Faces 2.3 implementation as specified by JSR-372.
 - Apache MyFaces Core v2.3.0-beta released http://myfaces.apache.org/

Apache Tomcat™ –an Open Source software implementation of the Java Servlet, JavaServer Pages, Java Unified Expression Language, Java WebSocket and Java Authentication Service Provider Interface for Containers technologies.
 - Apache Tomcat 8.5.16 and 9.0.0.M22 released http://tomcat.apache.org/

Did You Know?

 - Did you know that Apache Kafka v0.11 has support for exactly-once guarantees? http://kafka.apache.org/

 - Did you know that Apache Drill v1.10 has improved fault tolerance? http://drill.apache.org

 - Did you know that Taiwan High Speed Rail Online Booking uses Apache Wicket? http://wicket.apache.org

Apache Community Notices:

 - "Success at Apache" focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be

 - Presentations from ApacheCon https://s.apache.org/Hli7 and Apache: Big Data https://s.apache.org/tefE are available; as well as videos https://s.apache.org/AE3m and audio recordings https://feathercast.apache.org/

 - Check out the Apache Community Development blog https://blogs.apache.org/comdev/

 - Do friend and follow us on the Apache Community Facebook page https://www.facebook.com/ApacheSoftwareFoundation/ and Twitter account https://twitter.com/ApacheCommunity

 - Apache ActiveMQ Call For Logo https://blogs.apache.org/activemq/entry/apache-activemq-call-for-logo

 - The list of Apache project-related MeetUps can be found at http://apache.org/events/meetups.html

 - The CloudStack European User Group will be held 17 August in London https://www.eventbrite.co.uk/e/cloudstack-european-user-group-tickets-35565783215

 - Catch the Apache Ignite and Spark communities at the In-Memory Computing Summit 24-25 October in San Francisco https://imcsummit.org/

 - ASF Annual Report https://s.apache.org/FY2017AnnualReport

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

Forest Hill, MD —29 June 2017— The Apache® Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today the availability of the annual report for its 2017 fiscal year, which ended 30 April 2017. 

Now in its 18th year, the ASF's operational highlights include:

1. 35M page views per week across apache.org;
2. Web requests received from every Internet-connected country on the planet;
3. ASF's 100M functional lines of code (out of 150M+ total) have been developed over 65,000 person years, valued at US$7B;
4. 65+M lines of code committed over the past year;
5. 20% of all Apache lines of code are comments (nearly 3x the entire Linux codebase);
6. 3,300 Apache code Committers made 214,398 commits;
7. More than half of all contributions made by individuals new to Apache;
8. Nearly 300 new code contributors and 300-400 new people filing issues each month;
9. 25,154 authors sent 2,105,992 emails on 834,045 topics over the past year;
10. Approximately 9M source code downloads served from Apache mirrors on a yearly basis (excluding convenience binaries);
11. 64 new individual ASF Members elected, bringing the total to 684;
12. Exceeded 6,000 code Committers;
13. Remains all-volunteer for all code/development-related activities;
14. 182 Top-Level communities overseeing 300+ Apache projects and sub-projects;
15. Dozens of Apache projects continue to dominate the enterprise Big Data ecosystem
16. Record 64 "podlings" undergoing incubation during FY2017 (59 at end of FY2017);
17. New innovations include IoT, Microfinance, Machine Learning, and Cryptography;
18. 22nd anniversary of the Apache HTTP Server (18 years under the ASF umbrella);
19. Apache OpenOfficeTM exceeded 200M downloads (value to users $25M+ per day);
20. Apache GroovyTM downloaded 12M times during the first 4 months of 2017;
21. 976 Individual Contributor License Agreements (CLAs) signed;
22. 42 Corporate Contributor License Agreements signed (totalling 384);
23. 30 Software Grant Agreements signed;
24. Apache License remains one of the most popular Open Source licenses;
25. Apache Infrastructure services running 24x7x365 at near 100% uptime on an annual budget of less than US$5,000 per project;
26. New "Gitbox" service launched to allow communities to host their read/write Git repositories on GitHub;
27. Continued reduction in Infrastructure costs by moving select services to the Cloud;
28. Improved buildbot and Jenkins build farms for continuous integration, testing, and automated Website generation;
29. Increased trademarks, brand management, and legal support for dozens of projects;
30. Launched new Apache Community survey, newsletter, and social media resources;
31. ASF serves as a mentoring organization in Google Summer of Code for 12th consecutive year;
32. Participated in hundreds of events globally, including ApacheCon North America and Europe;
33. Increased corporate backing, with 51 ASF Sponsors and 11 Infrastructure partners;
34. Improved individual giving program and outreach tactics;
35. Completed first-time budgetary planning with five-year projections;
36. FY17 ended with a 15.3-month cash reserve (more than double the industry average);
37. ASF operational areas (Brand Management, Fundraising, Marketing and Publicity, Infrastructure, Conferences, and Travel Assistance) now supported by professional staff overseen by appointed ASF Members;
38. FY17 tapped as an investment year, with resources committed to core Infrastructure and Marketing & Publicity services.

The full report is available online at https://s.apache.org/FY2017AnnualReport

Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server --the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 680 individual Members and 6,000 Committers successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official user conference, trainings, and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Alibaba Cloud Computing, ARM, Bloomberg, Budget Direct, Capital One, Cash Store, Cerner, Cloudera, Comcast, Confluent, Facebook, Google, Hortonworks, HP, Huawei, IBM, InMotion Hosting, iSigma, LeaseWeb, Microsoft, ODPi, PhoenixNAP, Pivotal, Private Internet Access, Produban, Red Hat, Serenata Flowers, Target, WANdisco, and Yahoo. For more information, visit https://www.apache.org/ and https://twitter.com/TheASF

# # #

© The Apache Software Foundation. "Apache", "Apache Groovy", "Groovy", "Apache HTTP Server", "Apache OpenOffice", "OpenOffice", and "ApacheCon", are registered trademarks or trademarks of The Apache Software Foundation. All other brands and trademarks are the property of their respective owners.

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 19 July 2017. Board calendar and minutes http://apache.org/foundation/board/calendar.html

ASF Infrastructure –our distributed team on four continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield cheery performance at 97.28% uptime http://status.apache.org/

Apache Incubator –the entry path for codebases and their communities to become an official part of the ASF.
 - Welcome New Podlings: Livy, Pulsar, and Superset http://incubator.apache.org/

Apache HTTP Server™ –the world's most popular Web server.
 - Apache HTTP Server 2.4.26 released http://httpd.apache.org/

Apache HBase™ –an Open Source, distributed, versioned, non-relational database.
 - Apache HBase 1.1.11 released https://hbase.apache.org/

Apache Impala (incubating) –a high-performance C++ and Java SQL query engine for data stored in Apache Hadoop-based clusters.
 - Apache Impala 2.9.0 released https://impala.incubator.apache.org/

Apache Jackrabbit™ Oak –a scalable, high-performance hierarchical content repository designed for use as the foundation of modern world-class Web sites and other demanding content applications.
 - Apache Jackrabbit Oak 1.7.2 released https://jackrabbit.apache.org/

Apache Joshua (incubating) –a statistical machine translation decoder for phrase-based, hierarchical, and syntax-based machine translation, written in Java.
 - Apache Joshua 6.1 released https://joshua.apache.org

Apache Lucene™ Solr™ –the search server built on Apache Lucene.
 - Apache Solr Reference Guide for 6.6 released https://www.apache.org/dyn/closer.cgi/lucene/solr/ref-guide/apache-solr-ref-guide-6.6.pdf

Apache Mnemonic (incubating) –an advanced hybrid memory storage oriented library.
 - Apache Mnemonic-0.8.0-incubating released http://mnemonic.incubator.apache.org/

Apache Pig™ –provides a high-level data-flow language and execution framework for parallel computation on Apache Hadoop clusters.
 - Apache Pig 0.17.0 released http://pig.apache.org/

Apache RocketMQ (incubating) –a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.
 - Apache RocketMQ 4.1.0 released https://rocketmq.apache.org/

Did You Know?

 - Did you know that Apache Mahout has both Apache Flink and Apache Spark GPU compute? http://mahout.apache.org/

 - Did you know that Apache Tinkerpop's Gremlin is not just a graph traversal language, but a graph in itself? http://tinkerpop.apache.org/

 - Did you know that Xiaomi uses Apache HBase and Thrift across 4 data centers and 40+ clusters? http://hbase.apache.org/ http://thrift.apache.org/

Apache Community Notices:

 - "Success at Apache" focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be

 - Presentations from ApacheCon https://s.apache.org/Hli7 and Apache: Big Data https://s.apache.org/tefE are available; as well as videos https://s.apache.org/AE3m and audio recordings https://feathercast.apache.org/

 - Check out the Apache Community Development blog https://blogs.apache.org/comdev/

 - Do friend and follow us on the Apache Community Facebook page https://www.facebook.com/ApacheSoftwareFoundation/ and Twitter account https://twitter.com/ApacheCommunity

 - Apache ActiveMQ Call For Logo https://blogs.apache.org/activemq/entry/apache-activemq-call-for-logo

 - The list of Apache project-related MeetUps can be found at http://apache.org/events/meetups.html

 - The CloudStack European User Group will be held 17 August in London https://www.eventbrite.co.uk/e/cloudstack-european-user-group-tickets-35565783215

 - Catch the Apache Ignite and Spark communities at the In-Memory Computing Summit 24-25 October in San Francisco https://imcsummit.org/

 - ASF Operations Summary - Q3 FY2017 https://s.apache.org/NKFz

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 21 June 2017. Board calendar and minutes http://apache.org/foundation/board/calendar.html

ApacheCon™ –the official conference of the Apache Software Foundation. Tomorrow's Technology Today.
 - Presentations from ApacheCon https://s.apache.org/Hli7 and Apache: Big Data https://s.apache.org/tefE
 - Videos of keynotes + presentations https://s.apache.org/AE3m and Audio recordings https://feathercast.apache.org/

ASF Infrastructure –our distributed team on four continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield savvy performance at 98.54% uptime http://status.apache.org/

Apache Incubator –the entry path for codebases and their communities to become an official part of the ASF.
 - Welcome New Podlings: Livy, Pulsar, and Superset http://incubator.apache.org/

Apache Ant™ Compress –offers tasks and types for archive and compression formats.
 - Apache Compress Antlib 1.5 released http://ant.apache.org/antlibs/compress/

Apache Arrow™ –a columnar in-memory analytics layer designed to accelerate Big Data.
 - Apache Arrow 0.4.1 released http://arrow.apache.org/

Apache Commons™ FileUpload –parses HTTP requests which conform to RFC 1867, "Form-based File Upload in HTML".
 - Apache Commons FileUpload 1.3.3 released http://commons.apache.org/proper/commons-fileupload/

Apache Commons™ Lang –provides helper utilities for the java.lang API, notably String manipulation methods, basic numerical methods, object reflection, concurrency, creation and serialization and System properties.
 - Apache Commons Lang 3.6 released http://www.apache.org/dist/commons/lang/

Apache Directory™ DS –an extensible and embeddable directory server entirely written in Java, which has been certified LDAPv3 compatible by the Open Group.
 - ApacheDS 2.0.0-M24 released http://directory.apache.org/apacheds

Apache Fluo (incubating) –adds distributed transactions to Apache Accumulo, and provides an observer/notification framework to allow users to incrementally update large data sets stored in Accumulo.
 - Apache Fluo 1.1.0-incubating released https://fluo.apache.org/

Apache Groovy™ –a multi-facet programming language for the JVM.
 - Apache Groovy-2.5.0-beta-1 released https://groovy.apache.org/

Apache Jackrabbit™ –a fully compliant implementation of the Content Repository for Java(TM) Technology API, version 2.0 (JCR 2.0) as specified in the Java Specification Request 283 (JSR 283).
 - Apache Jackrabbit 2.15.3 and Jackrabbit Oak 1.6.2 and 1.7.1 released https://jackrabbit.apache.org/

Apache Kudu™ –an Open Source storage engine for structured data that supports low-latency random access together with efficient analytical access patterns.
 - Apache Kudu 1.4.0 released http://kudu.apache.org/

Apache NiFi™ –an easy to use, powerful, and reliable system to process and distribute data.
 - Apache NiFi 0.7.4 and 1.3.0 released https://nifi.apache.org/
 - Apache NiFi CVE-2017-7667 and CVE-2017-7665 http://mail-archives.apache.org/mod_mbox/www-announce/201706.mbox/%3CCAFddr25eFkXCOQGwyN4B4VVNjdVYLcKya_JCaW%3Dd%3D11%3DQkyd4g%40mail.gmail.com%3E

Apache Portable Runtime™ –provides cross-platform APIs that relieve developers of the need to deal with platform differences.
 - Apache Portable Runtime and Utilities 1.6 released https://apr.apache.org/

Apache Sling™ –a Web framework that uses a Java Content Repository, such as Apache Jackrabbit, to store and manage content. 
 - Apache Sling 9 released https://sling.apache.org/

Apache Zeppelin™ –a collaborative data analytics and visualization tool for distributed, general-purpose data processing system such as Apache Spark, Apache Flink, etc.
 - Apache Zeppelin 0.7.2 released http://zeppelin.apache.org/

Did You Know?

 - Did you know that the Symphony communications and messaging platform uses Apache Cassandra, HBase, Kafka, Solr, Tomcat, and the Apache License? http://cassandra.apache.org/ http://hbase.apache.org/ http://kafka.apache.org/ http://lucene.apache.org/solr http://tomcat.apache.org/ https://www.apache.org/licenses/LICENSE-2.0

 - Did you know that the Apache Drill distributed SQL engine brings flexibility and agility to data lakes across the Hadoop ecosystem? http://drill.apache.org/

 - Did you know that we have 10 project birthdays this month? Happy Apache Anniversary to SpamAssassin (13 yrs); Santuario (12 yrs); Commons and Wicket (10 yrs); Sling (8 yrs); Karaf (7 yrs); Flume and VCL (5 yrs); Mesos (4 yrs); and Twill (1 year) --many happy returns to all! https://projects.apache.org/

Apache Community Notices:

 - "Success at Apache" focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be

 - Check out the Apache Community Development blog https://blogs.apache.org/comdev/

 - Do friend and follow us on the Apache Community Facebook page https://www.facebook.com/ApacheSoftwareFoundation/ and Twitter account https://twitter.com/ApacheCommunity

 - Apache ActiveMQ Call For Logo https://blogs.apache.org/activemq/entry/apache-activemq-call-for-logo

 - Catch the Apache Ignite and Spark communities at the In-Memory Computing Summit 20-21 June in Amsterdam and 24-25 October in San Francisco https://imcsummit.org/

 - ASF Operations Summary - Q3 FY2017 https://s.apache.org/NKFz

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 21 June 2017. Board calendar and minutes http://apache.org/foundation/board/calendar.html

Success at Apache –monthly blog series that focuses on the processes behind why the ASF "just works".
 - Learning to Build a Stronger Community by John Ament https://s.apache.org/x9Be

ApacheCon™ –the official conference of the Apache Software Foundation. Tomorrow's Technology Today.
 - Presentations from ApacheCon https://s.apache.org/Hli7 and Apache: Big Data https://s.apache.org/tefE
 - Videos of keynotes + presentations https://s.apache.org/AE3m and Audio recordings + soundbites from the conference floor https://feathercast.apache.org/

ASF Infrastructure –our distributed team on four continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield skipping performance at 99.96% uptime http://status.apache.org/

Apache Directory™ LDAP API –an ongoing effort to provide an enhanced LDAP API, as a replacement for JNDI and the existing LDAP API (jLdap and Mozilla LDAP API).
 - Apache Directory LDAP API 1.0.0 released http://directory.apache.org/api

Apache Groovy™ –a multi-facet programming language for the JVM.
 - Apache Groovy-2.5.0-beta-1 released https://groovy.apache.org/

Apache Hadoop™ –the cornerstone of the Big Data ecosystem, from which dozens of Apache Big Data projects and countless industry solutions originate.
 - The Apache Software Foundation Announces Momentum With Apache® Hadoop® v2.8 https://s.apache.org/h0Tl

Apache HBase™ –an Open Source, distributed, versioned, non-relational database.
 - Apache HBase 1.2.6 released https://hbase.apache.org/

Apache Jackrabbit™ Oak –a scalable, high-performance hierarchical content repository designed for use as the foundation of modern world-class Web sites and other demanding content applications.
 - Apache Jackrabbit Oak 1.2.26 and 1.4.16 released http://jackrabbit.apache.org/

Apache Lucene™ –a high-performance, full-featured text search engine library written entirely in Java.
 - Apache Lucene 6.6.0 and Apache Solr 6.6.0 released http://lucene.apache.org/

Apache Tomcat™ –an Open Source software implementation of the Java Servlet, JavaServer Pages, Java Unified Expression Language, Java WebSocket and JASPIC technologies.
 - CVE-2017-5664 Apache Tomcat Security Constraint Bypass http://mail-archives.apache.org/mod_mbox/www-announce/201706.mbox/%3C3abc830a-69e1-9ce1-27c8-1eaf9c2d6739%40apache.org%3E

Did You Know?

 - Did you know that if it's not at *.apache.org, it's not from us? https://s.apache.org/QviH

 - Did you know that Apache CouchDB was one of the first Apache projects to use git? https://blog.couchdb.org/2017/06/06/couchdb-developer-profile-joan-touzet/

 - Did you know that nearly half of businesses' security breaches are due to the Internet Of Things? Apache Spot (incubating) can help! http://spot.incubator.apache.org/

Apache Community Notices:

 - "Success at Apache" focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be

 - The latest Apache Community Newsletter https://blogs.apache.org/comdev/entry/community-development-news-april-2017

 - Do friend and follow us on the Apache Community Facebook page https://www.facebook.com/ApacheSoftwareFoundation/ and Twitter account https://twitter.com/ApacheCommunity

 - Apache ActiveMQ Call For Logo https://blogs.apache.org/activemq/entry/apache-activemq-call-for-logo

 - Join members of the Apache Apex, Beam, Flink, Hadoop, Kafka, Lucene, Solr, and Spark communities at Berlin Buzzwords 11-13 June in Berlin https://berlinbuzzwords.de/17/

 - The Apache Phoenix community will be holding PhoenixCon on 13 June in San Francisco https://www.eventbrite.com/e/phoenixcon-2017-tickets-32872245772

 - Will we be seeing you at HBaseCon 12 June/Mountain View https://www.eventbrite.com/e/hbasecon-west-2017-tickets-33101238696 and PhoenixCon 13 June/San Francisco https://www.eventbrite.com/e/phoenixcon-2017-tickets-32872245772 ?

 - Meet members of Apache's Cloud community at Cloud Foundry Summit Silicon Valley 13-15 June in Santa Clara; enjoy 20% off registration rates using discount code CFSV17ASF20 https://goo.gl/Uq3g0t

 - Catch the Apache Ignite and Spark communities at the In-Memory Computing Summit 20-21 June in Amsterdam and 24-25 October in San Francisco https://imcsummit.org/

 - ASF Operations Summary - Q3 FY2017 https://s.apache.org/NKFz

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

by John Ament

As the next line in the series of "Success at Apache", I had to think about what kind of blog post I wanted to write.  Given my personal focus, it made sense to focus on new projects coming in and the incubator.  When I'm not busy dreaming up new ideas and working on personal projects, I'm helping new projects get in to Apache, keeping their goals in alignment with the Apache Way http://apache.org/foundation/governance/ . I'm a member of a few different PMCs here at Apache, notably the Incubator. I'm a mentor to five different podlings right now. While my primary programming focus is on programming models, my podlings are all over the place. Starting a new project here at Apache can be a daunting task: how do I get in? What if I don't build a diverse community?  Becoming a podling has more to do with the community than it does the technical aspects of the project. We don't expect you to be experts in it, but we do expect new projects to be experts in how their own software works. We want to teach you, and we want you to be receptive to learning about The Apache Software Foundation and its best practices.

I'm not sure if everyone does it, but I build a lot of parallels between how an ASF project works and how an Agile team works. Agile teams start off as a bunch of people who don't really know each other but have assembled themselves into an informal team focused on solving a problem, or some number of problems, knowing that they can only do it together. They have common goals and objectives, but lack camaraderie early on to be able to work together smoothly. Over time, they get to know one another, figure out strengths and weaknesses and can resolve issues together. A well-functioning team isn't one at the beginning. It takes time and practice for them to work well - both together and as an outwardly facing unit.

Projects here at Apache follow the same type of maturity progression. Whether it's learning The Apache Way or learning to work with one another, it takes them time to mature and get into a good groove. 

Open Communication
The ASF is pretty big on open communication, wherever it's a sensible solution. We want to discuss with each other what we're doing, ideas around how to solve it and come up with a good solution together, as a team, in an open manner.  

This all ties into agile practices. We host stand ups to talk about what we're doing and see if others have an opinion about what we're doing.

When a project comes to Apache, the original authors need to remember that they're bringing in a lot of experience, and the expectation is that those existing contributors must help get new contributors from the outside - outside their organization specifically - to contribute into the project. By driving towards open communication, outside of your own organization, you're encouraging more people to participate. This sort of governance model ensures that all parties who can participate are aware of decisions being made.

Open Communication isn't for everything though. We need to remember to be respectful in our communications with others and if it's felt that something’s awry - speak privately. But remember that isn't part of the decision making process. Likewise, anytime we're talking about individuals in either a positive or negative way that should be conducted on the private list for a project.

Turning Into a Well Oiled Machine
Once a project begins to grow, new people start to get attracted to it. As a community, you have to figure out how to work together. Building a community of diverse ideas and skills will ensure that new ideas keep flowing. Contributors can react quickly to a user's question on list and help them resolve the problem, put in an enhancement request or get a bug report squashed in a following commit. Time is of the essence right now because I have availability to work on this.

There can't be a long drawn out waterfall style process when dealing with Open Source. At the same time, making sure there's a documented decision process and in sometimes an in depth design is critical for both new contributors and existing alike to come to a shared understanding of what is being proposed.

Sustaining
Projects need to plan for longevity. Longevity comes in many forms. A strong backlog of features is important. Having a diverse set of committers is even more critical. You could even say that each helps create the other. Just like any feature set, we get to a point where the feature is complete enough that we can move on to another feature.  

How do you get there?
Apache's main way to go to these points is to incubate http://incubator.apache.org/ . You can't get to this point by yourselves, experiencing with first-hand from existing Foundation members will help get your community to turn a new leaf and adopt this way of working. We want you to be successful, as long as your project can dedicate itself to the practices that have been set forth within the Foundation.

New projects may be comfortable with a champion http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Champion that can work with them closely, answering their questions up front. While a lot of the pre-incubation chatter will happen off list, it is important that potential new podlings subscribe to the incubator general list http://incubator.apache.org/guides/lists.html#general+at+incubator.apache.org and understand both the goings on of a podling as well as try to build their list of mentors http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Mentor in the open. Mentors are extremely important to a podling, and understanding their roles and why you need to pick great mentors is something your champion and the rest of the Incubator community can help explain. Participating in our public discussion lists is sometimes the first step to joining the foundation at a deeper level.

Where do we go next?
If you're a potential new project, feel free to reach out on the Incubator mailing lists http://incubator.apache.org/guides/lists.html#general+at+incubator.apache.org to get started. We'd love to hear from you and get you acquainted with The Apache Software Foundation.

If you're on an existing project, we want to hear your perspectives on how the Foundation works. You may want to reach out to dev@community http://community.apache.org/lists.html to let others know your thoughts, or even just subscribe and see what others have to say. We're all working together to make the foundation better. The more input we receive, both the positive and the negative, will help shape everyone's actions in the community.

= = =

"Success at Apache" is a new monthly blog series that focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh

Major release of the cornerstone of the Big Data ecosystem, from which dozens of Apache Big Data projects and countless industry solutions originate.

Forest Hill, MD —5 June 2017— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today momentum with Apache® Hadoop® v2.8, the latest version of the Open Source software framework for reliable, scalable, distributed computing.

Now ten years old, Apache Hadoop dominates the greater Big Data ecosystem as the flagship project and community amongst the ASF's more than three dozen projects in the category.

"Apache Hadoop 2.8 maintains the project's momentum in its stable release series," said Chris Douglas, Vice President of Apache Hadoop. "Our community of users, operators, testers, and developers continue to evolve the thriving Big Data ecosystem at the ASF. We're committed to sustaining the scalable, reliable, and secure platform our greater Hadoop community has built over the last decade."

Apache Hadoop supports processing and storage of extremely large data sets in a distributed computing environment. The project has been regularly lauded by industry analysts worldwide for driving market transformation. Forrester Research estimates that firms will spend US$800M in Hadoop software and related services in 2017. According to Zion Market Research, the global Hadoop market is expected to reach approximately US$87.14B by 2022, growing at a CAGR of around 50% between 2017 and 2022.

Apache Hadoop 2.8 is the result of 2 years of extensive collaborative development from the global Apache Hadoop community. With 2,914 commits as new features, improvements and bug fixes since v2.7, highlights include:

• Several important security related enhancements, including Hadoop UI protection of Cross-Frame Scripting (XFS) which is an attack that combines malicious JavaScript with an iframe that loads a legitimate page in an effort to steal data from an unsuspecting user, and Hadoop REST API protection of Cross site request forgery (CSRF) attack which attempt to force an authenticated user to execute functionality without their knowledge.
  
  
• Support for Microsoft Azure Data Lake as a source and destination of data. This benefits anyone deploying Hadoop in Microsoft's Azure Cloud. The Azure Data Lake service was actually developed for Hadoop and analytics workloads.
  
  
• The "S3A" client for working with data stored in Amazon S3 has been radically enhanced for scalability, performance, and security. The performance enhancements were driven by Apache Hive and Apache Spark benchmarks. In Hive TCP-DS benchmarks, Apache Hadoop is currently faster working with columnar data stored in S3  than Amazon EMR's closed-source connector. This shows the benefit of collaborative Open Source development.
  
  
• Several WebHDFS related enhancements include integrated CSRF prevention filter in WebHDFS, support OAuth2 in WebHDFS, disallow/allow snapshots via WebHDFS, and more.
  
  
• Integration with other applications has been improved with a separate jar for the hdfs-client than the hadoop-hdfs JAR with all the server side code. Downstream projects that access HDFS can depend on the hadoop-hdfs-client module to reduce the amount of transitive classpath dependencies.
  
  
• YARN NodeManager Resource Reconfiguration through RM Admin CLI for a live cluster that allows YARN clusters to have a more flexible resource model especially for a Cloud deployment.

In addition to physical Hadoop clusters, where the majority of storage and computation lies, Apache Hadoop is very popular within Cloud infrastructures. Contributions from Apache Hadoop's diverse community includes improvements provided by Cloud infrastructure vendors and large Hadoop-in-Cloud users. These improvements include: Azure and S3 storage and YARN reconfiguration in particular, improve Hadoop's deployment on and integration with Cloud Infrastructures. The improvements in Hadoop 2.8 enable Cloud-deployed clusters to be more dynamic in sizing, adapting to demand by scaling up and down.

"My colleagues and I are happy that tests of Apache Hive and Hadoop 2.8 show that we are able to provide a similar experience reading data in from S3 as Amazon EMR, with its closed-source fork/rewrite of S3," said Steve Loughran, member of the Apache Hadoop Project Management Committee.

Hailed as a "Swiss army knife of the 21st century" by the Media Guardian Innovation Awards  and "the most important software you’ve never heard of…helped enable both Big Data and Cloud computing" by author Thomas Friedman, Apache Hadoop is used by an array of companies such as Alibaba, Amazon Web Services, AOL, Apple, eBay, Facebook, foursquare, IBM, HP, LinkedIn, Microsoft, Netflix, The New York Times, Rackspace, SAP,  Tencent, Teradata, Tesla Motors, Uber, and Twitter. Yahoo, an early pioneer, hosts the world's largest known Hadoop production environment to date, spanning more than 38,000 nodes.

Catch Apache Hadoop in action at DataWorks Summit 13-15 June 2017 in San Jose, CA.

Availability and Oversight
Apache Hadoop software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache Hadoop, visit http://hadoop.apache.org/ and https://twitter.com/hadoop

About The Apache Software Foundation (ASF)
Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server --the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 680 individual Members and 6,000 Committers successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official user conference, trainings, and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Alibaba Cloud Computing, ARM, Bloomberg, Budget Direct, Capital One, Cash Store, Cerner, Cloudera, Comcast, Confluent, Facebook, Google, Hortonworks, HP, Huawei, IBM, InMotion Hosting, iSigma, LeaseWeb, Microsoft, ODPi, PhoenixNAP, Pivotal, Private Internet Access, Produban, Red Hat, Serenata Flowers, Target, WANdisco, and Yahoo. For more information, visit http://www.apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Hadoop", "Apache Hadoop", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 21 June 2017. Board calendar and minutes http://apache.org/foundation/board/calendar.html

ApacheCon™ –the official conference of the Apache Software Foundation. Tomorrow's Technology Today.
 - Presentations from ApacheCon https://s.apache.org/Hli7 and Apache: Big Data https://s.apache.org/tefE
 - Videos of keynotes + presentations https://s.apache.org/AE3m and Audio recordings + soundbites from the conference floor https://feathercast.apache.org/

ASF Infrastructure –our distributed team on four continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield grand performance at 99.92% uptime http://status.apache.org/

Apache ActiveMQ™ –the most popular and powerful Open Source message broker.
 - Apache ActiveMQ Call For Logo https://blogs.apache.org/activemq/entry/apache-activemq-call-for-logo

Apache Calcite™ Avatica –a framework for building database drivers.
 - Apache Calcite Avatica 1.10.0 released https://calcite.apache.org/avatica/

Apache Fineract™ –an Open Source system for core banking as a platform to offer financial services to the world's 2B under-banked and unbanked.
 - Apache Fineract 1.0.0 released http://fineract.apache.org/

Apache Flink™ –an Open Source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications.
 - Apache Flink 1.3.0 released https://flink.apache.org/

Apache Jackrabbit™ –a fully compliant implementation of the Content Repository for Java(TM) Technology API, version 2.0 (JCR 2.0) as specified in the Java Specification Request 283 (JSR 283).
 - Apache Jackrabbit 2.14.1 and Jackrabbit Oak 1.7.0 released http://jackrabbit.apache.org/

Apache Knox™ –a free and Open Source Java framework for building semantic Web and Linked Data applications.
 - CVE-2017-5646: Apache Knox Impersonation Issue for WebHDFS http://mail-archives.apache.org/mod_mbox/www-announce/201705.mbox/%3CCACRbFyjtT7QQGHUzTRdbJoySbJb7tt4BDk5-r-VRn0GB0Kgvag%40mail.gmail.com%3E

Apache Tephra (incubating) –a transaction engine for distributed data stores like Apache HBase.
 - Apache Tephra-0.12.0-incubating released http://tephra.incubator.apache.org/

Apache Tika™ –a toolkit for detecting and extracting metadata and structured text content from various documents using existing parser libraries.
 - Apache Tika 1.15 released http://tika.apache.org/

Did You Know?

 - Did you know that the CFP for MesosCon North America closes on June 3rd? Submit your proposals to http://bit.ly/2rbVJhY

 - Did you know that ARGO Labs uses Apache Airflow (airflow) towards automating Extract, Transform, Load (ETL) workflows for California water data from The California Data Collaborative? http://incubator.apache.org/projects/airflow.html

 - Did you know Dutch multinational banking company Rabobank uses Apache Kafka for real-time financial alerts? http://kafka.apache.org/

Apache Community Notices:

 - "Success at Apache" is a blog series that focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh

 - The latest Apache Community Newsletter https://blogs.apache.org/comdev/entry/community-development-news-april-2017

 - Do friend and follow us on the Apache Community Facebook page https://www.facebook.com/ApacheSoftwareFoundation/ and Twitter account https://twitter.com/ApacheCommunity

 - The Apache Phoenix community will be holding PhoenixCon on 13 June in San Francisco https://www.eventbrite.com/e/phoenixcon-2017-tickets-32872245772

 - Join members of the Apache Apex, Beam, Flink, Hadoop, Kafka, Lucene, Solr, and Spark communities at Berlin Buzzwords 11-13 June in Berlin https://berlinbuzzwords.de/17/

 - Will be seeing you at HBaseCon 12 June/Mountain View https://www.eventbrite.com/e/hbasecon-west-2017-tickets-33101238696 and PhoenixCon 13 June/San Francisco https://www.eventbrite.com/e/phoenixcon-2017-tickets-32872245772 ?

 - Meet members of Apache's Cloud community at Cloud Foundry Summit Silicon Valley 13-15 June in Santa Clara; enjoy 20% off registration rates using discount code CFSV17ASF20 https://goo.gl/Uq3g0t

 - Catch the Apache Ignite and Spark communities at the In-Memory Computing Summit 20-21 June in Amsterdam and 24-25 October in San Francisco https://imcsummit.org/

 - ASF Operations Summary - Q3 FY2017 https://s.apache.org/NKFz

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

Open Source Big Data machine learning platform in use at Cadent Technology and IBM Watson Health, among other organizations.

Forest Hill, MD –31 May 2017– The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today that Apache® SystemML™ has graduated from the Apache Incubator to become a Top-Level Project (TLP), signifying that the project's community and products have been well-governed under the ASF's meritocratic process and principles.

Apache SystemML is a machine learning platform optimal for Big Data that provides declarative, large-scale machine learning and deep learning. SystemML can be run on top of Apache Spark, where it automatically scales data, line by line, to determine whether code should be run on the driver or an Apache Spark cluster.

"Today, the machine learning revolution is leading to thousands of life-altering innovations such as self-driving cars and computers that detect cancer," said Deron Eriksson, Vice President of Apache SystemML. "Apache SystemML enables and simplifies this process by executing optimized high-level algorithms on Big Data using proven technologies such as Apache Spark and Apache Hadoop MapReduce."

The core of Apache SystemML has been created from the ground up with the following design principles in mind: 

• Performance and Scalability, as SystemML scales up on single nodes, and scales out on large clusters using Apache Spark or Apache Hadoop;
• "Designed for data scientists", enabling data scientists to develop algorithms in a system with a strong foundation in linear algebra and statistical functions; and 
• Cost-based optimization for scalable execution plans, that significantly shortens and simplifies the development and deployment cycle of algorithms for varying data characteristics and system configurations.

Using Apache SystemML, data scientists are able to implement algorithms using high-level language concepts without knowledge of distributed programming. Depending on data characteristics such as data size/shape and data sparsity (dense/sparse), and cluster characteristics such as cluster size and memory configurations, SystemML's cost-based optimizing compiler automatically generates hybrid runtime execution plans that are composed of single-node and distributed operations on Apache Spark or Apache Hadoop clusters for best performance.

"SystemML allows Cadent to implement advanced numerical programming methods in Apache Spark, empowering us to leverage specialized algorithms in our predictive analysis software," said Michael Zargham, Chief Scientist at Cadent Technology.

"SystemML is like SQL for Machine Learning, it enables Data Scientists to concentrate on the problem at hand, working in a high-level script language like R, and all the optimizations and rewrites are handled by the very powerful SystemML optimizer that considers data and available resources to produce the best execution plan for the application," said Luciano Resende, Architect at the IBM Spark Technology Center and Apache SystemML Incubator Mentor.

"IBM Watson Health VBC is using Apache SystemML on Apache Spark to build risk models on a very large EHR data set to predict emergency department visits," said Steve Beier, Vice President of Value Based Care Platform and Analytics at IBM Watson Health. "The models identify high-risk patients so that they can be targeted with preemptive strategies, thus potentially reducing care costs while at the same time leading to optimal outcomes for patients."

SystemML originated at IBM Research - Almaden in 2010, and was submitted to the Apache Incubator in November 2015. SystemML initiated compressed linear algebra research, a differentiating feature in SystemML, which received the VLDB 2016 Best Paper.

"The Apache Incubator is all about open collaboration and communication and was invaluable for everyone involved in SystemML," added Eriksson. "The Apache SystemML community sincerely encourages everyone interested in machine learning and deep learning to help build our community around this revolutionary technology."

Catch Apache SystemML in action at the Big Data Developers Silicon Valley MeetUp on 8 June 2017 in San Francisco, CA.

Availability and Oversight
Apache SystemML software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache SystemML, visit http://systemml.apache.org/ and https://twitter.com/ApacheSystemML

About the Apache Incubator
The Apache Incubator is the entry path for projects and codebases wishing to become part of the efforts at The Apache Software Foundation. All code donations from external organizations and existing external projects wishing to join the ASF enter through the Incubator to: 1) ensure all donations are in accordance with the ASF legal standards; and 2) develop new communities that adhere to our guiding principles. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. For more information, visit http://incubator.apache.org/

About The Apache Software Foundation (ASF)
Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server --the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 620 individual Members and 6,000 Committers successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official user conference, trainings, and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Alibaba Cloud Computing, ARM, Bloomberg, Budget Direct, Capital One, Cash Store, Cerner, Cloudera, Comcast, Confluent, Facebook, Google, Hortonworks, HP, Huawei, IBM, InMotion Hosting, iSigma, LeaseWeb, Microsoft, ODPi, PhoenixNAP, Pivotal, Private Internet Access, Produban, Red Hat, Serenata Flowers, Target, WANdisco, and Yahoo. For more information, visit https://www.apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "SystemML", "Apache SystemML", "Hadoop", "Apache Hadoop", "Spark", "Apache Spark", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 21 June 2017. Board calendar and minutes http://apache.org/foundation/board/calendar.html
 - On the State of the Feather https://s.apache.org/Lz3t

ApacheCon™ –the official conference of the Apache Software Foundation. Tomorrow's Technology Today.
 - Presentations from ApacheCon https://s.apache.org/Hli7 and Apache: Big Data https://s.apache.org/tefE
 - Videos of keynotes + presentations are now available https://s.apache.org/AE3m
 - Soundbites from the conference floor https://feathercast.apache.org/
 - My First Experience of ApacheCon https://blogs.apache.org/comdev/entry/my-first-experience-of-apachecon + "epic" slideshow https://youtu.be/pKFfirqEgQ8

ASF Infrastructure –our distributed team on four continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield zippity performance at 99.95% uptime http://status.apache.org/

Apache Archiva™ –an extensible repository management tool that helps taking care of your own personal or enterprise-wide build artifact repository.
 - CVE-2017-5657: Apache Archiva CSRF vulnerability for REST endpoints http://mail-archives.apache.org/mod_mbox/www-announce/201705.mbox/%3C1622774.CTg74Sxca6%40golgafrichnam%3E

Apache Arrow™ –a columnar in-memory analytics layer designed to accelerate Big Data.
 - Apache Arrow 0.4.0 released http://arrow.apache.org/

Apache Commons™ Text –Open Source software library that provides a host of algorithms focused on working with strings and blocks of text.
 - Apache Commons Text 1.1 released http://commons.apache.org/proper/commons-text/

Apache Jena™ –a free and Open Source Java framework for building semantic Web and Linked Data applications.
 - Apache Jena 3.3.0 released http://jena.apache.org/

Apache NiFi™ –an easy to use, powerful, and reliable system to process and distribute data.
 - Apache NiFi 0.7.3 and MiNiFi 0.2.0 released https://nifi.apache.org/

Apache OFBiz™ –an Open Source product for the automation of enterprise processes that includes framework components and business applications.
 - Apache OFBiz 16.11.02 released http://ofbiz.apache.org/

Did You Know?

 - Did you know that Apache Metron helps make it possible for enterprise security personnel to more quickly detect and investigate costly threats using Big Data in real time? http://metron.apache.org/

 - Did you know that Apache Samza currently processes billions of messages per day, accounting for 100s of TB of data flowing through the system at Uber? http://samza.apache.org/

 - Did you know that over the past week, 482 contributors changed 766,603 lines of Apache code through 2,565 commits? Still productive during ApacheCon! http://status.apache.org/

Apache Community Notices:

 - "Success at Apache" is a blog series that focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh

 - The latest Apache Community Newsletter https://blogs.apache.org/comdev/entry/community-development-news-april-2017

 - Do friend and follow us on the Apache Community Facebook page https://www.facebook.com/ApacheSoftwareFoundation/ and Twitter account https://twitter.com/ApacheCommunity

 - The Apache Phoenix community will be holding PhoenixCon on 13 June in San Francisco https://www.eventbrite.com/e/phoenixcon-2017-tickets-32872245772

 - Catch the Apache Ignite and Spark communities at the In-Memory Computing Summit 20-21 June in Amsterdam and 24-25 October in San Francisco https://imcsummit.org/

 - ASF Operations Summary - Q3 FY2017 https://s.apache.org/NKFz

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby