The Apache Software Foundation Blog

Friday January 20, 2017

The Apache News Round-up: week ending 20 January 2017

Another week has skipped by, and there's no stopping the always-productive Apache community ... here's what's happened:

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 15 February 2017. Board calendar and minutes available at http://apache.org/foundation/board/calendar.html

ASF Infrastructure –our distributed team on four continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield performance back in the "three nines" at 99.94% uptime http://status.apache.org/

ApacheCon™ –the official conference series of The Apache Software Foundation.
 - CFP CLOSES 11 Feb: Apache: Big Data and ApacheCon North America 16-18 May 2017/Miami http://apachecon.com/
 - Recordings from ApacheCon Europe 2016 are being uploaded at Feathercast http://feathercast.org

Calling all Creatives: with the launch of the ASF's new visual identity last year, many Apache projects have been freshening up their looks. New logo calls are open for:
 - Apache Incubator https://s.apache.org/rFii
 - Apache OpenNLP https://issues.apache.org/jira/browse/OPENNLP-6

Apache Fineract (incubating) –an Open Source system for core banking as a platform.
 - Apache Fineract 0.6.0-incubating released https://dist.apache.org/repos/dist/release/incubator/fineract/0.6.0-incubating/

Apache Groovy™ –a multi-facet programming language for the JVM.
 - Apache Groovy 2.4.8 released http://www.groovy-lang.org/download.html
 - CVE-2016-6814: Apache Groovy Information Disclosure http://mail-archives.apache.org/mod_mbox/www-announce/201701.mbox/%3CCADRx3PMZ2hBCGDTY35zYXFGaDnjAs0tc5-upaVs6QN2sYUejyA%40mail.gmail.com%3E

Apache HBase™ –an Open Source, distributed, versioned, non-relational database.
 - Apache HBase 1.3.0 released http://www.apache.org/dyn/closer.lua/hbase/1.3.0

Apache HTTP Server™ –the #1 Web server on the planet since April 1996.
 - Apache HTTP Server 2.2.32 released http://httpd.apache.org/download.cgi

Apache Ignite™ –an integrated and distributed In-Memory Data Fabric for computing and transacting on large-scale data sets in real-time, orders of magnitude faster than possible with traditional disk-based or flash technologies.
 - The ASF asks: Have you met Apache Ignite? https://s.apache.org/Slah

Apache Jackrabbit™ –a fully compliant implementation of the Content Repository for Java(TM) Technology API, version 2.0 (JCR 2.0) as specified in the Java Specification Request 283 (JSR 283).
 - Apache Jackrabbit 2.10.5 released http://jackrabbit.apache.org/downloads.html

Apache Johnzon™ –a Java library for parsing and creating JSON.
 - Apache Johnzon-1.0.0 released http://johnzon.apache.org/

Apache NiFi™ –easy to use, powerful, and reliable system to process and distribute data.
 - CVE-2016-8748: Apache NiFi XSS vulnerability in connection details dialogue http://mail-archives.apache.org/mod_mbox/www-announce/201701.mbox/%3CCALJK9a4TNPvGav_UxwLQvqY0M2mRNWnvQBvu58p7%3D_ZfD1_AGg%40mail.gmail.com%3E

Apache Portals™ Pluto –the Reference Implementation of the Java Portlet Specification.
 - Apache Portals Pluto 3.0.0 released http://portals.apache.org/pluto/v30/deploying.html


Did You Know?

 - Did you know that whilst Big Data projects comprise 49% of the podlings in the Apache Incubator, it is only ~9% of the ASF's overall projects and initiatives? https://projects.apache.org/projects.html?category

 - Did you know that creating relationships between multiple ORM modules dynamically in runtime is possible only with Apache Cayenne? http://cayenne.apache.org/

 - Did you know that hundreds of thousands of software solutions are distributed under the Apache License, with Web requests from every UN-recognized nation? http://apache.org/licenses/


Apache Community Notices:

 - "Success at Apache" is a new monthly blog series that focuses on the processes behind why the ASF "just works". First article: Project Independence https://s.apache.org/CE0V January's post: "All Carrot and No Stick" https://s.apache.org/ykoG

 - Feedback from The Apache Software Foundation on the Free and Open Source Security Audit (FOSSA) https://s.apache.org/romf

 - ASF Operations Summary - Q2 FY2017 https://s.apache.org/oTOF

 - The list of Apache project-related MeetUps can be found at http://apache.org/events/meetups.html

 - Find out how you can participate with Apache community/projects/activities --opportunities open with Apache HTTP Server, Avro, ComDev (community development), Directory, Incubator, OODT, POI, Polygene, Syncope, Tika, Trafodion, and more! https://helpwanted.apache.org/

 - ApacheCon North America and Apache:BigData will be held 16-18 May 2017 in Miami http://apachecon.com/

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

 - Show your support for Apache with ASF-approved swag from http://www.zazzle.com/featherwearand http://s.apache.org/landsend--all proceeds benefit the ASF! 

= = =

For real-time updates, sign up for Apache-related news by sending mail to announce-subscribe@apache.org and follow @TheASF on Twitter. For a broader spectrum from the Apache community, https://twitter.com/PlanetApache provides an aggregate of Project activities as well as the personal blogs and tweets of select ASF Committers.

# # #

Wednesday January 18, 2017

The ASF asks: Have you met Apache Ignite?

Since 1999, The Apache Software Foundation (ASF) has been recognized as a leading source for an array of Open Source software and tools that meet the demand for interoperable, adaptable, and sustainable solutions. The all-volunteer ASF develops, stewards, and incubates dozens of enterprise-grade Open Source projects that power mission-critical applications in financial services, aerospace, publishing, government, healthcare, research, infrastructure, and more. From Abdera to ZooKeeper, the demand for ASF's reliable, community-driven software continues to grow dramatically across many categories, including Cloud, IoT, Artificial Intelligence and Machine Learning, Mobile, and Big Data, where the Apache Hadoop ecosystem dominates the marketplace.

Did you know that numerous Fortune 500 enterprises depend on Apache Ignite's in-memory data platform to process large-scale data sets in real-time, at orders of magnitude faster than traditional technologies?

We are pleased to showcase Apache Ignite, the high-performance In-Memory Data Fabric that provides in-memory data caching, partitioning, processing, and querying components.

Quick peek: Apache Ignite is an integrated and distributed In-Memory Data Fabric for computing and transacting on large-scale data sets in real-time, orders of magnitude faster than possible with traditional disk-based or flash technologies. It is designed to easily power both existing and new applications in a distributed, massively parallel architecture on affordable, industry-standard hardware.

Background: Originally created at GridGain as its flagship in-memory computing (IMC) platform, Ignite entered the Apache Incubator in September 2014 and graduated as an Apache Top-Level Project in August 2015.

Why Ignite: Apache Ignite addresses today's Fast Data and Big Data needs by providing a comprehensive in-memory data fabric, which includes a data grid with SQL and transactional capabilities, in-memory streaming, an in-memory file system, and more.

Heavily benchmarked, Ignite has been built from the ground up to linearly scale to hundreds of nodes with strong semantics for data locality and affinity data routing to reduce redundant data noise. Ignite data grid is lightning fast and is one of the fastest implementations of transactional or atomic data in distributed clusters today.

Unlike other Big Data processing solutions, Apache Ignite treats RAM as a primary storage facility (as opposed to being used exclusively for processing). As such, Ignite's memory-first approach is more efficient and faster: with improved system indexes, reduced data fetch time, and no delays in a stream content processing, among other benefits.

Additionally --and unique to Apache Ignite-- its SQL Grid eliminates the need for painful and challenging migration from relational database to in-memory data grid (IMDG), alleviating the need for developers to have to rewrite SQL based code to IMDG's native APIs. This means that developers can keep using existing applications and tools written for relational databases and based on SQL language with very little to no code modification. Ignite SQL Grid is horizontally scalable, fault tolerant, and SQL ANSI-99 compliant.

Using Apache Ignite, developers benefit from:
  • Data Grid --replicate or partition data in memory within the cluster;
  • SQL Grid --add in-memory distributed database capabilities;
  • Compute Grid --distribute computations across cluster nodes;
  • Service Grid -- implement fault-tolerant microservices based solutions;
  • Streaming & CEP --easily stream large volumes of data into Ignite processing them in real-time; and
  • Data Structures --distribute own data structure across the cluster.

To solve real-time business issues and meet application requirements for the highest performance and scale, Apache Ignite leverages and integrates a host of Apache projects including Spark, Hadoop, YARN, and Mesos.

Latest release: Apache Ignite v1.8 on 9 December 2016 under the Apache License v.2.0. More details can be found below and in the release notes.

What's under the hood: New in Apache Ignite v1.8:
  • SQL Grid now fully supports all DML commands including UPDATE, INSERT and DELETE queries. A full-fledged support of DML and SELECT statements allows to interact with Apache Ignite using standard SQL commands connecting via ODBC and JDBC drivers. This provides true cross-platform connectivity even from languages such as PHP and Ruby which are not natively supported by the project. 
  • Redis protocol implementation which enables users to store and retrieve distributed data from Apache Ignite cache using any Redis compatible client.
  • Ignite.NET provides .NET Entity Framework 2nd Level Cache solution that stores data in the distributed Ignite cache. This is ideal for scenarios with multiple application servers using a single SQL database via Entity Framework: cached queries are shared between all machines in the cluster.
  • Ignite.NET implements ASP.NET session caching provider that stores session data in the Ignite cache which distributes session state across multiple servers in order to provide high availability and fault tolerance.
  • Deadlock detection mechanism has been improved and now works for optimistic transaction and near caches.

Check out the Apache Ignite blog for articles, insight, how-tos, and additional resources at https://ignite.apache.org/blogs.html

For downloads, documentation, examples, use cases, and more information, visit http://ignite.apache.org/ .

# # #

Friday January 13, 2017

The Apache News Round-up: week ending 13 January 2017

It's Friday! Here's what the Apache community has been up to over the past week:

Success at Apache –the new monthly blog series that focuses on the processes behind why the ASF "just works".  
 - January's post: "All Carrot and No Stick" https://s.apache.org/ykoG

Notice: Apache Project Name Change –Apache Zest Renamed to Apache Polygene https://s.apache.org/4Klg

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 18 January 2017. Board calendar and minutes available at http://apache.org/foundation/board/calendar.html

ASF Infrastructure –our distributed team on four continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield swift performance at 99.65% uptime http://status.apache.org/

ApacheCon™ –the official conference series of The Apache Software Foundation.
 - CFP OPEN: Apache: Big Data and ApacheCon North America 16-18 May 2017/Miami http://apachecon.com/
 - 2016/Seville's session recordings are being processed and posted at Feathercast http://feathercast.org

Apache Incubator –projects and communities intending to become fully-fledged projects under the auspices of The Apache Software Foundation do so through the Apache Incubator.
 - Call for Entries --Apache Incubator Logo https://s.apache.org/rFii

Apache Beam™ –unified programming model for batch and streaming Big Data processing, handling data of any scale, and providing portability across multiple execution engines and environments.
 - The Apache Software Foundation Announces Apache Beam as a Top-Level Project https://s.apache.org/u67z

Apache Calcite™ –a dynamic data management framework.
 - Apache Calcite 1.11.0 released http://www.apache.org/dyn/closer.cgi/calcite/apache-calcite-1.11.0/

Apache CloudStack™ –an integrated Infrastructure-as-a-Service (IaaS) software platform that allows users to build feature-rich public and private cloud environments.
 - Apache CloudStack 4.9.2.0 released http://cloudstack.apache.org/downloads.html

Apache Eagle™ –intelligent Big Data monitoring and alerting solution in use at high volume, high demand Websites, platforms, and organizations such as eBay, PayPal, Dataguise, and YHD.com, among others.
 - The Apache Software Foundation Announces Apache Eagle as a Top-Level Project https://s.apache.org/lRU1

Apache HttpComponents™ Core – a set of low level HTTP transport components that can be used to build custom client and server side HTTP services with a minimal footprint.
 - Apache HttpComponents Core 4.4.6 GA released http://hc.apache.org/downloads.cgi

Apache Jackrabbit™ –a fully compliant implementation of the Content Repository for Java(TM) Technology API, version 2.0 (JCR 2.0) as specified in the Java Specification Request 283 (JSR 283).
 - Apache Jackrabbit 2.14.0 and 2.15.0, and Jackrabbit Oak 1.5.17 and 1.2.23 released http://jackrabbit.apache.org/downloads.html

Apache MyFaces™ Tobago – a component library for JavaServer Faces (JSF) that allows to write Web applications without the need of coding HTML, CSS and JavaScript.
 - Apache Tobago 3.0.0 released http://myfaces.apache.org/tobago/

Apache OpenJPA™ –a Java persistence project that can be used as a stand-alone POJO persistence layer or integrated into any Java EE compliant container and many other lightweight frameworks, such as Tomcat and Spring.
 - Apache OpenJPA 2.4.2 released http://openjpa.apache.org/downloads.html

Apache OpenMeetings™ –provides video conferencing, instant messaging, white board, collaborative document editing and other groupware tools using API functions of the Red5 Streaming Server for Remoting and Streaming.
 - Apache OpenMeetings 3.1.4 released http://openmeetings.apache.org/downloads.html


Did You Know?

 - Did you know that there are hundreds of *new* code contributors to Apache projects each month? https://twitter.com/TheASF/status/819220448625983488

 - Did you know that Ippon uses Apache Kafka, Spark, and ZooKeeper to analyze 25 million records per day? http://kafka.apache.org/ , http://spark.apache.org/ , and http://zookeeper.apache.org/

 - Did you know that hundreds of thousands of software solutions are distributed under the Apache License, with Web requests from every UN-recognized nation? http://apache.org/licenses/


Apache Community Notices:

 - "Success at Apache" is a new monthly blog series that focuses on the processes behind why the ASF "just works". First article: Project Independence https://s.apache.org/CE0V

 - Feedback from The Apache Software Foundation on the Free and Open Source Security Audit (FOSSA) https://s.apache.org/romf

 - ASF Operations Summary - Q2 FY2017 https://s.apache.org/oTOF

 - The list of Apache project-related MeetUps can be found at http://apache.org/events/meetups.html

 - Find out how you can participate with Apache community/projects/activities --opportunities open with Apache HTTP Server, Avro, ComDev (community development), Directory, Incubator, OODT, POI, Syncope, Tika, Trafodion, Zest, and more! https://helpwanted.apache.org/

 - ApacheCon North America and Apache:BigData will be held 16-18 May 2017 in Miami  http://apachecon.com/

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

 - Show your support for Apache with ASF-approved swag from http://www.zazzle.com/featherwearand http://s.apache.org/landsend--all proceeds benefit the ASF! 

= = =

For real-time updates, sign up for Apache-related news by sending mail to announce-subscribe@apache.org and follow @TheASF on Twitter. For a broader spectrum from the Apache community, https://twitter.com/PlanetApache provides an aggregate of Project activities as well as the personal blogs and tweets of select ASF Committers.

# # #

Wednesday January 11, 2017

The Apache Software Foundation Announces Apache® Zest™ Renamed to Apache Polygene

Rebranded Open Source Composite Oriented Programming platform reflects growing codebase and community.

Forest Hill, MD —11 January 2017— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today that Apache® Zest™, the Composite Oriented Programming platform, has been renamed Apache Polygene.

Apache Polygene is a platform to develop applications with large domain models and complex business logic for Java enterprise developers. Apache Polygene introduces multi-inheritence, aspect orientation (both typesafe and generic weaving) and persistence to both SQL and NoSQL storage systems. Apache Polygene also easily integrates with other technologies such as Spring Framework, REST, OSGi and many more.

"The name change was triggered to prevent confusion with other similarly named software such as the visualization toolkit from Eclipse," said Niclas Hedhman, Vice President of Apache Polygene. "Since becoming an official ASF project, our codebase and community continue to flourish. We are confident that our new identity will reflect ongoing innovation and increased productivity."

The resolution relating to the project's name change was approved at the ASF Board meeting in December 2016.

Project History
In 2007, Hedhman convinced Rickard Öberg to create an Open Source project based on Öberg’s Composite Oriented Programming (COP) concept, which launched as Qi4j. Since then, 28 people have contributed source to the project, with many others participating on mailing lists regarding direction, concepts and design. In 2015 the project arrived at the ASF as Apache Zest, along the unique designation as the first project to enter the ASF as al Top-Level Project– without entering the Apache Incubator (the official entry path for projects and codebases wishing to become part of the ASF’s efforts). As part of its eligibility, the project had to meet the rigorous requirements of the Apache Maturity Model http://s.apache.org/O4p , that addresses the integrity of a project's code, copyright, licenses, releases, community, consensus building, and independence, among other qualities. In March 2015 Apache Zest became an official ASF Top-Level Project, and renamed as Apache Polygene in December 2016.

Availability and Oversight
Apache Polygene software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For project updates, downloads, documentation, and ways to become involved with Apache Polygene, visit http://polygene.apache.org/

About The Apache Software Foundation (ASF)
Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server --the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 620 individual Members and 5,900 Committers successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official user conference, trainings, and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Alibaba Cloud Computing, ARM, Bloomberg, Budget Direct, Capital One, Cash Store, Cerner, Cloudera, Comcast, Confluent, Facebook, Google, Hortonworks, HP, Huawei, IBM, InMotion Hosting, iSigma, LeaseWeb, Microsoft, OPDi, PhoenixNAP, Pivotal, Private Internet Access, Produban, Red Hat, Serenata Flowers, Target, WANdisco, and Yahoo. For more information, visit http://www.apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Polygene", "Apache Polygene", "Zest", "Apache Zest", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Tuesday January 10, 2017

The Apache Software Foundation Announces Apache® Beam™ as a Top-Level Project

Unified programming model for batch and streaming Big Data processing, handling data of any scale, and providing portability across multiple execution engines and environments.

Forest Hill, MD —10 January 2017— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today that Apache® Beam™ has graduated from the Apache Incubator to become a Top-Level Project (TLP), signifying that the project's community and products have been well-governed under the ASF's meritocratic process and principles.

Apache Beam is a unified programming model for both batch and streaming data processing. It includes software development kits in Java and Python for defining the data processing pipelines, as well as runners to execute them on several execution engines, including Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow.

"Graduation is an exciting milestone for Apache Beam," said Davor Bonaci, Vice President of Apache Beam. "Becoming a top-level project is a recognition of the amazing growth of the Apache Beam community, both in terms of size and diversity. Together we are pushing forward the state of the art in distributed data processing and, at the same time, enhancing the ability to interconnect additional storage/messaging systems and execution engines."

The technology behind Apache Beam evolved in large part from Google's internal work on data processing, tracing its roots all the way back to the Google's initial MapReduce system and its fundamental changes to the science of distributed data processing. It also reflects modern advances in data processing, embodied in Google's FlumeJava and MillWheel systems, and culminating with the unified programming model of Google Cloud Dataflow, which became the heart of Apache Beam.

This unified programming model can easily and intuitively express data processing pipelines for everything from simple batch-based data ingestion to complex event-time-based stream processing. The abstractions in the model are designed to support efficient parallel execution, while also cleanly separating the user's processing logic from details of the underlying engine.

Raising the level of abstraction allows a single Apache Beam pipeline to run, without modification, on multiple execution engines. This portability across diverse execution engines is just one of many extensibility points that let Apache Beam integrate with the broader Apache and Big Data ecosystems. Beside runners, developers can already easily add support for additional IO connectors, libraries of transformations, SDKs, and even domain-specific extensions.

"Apache Beam helps us make stream processing accessible to a broad audience of data engineers, by offering an API which is comprehensive, easy to reason about and at the same time fully decoupled from the underlying execution engine," said Assaf Pinhasi, Director of Big Data Platform at PayPal. "Our data engineers can now focus on what they do best – i.e. express their processing pipelines easily, and not have to worry about how these get translated to the complex underlying engine they run on."

"The graduation of Apache Beam as a top-level project is a great achievement and, in the fast-paced Big Data world we live in, recognition of the importance of a unified, portable, and extensible abstraction framework to build complex batch and streaming data processing pipelines," said Laurent Bride, Chief Technology Officer at Talend. "Customers don't like to be locked-in, so they will appreciate the runtime flexibility Apache Beam provides. With four mature runners already available and I'm sure more to come, Beam represents the future and will be a key element of Talend's strategic technology stack moving forward."

"We applaud the Apache Beam working group for its success in creating a unified and consistent platform for building portable data processing pipelines," said Fausto Ibarra, Director of Product Management, Google Cloud Platform. "We believe that we all have a responsibility to share what we're learning, and we are proud and delighted to witness the successful collaboration to build not only a powerful programming model for processing data from bounded and unbounded sources, but also a portability layer for running pipelines on many processing engines, including Apache Spark, Apache Flink, Apache Apex, and Google Cloud Dataflow. Apache Beam's graduation to Top Level Project is a well-deserved recognition for the individuals and companies who contributed to the project."

"Apache Beam represents a principled approach for analyzing data streams, simplifying a range of complex data processing concepts and providing developers with a flexible, straightforward model," said Kostas Tzoumas, Co-founder and Chief Executive Officer at data Artisans. "The Apache Flink community wrote one of the first Beam runners, and those of us at data Artisans has been contributing to the Beam project since its inception."

"The Apache Beam community has quickly adapted the Apache Way and been very welcoming to new contributors and ideas. It also encourages communication across other projects that collaborate under the Beam umbrella," said Thomas Weise, Vice President of Apache Apex, and Chief Technology Officer/Co-Founder of Atrato. "Beam helps the wider ecosystem by establishing common terminology and well thought through concepts that reflect in multiple runners and even the native API of the underlying engines."

"In my work at Apache, I have rarely seen an incubating project build a community as well as the Apache Beam project has done," said Ted Dunning, Vice President of Apache Incubator, and Chief Application Architect at MapR Technologies. "The way that they have been able to complement and enhance other streaming data projects is really a credit to everyone involved."

"We'd like to invite you to consider joining us on this exciting ride, whether as a user or a contributor, as we work towards our first release with API stability," added Bonaci. "If you'd like to try out Apache Beam today, check out the latest 0.4.0 release. We welcome contribution and participation from anyone through our mailing lists, issue tracker, pull requests, and events."

Catch Apache Beam in action at numerous face-to-face meetups and conferences, including Apache: Big Data North America 2017, DataWorks Summit and Hadoop Summit Munich 2017, Strata + Hadoop World San Jose and London 2017.

Availability and Oversight
Apache Beam software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For project updates, downloads, documentation, and ways to become involved with Apache Beam, visit https://beam.apache.org/ and @ApacheBeam.

About the Apache Incubator
The Apache Incubator is the entry path for projects and codebases wishing to become part of the efforts at The Apache Software Foundation. All code donations from external organizations and existing external projects wishing to join the ASF enter through the Incubator to: 1) ensure all donations are in accordance with the ASF legal standards; and 2) develop new communities that adhere to our guiding principles. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. For more information, visit http://incubator.apache.org/

About The Apache Software Foundation (ASF)
Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server --the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 620 individual Members and 5,900 Committers successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official user conference, trainings, and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Alibaba Cloud Computing, ARM, Bloomberg, Budget Direct, Capital One, Cash Store, Cerner, Cloudera, Comcast, Confluent, Facebook, Google, Hortonworks, HP, Huawei, IBM, InMotion Hosting, iSigma, LeaseWeb, Microsoft, OPDi, PhoenixNAP, Pivotal, Private Internet Access, Produban, Red Hat, Serenata Flowers, Target, WANdisco, and Yahoo. For more information, visit http://www.apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Beam", "Apache Beam", "Apache Apex", "Apex", "Apache Flink", "Flink", "Apache Spark", "Spark", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

The Apache Software Foundation Announces Apache® Eagle™ as a Top-Level Project

Intelligent Big Data monitoring and alerting solution in use at high volume, high demand Websites, platforms, and organizations such as eBay, PayPal, Dataguise, and YHD.com, among others.

Forest Hill, MD —10 January 2017— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today that Apache® Eagle™ has graduated from the Apache Incubator to become a Top-Level Project (TLP), signifying that the project's community and products have been well-governed under the ASF's meritocratic process and principles.

Apache Eagle is an Open Source monitoring and alerting solution for instantly identifying security and performance issues on Big Data platforms such as Apache Hadoop, Apache Spark, and more.

"We are proud to complete the incubation process and graduate as an Apache Top-Level Project," said Edward Zhang, Vice President of Apache Eagle. "The community is actively improving product coverage for analyzing various performance and security issues in large Hadoop clusters."

Eagle was first developed at eBay to solve the monitoring problem for a large scale Hadoop cluster. The eBay team soon realized it would be useful to the whole community, and submitted the project to the Apache Incubator in October 2015. Since then, the project gained a lot of attraction from various developers and organizations for its broad usage scenarios, such as system/service monitoring, application performance monitoring, and security breach detection.

Apache Eagle features include:
  • Highly extensible - Apache Eagle builds its core framework around the application concept; the application itself includes the logic for monitoring source data collection, pre-processing and normalization. Developers can easily develop out-of-box monitoring applications using Eagle's application framework, and deploy into Eagle.
  • Scalable - the project’s fundamental runtime is based on proven Big Data technologies, and applies a scalable core to make it adaptive according to the throughput of the data stream as well as the number of monitored applications.
  • Real-time - provides state-of-the-art alert engine to identify security breaches and performance issues.
  • Dynamic - users can freely enable or disable a monitoring application and dynamically change their alert policies without any impact to the underlying runtime.

"It is exciting to see increasing deployments of Apache Eagle, along with great use cases and contributions back to the project," added Zhang.

"Apache Eagle is a highly scalable and extensible technology platform to support the ever growing needs of intelligent monitoring and alerting in a massively distributed computing environment," said Debashis Saha, CTO and EVP at Jiff Inc. "As the founding executive sponsor of this project at eBay, I am proud to see the community continue to expand the capabilities by supporting complex and diverse use cases for monitoring in security, infrastructure, networking and distributed services in Apache Eagle. Congratulations to the team and the community in graduating to a Apache top level project."

"As a leader in data-centric security with a focus on cloud and Big Data technologies, Dataguise is proud to be part of the Eagle committers group. DgSecure Monitor, our sensitivity-aware monitoring product, uses Apache Eagle as the core engine," said Subra Ramesh, VP of Products and Engineering at Dataguise Inc. "Apache Eagle's flexible architecture, proven scalability, and  cutting-edge design, have enabled DgSecure Monitor to be a highly responsive and scalable solution for both on-premises and cloud deployments. We look forward to continued involvement with Eagle as it has now become a top-level Apache project."

"We have been using Apache Eagle for about a year, and are very happy to see it graduate to a Top-Level Project. Apache Eagle and its low latency real-time alert engine can help us easily identify security and performance issues instantly on Hadoop platform," said Anson Zhong, Senior Vice President of Tech Department at YHD.com. "In addition, Eagle's architecture is highly extensible. We are looking forward to using it in real time risk management system."

"Apache Eagle is a great monitoring and alerting solution designed for large-scale distributed environment," said Chad Chun, Director of Analytics Data Infrastructure at eBay. "It was originally intended for security monitoring and quickly become a generic solution for allowing domain experts to create their own monitoring applications on top of Eagle. This is a wonderful design for easily leveraging the power of community to create and share applications. Looking forward to the tremendous adoption in the industry."

"The Apache Eagle community has done a tremendous job throughout the incubation process, and I'm thrilled to see it graduate to a Top-Level Project," said P. Taylor Goetz, ASF Member and Apache Eagle Project Mangement Committee member. "Eagle fills a very important role in providing top-notch security and performance monitoring and alerting for Big Data deployments. The Eagle project has built a robust, sustainable community and demonstrated a firm understanding of the Apache Way. I look forward to further innovation as the Eagle community marks this important milestone."

"It is great to see Apache Eagle graduate to a Top Level Project within a year of time," said Seshu Adunuthula, Senior Director of Data Platforms at eBay. "It is a great product with unique position to fill the gap of monitoring and alerting large-scale distributed computing environment which is well architected to allow communities to easily implement monitoring and alerting applications on different technical domains such as networking and database clusters.  I would love to see the community to grow fast in the next coming years!"

The project welcomes contributions and community participation through mailing lists, Slack channel, face-to-face Meetups, and other events.

Availability and Oversight
Apache Eagle software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For project updates, downloads, documentation, and ways to become involved with Apache Eagle, visit http://eagle.apache.org and @TheApacheEagle.

About the Apache Incubator
The Apache Incubator is the entry path for projects and codebases wishing to become part of the efforts at The Apache Software Foundation. All code donations from external organizations and existing external projects wishing to join the ASF enter through the Incubator to: 1) ensure all donations are in accordance with the ASF legal standards; and 2) develop new communities that adhere to our guiding principles. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. For more information, visit http://incubator.apache.org

About The Apache Software Foundation (ASF)
Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server --the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 620 individual Members and 5,900 Committers successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official user conference, trainings, and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Alibaba Cloud Computing, ARM, Bloomberg, Budget Direct, Capital One, Cash Store, Cerner, Cloudera, Comcast, Confluent, Facebook, Google, Hortonworks, HP, Huawei, IBM, InMotion Hosting, iSigma, LeaseWeb, Microsoft, OPDi, PhoenixNAP, Pivotal, Private Internet Access, Produban, Red Hat, Serenata Flowers, Target, WANdisco, and Yahoo. For more information, visit http://www.apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Eagle", "Apache Eagle", "Apache Hadoop", "Hadoop", "Apache Spark", "Spark", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #


Monday January 09, 2017

Success at Apache: "All Carrot and No Stick"

By Danny Angus

When the ASF launched their "Success at Apache" series I offered to share my own experiences. If you read on, remember that this is my personal experience and that others may disagree with me, but as you'll see, that's really part of the fun. 

For a bit background I’m currently the Project Management Committee (PMC) Chair of Apache Labs and in my day job I’m a "Divisional CTO" for a FTSE250 technology company. I first came to the ASF around 2000 when I was part of a startup - I was a CTO then too, it was the dot com boom, and it was just me and a couple of guys. We were considering a partnership with some researchers who wanted to commercialise their work, and were looking for a bit of software that we could use as the foundation for a product because a) we couldn’t afford to write it or buy it, and b) we didn’t have the knowledge anyway. What I found was Apache James http://james.apache.org , so I downloaded it, got it up and running, and did some prototyping, but we quickly realised that it needed work if we were going to be able to use it in production. I dug into it a little, subscribed to the mailing lists, asked questions and figured out what needed to be done to fix and extend what was already there, then started to modify it locally. Meantime I found myself answering other users’ questions on the user list, and one day noticed that I was actually answering more questions that I was asking. Shortly after that, that I was answering more questions than anyone else. Then I started submitting patches to the developer list (this was in the days of CVS: long before git!), which were reviewed and committed for me by the committers … but eventually they got bored with that and decided to extend commit privileges to me so I could do it all myself.


My experience illustrates an important characteristic of Apache projects: the fact that you can just turn up and get involved. Another very other important characteristic is that we are a meritocracy: demonstrating your capability is all you need to do in order to gain more responsibility; demonstrating your willingness and trustworthiness should be enough to get you the job. "Karma" is a word that is used to mean "access permission" in many Open Source projects, and we used to say that if you knew how to ask for karma properly, that was itself a sign that you could be trusted with it. Of course we were a much different organisation in those days, but the principles of a community built on merit and trust are still core to our identity. It's no coincidence that organisations cannot be part of our community: only individuals. Organisations are an important part of the world in which we exist, but we don't exist for their benefit, we only exist at all because as individuals we each bother to turn up and do stuff, from the guy who one time downloads and installs the Apache HTTP Web Server to Sam Ruby, our current (and can I just say excellent) President, everyone is contributing in their own way to the life of Apache and achieving benefits suited to their own, personal, motivations. So it was OK for me to focus on my own and my employer's priorities, which meant that I could learn from my new friends, develop the software we needed at work and become part of this amazing community all at the same time.


My experience of Apache is that it is what I would call "all carrot and no stick". I think that is the most healthy model of Open Source, as it is predicated on the fact that every participant will benefit from their participation without the need to contribute more than they are prepared to do. For me, focusing my contribution on the things I knew about was not only the most efficient use of my time, in terms of meeting our company's product goals, but it also allowed me to learn from others who had, and continue to have, way more knowledge and experience than I, and to benefit from their work. Mixing with these amazing people, many of whom are now real friends of mine, has taught me more than I would ever have learned any other way.


At this point in my involvement Apache went through a bit of what has diplomatically been described as "navel gazing", and settled on the idea that the organisational structure should be very very flat, and there should be no limit to our growth. As long as our standards were met by projects and people, we would welcome them both into our community. Those standards are partly about merit, partly about legal protection, one of the key roles Apache plays is to provide a degree of protection to projects and the people contributing to them, from the threat of bullies, trolls, and gorillas with expensive lawyers; and partly about ensuring that the behaviours and practices that define our identity and have contributed to the survival and the success of our organisation are continued by new generations of people in new projects using and creating technologies that we could hardly have dreamed about 16 years ago.


Before long the dust settled and I found myself voted to chair the Apache James Project http://apache.org/foundation/governance/pmcs.html , which was a whole new dimension of interesting. Chairing a project using only positive motivation teaches you a lot about people, including yourself, and I have a few observations about successful collaboration that I have found to be helpful both at work, where I strive to implement bottom-up decision making, and at the ASF where I want to make a positive contribution and see our communities flourish:


  • Free your mind.The collective sense of direction may not be what you expect, there have been times when I have been very sceptical about the reality of great sounding ideas, but I have also learned that it’s OK to go down the wrong road because most of the time it makes little difference in the end, usually you learn a lot regardless, and if people are really behind it you stand a much better chance of success than if the really good idea has all the fun of a death march. One phrase which is often used to summarise the spirit of Apache is “Community over code”, put the community first, and the code will follow.

  • Listen, and be supportive. There are a lot of different people involved in our projects with a lot of very different motivations. They are mostly all valid, and mostly all equally important if that even has an absolute scale. There are students studying our code, asking questions using our software and maybe fixing defects so that they can learn, there are employees of corporations who are being paid to protect their investment, to implement the product roadmap and maintain some predictable velocity, there are researchers who are pushing the boundaries of their chosen topic, there are people whose livelihood and success depends on a project, and those who are involved because it is a release from the pressure of things with names like "impact", "benefits", "deadlines" and "goals". Moderate or steer the discussion to ensure that all sides are heard, a meritocracy needs to listen to everyone not just the most vocal or assertive, and when I say listen that doesn’t mean formulating your own response while someone else is talking. Support people who you agree with, help to realise other people’s ideas, collaboration is only achieved by being truly committed to each other’s success, not just your own.

  • "A's hire A's B's hire C's". Find, support, and mentor the next generation, when your success depends upon the community it makes sense for you to put some effort into creating the best community you can.

  • Use Positive Language. When I was a kid being mean to my sister, adults used to say, "If you don't have anything nice to say, don't say anything at all". That's great advice if you’re involved in any collaborative venture, but doubly so when it is something like an Open Source project where you are usually communicating using written English, with people you don't know well, who might not have the same language skills as you do, who live in a different time zone and sometimes have very different cultural background than you. On top of all that you"re often debating the details of highly abstract technical concepts. The communication barrier itself can cause a kind of baseline of frustration so go easy on the negativity, one thing I like to do when I strongly disagree with someone is to write how I feel, then try to reword it using only positive language, it might sound like touchy-feely hippy nonsense to you, but you will be surprised how effective changing "I think you’re wrong and here’s why..." into "You have clearly thought a lot about this, I wonder if you have considered...". Alienating people is not the way to get your point across.

  • Learn to be a good loser. You don't own your projects, not here, and you're not the smartest person here either (OK so that’s not going to be 100% true, but there are 5,938 Committers today which makes it about 99.98%) recognising that and learning to embrace the collective view is hard for some people, but being able to step outside your subjective point of view and make a success of something you didn't believe in is a lesson in leadership that is definitely worth learning, because if not, your growth will be limited by the ideas that come from your own head, not accelerated by other people.

  • We are making it up as we go along http://apache.org/foundation/how-it-works.html . Yes, it sometimes seems from the outside like we have it all sorted and nailed down, and that we want to lawyer up and suck the life out of every fun thing (I mean we have a major software licence with our name on it, how grown up is that for goodness sakes?)  But the truth is that Apache, The Apache Software Foundation is, and probably always will be, a work in progress, hopefully will be at-least-good-enough to survive the next unexpected storm, and there have been several of those already, but the only way we ever find that out is when it hits us. Over a relatively long period we have figured out, adopted, borrowed, adapted, had donations of, and thunk out with nothing but our own brains, a whole load of ideas about effective Open Source collaboration, governance, legal shenanigans, marketing, community building, and so on. Things that work well, some that mostly work, and some that are sometimes rubbish, but better than nothing. We write these things down and propagate this good practice amongst projects because it is the bedrock on which our foundation rests, but that doesn’t mean that it can’t change, we correct, adapt and evolve our best practices all the time, this is how we adapt, this is how we have survived and remained relevant in a field that seems to change almost beyond recognition every four or five years. And, being a meritocracy, if you don’t agree with the way things are, if you think it is out of date or ineffective or pointless, don’t complain, stay and fix it. We have another saying which is that "you can scratch your own itch" - don’t be passive, if you care about it, do it.

    The important point about Apache is not that we have rules and committees but that we have these things because they have been shown to help us do the right thing, to help us to live by our principles and to provide a home for Open Source projects that will equip them to survive amongst the commercial sharks in the ocean of the software industry.

  • Finally: Define your own achievements. Whether you are doing it because you need some software, or because, like me, you just found it and it wasn't quite ready, whether you want to make friends, or to learn something new, whether it is because you are being paid to promote your employer's best interest, because you want to explore new ideas, or because you always wanted to write a book, Success at Apache is yours to define. Create your own measure of success and let us achieve it together.


# # #


"Success at Apache" is a new monthly blog series that focuses on the processes behind why The Apache Software Foundation (ASF) "just works". First article: Project Independence https://s.apache.org/CE0V

Friday January 06, 2017

The Apache News Round-up: week ending 6 January 2017

Happy New Year! The Apache community kicks off 2017 with the following activities:

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 18 January 2017. Board calendar and minutes available at http://apache.org/foundation/board/calendar.html

ASF Infrastructure –our distributed team on four continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield smashing performance at 99.92% uptime http://status.apache.org/

ApacheCon™ –the official conference series of The Apache Software Foundation.
 - CFP OPEN: Apache: Big Data and ApacheCon North America 16-18 May 2017/Miami http://apachecon.com/
 - 2016/Seville's session recordings continue to be processed and posted at Feathercast http://feathercast.org

Apache Incubator –projects and communities intending to become fully-fledged projects under the auspices of The Apache Software Foundation do so through the Apache Incubator.
 - Call for Entries --Apache Incubator Logo https://s.apache.org/rFii

Apache Attic –provides process and solutions to make it clear when an Apache project has reached its end of life.
 - Apache DeviceMap retired http://mail-archives.apache.org/mod_mbox/www-announce/201701.mbox/%3CCALGG8z3wZ3iSii15BdgVx6SnfVwVuNFMQD3mQuVOQCqWi5CG9A%40mail.gmail.com%3E

Apache Ant™ –a Java library and command-line tool that helps building software.
 - Apache Ant 1.9.8 and 1.10.0 released http://ant.apache.org/bindownload.cgi

Apache Commons™ JCS –a distributed, versatile caching system.
 - Apache Commons JCS 2.0 released https://commons.apache.org/proper/commons-jcs/download_jcs.cgi

Apache Guacamole –a clientless remote desktop gateway that supports standard protocols like VNC, RDP, and SSH.
 - Apache Guacamole 0.9.10-incubating released http://guacamole.incubator.apache.org/releases/0.9.10-incubating/

Apache log4net™ –a tool to help the programmer output log statements to a variety of output targets.
 - Apache log4net 2.0.7 released https://logging.apache.org/log4net/download_log4net.cgi

Apache OpenNLP™ –a machine learning based toolkit for the processing of natural language text.
 - Apache OpenNLP 1.7.0 released http://opennlp.apache.org/cgi-bin/download.cgi

Apache Tomcat™ –a Web server that is an Open Source software implementation of the Java Servlet, JavaServer Pages, Java Expression Language and Java WebSocket technologies.
 - CVE-2016-8745 Apache Tomcat Information Disclosure http://mail-archives.apache.org/mod_mbox/www-announce/201701.mbox/%3C04ead0cb-c989-1386-0fd1-a51ef80f7b57%40apache.org%3E


Did You Know?

 - Did you know that in 2016 Apache projects comprised 797 Repositories; 205,167 code commits by 3,314 Committers; and 60,327,418 lines changed. https://projects.apache.org/

 - Did you know that over the past year, Apache communities sent 2,003,919 emails by 27,940 authors on 1,127 lists with 789,825 topics. Prolific!

 - Did you know that ASF Infrastructure have upgraded and improved blogs.apache.org? https://blogs.apache.org/infra/entry/blogs-a-o-moved-upgraded


Apache Community Notices:

 - "Success at Apache" is a new monthly blog series that focuses on the processes behind why the ASF "just works". First article: Project Independence https://s.apache.org/CE0V

 - Feedback from The Apache Software Foundation on the Free and Open Source Security Audit (FOSSA) https://s.apache.org/romf

 - ASF Operations Summary - Q2 FY2017 https://s.apache.org/oTOF

 - The list of Apache project-related MeetUps can be found at http://apache.org/events/meetups.html

 - Find out how you can participate with Apache community/projects/activities --opportunities open with Apache HTTP Server, Avro, ComDev (community development), Directory, Incubator, OODT, POI, Syncope, Tika, Trafodion, Zest, and more! https://helpwanted.apache.org/

 - ApacheCon North America and Apache:BigData will be held 16-18 May 2017 in Miami  http://apachecon.com/

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

 - Show your support for Apache with ASF-approved swag from http://www.zazzle.com/featherwearand http://s.apache.org/landsend--all proceeds benefit the ASF! 

= = =

For real-time updates, sign up for Apache-related news by sending mail to announce-subscribe@apache.org and follow @TheASF on Twitter. For a broader spectrum from the Apache community, https://twitter.com/PlanetApache provides an aggregate of Project activities as well as the personal blogs and tweets of select ASF Committers.

# # #

Friday December 30, 2016

The Apache News Round-up: week ending 30 December 2016

It's a wrap! The Apache community's final activities of 2016 include:

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 18 January 2017. Board calendar and minutes available at http://apache.org/foundation/board/calendar.html

ASF Infrastructure –our distributed team on four continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield "three nines" performance at 99.92% uptime http://status.apache.org/

ApacheCon™ –the official conference series of The Apache Software Foundation.
 - CFP OPEN: Apache: Big Data and ApacheCon North America 16-18 May 2017/Miami http://apachecon.com/
 - Session recordings are being processed and posted at Feathercast http://feathercast.org

Apache Commons™ Compress –library that defines a Java API for working with ar, cpio, tar, zip, 7z, arj, dump, gzip, pack200, bzip2, lzma, snappy, Z, xz and deflate files.
 - Apache Commons Compress 1.13 released http://commons.apache.org/proper/commons-compress/download_compress.cgi

Apache HttpComponents™ –a set of HTTP/1.1 and HTTP/2 transport components that can be used to build custom client and server side HTTP services with a minimal footprint.
 - Apache HttpComponents Core 5.0 alpha2 released http://hc.apache.org/downloads.cgi

Apache Knox™ –a REST API Gateway for providing secure access to the data and processing resources of Hadoop clusters.
 - Apache Knox 0.11.0 released http://www.apache.org/dyn/closer.cgi/knox/0.11.0

Apache log4net™ –a tool to help the programmer output log statements to a variety of output targets.
 - Apache log4net 2.0.6 released https://logging.apache.org/log4net/download_log4net.cgi 

Apache NiFi™ –an easy to use, powerful, and reliable system to process and distribute data.
 - Apache NiFi 1.1.1 released https://nifi.apache.org/download.html

Apache Streams (incubating) –unifies a diverse world of digital profiles and online activities into common formats and vocabularies, and makes these datasets accessible across a variety of databases, devices, and platforms for streaming, browsing, search, sharing, and analytics use-cases.
 - Apache Streams 0.4.1-incubating released http://www.apache.org/dyn/closer.cgi/incubator/streams/releases/0.4.1-incubating/


Did You Know?

 - Did you know that 620 individual Members and 5,934 Committers drive 350+ Apache projects and global operations? All volunteer: no days off! http://apache.org/foundation/how-it-works.html

 - Did you know that the top 5 Committers in 2016 were Mark Thomas (3,032 commits), Claus Ibsen (2,890 commits), Gary Gregor (2,004 commits), Colm Ó hÉigeartaigh (1,900 commits), and Jean-Baptiste Onofré (1,825 commits)? http://community.apache.org/committers/

 - Did you know that Apache CloudStack powers large-scale Clouds with tens of thousands of nodes in production? http://cloudstack.apache.org/


Apache Community Notices:

 - "Success at Apache" is a new monthly blog series that focuses on the processes behind why the ASF "just works". First article: Project Independence https://s.apache.org/CE0V

 - Feedback from The Apache Software Foundation on the Free and Open Source Security Audit (FOSSA) https://s.apache.org/romf

 - ASF Operations Summary - Q2 FY2017 https://s.apache.org/oTOF

 - The list of Apache project-related MeetUps can be found at http://apache.org/events/meetups.html

 - Find out how you can participate with Apache community/projects/activities --opportunities open with Apache HTTP Server, Avro, ComDev (community development), Directory, Incubator, OODT, POI, Syncope, Tika, Trafodion, Zest, and more! https://helpwanted.apache.org/

 - ApacheCon North America and Apache:BigData will be held 16-18 May 2017 in Miami  http://apachecon.com/

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

 - Show your support for Apache with ASF-approved swag from http://www.zazzle.com/featherwear and http://s.apache.org/landsend--all proceeds benefit the ASF! 

= = =

For real-time updates, sign up for Apache-related news by sending mail to announce-subscribe@apache.org and follow @TheASF on Twitter. For a broader spectrum from the Apache community, https://twitter.com/PlanetApache provides an aggregate of Project activities as well as the personal blogs and tweets of select ASF Committers.

# # #

Friday December 23, 2016

The Apache News Round-up: week ending 23 December 2016

Happy holidays! The Apache community has worked hard this week on:

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 18 January 2017. Board calendar and minutes available at http://apache.org/foundation/board/calendar.html

ASF Infrastructure –our distributed team on four continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield steady performance at 99.23% uptime http://status.apache.org/

ApacheCon™ –the official conference series of The Apache Software Foundation.
 - CFP OPEN: Apache: Big Data and ApacheCon North America 16-18 May 2017/Miami http://apachecon.com/
 - Session recordings are being processed and posted at Feathercast http://feathercast.org

Apache Allura™ –an Open Source implementation of a software forge, a Web site that manages source code repositories, bug reports, discussions, wiki pages, blogs, and more for any number of individual projects.
 - Apache Allura 1.6.0 released https://allura.apache.org/

Apache Apex™ –an enterprise grade Big Data-in-motion platform that unifies stream and batch processing.
 - Apache Apex Core 3.5.0 released http://apex.apache.org/downloads.html

Apache Edgent (incubating) –a stream processing programming model and lightweight micro-kernel style runtime to execute analytics at devices on the edge or at the gateway.
 - Apache Edgent 1.0.0-incubating released https://edgent.apache.org/docs/downloads.html

Apache Fineract (incubating) –an Open Source system for core banking as a platform.
 - Apache Fineract 0.5.0-incubating released https://dist.apache.org/repos/dist/release/incubator/fineract/0.5.0-incubating/

Apache HTTP Server™ –the world's most popular Web server.
 - Apache HTTP Server 2.4.25 released http://httpd.apache.org/download.cgi

Apache Jackrabbit™ –a fully compliant implementation of the Content Repository for Java(TM) Technology API, version 2.0 (JCR 2.0) as specified in the Java Specification Request 283 (JSR 283).
 - Apache Jackrabbit 2.13.6 and 2.13.7, and Jackrabbit Oak 1.5.16 released http://jackrabbit.apache.org/downloads.html

Apache Kafka™ –a distributed, fault tolerant, publish-subscribe messaging.
 - Apache Kafka 0.10.1.1 released https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.1/kafka-0.10.1.1-src.tgz

Apache Struts™ –an elegant, extensible framework for creating enterprise-ready Java Web applications.
 - Apache Struts 2.5.8 GA released http://struts.apache.org/download.html#struts-ga


Did You Know?

 - Did you know that the top 5 Committers this week were Stefan Bodewig (83 commits), Claus Ibsen (77 commits), Philippe Mouawad (73 commits), Sterling Hughes (51 commits), and Colm Ó hÉigeartaigh (49 commits)? http://www.apache.org/foundation/how-it-works.html#roles

 - Did you know that Greenplum uses Apache Solr and MADlib (incubating) for scalable text analytics? http://lucene.apache.org/solr/ and http://incubator.apache.org/projects/madlib.html

 - Did you know that Apache NetBeans (incubating) began as a student project and has an active community of more than 1.5M users? http://incubator.apache.org/projects/netbeans.html

Apache Community Notices:

 - Introducing "Success at Apache" –a new monthly blog series that focuses on the processes behind why the ASF "just works". First article: Project Independence https://s.apache.org/CE0V

 - Feedback from The Apache Software Foundation on the Free and Open Source Security Audit (FOSSA) https://s.apache.org/romf

 - ASF Operations Summary - Q2 FY2017 https://s.apache.org/oTOF

 - The list of Apache project-related MeetUps can be found at http://apache.org/events/meetups.html

 - Find out how you can participate with Apache community/projects/activities --opportunities open with Apache HTTP Server, Avro, ComDev (community development), Directory, Incubator, OODT, POI, Syncope, Tika, Trafodion, Zest, and more! https://helpwanted.apache.org/

 - ApacheCon North America and Apache:BigData will be held 16-18 May 2017 in Miami  http://apachecon.com/

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

 - Show your support for Apache with ASF-approved swag from http://www.zazzle.com/featherwear and http://s.apache.org/landsend --all proceeds benefit the ASF! 

= = =

For real-time updates, sign up for Apache-related news by sending mail to announce-subscribe@apache.org and follow @TheASF on Twitter. For a broader spectrum from the Apache community, https://twitter.com/PlanetApache provides an aggregate of Project activities as well as the personal blogs and tweets of select ASF Committers.

# # #

Friday December 16, 2016

The Apache News Round-up: week ending 16 December 2016

As we're approaching the holidays, the Apache community has been busy this week on:

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - ASF Operations Summary - Q2 FY2017 https://s.apache.org/oTOF
 - Feedback from The Apache Software Foundation on the Free and Open Source Security Audit (FOSSA) https://s.apache.org/romf
 - Next Board Meeting: 21 December 2016. Board calendar and minutes available at http://apache.org/foundation/board/calendar.html

ASF Infrastructure –our distributed team on four continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield brisk performance at 99.85% uptime http://status.apache.org/

ApacheCon™ –the official conference series of The Apache Software Foundation.
 - CFP OPEN: Apache: Big Data and ApacheCon North America 16-18 May 2017/Miami http://apachecon.com/
 - Session slides + photos available at http://bit.ly/2gTgdYK; recordings are being processed and posted at Feathercast http://feathercast.org

Apache Apex™ –an enterprise-grade native YARN big data-in-motion platform that unifies stream and batch processing.
 - Apache Apex Malhar 3.6.0 released http://apex.apache.org/downloads.html

Apache Commons™ RNG –provides Java implementations of pseudo-random numbers generators.
 - Apache Commons RNG v1.0 released https://commons.apache.org/proper/commons-rng/download_rng.cgi

Apache Ignite™ –a high-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time, orders of magnitude faster than possible with traditional disk-based or flash-based technologies.

 - Apache Ignite 1.8.0 released https://ignite.apache.org/download.cgi

Apache Jackrabbit™ Oak –a scalable, high-performance hierarchical content repository designed for use as the foundation of modern world-class Web sites and other demanding content applications.
 - Apache Jackrabbit Oak 1.5.15 and 1.2.22 released http://jackrabbit.apache.org/downloads.html

Apache Lucy™ –search engine library provides full-text search for a variety of programming languages.
 - Apache Lucy 0.6.1 and Clownfish 0.6.1 released http://lucy.apache.org/download.html

Apache Mynewt (incubating) –a community-driven module OS for constrained, embedded applications.
 - Apache Mynewt 1.0.0-b1-incubating released http://www.apache.org/dyn/closer.lua/incubator/mynewt/apache-mynewt-1.0.0-b1-incubating

Apache Phoenix™ –enables OLTP and operational analytics for Apache Hadoop through SQL support using Apache HBase as its backing store and providing integration with other Apache projects in the ecosystem such as Spark, Hive, Pig, Flume, and MapReduce.
 - Apache Phoenix 4.9 released https://phoenix.apache.org/download.html

Apache Qpid™ Proton –a messaging library for the Advanced Message Queuing Protocol 1.0 (AMQP 1.0, ISO/IEC 19464, http://www.amqp.org).
 - Apache Qpid Proton 0.16.0 and Qpid C++ 1.36.0 released http://qpid.apache.org/download.html

Apache Tomcat™ –an Open Source software implementation of the Java Servlet, JavaServer Pages, Java Unified Expression Language, Java WebSocket and JASPIC technologies.
 - Apache Tomcat 8.5.9 and 9.0.0.M15 released http://tomcat.apache.org/download-80.cgi and http://tomcat.apache.org/download-90.cgi
 - CVE-2016-8745 Apache Tomcat Information Disclosure http://mail-archives.apache.org/mod_mbox/www-announce/201612.mbox/%3C76fe5f99-cc2c-4e48-b669-738f5dae7266%40apache.org%3E


Did You Know?

 - Did you know that recordings from Apache: BigData and ApacheCon Europe/Seville are available at FeatherCast? http://feathercast.apache.org/

 - Did you know that the German National Library of Science and Technology uses Apache Wicket? http://wicket.apache.org/

 - Did you know that Apache MADlib (incubating) can be used for principal component analysis such as image analysis? http://madlib.incubator.apache.org/

Apache Community Notices:

 - Introducing "Success at Apache" –a new monthly blog series that focuses on the processes behind why the ASF "just works". First article: Project Independence https://s.apache.org/CE0V

 - The list of Apache project-related MeetUps can be found at http://apache.org/events/meetups.html

 - Find out how you can participate with Apache community/projects/activities --opportunities open with Apache HTTP Server, Avro, ComDev (community development), Directory, Incubator, OODT, POI, Syncope, Tika, Trafodion, Zest, and more! https://helpwanted.apache.org/

 - ApacheCon North America and Apache:BigData will be held 16-18 May 2017 in Miami  http://apachecon.com/

 - The ASF Q1 FY2017 Report is available at https://s.apache.org/1BsV

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

 - Show your support for Apache with ASF-approved swag fromhttp://www.zazzle.com/featherwearand http://s.apache.org/landsend--all proceeds benefit the ASF! 

= = =

For real-time updates, sign up for Apache-related news by sending mail to announce-subscribe@apache.org and follow @TheASF on Twitter. For a broader spectrum from the Apache community, https://twitter.com/PlanetApache provides an aggregate of Project activities as well as the personal blogs and tweets of select ASF Committers.

# # #

Feedback from The Apache Software Foundation on the Free and Open Source Security Audit (FOSSA)

by Dirk-Willem van Gulik <dirkx(at)apache(punto)org>

December 2016, v1.09

Background

The important role of open source software in key infrastructures was brought to collective attention by two major security vulnerabilities in the core of the internet infrastructure. Heartbleed and Shellshock of 2014 caused significant concern. It made a lot of people realise how important the collective efforts around these open source infrastructures are. And how much key internet infrastructure relies on open source communities. Such as the Apache community.

Two of those people were Julia Reda and Max Andersson; Members of the European Parliament. As a result they proposed (and directed Europe to fund) a pilot project:  the "Free and Open Source Software Audit (FOSSA)" within a larger workstream that was about "€1 million to demonstrate security and freedom are not opposites".

One part of the money was about developing a methodology; the other about actually auditing some widely used open source software. After soliciting votes from the public - two projects "won": KeePass and the Apache Web Server.

Audit Process

The European Commission (easiest thought of as executive part of Europe) commissioned Spanish Aerospace and Defence company Everis to carry out the review on the Apache HTTPD server (and associated APR).  Their first draft had a considerable number of false positives and a fair bit of focus on some of the more arcane build tools (e.g. our libtool that is used on OS/2 where there is no gnu-libtool). At  Apache vulnerabilty scans are most valuable if we see analysis and at least a theory as to why something is vulnerable -- so we then worked with Everis to improve the report. Their final report on Apache HTTPD and APR has since gone live along with the other audits reports and results.

As none of the vulnerabilities found were particularly severe, we did not need to go through a responsible disclosure path; but could post the issues publicly to the developer mailing list.

Feedback on FOSSA

As part of this work, we were also asked for feedback - especially important now that Julia Reda and Max Andersson have managed to secure a recent vote in the the European Parliament for additional budget.

So in the remainder of this post I'll try to outline some of the conflicting forces around a security issue report v.s. a report of a vulnerability.

Security Reports

Infrastructure software needs constant maintenance to accommodate the evolving platforms; and to back port or propagate improvements and new learnings throughout the code. It is not a static piece of code with 'security holes' waiting to be found. `Fixing' a hole without `lifting the helicopter' is not net-positive by definition; in fact it can be negative. For example if a 'fix' makes the code more complex, if it reduces the number of people that understand it, or if it has an adverse effect on systems that use a different CPU architecture, build environment or operating system.

So in general terms, the main metric is whether security overall gets better - and indirectly about optimising efficient use of the available (existing and extra), but always limited, capacity and capabilities of the resources. At any given time there is both a known 1) backlog of deficiencies and known loose ends and 2) a reservoir of unknown issues. Tackling the first will generally make things more secure. Whereas searching in the latter space only makes things more secure if one finds issues that are severe enough to warrant the time spent on the unknown versus the time not spent on the known deficiencies.

To illustrate this with examples; a report from a somewhat outdated automated vulnerability tool often reduces overall security. Time that could be spent on fixing real issues and cleanups is instead spent on dealing with the false positives and minor stuff. The opposite is also true: bringing a verified security issue to us with a modest bit of analysis as to how such is exploitable, is virtually always a straight win. This obviously is even more true for a very severe issue (where it is immediately clear how it is exploitable). 

But it is also true for the case where someone bestows time on us on a small deficiency (e.g. initially found by a tool) - provided they spend significant time and engineering on handing us the 'fix' on a well tested silver platter. And it is even more useful if a class of issues is tackled throughout; with things like updated test cases.

Throughout this it is very important to consider the threat model and what or whom the bad actors are that you are protecting against. This includes questions like: Is it when the server runs in production? Or also during build? What is the attack surface?. This is particularly important when using (modern!) automated scanning tools (even after you laboriously winnow down the 1000's of false positives for the 1 nugget).

The reason for this is that it is common for constructs such as:
  ....
  results = (results_t *) mallocOrDie(sizeof(results_t));
  results->sum = 0;
  for(int i = 0; i < ptr->array_len; i++) {
    results->sum += ptr->array[i];
  ....
to be automatically flagged by (old-fashioned) tools. This is because there is seemingly no error trapping on mallocOrDie() and because there is no bound checking on ptr->array[i]. So in those cases you need to carefully analyse how this code is used; and what assumptions there are in the API; how exposed it is and so on (e.g is len public or private to the API). 

The last thing you want (when the situation is more complex) is to add a whole load of sentinels to the above code. That would make the code harder to maintain, harder to test and introduce things like the risk of a dangling else going unnoticed. As then you've just reduced security by tackling a non-existent issue. It would have been better to focus, for example, on making sure that mallocOrDie() always bombs out reliably when it fails to allocate.

People and Community versus tools

So specifically this means people, rather than tools, spending a lot of time analysing issues are the thing that is most valuable to Open Source communities.

By the time open source infrastructure code sees use in the market that is significant enough for the likes of FOSSA to consider it 'infrastructure and important' by some metric, it is likely that it is reasonably robust and secure.  As it is open source, it has some standing and is probably used by sizeable organisations that care about security or are regulated. Therefore, it has probably seen a fair bit of (automated and manual) security testing. 

In fact, once an open source project has become part of the landscape every security vendor worth their salt will probably test their tools on it - and try to use it as a wonderful (because you are public) example they can talk about in their sales pitches (that is, if they find something).

It also means that the issues that remain tend to be hard; and are more likely to require structural improvements (e.g. hardening an API) and large scale, systematic changes. Which result in totally disproportional amounts of time to be spent on updating test cases, testing and manual validation. As otherwise it would probably already have been done before. To some extent this also applies to automated tooling; we see that modern/complex tools that are hard to run; require a lot of manual work to update their rule bases for false positives or require sizeable investments (such as certain types of fuzzing, code coverage tools, automated condition testing/swaps) are used less often (but thus tend to sometimes yield promising new strains of issues).

Secondly there is the process of impact and the cost of dealing with the report and changes.  Often the report will find a lot of 'low' issues and perhaps one or two serious ones. For the latter it is absolutely warranted to 'light up' the security response of an open source project; and have people rush into action to do triage, fix and follow up with responsible disclosure.

Given that the code is already open source, the same cannot be said for the 'low' issues. Generally anyone (bad actors and good actors) can find these too. So in a lot of cases it is better to work with the community to file these as bug reports; or even better - as simple issues usually have simple non controversial fixes, submit the fixes and associated test cases as contributions. (It is often less work for the finder of the bug to submit a technical patch & test case than to fully write up a nicely formatted PDF report)

Bug Bounties - a Panacea ?

One 'solution' which is getting a lot of media attention is that of bug bounties; where the romantic concept of a lone open source volunteer coder code the internet is replaced by a lone bounty hunter - valiantly searching for holes & getting paid if they shoot first. 

If we review that solution against the needs of large, stable, communities that deal with relatively mature and stable infrastructure code (as opposed to commercial project or new code that is still evolving) we have seen a number of counter-indications stack up:
  • Fees are not high enough for the expert volunteers one would need to be enticed by the fee alone `in bulk'.

    Take the recent Azure-Linux update reporting or the Yahoo issue as examples. 5 to 10k is unlikely to come even close to the actual out of cost of a few weeks to a few months of engineering time at that quality level (or compensating the years invested in training) that was required to find, analyse and report that issue.

  • The same applies for the higher `competition' fees - topping out at 30-100k. In those cases only the first to report gets it. So your actual payment-per-issue found is lower on average; with some 4 to 8 top global teams at this level and with 2 to 4 high-value target events per year - that works out at well below 8k/teammember per year on average.
That in itself has a number of ramifications:
  • The very best people will only engage in this as a hobby and (hence) for personal credit and pride; OR when they work for a vulnerability company that wants the PR and marketing.

BUT that means that it is personal credit & marketing that is the real driving value, not the money itself. So what then happens if we introduce money into this (already credit and marketing driven) situation? 

  • Very large numbers of people without sufficient skill may be tempted --- but then one has to worry about the impact on the open source community: is dealing with reports at that level a better time spend for volunteers than having insiders look for things ? Will time spent on these fixes distract from the important things ?

    Should we ask people to pre-filter; or ask people managing bug hunting programmes to pre-vet or otherwise carry an administrative burden ? (Keep in mind that there are third party bug-hunting programmes for Apache code that the Apache Software Foundation has no control over).
Secondly - we know (from various dissertations and experience) that introducing money into a volunteer arrangement has an impact on group dynamics and how volunteers feel rewarded; or what work they seek to get rewarded for. 

With that - it may be so that:
  • It is likely that `grunt' and `boring' work in the security area will suffer --- `let that be done by paid folks';

  • It fundamentally shifts the non-monetary (and monetary - but not relevant as too low) reward from writing secure/good code and caring/maintaining --- to the negative - finding a flaw in (someone else) code. So feel-good, job-well-done and other feedback cycles now bypass primary production processes (that of writing good code), or at the very least, make that feedback loop involve a bug bounty party.
Finally - in complex/mature code - the class of vulnerabilities that we probably want to get fixed tend to be very costly to fix/find - and any avenue you go down has a high risk of not finding a security issue but a design/quality issue. 

Bug bounty finders, unlike the coding volunteers are NOT incentivised to report/fix these.  

On top of this, they are more likely to go for the higher reward/lower risk kind of niggle stuff. Stuff that, without digging deeper, is likely to cause higher layers of the code to get convoluted and messy. As these groups have no incentive to reduce complexity or fix deeper issues (in fact, if one were cynical - they have every reason to stay clear of such - as it means ripe hunting grounds during periods of drought).

So at some level Bug bounties are about the trade-off between rewarding, paying, a single person versus saddling a community of motivated volunteers with the fallout - not so much of genuine reports; but of everything else.

So ultimately - it is about the risks of what Economists call "Externalisation"; making a cost affects a party who did not choose to incur that cost - or denying that party a choice how to spend their resources most effectively.

Summary and suggestions for the next FOSSA Audits

In summary:
  1. Submitting the results of automated validation (even with some human vetting) is generally a negative contribution to security. 

  2. Submitting a specific detailed vulnerability that includes some sort of analysis as how this could be exploitable is generally a win. 

  3. Broad classes of issues which (perhaps rightly!) give you hits all over the code base are generally only worth the time spent on them if there are additional resources willing to work on the structural fixes, write the test cases and test them on the myriad of platforms and settings -- and if a lot of the analysis and planning for this work has been done prior to submitting the issue (to generally a public mailing list).

    From this it also follows that narrow and specific (and hence more "new" and "unique") is generally more likely to increase overall security; while making public the results of something broad and shallow is at best not going to decrease security.

  4. Lighting up the security apparatus of an open source project is not 'free'. People are volunteers. So consider splitting your issues into: ones that need a responsible disclosure path; and ones that can go straight to the public lists. Keep in mind that, as the code is open source, you generally can err towards the open path a bit - other (bad) actors can run the same tools and processes as you.

  5. Consider raising the bar; rather than report a potential vulnerability - analyse it; have the resources to (help) solve it and support the community with expensive things; such as the human manpower for subsequent regression testing, documentation, unit tests or searching the code for similar issues. 

  6. Security is a process; over very long periods of time. So consider if you can consistently spend resources over long periods on things which are hard to do for (isolated) volunteers. And if it is something like comprehensive fuzzing, code-coverage, condition/exchange testing -  then consider the fact that it is only valuable if it is; a) done over long periods of time and b) comes with a large block of human manpower that do things like analyses of the results and updates of test cases.

  7. Anything that increases complexity is a risk; and may have long term negative consequences. As it may lead to code which is harder to read, harder to maintain or where the pool of people that can maintain it becomes disproportionally smaller. A broad sweeping change that increases complexity may need to be backed by a significant (5.10+ years) commitment of maintenance in order to be safe to implement; especially if the security improvement it brings is modest.

  8. Carefully consider threat model and actors when you are classing things a security hole - especially around APIs.

  9. Carefully consider what type of resources you want to mobilise in the wider community; and what incentivises the people and processes that are most likely to improve the overall security and safety. And take the overall, longterm, health and social patterns of the receiving community into account when there such forces for good are "external".  It is all to easy to in essence to in effect cause a "Denial of Service" style effect; no mater how well intentioned.

  10. World-class expertise is rare; and by extension - the experts are often isolated. Bringing them together for long periods of time in relatively neutral settings gives synergy which is hard to get otherwise. Consider using a JRC or ENISA setting as a base for long term committed efforts. An effort that is perhaps more about strengthening and improving large scale (IT) infrastructures and (consumer) safety - rather than security.

  11. Bug bounties are not the only option. Some open source communities have benefited from "grants" or "stipend"; where a specific issue got tackled or addressed. In some cases, such as in for example Google its Summer of Code - it is focused on relatively young people; and helps train them up; in other cases it gives established experts room for a (few) year(s) to really bottom out some long standing issue.
With respect to the final point - security engineering (and its associated areas; such as privacy, trust and so on) is a "hard" thing to hire; the market generally lacks capacity and capability. Also in Europe. 

While open source its access to `lots of eyeball's does help; it does not magically give us access to a lot of the right eyeballs.

Yet increasing both Capacity and Capability in society does help. And that is a long process that starts early.

# # #

Thursday December 15, 2016

The Apache Software Foundation Operations Summary: August - October 2016

FOUNDATION OPERATIONS SUMMARY

Second Quarter, Fiscal Year 2017 (August-October 2016)

"With hundreds of projects and thousands of committers, the Apache Foundation has found stunning success without knuckling under to the software titans."
--Matt Asay, InfoWorld

> President's Statement: As a newly appointed President, my first priority has been to get a budget in place for the board to approve. Costs still slightly exceed revenue, but we have adequate reserve to cover this.

Focus items for both Brand Management and Fundraising include better tracking and prioritization. In the case of Fundraising, this likely means reaching out beyond the traditional technical sponsors.

The appointment of a paid Infrastructure Administrator is already showing results. Open Infrastructure positions have been backfilled and new hires are being onboarded. Priorities include resolving whether or not GitHub can be used as a master and finding ways to reduce the infrastructure costs per project. Meanwhile, uptime continues to be a point of pride for the infrastructure team. While we remain in a very healthy financial position, it never hurts to take the opportunity to ask for your support. As an individual you can donate to the Foundation (http://www.apache.org/foundation/contributing.html), as a corporation you can become a sponsor (http://www.apache.org/foundation/sponsorship.html).

Events and Community: Since our last quarterly report, we have not held any additional ApacheCon events. We do, however, have one coming up very soon, and another in the beginning stages of planning. 

We will hold Apache Big Data Europe 2016, and ApacheCon Europe 2016, in Seville, Spain, November 14-18th, at the Melia Sevilla hotel. The we will be announcing the schedules for these events mid September. Details about these events may be found on the ApacheCon Website, at http://apachecon.com/ . In 2017, we plan to hold ApacheCon North America in Miami, May 15-19, at the Intercontinental Miami. Details will be published to the ApacheCon Website very soon. Sponsorship opportunities are still available for both events.

Meanwhile, we continue, as a larger community, to plan and attend an enormous number of meetups and other small events. You can see the weekly list of meetups at http://apache.org/events/meetups.html or by searching for your favorite Apache project on meetup.com.

> Committers and Contributions: Over the past quarter, 1,721 contributors committed 48,551 changes that amount to 15,102,280 lines of code across Apache projects. The top 5 contributors during this timeframe are: Mark Thomas (729 commits), Gary Gregory (614 commits), Carsten Ziegeler (546 commits), Shad Storhaug (541 commits), and Maxim Solodovnik (491 commits).

The ASF Secretary processes new Apache Committers' paperwork so that they can continue contributing to our projects. All individuals who are granted write access to the Apache repositories must submit an Individual Contributor License Agreement (ICLA). Corporations that have assigned employees to work on Apache projects as part of an employment agreement may sign a Corporate CLA (CCLA) for contributing intellectual property via the corporation. Individuals or corporations donating a body of existing software or documentation to one of the Apache projects need to execute a formal Software Grant Agreement (SGA) with the ASF. 

During this timeframe, the Secretary processed 281 ICLAs, 17 CCLAs, and 7 Software Grants. The activity of Apache committers, and the community of contributors they serve, can be seen at http://status.apache.org/#commits

> Brand Management: The ASF continues to be at the forefront of what's really a new kind of organization, where our independently governed and distributed volunteer communities are in charge of managing not just their technologies and communities, but their trademarks and their whole brand and presence in the larger world.  We continue to build new educational materials to help our highly technical communities understand the larger implications of managing the brand and outward impact of their projects, including proper trademark maintenance.

The ASF is seen as a leader in trademark and brand policies, and our example is helping other FOSS communities as well as companies better understand how we can work together fairly and productively.  Our community-focused education and policy materials are the best available, and we recently expanded to provide a more generic module on Practical Trademark Law for FOSS projects.  We continue to work on improving education and mentoring for projects to ensure they understand how to best maintain their independent brand and image.

All of the ASF's education and policies around trademark law for Open Source as well as brand management is published online, and we urge project participants and software vendors alike to review and ask us questions about them: http://www.apache.org/foundation/marks/resources

On the registration front, we continue to get some projects who request registration of names or beloved logos in the US and internationally. We continue to exercise financial care with our budget by working with the relevant project communities to detail why registration is important for them to attract new project contributors around the world.

With the continued rise of prominent Apache brands and projects that power more business every year, we look to the many companies that profit from Apache software products to help respect Apache brands.

While many companies continue to properly give credit to our volunteer communities, sadly some companies continue to --or have started to-- take advantage of our non-profit work by unfairly co-opting Apache project brands or by interfering with Apache project governance. Reviewing and correcting these mis-uses is an ongoing effort for the ASF Board, the Brand Management Committee, and all Apache projects.

The Apache Brand Management team welcomes your questions on our private email list: trademarks@apache.org

> Infrastructure: The Infrastructure team has been continuing its work with puppet to create better resilience and repeatable deployment, for the set of machines and VMs under our management. Much of this work has been with the build slaves for our Jenkins and Buildbot systems, where we have added and streamlined the configuration of many new nodes. We continue to decommission our hardware, in favor of third-party hardware hosted in multiple cloud providers around the world.

The team has hired Freddy Barboza Oviedo and Chris Thistlethwaite, who will join the team in November. With Freddy, Chris, and (previously-reported) Greg joining the team this quarter, we hope to better serve the vast number of users of the Foundation infrastructure.

Beyond retiring technical debt and bringing puppet to our services, we continue to work on providing GitHub's toolset to our projects in a way that maintains our community and legal needs. This service will be rolled out incrementally for a limited set of test projects, and is expected to be available to all projects some time in 2017.

We saw 477 issues opened during the quarter, with 416 of those alerady closed. Another 38 issues were closed, leaving us with a net increase of a couple dozen issues. We are hopeful that our increased staffing levels will reverse this trend and provide better service to our users.

During the quarter, the services offered by the Infrastructure team maintained an uptime of 99.75%, beating our goal of 99.50% for critical services and easily beating the goals for less critical services. Our work with puppet and multiple cloud providers has greatly improved our ability to maintain a high level of uptime.

> Financial Statement:


> Fundraising:
 The ASF Fundraising team closes another strong quarter. Four more organization joined our family of sponsors. The growth in the number of sponsors is consistent with the overall growth of the fundation. We continue our efforts to engage with existing and potential sponsors and we are looking forward to more sponsors joining in the following quarters.

The ASF enjoys the support of the same 7 Platinum Sponsors: Cloudera, Facebook, Google, LeaseWeb, Microsoft, Pivotal and Yahoo. With Huawei upgrading to Gold we now benefit from the support of 9 Gold Sponsors: ARM, Bloomberg, Comcast, Hortonworks, HP, Huawei, IBM, ODPi, PhoenixNap and 14 Silver Sponsors: Alibaba Cloud Computing, Budget Direct, Capital One, Cerner, Confluent, InMotion Hosting, iSIGMA, Private Internet Access, Produban, Red Hat, Serenata Flowers Wandisco with the addition of Cash Store and Target, the ASF newest silver sponsors. The number of Bronze sponsors has also increased in the second quarter from 19 to 21 Bronze Sponsors. The number of Infrastructure sponsors remained unchanged, the ASF infra@ team continues to rely on the help and support of: The OSE Open Source Labs, SURFnet, Freie Universitat Berlin, Quenda, PagerDuty, Symantec, No-IP, Bintray, Hotwax Systems, Rackspace and Sonatype.

As we always do, we want to use this opportunity too to express our gratitude to our generous sponsors. Our operations continue uninterrupted because of our sponsors support and for that they deserve our most sincere thanks.

# # #

Report prepared by Sally Khudairi, Vice President Marketing & Publicity, with contributions by Sam Ruby, ASF President; Rich Bowen, Vice President Conferences; Shane Curcuru, Vice President Brand Management; Greg Stein, ASF Infrastructure Administrator; Tom Pappas, ASF Member and Vice President, Finance & Accounting at Virtual, Inc.; and Hadrian Zbarcea, Vice President Fundraising.


For more information, subscribe to the announce@apache.org mailing list and visit http://www.apache.org/, the ASF Blog at http://blogs.apache.org/, and the @TheASF on Twitter.

(c) The Apache Software Foundation 2016.

Friday December 09, 2016

The Apache News Round-up: week ending 9 December 2016

Another brilliant week with the following accomplishments from the Apache community:

ASF Board –management and oversight of the business and affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 21 December 2016. Board calendar and minutes available at http://apache.org/foundation/board/calendar.html

Introducing Success at Apache –a new monthly blog series that focuses on the processes behind why the ASF "just works". - Success at Apache: Project Independence  https://s.apache.org/CE0V

ASF Infrastructure –our distributed team on four continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield "three nines" performance at 99.91% uptime http://status.apache.org/

ApacheCon™ –the official conference series of The Apache Software Foundation.
 - CFP OPEN: Apache: Big Data and ApacheCon North America 16-18 May 2017/Miami http://apachecon.com/
 - Session slides + photos available at http://bit.ly/2gTgdYK; recordings are being processed and posted at Feathercast http://feathercast.org

Apache Community Development –helps those new to the ASF and Apache Projects take their first steps towards being a part of the Apache community.
 - REMINDER TO ASF COMMITTERS: please complete the Apache Community Development Diversity Survey (check your @apache.org email)

Apache Apex™ –an enterprise-grade native YARN big data-in-motion platform that unifies stream and batch processing.
 - Apache Apex Malhar 3.6.0 released http://apex.apache.org/downloads.html

Apache Hive™ –Big Data warehouse software that facilitates querying and managing large datasets residing in distributed storage.
 - Apache Hive 2.1.1 released https://hive.apache.org/downloads.html

Apache Jackrabbit™ –a fully compliant implementation of the Content Repository for Java(TM) Technology API, version 2.0 (JCR 2.0) as specified in the Java Specification Request 283 (JSR 283).
 - Apache Jackrabbit 2.12.6, 2.13.5, and Jackrabbit Oak 1.5.14 released http://jackrabbit.apache.org/downloads.html

Apache NiFi™ MiNiFi –a complementary data collection approach that supplements the core tenets of NiFi in dataflow management, focusing on the collection of data at the source of its creation.
- Apache NiFi MiNiFi 0.1.0 and C++ 0.0.1 released http://nifi.apache.org/minifi/download.html

Apache PDFBox™ –an Open Source Java tool for working with PDF documents.
 - Apache PDFBox 1.8.13 released http://pdfbox.apache.org/download.cgi


Did You Know?

 - Did you know that the following Apache projects are celebrating anniversaries in December? Apache Portable Runtime (16 years); Logging Services (13 years); Cayenne, OFBiz, and Tiles (10 years); Synapse (9 years); Camel (8 years); Aries (6 years); ACE (5 years); Flex and Wink (4 years); Helix (3 years); Falcon and Flink (2 years) --many happy returns! https://projects.apache.org/

 - Did you know that an immersive introduction to the ASF for newcomers is available at the Community Development (ComDev) site? http://community.apache.org/

 - Did you know that PayPal cuts costs tenfold by using continuous integration tools including Apache Aurora and Apache Mesos? http://aurora.apache.org/ and http://mesos.apache.org/


Apache Community Notices:

 - The list of Apache project-related MeetUps can be found at http://apache.org/events/meetups.html

 - Find out how you can participate with Apache community/projects/activities --opportunities open with Apache HTTP Server, Avro, ComDev (community development), Directory, Incubator, OODT, POI, Syncope, Tika, Trafodion, Zest, and more! https://helpwanted.apache.org/

 - ApacheCon North America and Apache:BigData will be held 16-18 May 2017 in Miami  http://apachecon.com/

 - The ASF Q1 FY2017 Report is available at https://s.apache.org/1BsV

 - Are your software solutions Powered by Apache? Download & use our "Powered By" logos http://www.apache.org/foundation/press/kit/#poweredby

 - Show your support for Apache with ASF-approved swag fromhttp://www.zazzle.com/featherwearand http://s.apache.org/landsend--all proceeds benefit the ASF! 

= = =

For real-time updates, sign up for Apache-related news by sending mail to announce-subscribe@apache.org and follow @TheASF on Twitter. For a broader spectrum from the Apache community, https://twitter.com/PlanetApache provides an aggregate of Project activities as well as the personal blogs and tweets of select ASF Committers.

# # #

Monday December 05, 2016

Success at Apache: Project Independence

By Mark Thomas

I've been involved in The Apache Software Foundation (ASF) since 2003. I was using Apache Tomcat at work and I hit a problem that needed a new feature to be implemented. There was already an enhancement request in Bugzilla so I submitted a patch. After some re-work by the project committers, the patch was applied and the feature available in the next release. I enjoy problem solving, so I started to take a look at the other open Tomcat bug reports and my involvement grew from there to include Apache Commons, the Infrastructure Team, the Security Team and, most recently, the Board of Directors to which I was elected in March 2016.

Apache Tomcat has always been at the heart of my involvement and is where I spend most of my time. Tomcat started with a donation to the ASF by Sun in 1999 and, some seven major versions later, the project continues to be very successful. A significant part that success is due to the involvement of a wide range of individuals from different companies. The reason those companies are happy co-operating on Tomcat is because of the importance the ASF places on project independence.

There are many aspects to project independence but, for me, the most important is that committers and Project Management Committee (PMC) members contribute to the project as individuals and do so with the intention of doing what is best for the community as a whole. Some committers contribute in their free time – I did for the first five years or so with Tomcat – and some are allowed /directed to spend time contributing to Apache projects by their employer. However, those committers contributing on their employer's time still need to act in the best interests of the community rather than the best interest of their employer.

To give a specific example, my employer has a product that is built around Apache Tomcat. The sales folks at my employer asked if I could add a feature to this product. The problem was that this feature required access to low-level Tomcat internals in order to implement it effectively. For this to be possible, I would have needed to make some ugly API changes to Tomcat to provide the integration points required. Rather than try and push those changes through, I persuaded my employer that it would better to donate the entire feature to the Apache Tomcat project.

This feature also demonstrates other important elements of a successful ASF project: the ability to make decisions in public and always aiming to achieve community consensus with those decisions. As the development of this new feature progressed, the design evolved as the community reviewed the commits and suggested improvements. This isn't always the quickest way of working but the quality of the end result – both technically but more importantly in terms of community health - more than makes up for that.

The perception of project independence is as important as projects actually being independent. It is a key factor in many projects choosing the ASF as their home so projects need to ensure that the perception agrees with reality.

Things can and do go wrong. With 350 projects it is pretty much a given that there will be a handful of ongoing issues at any given time. For example, there might be an attempt to push a project in a particular direction or to suggest that some external entity controls / leads / manages the project. Typically these are self-corrected by the PMC. Sometimes the PMC needs help to resolve the issue e.g. from V.P. Brand Management or possibly the ASF Board.

Being a board member is often viewed as more significant than it is. I have no more status in Apache Tomcat, Apache Commons or any other project as a board member than I did before my election to the board. I can still have bad ideas and my fellow community members still point it out when it happens. I don't get to always have my way just because I am board member. It is the board as a whole, rather than the individual board members, whose voice carries significant weight. It is fairly rare for any board member to speak on behalf of the board. To give that some context, I've probably done it no more than once a month since joining the board. It is sufficiently rare that board members always include an explicit "on behalf of the board" when speaking for the board rather than as an individual. Sometimes this point isn't appreciated and the views of an individual board member are incorrectly taken to be the views of the board.

The ASF board is also very different to a corporate board. The board manages the Foundation but it is the PMC that manages the project and sets the direction. The board has no role in the technical direction of a project. The board has responsibility for corporate governance, finance, legal etc., but its primary role is monitoring, mentoring and coaching our project communities to help keep them healthy. As part of this, the board reviews all projects on a regular basis. Newly graduated projects are reviewed monthly for typically 3 months before moving to quarterly reviews. The project V.P. (PMC Chair) is an important part of this. They are the eyes and ears of the board. While the board will look for warning signs as part of its regular review, the V.P. has much more in depth knowledge of the project and can flag specific issues early. Where issues are identified, the aim is to get the PMC to self-correct. The board will provide mentoring / coaching / guidance as necessary but it will be the PMC members who do the work to correct the issue.

As an example of the board working with a PMC, earlier this year the V.P. for a particular project became unavailable. The board became concerned because the regular reports were not being produced for the project. In this instance, no-else on the PMC had experience of being a project V.P so the board worked with the PMC to identify a new V.P. and to then mentor the new V.P. as they found their way in their new role.

For the last 17 years, the ASF has provided a home for a large and diverse set of open source projects. Key to this success has been the importance the ASF places on project independence as part of the Apache Way. By continuing to adhere to the principles of the Apache Way, I am confident that the ASF will continue to be successful for another 17 years and a long way beyond.

Calendar

Search

Hot Blogs (today's hits)

Tag Cloud

Categories

Feeds

Links

Navigation