Apache Ranger response to incorrect analyst report on Cloud data security

Introduction

A recent industry analyst report by GigaOm and sponsored by Immuta comparing Apache Ranger to Immuta paints an incorrect picture on the complexities of using Apache Ranger. We believe the report contains a number of errors and inconsistencies. Unfortunately the Apache Ranger Project Management Committee (PMC) was not contacted by the analyst firm during preparation of the report.

We have attempted to contact the authors and members of the research team several times, requesting the opportunity to review the inaccuracies and have them corrected. Despite our many attempts to rectify the misinformation, no-one from the analyst firm responded.

For the benefit of existing and potential users of Apache Ranger, it is important for Apache Ranger PMC to respond to this report with facts.

Use cases

Let us now go through the scenarios covered in the report, and see how the numbers reported change with appropriate use of Apache Ranger to address the requirements.

• Scenario 1b: Mask All PII Data

• lists 2 policy changes in Immuta vs 5 in Apache Ranger. In fact, only one Apache Ranger policy would be needed to address this requirement. 

• Shows author's lack of understanding of Apache Ranger policy model. Series of steps to allow/deny/deny-exception listed are applicable only for an access policy but not for a masking policy. Also, in access policies, allow/deny/deny-exception can be replaced by a switch named denyAllElse, as shown in the image below.

• With use of user-groups or roles, a time-tested best practice followed universally by access control systems, this requirement can be met by a single Apache Ranger policy, as shown below.
  Masking policy:

Access policy:

• Scenario 1c: Allow Email Domains Through the Masking Policy

• lists 2 policy changes in Immuta vs 5 in Apache Ranger. In fact, only one Apache Ranger masking policy would be needed to address this requirement. Same as the previous scenario.

• Claim: Apache Ranger does not have a regular expression masking policy

• Truth: instead of building a virtualization layer that can introduce significant complexities and performance penalties, Apache Ranger uses native capabilities of the data processing application to perform masking and filtering. Given regular expressions are supported by such applications, it will be simpler to create a custom expression to suit your needs like email address, account numbers, credit card numbers; importantly without having to drag security software vendor.

• Scenario 1d: Add Two Users Access to All PII Data

• lists 1 policy change in Immuta vs 4 in Apache Ranger. However, the following suggests that each user must be updated in Immuta UI to add necessary attributes. Wouldn't the number of steps be as large as the number of users?

• Added the AuthorizedSensitiveData > All attribute to each user in the Immuta UI.

• counts 4 policy changes in Apache Ranger policies, while the only change needed is to add users (2 or 200 users!) to a group or role. No policy changes are needed if time tested best practices are followed - by referencing groups or roles in policies instead of individual users.

• Scenario 2a: Share Data With Managers

• lists 1 policy change in Immuta vs 101 in Apache Ranger. With use of lookup tables, which is a common practice in enterprises, the requirement can be met with a single row-filter policy in Apache Ranger.

ss_store_sk in (select store_id from store_authorization where user_name=current_user())

• Scenario 2b: Merging Groups

• lists 0 policy change in Immuta vs 1 in Apache Ranger. This is the same as the previous scenario, where the author chose to not follow common practice of using lookup tables. With use of a lookup table, as detailed above, no policy changes will be needed in Apache Ranger.

• Scenario 2c: Share Additional Data With Managers

• lists 0 policy changes in Immuta vs 102 in Apache Ranger. Once again, with use of a lookup table, only 2 policies would be required in Apache Ranger:

table store:
s_store_sk in (select store_id from store_authorization where user_name=current_user())

table store_returns:
sr_store_sk in (select store_id from store_authorization where user_name=current_user())

• Scenario 2d: Reorganize Managers Into Regions

• lists 0 policy changes in Immuta vs 40 in Apache Ranger. Same as previous scenarios - with use of a lookup table, no policy changes will be needed in Apache Ranger.

• Scenario 2e: Restrict Data Access to Specific Countries

• lists 1 policy change in Immuta vs 71 in Apache Ranger. With use of a lookup table, only one row-filter policy is needed in Apache Ranger.

• Scenario 2f: Grant New User Group Access to All Rows by Default

• lists 0 policy change in Immuta vs 30 in Apache Ranger. With use of a lookup table, no additional policy would be needed in Apache Ranger.

• Scenario 2g: Apply Policies to a Derived Data Mart

• lists 0 policy changes in Immuta vs 140 in Apache Ranger for the addition of 15 tables. With Apache Ranger, new tables can either be added to existing policies, or new policies can be created. It will require 15 policy updates in Apache Ranger - not 140 as claimed by the author. Also, no details on the changes to be done in Immuta (other than ‘0 policy changes’) are provided.

• Scenario 3a: "AND" logic policy

• says "unable to meet requirement" in Apache Ranger - which is incorrect. The author does suggest a good approach to meet this requirement in Apache Ranger - by creating a role with users who are both the groups, and referencing this role in policies. However, the point about Apache Ranger not supporting policies based on a user belonging to multiple groups is correct. However, this can easily be addressed with a custom condition extension. If there is enough interest from the user community, an enhancement to support this condition out of the box would be considered.

• Scenario 3b: Conditional Policies

• says "unable to meet requirement" in Apache Ranger - which is incorrect. As mentioned earlier, Apache Ranger leverages expressions supported by underlying data processing engine for masking and row-filtering. The requirement can easily be met with following expression in the masking policy:
  
  CASE WHEN (extract(year FROM current_date()) - birth_year) > 16) THEN {col} ELSE NULL END

There is no need to create views as suggested in the report.

• Scenario 3c: Minimization Policies

• as mentioned in the report Apache Ranger doesn't support policies to limit the number of records accessed. If there is enough interest from the user community, this enhancement would be considered.

• Scenario 3d: De-Identification Policies

• Says “unable to meet requirement” in Apache Ranger - which is incorrect. While Apache Ranger doesn’t talk about k-anonymity directly, the requirements can be implemented using Apache Ranger data masking policies - by setting up appropriate masking expressions for columns.

• for columns that require NULL value to be returned, setup a mask policy with type as MASK_NULL

• for columns that require a constant value, setup a mask policy with type as CONSTANT and specify desired value - like “NONE”

• for columns that require a ‘generalized’ value based on the existing value of the column, use custom expressions as shown below. This does require analyzing the table to arrive at generalized values:
  CASE WHEN {col} < 20 THEN 16
      WHEN {col} BETWEEN 20 AND 29 THEN 26
      WHEN {col} BETWEEN 30 AND 39 THEN 36
      WHEN {col} BETWEEN 40 AND 49 THEN 46
      WHEN {col} BETWEEN 50 AND 59 THEN 56
      WHEN {col} BETWEEN 60 AND 69 THEN 66
      WHEN {col} BETWEEN 70 AND 79 THEN 76
      WHEN {col} BETWEEN 80 AND 89 THEN 86
      WHEN {col} BETWEEN 90 AND 99 THEN 96
      ELSE 106
  END

What the report doesn't talk about?

It is important to take note of what the report doesn’t talk about. For example:

Extendability: Apache Ranger’s open policy model and plugin architecture enable extending access control to other applications, including custom applications within an enterprise.

Wider acceptance of Apache Ranger by major cloud vendors like AWS, Azure, GCP; and availability of support from seasoned industry experts who continue to contribute to Apache Ranger and extend its reach.

Performance: Apache Ranger policy-engine is highly optimized for performance, which results in only a very small overhead (mostly around 1 millisecond) to authorize accesses; and importantly, there are no overheads in the data access path.

Apache Ranger features like security zones that allow different sets of policies to be applied to data in landing, staging, temp, production zones. A security zone can consist of resources across applications, for example: S3 buckets/paths, Solr collections, Snowflake tables, Presto catalogs/schemas/tables, Trino catalogs/schemas/tables, Apache Kafka topics, Synapse database/schemas/tables.

Posted at 06:41PM Sep 21, 2021 by Madhan in General  |   | 

The Apache News Round-up: week ending 17 September 2021

We're wrapping up another great week with the following activities from the Apache community:

Success at Apache

This series focuses on the people and processes behind why the ASF "just works." The most recent entry is  "From Mentee to PMC" by Ephraim Anierobi. 

ASF Annual Report

The ASF annual report is a look back at our many achievements during the 2021 Fiscal Year.

• Press release: The Apache Software Foundation Announces Annual Report for 2021 Fiscal Year
• ASF FY2021 Annual Report (PDF)

ApacheCon™ – the ASF's official global conference series, bringing Tomorrow's Technology Today since 1998. ApacheCon Asia was held online 6-8 August; ApacheCon@Home is coming up next on 21-23 September:
 - The Apache® Software Foundation Announces Program for ApacheCon@Home 2021
 - Learn all about ApacheCon with Rich Bowen and Swapnil M Mane
 - Learn about the Community Track from Sharan Foga and Swapnil M Mane 
 - Register for ApacheCon@Home 

ASF Infrastructure – our distributed team on three continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield uptime at 99.99%. Performance checks across 50 different service components spread over more than 250 machines in data centers around the world. View the Apache Infrastructure Uptime site to see the most recent averages.

Apache Code Snapshot – Over the past week, 356 Apache Committers changed 2,986,797 lines of code over 3,104 commits. Top 5 contributors, in order, are: Alex Heneveld, Andrea Cosentino, Stephen Mallette, Andi Huber, and Claus Ibsen.    

Apache Project Announcements – the latest updates by category.

Content --
 - Apache PDFBox 3.0.0-alpha2 released 
 - Apache Any23 2.5 released 
   -- CVE-2021-38555: An XML external entity (XXE) injection vulnerability exists in StreamUtils.java
   -- CVE-2021-40146: A Remote Code Execution (RCE) vulnerability exists in YAMLExtractor.java

Cloud Computing --
 - Apache jclouds 2.4.0 released 

Integration --
 - Apache Camel 3.11.2 (LTS) released 

IoT --
 - Apache IoTDB 0.12.2 released 

Libraries --
 - Apache Log4j Kotlin API 1.1.0 released 
 - Apache Commons RNG 1.4 released 
 - Apache Jena CVE-2021-39239: XML External Entity (XXE) vulnerability

Observability --
 - Apache SkyWalking Python Agent 0.7.0 and Satellite 0.2.0 released 

Search --
 - Apache Solr Operator v0.4.0 released 

Security Framework --
 - Apache Shiro CVE-2021-41303: Before 1.8.0, when using Apache Shiro with Spring Boot, a specially crafted HTTP request may cause an authentication bypass

Servers --
 - Apache Tomcat 8.5.71, 9.0.53, 10.0.11, and 10.1.0-M5 (alpha) released 
   -- CVE-2021-41079: Denial of Service
 - Apache HttpComponents Core 5.2-alpha1 released 

Web Frameworks -
 - Apache Wicket 9.5.0 released 

Did You Know?

 - Did you know that NASA JPL uses Apache Kafka to enable real-time data feeds from Mars?

 - Did you know that ASF Infrastructure's uptime during FY2021 was 99.75%?

 - Did you know that Shopify uses Apache Flink to analyze 10 TB+ of data? 

Apache Community Notices

- Watch "Trillions and Trillions Served", the documentary on the ASF 1) full feature [49 min] 2) "Apache Everywhere" [6 min] 3) "Why Apache" [2.5 min] 4) “Apache Innovation” [40 min] 

 - The Apache Month in Review: August 2021 

 - The Apache Way to Sustainable Open Source Success 

 - Foundation Reports and Statements

 - Presentations from ApacheCon Asia are available on YouTube

 - "Success at Apache" focuses on the people and processes behind why the ASF "just works." 

Posted at 02:36PM Sep 20, 2021 by Swapnil M Mane in Newsletter  |   | 

Success at Apache: from Mentee to PMC

by Ephraim Anierobi

This post is about how I became a committer and a Project Management Committee (PMC) member of Apache Airflow, and provides guidance to those new to programming, are new to contributing to open-source projects, and want to become committers and PMC members in their respective Apache projects.

About a year and a half after changing my career from electrical engineering to software development, I became a committer and a Project Management Committee member of Apache Airflow. Becoming a committer and a PMC member is a reward and a kind of validation that you are on the right part of your journey.

On February 16, 2021, I accepted an invitation to become a committer in Apache Airflow. It came as a surprise, as I was not expecting it. Six months down the line, I received another surprise invitation to become a PMC member in Apache Airflow.

These are impressive feats for me because before contributing to Apache Airflow, I didn't have experience working with other programmers. I was making websites and taught a few friends of mine how to make their own. I didn't have a mentor, and no one has ever seen my code to advise whether to continue on my journey or drop the idea of becoming a programmer.

While I desired to work with experienced programmers to improve my skills, I feared people seeing my code would talk me down. I almost gave up on my journey only to come across an Outreachy post on Twitter looking for interns for open source projects. Outreachy is a tech diversity program that provides three months of paid, remote internships to people underrepresented in tech.

I was ready to change my career and was looking for mentorship, but couldn't find an internship that could help me get started in my journey. In Nigeria where I'm living, your location affects your chances of getting an entry-level job. I was not close to the major cities. 

So I applied for an internship through Outreachy. 

There are two application processes. The initial application involves explaining your background and why you should be accepted into the program. You must pass the initial application before you could proceed to the next. The second application process (called the contribution period) is where you choose an open source project that matches your skill sets and then contribute to it. You must have some minimum contributions before you could be accepted.

That was how I found Apache Airflow.

You could imagine the joy I had when I was accepted into the program.

Here are things I did which I believe would help you in your journey to becoming an Apache committer and a PMC member.

Asking Questions

Asking questions is the fastest way to learn. Don't be afraid to ask questions if you do not understand something. I ask questions a lot and I always get answers, but I didn't start by asking questions: I made 40 commits to the repository without understanding what Airflow does. It was not until I joined my new employer Astronomer that I learned what DAG is and what a data pipeline is. Now I can easily reproduce issues following someone's descriptions. I wish I had asked questions earlier --I could have had more experience by now!

Start small

If you are like me, with little experience, start contributing from the minor issues. Find good first issues and work on them. You don't have to wait to contribute a large change before contributing.

While working on the REST API project, which I got hired by Outreachy to do, I was looking at the codebase. I started with Airflow providers because it was easy for me to understand. There were so many requests about providers at the time and I started looking into it, reading the code base, and helping with the providers. I didn't go into the core straight up; I avoided it. My first PR was on simple database migration during the Outreachy contribution period.

Refactor codes

Airflow is complex. Till now, I'm still learning it. Just last week I learned about how the execution date works. I know there are a lot of other things I have not understood very well but refactoring helped me to understand a lot.

When I was to work in the scheduler, I found the file was so large that I went back and forth without progress. I worked on separating the files and I'm glad I did because after that I could contribute. I recommend refactoring code but do not go into large refactoring. A little at a time, with the hope to understand the project. Avoid the core of the project if you are just starting.

Issues

One thing about issues is that most reporters would tell you how to reproduce them. Most times, you would find that the issue is quite easy to fix. I usually jump on those and fix them. Other times, I had to contact my superiors before I could fix it.

Looking at reported issues gives an added advantage that you could learn how the software works in the real world. Try to reproduce as many issues as possible. It adds to your knowledge.

Pull Requests

Here's where you can learn a great deal. I start my day by looking at the PRs. Most PRs link to issues. I read the issues and study PRs. I must admit that some of these PRs are just too complex for me. If I don't understand it, sometimes I ask questions, other times I go to the next PR. When I jump to the next PR, I record the topic that made me jump to the next and plan on reading about it some other time.

When you make a PR, ask for reviews in the community channel of communication. Airflow uses Slack and the mailing list for communications. You should ask for reviews in the slack channel and not the mailing list. The reviews not only give information on how to fix the problem but also teach you best practices in programming.

Culture

The ASF has a code of conduct that covers the Foundations activities as well as the projects. Read it first.

Among many other things, you would learn in Apache Airflow is communication. How to communicate with people in a civil manner. Spend time reading PR reviews, you will learn a lot and especially how to ask people to make changes to their code.

Conclusion

You don't have to wait for an invitation to contribute to an Apache project. You don't have to become an Outreachy intern to get involved with something you're interested in.

Don't be afraid to make a PR because nobody will penalize you if you're wrong. I know the feeling that people may think you are not good enough, forget it, they know you are new to the field and if you are thinking that they don't know your level in the language, forget it too, they know you are still a junior because it says so in your code. I can't count how many times I have had code reviews that showed me a better way to implement the code. Be open-minded, make mistakes, and excel.

Ephraim Anierobi started to work on the Apache Airflow project as an Outreachy Intern in May 2020. He became a committer in February 2021 and a member of the Apache Airflow Project Management Committee (PMC) in August 2021. He is a software engineer at Astronomer.

Posted at 02:32PM Sep 16, 2021 by Sally Khudairi in SuccessAtApache  |   | 

The Apache News Round-up: week ending 10 September 2021

We're wrapping up another great week with the following activities from the Apache community:

ASF Annual Report – a look back at our many achievements during the 2021 Fiscal Year
 - Press release https://s.apache.org/FY2021AnnualReport-pressrelease
 - Full report https://s.apache.org/FY2021AnnualReport

ApacheCon™ – the ASF's official global conference series, bringing Tomorrow's Technology Today since 1998. ApacheCon Asia was held online 6-8 August; ApacheCon@Home is coming up next on 21-23 September:
 - The Apache® Software Foundation Announces Program for ApacheCon@Home 2021 https://s.apache.org/ACHome2021
 - Learn all about ApacheCon with Rich Bowen and Swapnil M Mane https://youtu.be/m_c7NJ5yMOg
 - Learn about the Community Track from Sharan Foga and Swapnil M Mane https://youtu.be/8cZF-gaE3a4
 - Register for ApacheCon@Home https://www.apachecon.com/acah2021/

ASF Infrastructure – our distributed team on three continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield uptime at 100.00%. Performance checks across 50 different service components spread over more than 250 machines in data centers around the world. http://www.apache.org/uptime/

Apache Code Snapshot – Over the past week, 326 Apache Committers changed 9,187,985 lines of code over 2,985 commits. Top 5 contributors, in order, are: Harikrishna Patnala, Andi Huber, Yann Ylavic, Andrea Cosentino, and Benoit Tellier.     

Apache Project Announcements – the latest updates by category.

APIs --
 - Apache APISIX Go Plugin Runner 0.2.0 released https://apisix.apache.org/

Application Servers --
 - Apache Geronimo Arthur 1.0.3 released https://geronimo.apache.org/arthur

Content --
 - Apache Jackrabbit 2.14.10 released http://jackrabbit.apache.org/

Workflow --
 - Apache DolphinScheduler 1.3.8 released https://dolphinscheduler.apache.org/
 - Apache Airflow CVE-2021-38540: Variable Import endpoint missed authentication check https://s.apache.org/88ww5

Did You Know?

 - Did you know that ByteDance uses Apache Hudi to build exabyte-scale data lakes for services such as TikTok? http://hudi.apache.org/

 - Did you know that the Netherlands and Japan Pulsar MeetUp groups are having meetups in September? https://pulsar.apache.org/en/events/

 - Did you know that Kafka Summit will be held online and free of charge 14-15 September? http://kafka.apache.org/events

Apache Community Notices

- Watch "Trillions and Trillions Served", the documentary on the ASF 1) full feature [49 min] https://s.apache.org/Trillions-Feature 2) "Apache Everywhere" [6 min] https://s.apache.org/ApacheEverywhere 3) "Why Apache" [2.5 min] https://s.apache.org/ASF-Trillions-WhyApache 4) “Apache Innovation” [40 min] https://s.apache.org/ApacheInnovation

 - The Apache Month in Review: August 2021 https://s.apache.org/August2021

 - The Apache Way to Sustainable Open Source Success https://s.apache.org/GhnI

 - Foundation Reports and Statements http://www.apache.org/foundation/reports.html

 - Presentations from ApacheCon Asia are available at https://s.apache.org/ApacheConAsia2021-talks

 - "Success at Apache" focuses on the people and processes behind why the ASF "just works". https://blogs.apache.org/foundation/category/SuccessAtApache

Posted at 12:16PM Sep 10, 2021 by Swapnil M Mane in Newsletter  |   | 

The Apache News Round-up: week ending 3 September 2021

Welcome, September --we're opening the month with another great week. Here's what the Apache community has been up to:

ASF Annual Report – a look back at our many achievements during the 2021 Fiscal Year
 - Press release https://s.apache.org/FY2021AnnualReport-pressrelease
 - Full report https://s.apache.org/FY2021AnnualReport

Apache Month in Review – a round-up of our Round-ups and other newsworthy bits over the past month.
 - August Month in Review https://s.apache.org/August2021

ASF Board – management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 15 September 2021. Board calendar and minutes https://apache.org/foundation/board/calendar.html

ApacheCon™ – the ASF's official global conference series, bringing Tomorrow's Technology Today since 1998. ApacheCon Asia was held online 6-8 August; ApacheCon@Home is coming up next on 21-23 September:
 - The Apache® Software Foundation Announces Program for ApacheCon@Home 2021 https://s.apache.org/ACHome2021
 - Learn about the Community Track from Sharan Foga and Swapnil M Mane https://youtu.be/8cZF-gaE3a4
 - Register for ApacheCon@Home https://www.apachecon.com/acah2021/

ASF Infrastructure – our distributed team on three continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield uptime at 100.00%. Performance checks across 50 different service components spread over more than 250 machines in data centers around the world. http://www.apache.org/uptime/

Apache Code Snapshot – Over the past week, 340 Apache Committers changed 2,033,185 lines of code over 3,224 commits. Top 5 contributors, in order, are: Jean-Baptiste Onofré, Mark Thomas, Andrea Cosentino, Andi Huber, Harikrishna Patnala, and Albumen Kevin.         

Apache Project Announcements – the latest updates by category.

APIs --
 - Apache APISIX 2.9 released https://apisix.apache.org/

Big Data --
 - The Apache Drill Project Announces Apache® DrillTM v1.19 Milestone Release https://s.apache.org/bfhy6
 - Apache Qpid Broker-J 8.0.6 and Proton-J 0.33.9 released https://qpid.apache.org/
 - Apache Hudi 0.9.0 released https://hudi.apache.org/
 - Apache Zeppelin CVE-2021-36090: Bash command injection in spark interpreter https://s.apache.org/njaui
   -- CVE-2020-13929: Notebook permissions bypass https://s.apache.org/tx1s1
   -- CVE-2021-27578: Cross Site Scripting in markdown interpreter https://s.apache.org/701t0

Content --
 - Apache Tika 2.1.0 released https://tika.apache.org/

Servers --
 - Apache Tomcat Native 1.2.31 released https://tomcat.apache.org/

Did You Know?

 - Did you know that the following projects are celebrating anniversaries in September? Congratulations to Apache ServiceMix (14 years); Hive, Pig, and Shiro (11 years); Airavata, Bigtop, and SIS (9 years); Curator (8 years); Storm (7 years); Yetus (6 years); RocketMQ and Royale (4 years); Pulsar (3 years); Rya (2 years); IoTDB (1 year) https://projects.apache.org/committees.html?date

 - Did you know that Apache Cordova will be archiving all older translated documentation? https://cordova.apache.org/announcements/2021/08/25/translations.html

 - Did you know that Flink Forward will be held 26-27 October? http://flink.apache.org/

Apache Community Notices

- Watch "Trillions and Trillions Served", the documentary on the ASF 1) full feature [49 min] https://s.apache.org/Trillions-Feature 2) "Apache Everywhere" [6 min] https://s.apache.org/ApacheEverywhere 3) "Why Apache" [2.5 min] https://s.apache.org/ASF-Trillions-WhyApache 4) “Apache Innovation” [40 min] https://s.apache.org/ApacheInnovation

 - The Apache Way to Sustainable Open Source Success https://s.apache.org/GhnI

 - Foundation Reports and Statements http://www.apache.org/foundation/reports.html

 - Presentations from ApacheCon Asia are available at https://s.apache.org/ApacheConAsia2021-talks

 - "Success at Apache" focuses on the people and processes behind why the ASF "just works". https://blogs.apache.org/foundation/category/SuccessAtApache

Posted at 11:15AM Sep 03, 2021 by Swapnil M Mane in Newsletter  |   | 

Apache Month in Review: August 2021

Welcome to the latest monthly overview of events from the Apache community. Here's a summary of what happened in August:

New this month --

- The Apache Software Foundation Announces Apache® Pinot™ as a Top-Level Project https://s.apache.org/ft8p6

- The Apache® Software Foundation Announces Program for ApacheCon@Home 2021 https://s.apache.org/ACHome2021

- The Apache Drill Project Announces Apache® Drill™ v1.19 Milestone Release https://s.apache.org/bfhy6

- The Apache® Software Foundation Announces Annual Report for 2021 Fiscal Year https://s.apache.org/FY2021AnnualReport-pressrelease

- Apache Month in Review: July 2021 https://s.apache.org/July2021 + Video highlights https://youtu.be/KIYB1g6SKhg

Important Dates --

- Next Board Meeting: 15 September 2021. Board calendar and minutes http://apache.org/foundation/board/calendar.html

- ApacheCon™ --the ASF's official global conference series, bringing Tomorrow's Technology Today since 1998-- is being held twice in 2021:
  - UPCOMING: Register for ApacheCon@Home - 21-23 September https://www.apachecon.com/acah2021/
    -- Learn about the Community Track from Sharan Foga and Swapnil M Mane https://youtu.be/8cZF-gaE3a4
  - Completed: ApacheCon Asia - 6-8 August --presentations available at https://s.apache.org/37n3z

Infrastructure --

In August, 764 Apache Committers changed 15,185,996 lines of code over 17,295 commits. The Committers with the top 5 highest contributions, in order, were: Claus Ibsen, Alex Herbert, Andrea Cosentino, Harikrishna Patnala, and Kaxil Naik.  

Project Releases and Updates --

New releases from Apache ActiveMQ (Messaging); Airflow (Workflow); APISIX (APIs); Camel (Integration); Commons (Libraries); EventMesh (incubating; Eventing); Flink (Big Data); Geode (Databases); Hop (incubating; Orchestration); Jackrabbit (Content); NiFi (Big Data); OFBiz (ERP/Enterprise Resource Planning); Portable Runtime (Libraries); Pulsar (Messaging); Qpid (Messaging); Roller (Content); ServiceComb (Libraries); Teaclave (incubating; Confidential Computing); Tika (Content); Tomcat (Servers); Traffic Server (Servers).

Apache Project Anniversaries in August: jUDDI (11 years); Any23, Lucene.Net, and Oozie (9 years); Ignite, Serf, and Usergrid (6 years); HAWQ (3 years). Many happy returns!

The Apache Incubator is the primary entry path for projects wishing to become an official part of the ASF. Linkis (Middleware) entered the Apache Incubator in August. More than three dozen projects are currently undergoing development in the Apache Incubator http://incubator.apache.org/ .

# # #

To see our Weekly News Round-ups (published every Friday), visit https://blogs.apache.org/foundation/ and click on the calendar or hop directly to https://blogs.apache.org/foundation/category/Newsletter . For real-time updates, sign up for Apache-related news by sending mail to announce-subscribe@apache.org and follow @TheASF on Twitter. We appreciate your support!

Posted at 02:11PM Sep 01, 2021 by Sally Khudairi in Newsletter  |   | 

The Apache® Software Foundation Announces Annual Report for 2021 Fiscal Year

World's largest Open Source foundation's community rallies during pandemic; uptick in project activity, participation, and sponsor support advances Foundation to continue to provide $22B+ worth of software to the public-at-large at 100% no cost.

Wilmington, DE —31 August 2021— The Apache® Software Foundation (ASF), the world’s largest Open Source foundation, announced today the availability of the annual report for its 2021 fiscal year (1 May 2020 – 30 April 2021).

The all-volunteer ASF stewards 227M+ lines of code —valued conservatively at more than $22B (constructive cost model - CoCoMo)— all available to the public at 100% no cost. Apache software is used in every Internet-connected country on the planet, and is integral to nearly every aspect of modern computing.

FY2021 highlights include:

1. 40 new individual Members elected, totalling 853 
2. Exceeded 8,200 individual Committers
3. 200 Project Management Committees overseeing 351 Apache Projects, plus dozens of sub-projects and initiatives
4. 14 Top-Level Projects graduated from the Apache Incubator
5. 35 projects (a.k.a. “podlings”) undergoing development in the Apache Incubator
6. Top 5 most active Apache Projects accessed: Kafka, Hadoop, ZooKeeper, POI, Logging 
7. Top 5 Apache Project repositories by commits: Camel, Flink, Airflow, Lucene-Solr, NuttX (incubating)
8. Top 5 most visited Apache Projects on GitHub: Spark, Flink, Kafka, Arrow, Beam
9. 17,758 authors sent 2,184,671 emails on 780,274 topics
10. Top 5 Apache Project user and developer email lists by activity: Flink (user), Tomcat (developer), James (developer), Flink (developer), Kafka (developer)  
11. 17,000+ emails sent to ASF Security team
12. 3,058 Committers changed 134,517,884 lines of code over 258,860 commits
13. 672 Individual Contributor License Agreements signed
14. 23 Corporate Contributor License Agreements signed
15. 32 Software Grant Agreements executed
16. ASF's seven-member Infrastructure team on three continents supports all Apache projects, initiatives, and their communities across ~200 individual machines, 1,400+ repositories, 5-6PB in traffic annually, ~75M downloads per month, and 2-3M daily emails on 2,000+ lists. Average uptime in FY2021 was 99.75%
17. New Infrastructure-developed tools and services enable Apache Projects and their communities to self-administer numerous features around their code repositories
18. Fundraising yielded a positive net income, exceeding FY2021 targets
19. Foundation operations supported by contributions from 9 Platinum Sponsors, 10 Gold Sponsors, 8 Silver Sponsors, 30 Bronze Sponsors, 10 Platinum Targeted Sponsors, 5 Gold Targeted Sponsors, 3 Silver Targeted Sponsors, 12 Bronze Targeted Sponsors, and more than 630 individual donors;
20. Less than 10% of income spent on overhead
21. Published new Website dedicated to data privacy https://privacy.apache.org
22. Produced and released "Trillions and Trillions Served" documentary series 
23. Launched "Inside Infra" interview series with members of ASF Infrastructure team
24. Held first ApacheCon@Home event online to thousands of attendees from around the world
25. Advanced Diversity & Inclusion goals, including securing internships, conducting Community Survey and User Experience Research, and easing barriers to entry for contributors from underrepresented groups
26. ASF was a mentoring organization in Google Summer of Code for the 16th consecutive year
    
    

The full report is available online at https://s.apache.org/FY2021AnnualReport

About The Apache Software Foundation (ASF)
Established in 1999, The Apache Software Foundation (ASF) is the world’s largest Open Source foundation, stewarding 227M+ lines of code and providing more than $22B+ worth of software to the public at 100% no cost. The ASF’s all-volunteer community grew from 21 original founders overseeing the Apache HTTP Server to 850+ individual Members and 206 Project Management Committees who successfully lead 350+ Apache projects and initiatives in collaboration with 8,200+ Committers through the ASF’s meritocratic process known as "The Apache Way". Apache software is integral to nearly every end user computing device, from laptops to tablets to mobile devices across enterprises and mission-critical applications. Apache projects power most of the Internet, manage exabytes of data, execute teraflops of operations, and store billions of objects in virtually every industry. The commercially-friendly and permissive Apache License v2 is an Open Source industry standard, helping launch billion dollar corporations and benefiting countless users worldwide. The ASF is a US 501(c)(3) not-for-profit charitable organization funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, Amazon Web Services, Anonymous, ARM, Baidu, Bloomberg, Budget Direct, Capital One, Cloudera, Comcast, Facebook, Google, Handshake, Huawei, IBM, Indeed, Inspur, Leaseweb, Pineapple Fund, Private Internet Access, Red Hat, Target, Tencent, Union Investment, Verizon Media, and Workday. For more information, visit http://apache.org/  and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Arrow", "Apache Arrow", "AsterixDB", "Apache AsterixDB", "Beam", "Apache Beam", "Camel", "Apache Camel", "Cordova", "Apache Cordova", "Flex", "Apache Flex", "Flink", "Apache Flink", "Geode", "Apache Geode", "Apache GitBox", "Hadoop", "Apache Hadoop", "HBase", "Apache HBase", "Apache HTTP Server", "Ignite", "Apache Ignite", "Kafka", "Apache Kafka", "Lucene Solr", "Apache Lucene Solr", "Mynewt", "Apache Mynewt", "NetBeans", "Apache NetBeans", "OpenOffice", "Apache OpenOffice", "POI", "Apache POI", "Royale", "Apache Royale", "Spark", "Apache Spark", "Thrift", "Apache Thrift", "Tomcat", "Apache Tomcat", "Trafodion", "Apache Trafodion", "Whimsy", "Apache Whimsy", "ZooKeeper", "Apache ZooKeeper", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 01:00PM Aug 31, 2021 by Sally Khudairi in Milestones  |   | 

The Apache Drill Project Announces Apache® Drill(TM) v1.19 Milestone Release

Open Source, enterprise-grade, schema-free Big Data SQL query engine used by thousands of organizations, including Ant Group, Cisco, Ericsson, Intuit, MicroStrategy, Tableau, TIBCO, TransUnion, Twitter, and more.

Wilmington, DE —30 August 2021— The Apache Drill Project announced the release of Apache® DrillTM v1.19, the schema-free Big Data SQL query engine for Apache Hadoop®, NoSQL, and Cloud storage.

"Drill 1.19 is our biggest release ever," said Charles Givre, Vice President of Apache Drill. "With an already short learning curve, Drill 1.19 makes it even easier for users to quickly query, analyze, and visualize data from disparate sources and complex data sets.”

An "SQL-on-Hadoop" engine, Apache Drill is easy to deploy, highly performant, able to quickly process trillions of records, and scalable from a single laptop to a 1000-node cluster. With its schema-free JSON model (the first distributed SQL query engine of its kind), Drill is able to query complex semi-structured data in situ without requiring users to define schemas or transform data. It provides plug-and-play integration with existing Hive and HBase deployments, and is extensible out-of-the-box to access multiple data sources, such as S3 and Apache HDFS, HBase, and Hive. Additionally, Drill can directly query data from REST APIs to include platforms like SalesForce and ServiceNow. 

Drill supports the ANSI SQL 2003 standard syntax ecosystem as well as dozens of NoSQL databases and file systems, including Apache HBase, MongoDB, Elasticsearch, Cassandra, REST APIs, , HDFS, MapR-FS, Amazon S3, Azure Blob Storage, Google Cloud Storage, NAS,  local files, and more. Drill leverages familiar BI tools (such as Apache Superset, Tableau, MicroStrategy, QlikView and Excel) as well as data virtualization and visualization tools, and runs interactive queries on Hive tables with different Hive metastores.

Apache Drill v1.19
Drill is designed from the ground up to support high-performance analysis on rapidly evolving data on modern Big Data applications. v1.19 reflects more than 100 changes, improvements, and new features that include:

• New Connectors for Apache Cassandra, Elasticsearch, and Splunk.
  
  
• New Format Reader for XML without schemas
  
  
• Added Avro support for Kafka plugin
  
  
• Integrated password vault for secure credential storage
  
  
• Support for Linux ARM64 systems
  
  
• Added limit pushdowns for file systems, HTTP REST APIs and MongoDB
  
  
• Added streaming for Drill's REST API
  
  
• Integration with Apache Airflow

Developers, analysts, business users, and data scientists use Apache Drill for data exploration and analysis for its enterprise-grade reliability, security, and performance. Drill's flexibility and ease-of-use have attracted thousands of users that include Ant Group, Cardlytics, Cisco, Ericsson, Intuit, MicroStrategy, Qlik, Tableau, TIBCO, TransUnion, Twitter, National University of Singapore, and more.

"Individuals, businesses, and organizations of all types rely on Apache Drill's rich functionality," added Givre. "We invite everyone to participate in our user and developer lists as well as our Slack channel, and contribute to the project to build on our momentum and help improve the future experience for all Drill users."

Catch Apache Drill in action at ApacheCon@Home, taking place online 21-23 September 2021. For more information and to register, visit https://www.apachecon.com/ .

Availability and Oversight
Apache Drill software is released under the Apache License v2.0 and is overseen by a volunteer, self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases.

About Apache Drill
Apache Drill is the Open Source, schema-free Big Data SQL query engine for Apache Hadoop, NoSQL, and Cloud storage. For more information, including documentation and ways to become involved with Apache Drill, visit http://drill.apache.org/ , https://twitter.com/ApacheDrill , and https://apache-drill.slack.com/ .

© The Apache Software Foundation. "Apache", "Drill", "Apache Drill", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

#  #  #

Posted at 01:00PM Aug 30, 2021 by Sally Khudairi in General  |   | 

The Apache News Round-up: week ending 27 August 2021

Hello, Friday --let's take a look at the Apache community's activities from the past week:

ASF Board – management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 15 September 2021. Board calendar and minutes https://apache.org/foundation/board/calendar.html

ApacheCon™ – the ASF's official global conference series, bringing Tomorrow's Technology Today since 1998. ApacheCon Asia was held online 6-8 August; ApacheCon@Home is coming up next on 21-23 September:
 - The Apache® Software Foundation Announces Program for ApacheCon@Home 2021 https://s.apache.org/ACHome2021
 - Learn about the Community Track from Sharan Foga and Swapnil M Mane https://youtu.be/8cZF-gaE3a4
 - Register for ApacheCon@Home https://www.apachecon.com/acah2021/

ASF Infrastructure – our distributed team on three continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield uptime at 98.52%. Performance checks across 50 different service components spread over more than 250 machines in data centers around the world. http://www.apache.org/uptime/

Apache Code Snapshot – Over the past week, 307 Apache Committers changed 2,747,090 lines of code over 2,669 commits. Top 5 contributors, in order, are: Andrea Cosentino, Andi Huber, Harikrishna Patnala, Claus Ibsen, and Gary Gregory.        

Apache Project Announcements – the latest updates by category.

Big Data --
 - Apache NiFi - MiNiFi C++ CVE-2021-33191: MiNiFi CPP arbitrary script execution is possible on the agent's host machine through the c2 protocol https://s.apache.org/t1okn

Libraries --
 - Apache Geometry 1.0 released https://commons.apache.org/proper/commons-geometry/
 - Apache Portable Runtime CVE-2021-35940: Regression of CVE-2017-12613 https://s.apache.org/rvqdx

Messaging --
 - Apache Qpid Dispatch 1.17.0 released https://qpid.apache.org/

Did You Know?

 - Did you know that Dropbox uses Apache Superset as their data exploration platform? https://superset.apache.org/ 

 - Did you know that the CFP for Pulsar Summit Asia in November closes on 8 September? https://pulsar.apache.org/en/events/ 

 - Did you know that the CFP is open for the CloudStack Collaboration Conference in November? https://cloudstack.apache.org/ 

Apache Community Notices

- Watch "Trillions and Trillions Served", the documentary on the ASF 1) full feature [49 min] https://s.apache.org/Trillions-Feature 2) "Apache Everywhere" [6 min] https://s.apache.org/ApacheEverywhere 3) "Why Apache" [2.5 min] https://s.apache.org/ASF-Trillions-WhyApache 4) “Apache Innovation” [40 min] https://s.apache.org/ApacheInnovation

 - The Apache Way to Sustainable Open Source Success https://s.apache.org/GhnI

 - Foundation Reports and Statements http://www.apache.org/foundation/reports.html

 - Presentations from ApacheCon Asia are available at https://s.apache.org/ApacheConAsia2021-talks

 - "Success at Apache" focuses on the people and processes behind why the ASF "just works". https://blogs.apache.org/foundation/category/SuccessAtApache

Posted at 06:51AM Aug 27, 2021 by Swapnil M Mane in Newsletter  |   | 

The Apache News Round-up: week ending 20 August 2021

Happy Friday! We've had another great week --let's review the Apache community's activities:

ASF Board – management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 15 September 2021. Board calendar and minutes https://apache.org/foundation/board/calendar.html

ApacheCon™ – the ASF's official global conference series, bringing Tomorrow's Technology Today since 1998. ApacheCon Asia was held online 6-8 August; ApacheCon@Home is coming up next on 21-23 September:
 - The Apache® Software Foundation Announces Program for ApacheCon@Home 2021 https://s.apache.org/ACHome2021
 - Register for ApacheCon@Home https://www.apachecon.com/acah2021/

ASF Infrastructure – our distributed team on three continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield uptime at 100.00%. Performance checks across 50 different service components spread over more than 250 machines in data centers around the world. http://www.apache.org/uptime/

Apache Code Snapshot – Over the past week, 321 Apache Committers changed 32,652,622 lines of code over 3,047 commits. Top 5 contributors, in order, are: Jarek Potiuk, Alex Herbert, Claus Ibsen, Ephraim Anierobi, and Gary Gregory.        

Apache Project Announcements – the latest updates by category.

Messaging --
 - Apache ActiveMQ 5.16.3 released https://activemq.apache.org/

Servers --
 - Apache Tomcat 8.5.70 released https://tomcat.apache.org/
 - Apache Traffic Server 9.1.0 released https://trafficserver.apache.org/

Did You Know?

- Did you know that recent projects undergoing development in the Apache Incubator include EventMesh (eventing), Hop (orchestration), Kyuubi (Big Data), Linkis (middleware), Sedona (geospatial data), and Wayang (Big Data)? http://incubator.apache.org/

- Did you know that Uber's Big Data platform is powered by Apache Hadoop, Hive, Kafka, ORC, Parquet, Spark, YARN, and more? https://projects.apache.org/projects.html?category

- Did you know that Apache Maven's 125kb XSD file causes 200TB worth of traffic per year at http://apache.org? https://maven.apache.org/

Apache Community Notices

- Watch "Trillions and Trillions Served", the documentary on the ASF 1) full feature [49 min] https://s.apache.org/Trillions-Feature 2) "Apache Everywhere" [6 min] https://s.apache.org/ApacheEverywhere 3) "Why Apache" [2.5 min] https://s.apache.org/ASF-Trillions-WhyApache 4) “Apache Innovation” [40 min] https://s.apache.org/ApacheInnovation

 - The Apache Way to Sustainable Open Source Success https://s.apache.org/GhnI

 - Foundation Reports and Statements http://www.apache.org/foundation/reports.html

 - Presentations from select Apache events and 2020's ApacheCon@Home are available at https://www.youtube.com/c/TheApacheFoundation/ 

 - "Success at Apache" focuses on the people and processes behind why the ASF "just works". https://blogs.apache.org/foundation/category/SuccessAtApache

Posted at 10:28AM Aug 20, 2021 by Swapnil M Mane in Newsletter  |   | 

The Apache® Software Foundation Announces Program for ApacheCon@Home 2021

Official Apache global conference series to be held virtually for second year, featuring 140+ sessions, and keynotes from GitHub, NYU and Open Source Hardware Association, Red Hat, and the Wilson Center.

Wilmington, DE —17 August 2021— The Apache® Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced keynotes, sponsors, and program for ApacheCon@Home, taking place online 21-23 September 2021. Registration is open and free for all attendees.

"With dozens of compelling talks on the latest Apache innovations, presented by noted community experts and supported by an impressive roster of sponsors, we expect a record turnout of attendees," said Ruth Suehle, ASF Executive Vice President and ApacheCon@Home co-Chair. "We're excited to enable participants to join us at their own convenience online and to build on the success of last year's inaugural ApacheCon@Home, with nearly 3,500 attendees."

ApacheCon is the ASF's official global conference series, first held in 1998. ApacheCon draws attendees from more than 130 countries to experience "Tomorrow's Technology Today" independent of business interests, corporate biases, or sales pitches.

ApacheCon showcases the latest breakthroughs from dozens of Apache projects, with content selected entirely by Apache projects and their communities. ApacheCon@Home joins ApacheCon Asia, which took place online 6-8 August, to meet the educational demands of the growing Apache community of developers, users, and enthusiasts worldwide.

"The ApacheCon@Home format has been a game changer by letting many members of the Apache community experience ApacheCon for the first time —all from the comfort of their homes or any location,” said Rich Bowen, Vice President of Conferences at the ASF. "We continue to grow and evolve our events to enable participants from around the globe to experience the full event as it happens, and interact with community members and sponsors in real-time. Post-event session recordings enable thousands to enjoy ApacheCon year-round."

ApacheCon@Home features more than 140 sessions, all held virtually and online. Participants at all levels will learn about innovations in Apache projects and categories that include: ActiveMQ; AGE (incubating); APISIX; API and Microservice; Beam; Big Data; Calcite; Camel; Cassandra, Community; Content Delivery; Content Management; CXF; Daffodil; DataSketches; Data Visualization; DolphinScheduler; Druid; Federated Data; Fineract and FinTech; Flink; Geode; Geospatial; Groovy; Hadoop; Hadoop YARN; HBase; HDFS; Hive; Hop (incubating); HTTP Server; Hudi Ignite; Impala; Incubator; Integration; Internet of Things; James; Jena; Kafka; Karaf; Kudu; Liminal (incubating); Lucene; MADlib; MXNet (incubating); NiFi; NLPCraft (incubating); OpenNLP; Ozone; PLC4X; POI; Pulsar; Search; Sedona (incubating); SIS; SkyWalking; Sling; Solr; Spark; Storm; Streampipes (incubating); Teaclave (incubating); Thrift; Tika; Tomcat; Traffic Control; Traffic Server; Wayang (incubating); YuniKorn (incubating); Zookeeper; and more.

Keynote presentations will be delivered by Ashley Wolf, head of GitHub’s Open Source Program Office; Mark Cox, ASF Vice President Security, and Distinguished Software Engineer at Red Hat’s Open Source Program Office; Dr. Alison Parker, Senior Program Associate in the Science and Technology Innovation Program at the Wilson Center; and Michael Weinberg, Executive Director of the Engelberg Center on Innovation Law & Policy at NYU Law and the Board President of the Open Source Hardware Association.

The full program is available at https://www.apachecon.com/acah2021/tracks/

ApacheCon@Home sponsors include Strategic Sponsor Google; Platinum Sponsors Apple, Huawei, Instaclustr, and Tencent Cloud; Gold Sponsors AWS, Baidu, Cerner, Didi Chuxing, Dremio, Fiter, Red Hat, and Replicated; Silver Sponsors Imply, Securonix, and SphereEx; and Bronze Sponsor Technical Arts & Engineering. AWS, Huawei, Didi Chuxing, Tencent, Baidu, and SphereEx were also Sponsors of ApacheCon Asia.

To sponsor ApacheCon@Home, visit https://www.apachecon.com/acah2021/2021_ApacheCon_prospectus.pdf

Register today at https://www.apachecon.com/acah2021/register.html .

About ApacheCon
ApacheCon is the official global conference series of The Apache Software Foundation. Since 1998 ApacheCon has been drawing participants at all levels to explore "Tomorrow's Technology Today" across 350+ Apache projects and their diverse communities. In 2020 and 2021 ApacheCon events showcase ubiquitous Apache projects and emerging innovations virtually through sessions, keynotes, real-world case studies, community events, and more, all online and free of charge. For more information, visit http://apachecon.com/ and https://twitter.com/ApacheCon .

About The Apache Software Foundation (ASF)
Established in 1999, The Apache Software Foundation is the world’s largest Open Source foundation, stewarding 227M+ lines of code and providing more than $20B+ worth of software to the public at 100% no cost. The ASF’s all-volunteer community grew from 21 original founders overseeing the Apache HTTP Server to 850+ individual Members and 200 Project Management Committees who successfully lead 350+ Apache projects and initiatives in collaboration with 8,200+ Committers through the ASF’s meritocratic process known as "The Apache Way". Apache software is integral to nearly every end user computing device, from laptops to tablets to mobile devices across enterprises and mission-critical applications. Apache projects power most of the Internet, manage exabytes of data, execute teraflops of operations, and store billions of objects in virtually every industry. The commercially-friendly and permissive Apache License v2 is an Open Source industry standard, helping launch billion dollar corporations and benefiting countless users worldwide. The ASF is a US 501(c)(3) not-for-profit charitable organization funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, Amazon Web Services, Anonymous, Baidu, Bloomberg, Budget Direct, Capital One, Cloudera, Comcast, Confluent, Didi Chuxing, Facebook, Google, Handshake, Huawei, IBM, Microsoft, Namebase, Pineapple Fund, Red Hat, Reprise Software, Target, Tencent, Union Investment, Verizon Media, and Workday. For more information, visit http://apache.org/ and https://twitter.com/TheASF .

© The Apache Software Foundation. "Apache", "Airflow", "Apache Airflow", "APISIX", "Apache APISIX", "Arrow", "Apache Arrow", "Atlas", "Apache Atlas", "Bigtop", "Apache Bigtop", "BookKeeper", "Apache BookKeeper", "Camel", "Apache Camel", "CarbonData", "Apache CarbonData", "Cassandra", "Apache Cassandra", "Commons", "Apache Commons", "DolphinScheduler", "Apache DolphinScheduler", "Druid", "Apache Druid", "Dubbo", "Apache Dubbo", "ECharts", "Apache ECharts", "Flink", "Apache Flink", "Hadoop", "Apache Hadoop", "HBase", "Apache HBase", "Hive", "Apache Hive", "HUDI", "Apache HUDI", "Ignite", "Apache Ignite", "Impala", "Apache Impala", "IoTDB", "Apache IoTDB", "Kafka", "Apache Kafka", "Kudu", "Apache Kudu", "Kylin", "Apache Kylin", "Ozone", "Apache Ozone", "PLC4X", "Apache PLC4X", "Pulsar", "Apache Pulsar", "RocketMQ", "Apache RocketMQ", "ServiceComb", "Apache ServiceComb", "ShardingSphere", "Apache ShardingSphere", "SkyWalking", "Apache SkyWalking", "Sling", "Apache Sling", "Spark", "Apache Spark", "Superset", "Apache Superset", "Tomcat", "Apache Tomcat", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 12:00PM Aug 17, 2021 by Sally Khudairi in ApacheCon  |   | 

The Apache News Round-up: week ending 13 August 2021

We're wrapping up another great week with the following activities from the Apache community:

ASF Board – management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 18 August 2021. Board calendar and minutes https://apache.org/foundation/board/calendar.html

ApacheCon™ – the ASF's official global conference series, bringing Tomorrow's Technology Today since 1998. ApacheCon Asia was held online 6-8 August; ApacheCon@Home is coming up next on 21-23 September:
 - Register for ApacheCon@Home https://www.apachecon.com/acah2021/

ASF Infrastructure – our distributed team on three continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield uptime at 99.98%. Performance checks across 50 different service components spread over more than 250 machines in data centers around the world. http://www.apache.org/uptime/

Apache Code Snapshot – Over the past week, 313 Apache Committers changed 2,040,937 lines of code over 2,770 commits. Top 5 contributors, in order, are: Claus Ibsen, Gary Gregory, Harikrishna Patnala, Alex Herbert, and James Netherton.      

Apache Project Announcements – the latest updates by category.

Big Data --
 - Apache Flink 1.12.5 released https://flink.apache.org/

Databases --
 - Apache Geode 1.13.4 released http://geode.apache.org/

Enterprise Resource Planning (ERP) --
 - Apache OFBiz 17.12.08 released and CVE-2021-37608: Arbitrary file upload vulnerability in OFBiz https://ofbiz.apache.org/

Eventing  --
 - Apache EventMesh (incubating) 1.2.0 released https://eventmesh.apache.org/

Libraries --
 - Apache ServiceComb CVE-2021-21501: ServiceComb ServiceCenter Directory Traversal http://servicecomb.apache.org

Messaging --
 - Apache Pulsar 2.7.3 released https://pulsar.apache.org/

Orchestration --
 - Apache Hop (Incubating) 0.99 released https://hop.apache.org/

Servers --
 - Apache Tomcat 9.0.52, 10.1.0-M4 (alpha), and 10.0.10 released https://tomcat.apache.org/

Workflow --
 - Apache Airflow CVE-2021-35936: No Authentication on Logging Server https://s.apache.org/r1i0m

Did You Know?

- Did you know that Shopify's 400TB+ data platform is powered by Apache Avro, Beam, Flink, Kafka, and Spark https://projects.apache.org/projects.html?category

- Did you know that Imply's real-time analytics is powered by Apache Druid? https://druid.apache.org/

- Did you know that Flink Forward will be held 26-27 October? http://flink.apache.org/

Apache Community Notices

- Watch "Trillions and Trillions Served", the documentary on the ASF 1) full feature [49 min] https://s.apache.org/Trillions-Feature 2) "Apache Everywhere" [6 min] https://s.apache.org/ApacheEverywhere 3) "Why Apache" [2.5 min] https://s.apache.org/ASF-Trillions-WhyApache 4) “Apache Innovation” [40 min] https://s.apache.org/ApacheInnovation

 - The Apache Way to Sustainable Open Source Success https://s.apache.org/GhnI

 - Foundation Reports and Statements http://www.apache.org/foundation/reports.html

 - Presentations from ApacheCon@Home and other events are available at https://www.youtube.com/c/TheApacheFoundation/ 

 - "Success at Apache" focuses on the people and processes behind why the ASF "just works". https://blogs.apache.org/foundation/category/SuccessAtApache

Posted at 12:59PM Aug 13, 2021 by Sally Khudairi in Newsletter  |   | 

The Apache News Round-up: week ending 6 August 2021

Welcome, August --we're opening the month with another great week. Here's what the Apache community has been up to:

Apache Month in Review – a round-up of our Round-ups and other newsworthy bits over the past month.
 - July Month in Review https://s.apache.org/July2021 --video highlights at https://youtu.be/KIYB1g6SKhg

ASF Board – management and oversight of the business affairs of the corporation in accordance with the Foundation's bylaws.
 - Next Board Meeting: 18 August 2021. Board calendar and minutes https://apache.org/foundation/board/calendar.html

ApacheCon™ – the ASF's official global conference series, bringing Tomorrow's Technology Today since 1998. ApacheCon Asia (6-8 August) and ApacheCon@Home (21-23 September) are being held online and free-of-charge:
 - LIVE: ApacheCon Asia https://s.apache.org/ACAsia2021
    -- Learn more about ApacheCon Asia from Sheng Wu and Willem Jiang at https://youtube.com/watch?v=hfRCrpnbDhc
 - Register for ApacheCon@Home https://www.apachecon.com/acah2021/

ASF Infrastructure – our distributed team on three continents keeps the ASF's infrastructure running around the clock.
 - 7M+ weekly checks yield uptime at 100.00%. Performance checks across 50 different service components spread over more than 250 machines in data centers around the world. http://www.apache.org/uptime/

Apache Code Snapshot – Over the past week, 305 Apache Committers changed 1,999,558 lines of code over 3,149 commits. Top 5 contributors, in order, are: Bertrand Delacretaz, Alex Herbert, Tellier Benoit, Claus Ibsen, and Jarek Potiuk.       

Apache Project Announcements – the latest updates by category.

Integration --
 - Apache Camel 3.11.1 released https://camel.apache.org/

Libraries --
 - Apache Commons CSV 1.9.0 released https://commons.apache.org/proper/commons-csv/
 - Apache Commons DBCP 2.9.0 released http://commons.apache.org/dbcp/

Did You Know?

- Did you know that the following Apache Projects are celebrating anniversaries in August? Three cheers to jUDDI (11 years); Any23, Lucene.Net, Oozie (9 years); Ignite, Serf, Usergrid (6 years); HAWQ (3 years). https://projects.apache.org/committees.html?date

- Did you know that the presentations from the Airflow Summit 2021 are now online? https://airflowsummit.org/sessions/2021/

- Did you know that you can learn the latest on Apache TVM (incubating) in their monthly community report? https://discuss.tvm.apache.org/t/tvm-monthly-july-2021/10701

Apache Community Notices

- The Apache® Software Foundation Celebrates 22 Years of Open Source Leadership – world’s largest Open Source foundation advances community-led innovation "The Apache Way" https://s.apache.org/22ndAnniversay

- The Apache Software Foundation Operations Summary: Q3 FY2021 (November 2020 - January 2021) https://s.apache.org/Q3FY2021 + Video highlights https://youtu.be/S6FWqAuA_8M

- Apache in 2020 - By The Digits https://s.apache.org/Apache2020Digits + Video highlights https://s.apache.org/Apache2020Digits-vid

- ASF Security Report 2020 https://s.apache.org/SecurityReport2020 + Video highlights https://youtu.be/Z7yudar_da0

- ASF FY2020 Annual Report https://s.apache.org/FY2020AnnualReport

- "Trillions and Trillions Served" documentary on the ASF: 1) full feature https://s.apache.org/Trillions-Feature 2) "Apache Everywhere" https://s.apache.org/ApacheEverywhere 3) "Why Apache" https://s.apache.org/ASF-Trillions-WhyApache 4) “Apache Innovation” https://s.apache.org/ApacheInnovation 

 - The Apache Way to Sustainable Open Source Success https://s.apache.org/GhnI

 - Foundation Reports and Statements http://www.apache.org/foundation/reports.html

 - All presentations from ApacheCon@Home are available at https://www.youtube.com/c/TheApacheFoundation/ 

 - "Success at Apache" focuses on the people and processes behind why the ASF "just works". https://blogs.apache.org/foundation/category/SuccessAtApache

Posted at 10:58AM Aug 06, 2021 by Swapnil M Mane in Newsletter  |   | 

The Apache Software Foundation Announces Apache® Pinot™ as a Top-Level Project

Open Source distributed real-time Big Data analytics infrastructure in use at Amazon-Eero, Doordash, Factual/FourSquare, LinkedIn, Stripe, Uber, Walmart, Weibo, and WePay, among others.

Wilmington, DE —2 August 2021— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today Apache® Pinot™ as a Top-Level Project (TLP).

Apache Pinot is a distributed Big Data analytics infrastructure created to deliver scalable real-time analytics at high throughput with low latency. The project was first created at LinkedIn in 2013, open-sourced in 2015, and entered the Apache Incubator in October 2018.

"We are pleased to successfully adopt 'the Apache Way' and graduate from the Apache Incubator," said Kishore Gopalakrishna, Vice President and original co-creator of Apache Pinot. "Pinot initially pushed the boundaries of real-time analytics by delivering insights to millions of Linkedin users. Today, as an Apache Top-Level Project, Pinot is in the hands of developers across the globe who are building it to power several user-facing  analytical applications and unlock the value of data within their organizations."

Scalable to trillions of records, Apache Pinot’s online analytical processing (OLAP) ingests both online and offline data sources from Apache Kafka, Apache Spark, Apache Hadoop HDFS, flat files, and Cloud storages in real time. Pinot is able to ingest millions of events and serve thousands of queries per second, and provide unified analytics in a distributed, fault-tolerant fashion. Features include:

• Speed —answers OLAP queries with low latency on real-time data
  
  
• Pluggable indexing —Sorted, Inverted, Text Index, Geospatial Index, JSON Index, Range Index, Bloom filters
  
  
• Smart Materialized Views - Fast Aggregations via star-tree index
  
  
• Supports different stream systems with near real-time ingestion —with Apache Kafka, Confluent Kafka, and Amazon Kinesis, as well as customizable input format, with out-of the box support for Avro and JSON formats
  
  
• Highly available, horizontally scalable, and fault tolerant
  
  
• Supports lookup joins natively and full joins using PrestoDB/Trino
  
  

Apache Pinot is used to power internal and external analytics at Adbeat, Amazon-Eero, Cloud Kitchens, Confluera, Doordash, Factual/FourSquare, Guitar Center, LinkedIn, Publicis Sapient, Razorpay, Scale Unlimited, Startree, Stripe, Traceable, Uber, Walmart, Weibo, WePay, and more.

Examples of how Apache Pinot helps organizations across numerous verticals include: 1) a fintech company uses Pinot to achieve financial data visibility across 500+ terabytes of data and sustain half million queries per second with financial transactions; 2) a food delivery service leveraged Pinot in the midst of the COVID-19 pandemic to analyze real-time data to provide a socially-distanced pick-up experience for its riders and restaurants; and 3) a large retail chain with geographically distributed franchises and stores uses Pinot for revenue-generating opportunities by analyzing real-time data for internal use cases, as well as real-time cart analysis to increase sales.

"We rely on Apache Pinot for all our real-time analytics needs at LinkedIn," said Kapil Surlaker, Vice President of Engineering at LinkedIn. "It's battle-tested at LinkedIn scale for hundreds of our low-latency analytics applications. We believe Apache Pinot is the best tool out there to build site-facing analytics applications and we will continue to contribute heavily and collaborate with the Apache Pinot community. We are very happy to see that it's now a Top-level Apache project."

"We use Apache Pinot in our real-time analytics platform to power external user-facing applications and critical operational dashboards," said Ujwala Tulshigiri, Engineering Manager at Uber. "With Pinot's multi-tenancy support and horizontal scalability, we have scaled to hundreds of use cases that run complex aggregations queries on terabytes of data at millisecond latencies, with the minimal overhead of cluster management."

"We've been using Apache Pinot since last year, and it's been a huge win for our client’s dashboard project," said Ken Krugler, President of Scale Unlimited. "Pinot's ability to rapidly generate aggregation results over billions of records, with modest hardware requirements, was critical for the success of the project. We've also been able to provide patches to add functionality and fix issues, which the Pinot community has quickly integrated and released. There was never any doubt in our minds that Pinot would graduate from the Apache incubator and become a successful top-level project."

"Last year, we started without analytics built into our product," said Pradeep Gopanapalli, technical staff member at Confluera. "By the end of the year, we were using Apache Pinot for real-time analytics in production. Not many of our competitors can even dream of having such results. We are very happy with our choice."

"Pinot is critical to our real-time analytics platform and allowed us to scale without degrading latency," said software engineer Elon Azoulay. "Pinot enables us to onboard large datasets effortlessly, run complex queries which return in milliseconds and is super reliable. We would like to emphasize how helpful and engaged the community is and are certain that we made the right choice with Pinot, it continues to impress us and satisfy our real-time analytics needs."

"We created Pinot at LinkedIn with the goal of tackling the low-latency OLAP problem for site-facing use cases at scale. We evolved it to solve numerous OLAP use cases, and open-sourced it because there aren't many technologies in that domain," said Subbu Subramaniam, member of the Apache Pinot Project Management Committee, and Senior Staff Engineer at LinkedIn. "It is heart-warming to see such a wide adoption and great contributions from the community in improving Pinot over time."

"We are at the beginning of this transformation and we cannot wait to see every software company build real-time applications using Apache Pinot," added Gopalakrishna. "We welcome everyone to join our community Slack channel and contribute to the project."

Catch Apache Pinot in action at ApacheCon Asia online on 7 August 2021. For more information and to register, visit https://www.apachecon.com/acasia2021/

Availability and Oversight
Apache Pinot software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache Pinot, visit http://pinot.apache.org/ and https://twitter.com/ApachePinot

About the Apache Incubator
The Apache Incubator is the primary entry path for projects and codebases wishing to become part of the efforts at The Apache Software Foundation. All code donations from external organizations and existing external projects enter the ASF through the Incubator to: 1) ensure all donations are in accordance with the ASF legal standards; and 2) develop new communities that adhere to our guiding principles. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. For more information, visit http://incubator.apache.org/

About The Apache Software Foundation (ASF)
Established in 1999, The Apache Software Foundation is the world’s largest Open Source foundation, stewarding 227M+ lines of code and providing more than $22B+ worth of software to the public at 100% no cost. The ASF’s all-volunteer community grew from 21 original founders overseeing the Apache HTTP Server to 850+ individual Members and 200 Project Management Committees who successfully lead 350+ Apache projects and initiatives in collaboration with 8,200+ Committers through the ASF’s meritocratic process known as "The Apache Way". Apache software is integral to nearly every end user computing device, from laptops to tablets to mobile devices across enterprises and mission-critical applications. Apache projects power most of the Internet, manage exabytes of data, execute teraflops of operations, and store billions of objects in virtually every industry. The commercially-friendly and permissive Apache License v2 is an Open Source industry standard, helping launch billion dollar corporations and benefiting countless users worldwide. The ASF is a US 501(c)(3) not-for-profit charitable organization funded by individual donations and corporate sponsors that include Aetna, Alibaba Cloud Computing, Amazon Web Services, Anonymous, Baidu, Bloomberg, Capital One, Cloudera, Comcast, Confluent, Didi Chuxing, Facebook, Google, Huawei, IBM, Indeed, Microsoft, Namebase, Pineapple Fund, Red Hat, Reprise Software, Talend, Tencent, Target, Union Investment, Verizon Media, and Workday. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Pinot", "Apache Pinot", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 12:00PM Aug 02, 2021 by Sally Khudairi in General  |   | 

Apache Month in Review: July 2021

Welcome to the latest monthly overview of events from the Apache community. Here's a summary of what happened in July (video highlights available at https://youtu.be/KIYB1g6SKhg ):

New this month --

 - The Apache Cassandra Project Releases Apache Cassandra v4.0, the Fastest, Most Scalable and Secure Cassandra Yet https://s.apache.org/d30v9

 - Apache Attic --provides process and solutions when an Apache project has reached its end of life
  -- Apache Sqoop is now retired https://s.apache.org/0e51t

 - Apache Month in Review: June 2021 https://s.apache.org/June2021 + Video highlights https://youtu.be/yIE8SSHw2iw 

Important Dates --

 - Next Board Meeting: 18 August 2021. Board calendar and minutes http://apache.org/foundation/board/calendar.html

 - ApacheCon™ --the ASF's official global conference series, bringing Tomorrow's Technology Today since 1998-- will be held twice in 2021:
   --ApacheCon Asia - 6-8 August and ApacheCon@Home - 21-23 September
  The Apache® Software Foundation Welcomes its Global Community Online at ApacheCon Asia 2021 https://s.apache.org/ACAsia2021
  Program, Registration, and Sponsorship available for both events https://www.apachecon.com/

Infrastructure --

In July, 789 Apache Committers changed 13,194,378 lines of code over 17,560 commits. The Committers with the top 5 highest contributions, in order, were: Bertrand Delacrétaz, Andrea Cosentino, Gary Gregory, Mark Thomas, and Xiang Xiao.  

Project Releases and Updates --

New releases from Apache Ant (Build Management); Arrow (Big Data); APISIX (APIs); Beam (Big Data); Camel (Integration); Cassandra (Databases); CloudStack (Cloud Computing); Commons (Libraries); Curator (Messaging); Directory (Identity Management); Druid (Big Data); Fortress (Identity Management); Geode (Databases); HBase (Big Data); Impala (Databases); Jackrabbit (Content); James (Mail); Jena (Libraries); MINA (Network Client/Server); NiFi (Big Data); OpenMeetings (Web Conferencing); Qpid (Messaging); ShardingSphere (Big Data); SkyWalking (Observability); Streampipes (incubating; IoT); Tika (Content); Tomcat (Servers); Tuweni (incubating; Blockchain); UIMA (Content); Unomi (Data Management); Wicket (Web Frameworks); XML Beans (Big Data)

The Apache Incubator is the primary entry path for projects wishing to become an official part of the ASF. No new projects entered the Apache Incubator in July. More than three dozen projects are currently undergoing development in the Apache Incubator http://incubator.apache.org/ .

# # #

To see our Weekly News Round-ups (published every Friday), visit https://blogs.apache.org/foundation/ and click on the calendar or hop directly to https://blogs.apache.org/foundation/category/Newsletter . For real-time updates, sign up for Apache-related news by sending mail to announce-subscribe@apache.org and follow @TheASF on Twitter. We appreciate your support!

Posted at 12:47PM Aug 01, 2021 by Sally Khudairi in Newsletter  |   |