The Apache Software Foundation Blog: Entries tagged [bigdata]

Entries tagged [bigdata]

Main | Next page »

Monday August 02, 2021

The Apache Software Foundation Announces Apache® Pinot™ as a Top-Level Project

Open Source distributed real-time Big Data analytics infrastructure in use at Amazon-Eero, Doordash, Factual/FourSquare, LinkedIn, Stripe, Uber, Walmart, Weibo, and WePay, among others.

Wilmington, DE —2 August 2021— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today Apache® Pinot™ as a Top-Level Project (TLP).

Apache Pinot is a distributed Big Data analytics infrastructure created to deliver scalable real-time analytics at high throughput with low latency. The project was first created at LinkedIn in 2013, open-sourced in 2015, and entered the Apache Incubator in October 2018.

"We are pleased to successfully adopt 'the Apache Way' and graduate from the Apache Incubator," said Kishore Gopalakrishna, Vice President and original co-creator of Apache Pinot. "Pinot initially pushed the boundaries of real-time analytics by delivering insights to millions of Linkedin users. Today, as an Apache Top-Level Project, Pinot is in the hands of developers across the globe who are building it to power several user-facing analytical applications and unlock the value of data within their organizations."

Scalable to trillions of records, Apache Pinot’s online analytical processing (OLAP) ingests both online and offline data sources from Apache Kafka, Apache Spark, Apache Hadoop HDFS, flat files, and Cloud storages in real time. Pinot is able to ingest millions of events and serve thousands of queries per second, and provide unified analytics in a distributed, fault-tolerant fashion. Features include:

Speed —answers OLAP queries with low latency on real-time data
Pluggable indexing —Sorted, Inverted, Text Index, Geospatial Index, JSON Index, Range Index, Bloom filters
Smart Materialized Views - Fast Aggregations via star-tree index
Supports different stream systems with near real-time ingestion —with Apache Kafka, Confluent Kafka, and Amazon Kinesis, as well as customizable input format, with out-of the box support for Avro and JSON formats
Highly available, horizontally scalable, and fault tolerant
Supports lookup joins natively and full joins using PrestoDB/Trino

Apache Pinot is used to power internal and external analytics at Adbeat, Amazon-Eero, Cloud Kitchens, Confluera, Doordash, Factual/FourSquare, Guitar Center, LinkedIn, Publicis Sapient, Razorpay, Scale Unlimited, Startree, Stripe, Traceable, Uber, Walmart, Weibo, WePay, and more.

Examples of how Apache Pinot helps organizations across numerous verticals include: 1) a fintech company uses Pinot to achieve financial data visibility across 500+ terabytes of data and sustain half million queries per second with financial transactions; 2) a food delivery service leveraged Pinot in the midst of the COVID-19 pandemic to analyze real-time data to provide a socially-distanced pick-up experience for its riders and restaurants; and 3) a large retail chain with geographically distributed franchises and stores uses Pinot for revenue-generating opportunities by analyzing real-time data for internal use cases, as well as real-time cart analysis to increase sales.

"We rely on Apache Pinot for all our real-time analytics needs at LinkedIn," said Kapil Surlaker, Vice President of Engineering at LinkedIn. "It's battle-tested at LinkedIn scale for hundreds of our low-latency analytics applications. We believe Apache Pinot is the best tool out there to build site-facing analytics applications and we will continue to contribute heavily and collaborate with the Apache Pinot community. We are very happy to see that it's now a Top-level Apache project."

"We use Apache Pinot in our real-time analytics platform to power external user-facing applications and critical operational dashboards," said Ujwala Tulshigiri, Engineering Manager at Uber. "With Pinot's multi-tenancy support and horizontal scalability, we have scaled to hundreds of use cases that run complex aggregations queries on terabytes of data at millisecond latencies, with the minimal overhead of cluster management."

"We've been using Apache Pinot since last year, and it's been a huge win for our client’s dashboard project," said Ken Krugler, President of Scale Unlimited. "Pinot's ability to rapidly generate aggregation results over billions of records, with modest hardware requirements, was critical for the success of the project. We've also been able to provide patches to add functionality and fix issues, which the Pinot community has quickly integrated and released. There was never any doubt in our minds that Pinot would graduate from the Apache incubator and become a successful top-level project."

"Last year, we started without analytics built into our product," said Pradeep Gopanapalli, technical staff member at Confluera. "By the end of the year, we were using Apache Pinot for real-time analytics in production. Not many of our competitors can even dream of having such results. We are very happy with our choice."

"Pinot is critical to our real-time analytics platform and allowed us to scale without degrading latency," said software engineer Elon Azoulay. "Pinot enables us to onboard large datasets effortlessly, run complex queries which return in milliseconds and is super reliable. We would like to emphasize how helpful and engaged the community is and are certain that we made the right choice with Pinot, it continues to impress us and satisfy our real-time analytics needs."

"We created Pinot at LinkedIn with the goal of tackling the low-latency OLAP problem for site-facing use cases at scale. We evolved it to solve numerous OLAP use cases, and open-sourced it because there aren't many technologies in that domain," said Subbu Subramaniam, member of the Apache Pinot Project Management Committee, and Senior Staff Engineer at LinkedIn. "It is heart-warming to see such a wide adoption and great contributions from the community in improving Pinot over time."

"We are at the beginning of this transformation and we cannot wait to see every software company build real-time applications using Apache Pinot," added Gopalakrishna. "We welcome everyone to join our community Slack channel and contribute to the project."

Catch Apache Pinot in action at ApacheCon Asia online on 7 August 2021. For more information and to register, visit https://www.apachecon.com/acasia2021/

Availability and Oversight
Apache Pinot software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache Pinot, visit http://pinot.apache.org/ and https://twitter.com/ApachePinot

About the Apache Incubator
The Apache Incubator is the primary entry path for projects and codebases wishing to become part of the efforts at The Apache Software Foundation. All code donations from external organizations and existing external projects enter the ASF through the Incubator to: 1) ensure all donations are in accordance with the ASF legal standards; and 2) develop new communities that adhere to our guiding principles. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. For more information, visit http://incubator.apache.org/

About The Apache Software Foundation (ASF)
Established in 1999, The Apache Software Foundation is the world’s largest Open Source foundation, stewarding 227M+ lines of code and providing more than $22B+ worth of software to the public at 100% no cost. The ASF’s all-volunteer community grew from 21 original founders overseeing the Apache HTTP Server to 850+ individual Members and 200 Project Management Committees who successfully lead 350+ Apache projects and initiatives in collaboration with 8,200+ Committers through the ASF’s meritocratic process known as "The Apache Way". Apache software is integral to nearly every end user computing device, from laptops to tablets to mobile devices across enterprises and mission-critical applications. Apache projects power most of the Internet, manage exabytes of data, execute teraflops of operations, and store billions of objects in virtually every industry. The commercially-friendly and permissive Apache License v2 is an Open Source industry standard, helping launch billion dollar corporations and benefiting countless users worldwide. The ASF is a US 501(c)(3) not-for-profit charitable organization funded by individual donations and corporate sponsors that include Aetna, Alibaba Cloud Computing, Amazon Web Services, Anonymous, Baidu, Bloomberg, Capital One, Cloudera, Comcast, Confluent, Didi Chuxing, Facebook, Google, Huawei, IBM, Indeed, Microsoft, Namebase, Pineapple Fund, Red Hat, Reprise Software, Talend, Tencent, Target, Union Investment, Verizon Media, and Workday. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Pinot", "Apache Pinot", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 12:00PM Aug 02, 2021 by Sally Khudairi in General | |

Tuesday July 27, 2021

The Apache Cassandra Project Releases Apache® Cassandra™ v4.0, the Fastest, Most Scalable and Secure Cassandra Yet

Open Source enterprise-grade Big Data distributed database powers mission-critical deployments with improved performance and unparalleled levels of scale in the Cloud

Wilmington, DE —27 July 2021— The Apache Cassandra Project released today v4.0 of Apache® Cassandra™, the Open Source, highly performant, distributed Big Data database management platform.

"A long time coming, Cassandra 4.0 is the most thoroughly tested Cassandra yet," said Nate McCall, Vice President of Apache Cassandra. "The latest version is faster, more scalable, and bolstered with enterprise security features, ready-for-production with unprecedented scale in the Cloud."

As a NoSQL database, Apache Cassandra handles massive amounts of data across load-intensive applications with high availability and no single point of failure. Cassandra’s largest production deployments include Apple (more than 160,000 instances storing over 100 petabytes of data across 1,000+ clusters), Huawei (more than 30,000 instances across 300+ clusters), and Netflix (more than 10,000 instances storing 6 petabytes across 100+ clusters, with over 1 trillion requests per day), among many others. Cassandra originated at Facebook in 2008, entered the Apache Incubator in January 2009, and graduated as an Apache Top-Level Project in February 2010.

Apache Cassandra v4.0
Cassandra v4.0 effortlessly handles unstructured data, with thousands of writes per second. Three years in the making, v4.0 reflects more than 1,000 bug fixes, improvements, and new features that include:

Increased speed and scalability – streams data up to 5 times faster during scaling operations, and up to 25% faster throughput on reads and writes, that delivers a more elastic architecture, particularly in Cloud and Kubernetes deployments.
Improved consistency – keeps data replicas in sync to optimize incremental repair for faster, more efficient operation and consistency across data replicas.
Enhanced security and observability – audit logging tracks users access and activity with minimal impact to workload performance. New capture and replay enables analysis of production workloads to help ensure regulatory and security compliance with SOX, PCI, GDPR, or other requirements.
New configuration settings – exposed system metrics and configuration settings provides flexibility for operators to ensure they have easy access to data that optimize deployments.
Minimized latency – garbage collector pause times are reduced to a few milliseconds with no latency degradation as heap sizes increase.
Better compression – improved compression efficiency eases unnecessary strain on disk space and improves read performance.

Cassandra 4.0 is community-hardened and tested by Amazon, Apple, DataStax, Instaclustr, iland, Netflix, and others that routinely run clusters as large as 1,000 nodes and with hundreds of real-world use cases and schemas.

The Apache Cassandra community deployed several testing and quality assurance (QA) projects and methodologies to deploy the most stable release yet. During the testing and QA period, the community generated reproducible workloads that are as close to real-life as possible, while effectively verifying the cluster state against the model without pausing the workload itself.

"In our experience, nothing beats Apache Cassandra for write scaling, and we're looking forward to the performance and management improvements in the 4.0 release," said Elliott Sims, Senior Systems Administrator at Backblaze. "We rely on Cassandra to manage over one exabyte of customer data and serve over 50 billion files for our customers across 175 countries so optimizing Cassandra's capabilities and performance means a lot to us."

"Since 2016, software engineers at Bloomberg have turned to Apache Cassandra because it’s easy to use, easy to scale, and always available," said Isaac Reath, Software Engineering Team Lead, NoSQL Infrastructure at Bloomberg. "Today, Cassandra is used to support a variety of our applications, from low-latency storage of intraday financial market data to high-throughput storage for fixed income index publication. We serve up more than 20 billion requests per day on a nearly 1 PB dataset across a fleet of 1,700+ Cassandra nodes."

"Netflix uses Apache Cassandra heavily to satisfy its ever-growing persistence needs on its mission to entertain the world. We have been experimenting and partially using the 4.0 beta in our environments and its features like Audit Logging and backpressure," said Vinay Chella, Netflix Engineering Manager and Apache Cassandra Committer. "Apache Cassandra 4.0's improved performance helps us reduce infrastructure costs. 4.0's stability and correctness allow us to focus on building higher-level abstractions on top of data store compositions, which results in increased developer velocity and optimized data store access patterns. Apache Cassandra 4.0 is faster, secure, and enterprise-ready; I highly suggest giving it a try in your environments today."

"Apache Cassandra's contributors have worked hard to deliver Cassandra 4.0 as the project's most stable release yet, ready for deployment to production-critical Cloud services," said Scott Andreas, Apache Cassandra Contributor. "Cassandra 4.0 also brings new features, such as faster host replacements, active data integrity assertions, incremental repair, and better compression. The project's investment in advanced validation tooling means that Cassandra users can expect a smooth upgrade. Once released, Cassandra 4.0 will also provide a stable foundation for development of future features and the database's long-term evolution."

Apache Cassandra is in use at Activision, Apple, Backblaze, BazaarVoice, Best Buy, Bloomberg Engineering, CERN, Constant Contact, Comcast, DoorDash, eBay, Fidelity, GitHub, Hulu, ING, Instagram, Intuit, Macy's, Macquarie Bank, Microsoft, McDonalds, Netflix, New York Times, Monzo, Outbrain, Pearson Education, Sky, Spotify, Target, Uber, Walmart, Yelp, and thousands of other companies that have large, active data sets. In fact, Cassandra is used by 40% of the Fortune 100. Select Apache Cassandra case studies are available at https://cassandra.apache.org/case-studies/

In addition to Cassandra 4.0, the Project also announced a shift to a yearly release cycle, with releases to be supported for a three-year term.

Catch Apache Cassandra in action through presentations from the April 2021 Cassandra World Party https://s.apache.org/jjv2d .

Availability and Oversight
Apache Cassandra software is released under the Apache License v2.0 and is overseen by a volunteer, self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache Cassandra, visit https://cassandra.apache.org/ and https://twitter.com/cassandra .

About Apache Cassandra
Apache Cassandra is an Open Source, distributed, wide column store, NoSQL database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure. Cassandra offers robust support for clusters spanning multiple datacenters, with asynchronous masterless replication allowing low latency operations for all clients. Apache Cassandra is used in some of the largest data management deployments in the world, including nearly half of the Fortune 100.

© The Apache Software Foundation. "Apache", "Cassandra", "Apache Cassandra", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 03:00PM Jul 27, 2021 by Sally Khudairi in General | |

Wednesday June 09, 2021

The Apache® Software Foundation Welcomes its Global Community Online at ApacheCon(TM) Asia 2021

Asia edition of the official Apache global conference series to be held virtually, with 140+ sessions, and keynote and plenary sessions by luminaries from AliCloud, API7, DiDi Chuxing, Huawei, Kyligence, PingCAP, Tencent Cloud, Tsinghua University, and more.

Wilmington, DE —9 June 2021— The Apache^® Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced keynotes, sponsors, and program for ApacheCon^TM Asia, taking place online 6-8 August 2021. Registration is open and free for all attendees.

"We’re excited to hold ApacheCon Asia online following last year’s highly successful ApacheCon@Home," said Sheng Wu, ApacheCon Asia co-Chair and member of the ASF Board of Directors. "The pandemic mobilized the global Apache community to collectively produce a first-rate online event, supported by an outstanding group of sponsors. We are proud to build on ApacheCon’s new virtual format and bring the ApacheCon Asia program to participants joining us from any location."

ApacheCon is the ASF's official global conference series, first held in 1998. ApacheCon draws attendees from more than 130 countries to experience "Tomorrow's Technology Today" independent of business interests, corporate biases, or sales pitches.

ApacheCon showcases the latest breakthroughs from dozens of Apache projects, with content selected entirely by Apache projects and their communities. ApacheCon Asia joins ApacheCon@Home, taking place online 21-23 September, to meet the educational demands of the growing Apache community of developers, users, and enthusiasts worldwide.

"Tune in to ApacheCon Asia's 140+ sessions to learn the latest developments, best practices, and lessons learned with Apache projects, incubating podlings, and community-led development 'The Apache Way',” said Willem Jiang, ApacheCon Asia co-Chair and initiator of Apache Local Community Beijing. "Participants can also connect and network virtually with attendees, speakers, and sponsors in real-time, as well as revisit presentations and explore additional tracks after the event."

Participants at all levels will learn about Apache project innovations in categories that include: APIs and Microservices; Big Data; Community; Culture; Data Visualization; Incubator; Integration; IoT and IIoT; Messaging; Middleware; Observability; Streaming; Servers; Workflow and Data Governance.

Featured Apache projects include Airflow, APISIX, Arrow, Atlas, Bigtop, BookKeeper, brpc (incubating), Camel, CarbonData, Cassandra, Commons, DolphinScheduler, Doris (incubating), Druid, Dubbo, ECharts, Flink, Hadoop, HBase, Hive, HUDI, Ignite, Impala, InLong (incubating), IoTDB, Kafka, Kudu, Kylin, Liminal (incubating), MXNet (incubating), Nemo (incubating), Ozone, Pegasus (incubating), Pinot (incubating), PLC4X, Pulsar, RocketMQ, ServiceComb, ShardingSphere, SkyWalking, Sling, Spark, StreamPipes (incubating), Superset, Teaclave (incubating), Tomcat, YuniKorn (Incubating), and more.

Keynote presentations will be delivered by Dongxu Huang, CTO of PingCAP; Jianmin Wang, Dean, School of Software at Tsinghua University; Sharan Foga, ASF Board Member; and Sheng Wu, ASF Board Member. Plenary sessions will be presented by AliCloud, API7, DiDi Chuxing, Huawei, Kyligence, and Tencent Cloud.

The full program is available at https://apachecon.com/acasia2021/tracks.html

ApacheCon Asia sponsors include Strategic Sponsor Huawei; Platinum Sponsors AliCloud, API7, DiDi Chuxing, Kyligence, and Tencent Cloud; Gold Sponsors AWS and Baidu; and Silver Sponsors Imply and SphereEx. Huawei, Tencent, DiDi, AWS, Baidu, Imply, and SphereEx are also Sponsors of ApacheCon@Home at the above levels.

To sponsor ApacheCon Asia and/or ApacheCon@Home, visit https://www.apachecon.com/acah2021/2021_ApacheCon_prospectus.pdf

About ApacheCon
ApacheCon is the official global conference series of The Apache Software Foundation. Since 1998 ApacheCon has been drawing participants at all levels to explore "Tomorrow's Technology Today" across 350+ Apache projects and their diverse communities. In 2020 and 2021 ApacheCon events showcase ubiquitous Apache projects and emerging innovations virtually through sessions, keynotes, real-world case studies, community events, and more, all online and free of charge. For more information, visit http://apachecon.com/ and https://twitter.com/ApacheCon .

About The Apache Software Foundation (ASF)
Established in 1999, The Apache Software Foundation is the world’s largest Open Source foundation, stewarding 227M+ lines of code and providing more than $20B+ worth of software to the public at 100% no cost. The ASF’s all-volunteer community grew from 21 original founders overseeing the Apache HTTP Server to 850+ individual Members and 200 Project Management Committees who successfully lead 350+ Apache projects and initiatives in collaboration with 8,200+ Committers through the ASF’s meritocratic process known as "The Apache Way". Apache software is integral to nearly every end user computing device, from laptops to tablets to mobile devices across enterprises and mission-critical applications. Apache projects power most of the Internet, manage exabytes of data, execute teraflops of operations, and store billions of objects in virtually every industry. The commercially-friendly and permissive Apache License v2 is an Open Source industry standard, helping launch billion dollar corporations and benefiting countless users worldwide. The ASF is a US 501(c)(3) not-for-profit charitable organization funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, Amazon Web Services, Anonymous, Baidu, Bloomberg, Budget Direct, Capital One, Cloudera, Comcast, Confluent, Didi Chuxing, Facebook, Google, Handshake, Huawei, IBM, Microsoft, Namebase, Pineapple Fund, Red Hat, Reprise Software, Target, Tencent, Union Investment, Verizon Media, and Workday. For more information, visit http://apache.org/ and https://twitter.com/TheASF .

© The Apache Software Foundation. "Apache", "Airflow", "Apache Airflow", "APISIX", "Apache APISIX", "Arrow", "Apache Arrow", "Atlas", "Apache Atlas", "Bigtop", "Apache Bigtop", "BookKeeper", "Apache BookKeeper", "Camel", "Apache Camel", "CarbonData", "Apache CarbonData", "Cassandra", "Apache Cassandra", "Commons", "Apache Commons", "DolphinScheduler", "Apache DolphinScheduler", "Druid", "Apache Druid", "Dubbo", "Apache Dubbo", "ECharts", "Apache ECharts", "Flink", "Apache Flink", "Hadoop", "Apache Hadoop", "HBase", "Apache HBase", "Hive", "Apache Hive", "HUDI", "Apache HUDI", "Ignite", "Apache Ignite", "Impala", "Apache Impala", "IoTDB", "Apache IoTDB", "Kafka", "Apache Kafka", "Kudu", "Apache Kudu", "Kylin", "Apache Kylin", "Ozone", "Apache Ozone", "PLC4X", "Apache PLC4X", "Pulsar", "Apache Pulsar", "RocketMQ", "Apache RocketMQ", "ServiceComb", "Apache ServiceComb", "ShardingSphere", "Apache ShardingSphere", "SkyWalking", "Apache SkyWalking", "Sling", "Apache Sling", "Spark", "Apache Spark", "Superset", "Apache Superset", "Tomcat", "Apache Tomcat", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 04:00PM Jun 09, 2021 by Sally Khudairi in ApacheCon | |

Tuesday February 16, 2021

The Apache Software Foundation Announces Apache® Gobblin™ as a Top-Level Project

Open Source distributed Big Data integration framework in use at Apple, CERN, Comcast, Intel, LinkedIn, Nerdwallet, PayPal, Prezi, Roku, Sandia National Labs, Swisscom, Verizon, and more.

Wilmington, DE —16 February 2021— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today Apache® Gobblin™ as a Top-Level Project (TLP).

Apache Gobblin is a distributed Big Data integration framework used in both streaming and batch data ecosystems. The project originated at LinkedIn in 2014, was open-sourced in 2015, and entered the Apache Incubator in February 2017.

"We are excited that Gobblin has completed the incubation process and is now an Apache Top-Level Project," said Abhishek Tiwari, Vice President of Apache Gobblin and software engineering manager at LinkedIn. "Since entering the Apache Incubator, we have completed four releases and grown our community the Apache Way to more than 75 contributors from around the world."

Apache Gobblin is used to integrate hundreds of terabytes and thousands of datasets per day by simplifying the ingestion, replication, organization, and lifecycle management processes across numerous execution environments, data velocities, scale, connectors, and more.

"Originally creating this project, seeing it come to life and solve mission-critical problems at many companies has been a very gratifying experience for me and the entire Gobblin team," said Shirshanka Das, Founder and CTO at Acryl Data, and member of the Apache Gobblin Project Management Committee.

As a highly scalable data management solution for structured and byte-oriented data in heterogeneous data ecosystems, Apache Gobblin makes the arduous task of creating and maintaining a modern data lake easy. It supports the three main capabilities required by every data team:

Ingestion and export of data from a variety of sources and sinks into and out of the data lake while supporting simple transformations.
Data Organization within the lake (e.g. compaction, partitioning, deduplication).
Lifecycle and Compliance Management of data within the lake (e.g. data retention, fine-grain data deletions) driven by metadata.

"Apache Gobblin supports deployment models all the way from a single-process standalone application to thousands of containers running in cloud-native environments, ensuring that your data plane can scale with your company’s growth," added Das.

Apache Gobblin is in use at Apple, CERN, Comcast, Intel, LinkedIn, Nerdwallet, PayPal, Prezi, Roku, Sandia National Laboratories, Swisscom, and Verizon, among many others.

"We chose Apache Gobblin as our primary data ingestion tool at Prezi because it proved to scale, and it is a swiss army knife of data ingestion," said Tamas Nemeth, Tech Lead and Manager at Prezi. "Today, we ingest, deduplicate, and compact more than 1200 Apache Kafka topics with its help, and this number is still growing. We are looking forward to continuing to contribute to the project and helping the community enable other companies to use Apache Gobblin."

"Apache Gobblin has been at the center stage of the data management story at LinkedIn. We leverage it for various use-cases ranging from ingestion, replication, compaction, retention, and more," said Kapil Surlaker, Vice President of Engineering at LinkedIn. "It is battle-tested and serves us well at exabyte scale. We firmly believe in the data wrangling capabilities that Gobblin has to offer, and we will continue to contribute heavily and collaborate with the Apache Gobblin community. We are happy to see that Gobblin has established itself as an industry standard and is now an Apache Top-Level Project."

"Open community and meritocracy are the key drivers for Apache Gobblin's success," added Tiwari. "We invite everyone interested in the data management space to join us and help shape the future of Gobblin."

Catch Apache Gobblin in action in the upcoming hackathon planned for late Q1 2021. Details will be posted on the Apache Gobblin mailing lists and Twitter feed listed below.

Availability and Oversight
Apache Gobblin software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache Gobblin, visit https://gobblin.apache.org/ and https://twitter.com/ApacheGobblin

About The Apache Software Foundation (ASF)
Established in 1999, The Apache Software Foundation is the world’s largest Open Source foundation, stewarding 227M+ lines of code and providing more than $20B+ worth of software to the public at 100% no cost. The ASF’s all-volunteer community grew from 21 original founders overseeing the Apache HTTP Server to 813 individual Members and 206 Project Management Committees who successfully lead 350+ Apache projects and initiatives in collaboration with nearly 8,000 Committers through the ASF’s meritocratic process known as "The Apache Way". Apache software is integral to nearly every end user computing device, from laptops to tablets to mobile devices across enterprises and mission-critical applications. Apache projects power most of the Internet, manage exabytes of data, execute teraflops of operations, and store billions of objects in virtually every industry. The commercially-friendly and permissive Apache License v2 is an Open Source industry standard, helping launch billion dollar corporations and benefiting countless users worldwide. The ASF is a US 501(c)(3) not-for-profit charitable organization funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, Amazon Web Services, Anonymous, Baidu, Bloomberg, Budget Direct, Capital One, Cloudera, Comcast, Didi Chuxing, Facebook, Google, Handshake, Huawei, IBM, Microsoft, Pineapple Fund, Red Hat, Reprise Software, Target, Tencent, Union Investment, Verizon Media, and Workday. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Gobblin", "Apache Gobblin", "Hadoop", "Apache Hadoop", "MapReduce", "Apache MapReduce", "Mesos", "Apache Mesos", "YARN", "Apache YARN", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 02:00PM Feb 16, 2021 by Sally Khudairi in General | |

Thursday January 21, 2021

The Apache Software Foundation Announces Apache® Superset™ as a Top-Level Project

Open Source enterprise-grade Big Data visualization and business intelligence Web application in use at Airbnb, American Express, Dropbox, Lyft, Netflix, Nielsen, Rakuten Viki, Twitter, and Udemy, among others.

Wilmington, DE —21 January 2021— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today Apache® Superset™ as a Top-Level Project (TLP).

Apache Superset is a modern, Open Source data exploration and visualization platform that enables users to easily and quickly build and explore dashboards using its simple no-code visualization builder and state-of-the-art SQL editor. The project originated at Airbnb in 2015 and entered into the Apache Incubator program in May 2017.

"It's been amazing to be an active part of growing a welcoming, diverse and engaged community over the past five years while following the ASF principles around inclusion, openness and collaboration," said Maxime Beauchemin, Vice President of Apache Superset. "At the scale and level of diversity that the Superset project has achieved, it's critical to have a solid governance model in place like the one prescribed by the ASF."

Apache Superset v1.0
Superset helps streamline the analytics process by providing an intuitive interface to rapidly explore and visualize datasets, create interactive dashboards, and model real-time business intelligence insights at scale. The platform integrates with most SQL speaking data sources, including modern cloud-native databases, data warehouses, and engines at petabyte scale.

The Project also celebrates a major milestone with the release of Apache Superset 1.0. Features include:

Rich library of visualizations with support for integrating custom visualizations
Thin caching layer to optimize performance of charts and dashboards
Code-free visualization builder
State-of-the-art SQL editor and metadata workflow
Extensible enterprise authentication and security model
Easy-to-use, lightweight semantic layer
Notification alerts and scheduled reports

"Apache Superset 1.0 is a solid, mature, self-standing solution that fully solves business intelligence and data visualization needs for modern data teams," added Beauchemin. "Superset not only covers the table stakes, but also offers guarantees, features and a fresh approach that existing BI solutions can't match."

Apache Superset is in use at Airbnb, American Express, Dropbox, Lyft, Netflix, Nielsen, Rakuten Viki, Twitter, and Udemy, among others. A list of known users is available at https://github.com/apache/superset/blob/master/INTHEWILD.md .

"Apache Superset helps Airbnb democratize data insights and make data-informed decisions," said Jeff Feng, Product Lead at Airbnb and member of the Apache Superset Project Management Committee. "Superset uniquely connects SQL analysis with data exploration for thousands of our employees each week. It also serves as a flexible and reliable platform for visualizing metrics, helping executives and knowledge workers see and understand data."

"We had an amazing journey with Superset at Dropbox," said Chloe Wang, Senior Product Manager, Data Insights Platform at Dropbox. "Superset got introduced in 2019 and soon became the most widely adopted query engine within the analytical organization. As a result, our analysts are able to make timely and high confidence product decisions."

"Before Superset, we were paying for a patchwork of proprietary tools and we kept running into limitations when it came to customizing charts and dashboards," said Amit Miran, Software Team Lead for Media Application Framework group at Nielsen. "Once the Superset project supported adding of custom visualizations, that was the turning point for us at Nielsen to start adopting Superset in large projects. We’re very excited about native dashboard filters and future support for cross filtering, which will make our viz plugins even more powerful. The excitement for the project drove me to become involved in my first open source project."

"Apache Superset is an amazing project that enables engineers to easily execute data analysis," said Grace Guo, member of the Apache Superset Project Management Committee. "I have been a Superset user and a Superset builder for a few years. I run queries in SQL Lab, visualize data using one of the many supported chart types, and build dashboards, specifically focusing on performance and product adoption metrics. As an engineer, I appreciate the ability to contribute to the product. If I see some area to improve, or need a feature which doesn’t exist, I am happy to create a PR to fix it for myself and benefit other users."

"Apache Superset’s strength lies in its community," added Beauchemin. "We invite those interested in data visualization to join our mailing lists and help shape future versions of Superset."

Learn more about the latest in v1.0 at the Apache Superset community global MeetUp on 28 January. Registration is open to all and free of charge https://s.apache.org/3cm4f

Availability and Oversight
Apache Superset software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache Superset, visit https://superset.apache.org/

About The Apache Software Foundation (ASF)
Established in 1999, The Apache Software Foundation is the world’s largest Open Source foundation, stewarding 227M+ lines of code and providing more than $20B+ worth of software to the public at 100% no cost. The ASF’s all-volunteer community grew from 21 original founders overseeing the Apache HTTP Server to 813 individual Members and 206 Project Management Committees who successfully lead 350+ Apache projects and initiatives in collaboration with nearly 8,000 Committers through the ASF’s meritocratic process known as "The Apache Way". Apache software is integral to nearly every end user computing device, from laptops to tablets to mobile devices across enterprises and mission-critical applications. Apache projects power most of the Internet, manage exabytes of data, execute teraflops of operations, and store billions of objects in virtually every industry. The commercially-friendly and permissive Apache License v2 is an Open Source industry standard, helping launch billion dollar corporations and benefiting countless users worldwide. The ASF is a US 501(c)(3) not-for-profit charitable organization funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, Amazon Web Services, Anonymous, Baidu, Bloomberg, Budget Direct, Capital One, Cloudera, Comcast, Didi Chuxing, Facebook, Google, Handshake, Huawei, IBM, Microsoft, Pineapple Fund, Red Hat, Reprise Software, Target, Tencent, Union Investment, Verizon Media, and Workday. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Superset", "Apache Superset", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 02:00PM Jan 21, 2021 by Sally Khudairi in General | |

Wednesday September 23, 2020

The Apache Software Foundation Announces Apache® IoTDB™ as a Top-Level Project

Open Source Internet of Things-native database integrates with the Apache Big Data ecosystem for high-speed data ingestion, massive data storage, and complex data analysis in the cloud, in the field, and on the edge.

Wakefield, MA —23 September 2020— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today Apache^® IoTDB™ as a Top-Level Project (TLP).

Apache IoTDB is an Open Source IoT database designed to meet the rigorous data, storage, and analytics requirements of large-scale Internet of Things (IoT) and Industrial Internet of Things (IIoT) applications. The project was first developed as a research project at Tsinghua University and entered the Apache Incubator in November 2018.

"The Internet of Things, especially Industrial IoT, has swept the globe with unimaginable volumes of data,” said Xiangdong Huang, Vice President of Apache IoTDB. "To date, both Relational and Key Value-based database solutions struggle to meet the demands of IoT data management. Apache IoTDB is the missing link between current IoT data and IoT applications, and is redefining how IoT data is managed, both in the cloud and on the edge. We are proud to graduate as an Apache Top-Level Project, which is an important milestone in our project’s maturity."

Apache IoTDB provides a compact and time series optimized columnar data file, which is able to efficiently store and access time series data. The database engine is specially optimized for time series-oriented operations, such as aggregations query, down-sampling, and time alignment query. Due to its lightweight structure, high performance, and deep integration with Apache Big Data ecosystem projects (such as Flink, Hadoop, and Spark), Apache IoTDB easily meets the requirements of storing massive data sets, ingesting high-speed data, and analyzing complex data, both on the edge and the cloud. Features include:

High-throughput read and write: supports high-speed write access for millions of low-power and intelligently networked devices, and provides lightning-quick read access for retrieving data on billions of data points.
Efficient directory structure: organizes complex metadata structure from IoT devices and large scale time series data, with fuzzy searching strategy for complex directory of time series data.
Rich query semantics: supports time alignment for time series data across devices and sensors, computation in time series field, and abundant aggregation functions in time dimension.
Flexible deployment: supports running on the edge (e.g., running on a Raspberry Pi), as well as forming a cluster in the cloud. It also provides a bridge tool between cloud platforms and data synchronization on premise machines.
Deep integration with Open Source Big Data projects: supports analysis ecosystems, including Apache Flink, Hadoop, PLC4X and Spark, as well as other Open Source applications.
Low hardware cost: reaches a high compression ratio of disk storage.

Apache IoTDB is in use at dozens of organizations that include ArcelorMittal AMERICA, BONC Ltd., the China Meteorological Administration, Datang Xianyi, Goldwind, Haier, Lenovo, NAVINFO, pragmatic industries GMBH, Shanghai Metro, Tsinghua University, Yangtze Optical Fiber and Cable Company, and more.

"IoTDB has attained Apache Top Level project status at a time of confluence of database, IoT and AI technologies in conjunction with a wider adoption of Industry 4.0 and automation approaches to further enable remote work and increased efficiencies," said Prof. C. Mohan, recently retired IBM Fellow, Former Chief Scientist of IBM India, and a member of the US National Academy of Engineering. "I am excited since this is the first Chinese University originated open-source project to reach this status. While I have been associated with the researchers behind IoTDB as a Distinguished Visiting Professor of the School of Software at China's prestigious Tsinghua University, I have seen this project reach maturity and build up a vibrant OSS community around it. It has a bright future ahead of it and I plan to collaborate on it."

"Apache IoTDB is a perfect fit for edge computing," said Dr. Julian Feinauer, CEO at pragmatic industries GmbH. "The high compression helps to use the (limited) amount of memory we have very efficiently. IoTDB is a perfect fit, especially in IIoT use cases, where network and compute capabilities are limited on the edge."

"Apache IoTDB was initially launched by a Chinese University and then incubated successfully in the Apache Community," said Prof. Hong Mei, an academician of the Chinese Academy of Sciences. "Following the Apache Way, it has created a healthy and active international open source community. It is a successful practice of open source education and culture advancement in China."

"Apache IoTDB has made many optimizations for different runtime environments, operating systems, and workloads in both the edge and the cloud. As a core infrastructure software in Industrial Internet, it innovates a series of IoT data management and analysis techniques," said Prof. Xiangke Liao, an academician of the Chinese Academy of Engineering. "Through the open source model, Apache IoTDB shares its creative techniques to the world."

"With the continuous growth of intelligent devices, machine-generated data is growing day by day, which poses extraordinary challenges on storing process, query speed, and storage space," said Dawei Liu, architect at AutoAI Inc., a subsidiary of NAVINFO, and member of the Apache IoTDB Project Management Committee. "We tried and tested a variety of solutions and finally chose IoTDB as our core database for its high performance, openness to the enterprise, and its active community. We built our Wecloud platform based on Apache IoTDB, which has served well for BMW, Toyota, and Great Wall Motors, among other auto manufacturers. The project deeply attracted me to become a part of the community. The coolest thing is that I finally became an IoTDB committer and now share our ideas to the community."

"Apache IoTDB is an open source project and software technology innovation developed for the need of AIoT Big Data applications," said Prof. Jianmin Wang, Dean of the Tsinghua University School of Software, who originally decided to donate the project to the ASF. "It is also a very beneficial attempt for training leading talents. There will be a long way to go and the future is promising."

"Apache IoTDB is on its way to becoming a standard IoT data management and analysis solution, and we’re excited to build upon our work thus far," added Huang. "We believe Apache IoTDB will help more users and companies to solve their real problems. The process to achieve the goal is exciting and honorable, and we invite more contributors to join us. Following the Apache Way, let's bring this interesting, meaningful, and powerful software to the whole world."

A published paper on Apache IoTDB written by members of the Apache IoTDB Project Management Committee is available at http://www.vldb.org/pvldb/vol13/p2901-wang.pdf . An introduction to Apache IoTDB from ApacheCon Europe 2019 is available on Feathercast https://feathercast.apache.org/2019/09/12/hello-world-introducing-apache-iotdb-a-database-for-the-internet-of-things-xiangdong-huang-julian-feinauer/

Catch Apache IoTDB in action at ApacheCon@Home, 29 September-1 October 2020 https://www.apachecon.com/acah2020/tracks/iot.html

Availability and Oversight
Apache IoTDB software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache IoTDB, visit http://iotdb.apache.org/ and https://twitter.com/ApacheIoTDB

About The Apache Software Foundation (ASF)
Established in 1999, The Apache Software Foundation (ASF) is the world’s largest Open Source foundation, stewarding 227M+ lines of code and providing more than $20B+ worth of software to the public at 100% no cost. The ASF’s all-volunteer community grew from 21 original founders overseeing the Apache HTTP Server to 813 individual Members and 206 Project Management Committees who successfully lead 350+ Apache projects and initiatives in collaboration with 7,800+ Committers through the ASF’s meritocratic process known as "The Apache Way". Apache software is integral to nearly every end user computing device, from laptops to tablets to mobile devices across enterprises and mission-critical applications. Apache projects power most of the Internet, manage exabytes of data, execute teraflops of operations, and store billions of objects in virtually every industry. The commercially-friendly and permissive Apache License v2 is an Open Source industry standard, helping launch billion dollar corporations and benefiting countless users worldwide. The ASF is a US 501(c)(3) not-for-profit charitable organization funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, Amazon Web Services, Anonymous, Baidu, Bloomberg, Budget Direct, Capital One, Cerner, Cloudera, Comcast, Facebook, Google, Handshake, Huawei, IBM, Inspur, Pineapple Fund, Red Hat, Target, Tencent, Union Investment, Verizon Media, and Workday. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "IoTDB", "Apache IoTDB", "Flink", "Apache Flink", "Hadoop", "Apache Hadoop", "Spark", "Apache Spark", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 11:01AM Sep 23, 2020 by Sally Khudairi in General | |

Wednesday May 13, 2020

The Apache Software Foundation Announces the 10th Anniversary of Apache® HBase™

Open Source distributed, scalable Big Data store celebrates a decade of processing zettabytes of data across highly scalable large tables for the Apache Hadoop ecosystem

Wakefield, MA —13 May 2020— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today the tenth Anniversary of Apache^® HBase^™, the distributed, scalable data store for the Apache Hadoop Big Data ecosystem.

"The success of Apache HBase is the success of Open Source," said Duo Zhang, Vice President of Apache HBase. "Ten years after graduating as a TLP, HBase is still among the most active projects at the ASF. We have hundreds of contributors all around the world. We speak different languages, we have different skills, but we all work together to make HBase better and better. Ten year anniversary is not the end, but a new beginning, I believe our strong community will lead the project to a bright future."

HBase originated at Powerset in 2006 as an Open Source system to run on Apache Hadoop’s Distributed File System (HDFS), similar to how BigTable ran on top of the Google File System. In 2007, a significant code contribution was added to the Apache Hadoop codebase and was integrated into the Apache Hadoop 0.15.0 release later that year. Development on HBase continued as a sub-project of Apache Hadoop, and graduated as an Apache Top-Level Project (TLP) in April 2010.

An Open Source, versioned, non-relational database, Apache HBase provides low latency random access to very large tables —billions of rows and millions of columns— atop clusters of non-specialized, commodity hardware. HBase reads, writes, and processes structured, semi-structured, and unstructured data in real-time environments.

Apache HBase is in use at thousands of organizations, including Adobe, Airbnb, Alibaba, Bloomberg, Flipkart, Huawei, HP, Hubspot, IBM, Microsoft, NetEase, Pinterest, Salesforce, Shopee, Tencent, Twitter, Xiaomi, and Yahoo! (now Verizon Media), among others.

Testimonials

"Congratulations on the 10th birthday of Apache HBase! Alibaba started to use HBase since January 2011 and has witnessed its growth and come along with the community through the years. The Apache HBase community has always been an open and powerful team that produced many stable, production-ready and widely used versions. Today at Alibaba, we have HBase clusters with more than 10k nodes serving hundreds of petabytes of data, as well as more than 1,000 enterprise HBase users on Alibaba Cloud. We will continue collaborating with and contributing to the HBase community and wish us all ongoing success in future!"
—Chunhui Shen and Yu Li, members of the HBase team at Alibaba

"I have worked with Apache HBase for many years and I think it is a great product. it does what it says on the tin so to speak. Ironically if you look around the NoSQL competitors, most of them are supported by start-ups, whereas HBase is only supported as part of Apache suite of products by vendors like Cloudera, Hortonworks, MapR, etc. For those who would prefer to use SQL on top, there is Apache Phoenix around which makes life easier for the most SQL-savvy world to work on HBase: problem solved. For TCO, HBase is still value for money compared to others. You don't need expensive RAM or SSD with HBase. That makes it easy to onboard it in no time. Also HBase can be used in a variety of different business applications, whereas other commercial ones are focused on narrower niche markets. Least but last happy 10th anniversary and hope HBase will go from strength to strength and we will keep using it for years to come!"
—Dr. Mich Talebzadeh, Chief Data Architect, Big Data

"Congratulations on the 10th anniversary of Apache HBase! Xiaomi started to use HBase in 2012, when our business started booming. Many key Xiaomi products and services, as well as Xiaomi's data analytics platform, require a new system to provide quick and random access to billions of rows of structured and semi-structured data. Traditional solutions are not able to handle the large volume of data brought by the quickly increasing Xiaomi user base. Among several available options, we choose HBase not only because it provides a rich set of features and excellent performance specs, but also because it has a very active, open and friendly community. Embracing open source has been part of Xiaomi's engineering culture, and our deep involvement in the development of Apache HBase demonstrates the best practices of Xiaomi's open source strategy. In the past several years, we have contributed tons of bug fixes and important features to HBase, and, in the meantime, we have contributed 9 committers and 3 PMC members to the HBase community. Looking forward, we will continue to work closely with the Apache HBase community to help the project grow, and we wish Apache HBase a wonderful future!"
—Dr. Baoqiu Cui, Vice President of Xiaomi Corporation and Technical Committee Chairman

“Congratulations on the 10th anniversary of Apache HBase, it’s great to see how the project has developed over the years and continues to have good community support around it! Salesforce has a large global footprint of Apache HBase in production storing multiple petabytes of customer data and serving several billions of queries per day for a wide variety of use cases including security, monitoring, collaboration portals, and performance caches to scale over RDBMS limitations. HBase has played a major role in Salesforce’s customer success in the BigData storage space and we continue to invest in it as one of the pillars of our multi-substrate database strategy along with Apache Phoenix for SQL access to data stored in HBase. We have contributed many features and bug fixes to HBase over the last several years, and we look forward to continue working with the Apache HBase community to develop the project further. Here’s to many more successful years for Apache HBase!”
—Sanjeev Lakshmanan, Senior Director, Software Development, Salesforce

“Happy 10th Apache HBase! It was around 8 years ago that we started looking at HBase to include as part of our Hosted Big Data Services stack. Fast-forward to today and it continues to be a critical offering in our stack, powering a diverse set of use cases and workloads such as ad targeting, content personalization, analytics, security, monitoring, etc. HBase enables these diverse workloads thanks to it’s high-scalability, feature set and performance, all of which have been continuously refined through the years. In turn our footprint continues to grow storing petabytes of data across thousands of machines. Our success is in part thanks to the project’s success as we benefit from our collaborations, the contributions and other efforts by the community (eg mailing list, meetups, HBaseCon, etc). This is a testament to the open, friendly and dedicated community around Apache HBase which is necessary for the success of any open source project. We wish the project continued success for years to come as we continue to collaborate with and be part of the community cultivating the project.”
—Francis Liu and Thiruvel Thirumoolan, HBase Big Data Team Members, Yahoo! (now Verizon Media)

“Congratulations on the 10th anniversary of Apache HBase! It’s great to see how this project has evolved from a big data project to one that runs business critical systems and continues to accelerate with a growing community and increasing pace of development! Cloudera has over 500 customers in production using it for a range of use cases ranging from mission critical transactional applications to supporting data warehousing. Our largest customers have footprints in excess of 7,000 nodes storing over 70PB of data. Our customers choose HBase because of its resilience with some customers able to realize 100% application uptime using HBase (over the past 3 years). We plan to continue to invest in HBase (and Apache Phoenix) to ensure that we can continue to both broaden support for a variety of hybrid transactional and analytical use cases and deepen support for existing use cases. Here's to many more successful years!"
—Arun C. Murthy, Chief Product Officer, Cloudera

“Many Congratulations to the Apache HBase community on the 10th anniversary. Apache HBase provides rich functions and excellent performance, and has an open and friendly community. Huawei started using HBase since 2010: HBase is widely used by multiple solutions of Huawei running on more than 10,000 nodes, storing hundreds of PBs data to meet our requirements. Huawei FusionInsight provides the Best Practices of Huawei for HBase, which serves a lot of customers across many industries such as finance, operators, government, energy, medical, manufacturing, and transportation. Meanwhile, Huawei team members contributed a lot of bug fixes and features to HBase, successfully hosted the first HBase Asia Technology Conference HBaseCon Asia 2017 at Shenzhen. Going forward, Huawei will continue to work closely with the Apache HBase community to promote community development.”
—Wei Zhi, Kai Mo and Pankaj Kumar, members of the HBase team at Huawei

“Happy 10th anniversary, HBase! At Ultra Tendency, you have been the backbone of our Dual Lambda Streaming Architecture for many years! You have served billions of queries to our customers without interruption and at low latency. Your architecture guaranteed that you were always there when we needed you, never letting us or our customers down. You are the reason why our European clients today are running flourishing new business models backed by low-latency streaming products. Our committers and contributors will continue to fix bugs and provide feature enhancements. Ultra Tendency wishes you a bright and successful future!”
—Jan Hentschel, Chief Information Officer, Ultra Tendency

“Congratulations on the 10th anniversary of Apache HBase, I can't believe it's been 10 years since the first day when I tried to use Apache HBase and its ecosystem to help the business and company. Also, it is so great to see many colleagues and friends work, discuss, cooperate together to make this system become better. Some of them also make great career development and some are still progress. Shopee, one of the biggest e-commerce platforms in Southeast Asia, has several large Apache HBase clusters in production to support businesses that depend on several billions of queries per day. Apache HBase has played a significant role in Shopee and it is still in expansion along with the business growth of Shopee. Apache HBase, as well as the community, helps us a lot and we also will continue to make contributions to Apache HBase. Looking forward to keeping working with the Apache HBase community to develop the project and its ecosystem further.”
—Li Luo, Manager of Data Infra department, Shopee

”At Microsoft, our mission is to empower every person and every organization on the planet to achieve more, and it’s this mission that drives our commitment to open source. Congratulations to the Apache HBase community on its 10th anniversary. Microsoft has been part of the vibrant HBase community since 2014, today we are proud to serve the numerous enterprise customers across industries who are leveraging HBase in Azure HDInsight for their most critical business applications.”
—Tomas Talius, Director of Engineering, Azure Data Services, Microsoft

Availability and Oversight
Apache HBase software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache HBase, visit http://hbase.apache.org/ and https://twitter.com/HBase

About The Apache Software Foundation (ASF)
Established in 1999, The Apache Software Foundation (ASF) is the world’s largest Open Source foundation, stewarding 200M+ lines of code and providing more than $20B+ worth of software to the public at 100% no cost. The ASF’s all-volunteer community grew from 21 original founders overseeing the Apache HTTP Server to 813 individual Members and 206 Project Management Committees who successfully lead 350+ Apache projects and initiatives in collaboration with 7,600+ Committers through the ASF’s meritocratic process known as "The Apache Way". Apache software is integral to nearly every end user computing device, from laptops to tablets to mobile devices across enterprises and mission-critical applications. Apache projects power most of the Internet, manage exabytes of data, execute teraflops of operations, and store billions of objects in virtually every industry. The commercially-friendly and permissive Apache License v2 is an Open Source industry standard, helping launch billion dollar corporations and benefiting countless users worldwide. The ASF is a US 501(c)(3) not-for-profit charitable organization funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, Amazon Web Services, Anonymous, ARM, Baidu, Bloomberg, Budget Direct, Capital One, CarGurus, Cerner, Cloudera, Comcast, Facebook, Google, Handshake, Huawei, IBM, Indeed, Inspur, Leaseweb, Microsoft, ODPi, Pineapple Fund, Private Internet Access, Red Hat, Target, Tencent, Union Investment, Verizon Media, and Workday. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "HBase", "Apache HBase", "Hadoop", "Apache Hadoop", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 01:00PM May 13, 2020 by Sally Khudairi in General | |

Thursday April 16, 2020

The Apache Software Foundation Announces Apache® ShardingSphere™ as a Top-Level Project

Open Source distributed Big Data middleware ecosystem used for partitioning data, distributed transactions, and database orchestration by more than 120 organizations, including video sharing site Bilibili, commercial bank China MINSHENG Bank, telecommunications and mobile provider China Telecom, eCommerce retailer JD.com, and delivery courier ZTO Express, among many others.

Wakefield, MA —16 April 2020— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today Apache® ShardingSphere™ as a Top-Level Project (TLP).

Apache ShardingSphere is a distributed Big Data middleware ecosystem. The project was originally developed at Dangdang Information Technology, and was submitted to the Apache Incubator in November 2018.

"Graduating as a Top-Level Project reflects the efforts of the Apache ShardingSphere community over the past year and a half," said Liang Zhang, Vice President of Apache ShardingSphere. "Since entering the Apache Incubator, ShardingSphere has evolved from a JDBC driver for sharding into a distributed ecosystem. We thank our mentors, contributors, and the Apache Incubator for their support, especially during the challenges with the coronavirus outbreak. Moreover, the community has been active and diverse, with more than 120 contributors from all over the world involved with the project."

The Apache ShardingSphere ecosystem has 3 sub-projects that form the database solutions, nicknamed “JPS”, for:

ShardingSphere-JDBC —a lightweight Java framework that provides extra service at the Java JDBC (“Java Database Connectivity”) layer. It provides service in the form of JAR (“Java ARchive”) that requires no additional deployment or dependencies. It can be considered as an enhanced JDBC driver, which is fully compatible with JDBC and all kinds of ORM (Object/Relational Mapping) frameworks.
ShardingSphere-Proxy —database proxy that provides a database server that encapsulates database binary protocol to support all developed languages and any terminal.
ShardingSphere-Sidecar (TODO) —a Cloud-native database agent of the Kubernetes environment that controls the access to the database in the form of sidecar (supporting services deployed with the main application). It provides a mesh layer interacting with the database, known as “Database Mesh”.

Apache ShardingSphere's highlights include:

Completely distributed database solution that provides data sharding, distributed transactions, data migration, as well as database and data governance features.
Independent SQL parser for multiple SQL dialects that can be used independently of ShardingSphere.
Pluggable micro-kernel that enables all SQL dialects, database protocols and features to be plugged-in and pulled-out by service provider interfaces.

Apache ShardingSphere is in use at more than 120 organizations, including Bilibili video sharing site, China MINSHENG Bank, China Telecom Bestpay, DaoCloud, JD.com, Tingyun, and ZTO Express, among others.

"Glad to see the ShardingSphere community and contributors grow actively," said Hao Zheng, Senior Director of Jingdong Digital Technology Center. "It has already promoted and pushed the IT architecture of many enterprises to improve rapidly. ShardingSphere is widely used across JD.com, which validates its power and flexibility. Congratulations on graduation of Apache ShardingSphere from the Incubator!"

"In the past two years, we have witnessed ShardingSphere grow from small to large," said XiaoHu Zhang, General Manager and Senior Director of China Telecom Bestpay Technology Innovation Center, and Apache ShardingSphere committer. "It's a vibrant community with a group of contributors who are constantly contributing to it. Congratulations! We graduated!"

"Today, the number of customers and scenarios faced by the enterprise is increasing exponentially," said Grissom Wang, Products Vice President of DaoCloud. "Therefore, application architecture needs to transform from a traditional monolithic architecture to a microservice architecture. At the same time, more flexible data governance capability is needed, which can inherit the most familiar relationship database technology to meet the increasing data volume or new data usage scenarios. Relational database middleware is a suitable solution: it allows applications to continue to use the relational database access method, and at the same time fully and reasonably utilize the computing and storage capabilities of multiple relational databases in a distributed scenario. We researched many Open Source technologies, and chose ShardingSphere as the core component of DaoCloud database governance because of its functional characteristics, openness, scalability, and active community that meet the needs of the enterprise."

"Congratulations to the Apache ShardingSphere community," said Von Gosling, Apache ShardingSphere Incubator Mentor, original developer of Apache RocketMQ and OpenMessaging. "Graduation from the Incubator marks an important milestone for the Apache ShardingSphere project. This is recognition of the focus and hard work of the project members to learn The Apache Way and drive community around ShardingSphere. I am honored to have helped the project to successfully graduate, and wish its continued development in Cloud-Native Era."

"I am glad to see the ShardingSphere community becomegraduate from the Apache Incubator," said Dongxu Huang, Founder and CTO of PingCAP. "The community should be very proud of their abiity to develop such good Open Source software. With the continued efforts of the Apache ShardingSphere community, I am confident of their continued success in the future!"

"Apache ShardingSphere is a good Open Source distributed database middleware solution," said Lixun Peng, Member of MariaDB Foundation, Oracle ACE Director, and Vice President of ACMUG, the All China MySQL User Group. "Open Source is the mainstream of the world's software development. It's nice to see Chinese enterprises and developers become more invested in Open Source. I hope ShardingSphere continues to grow as part of the family of excellent Apache Open Source products."

"The construction and growth of the Apache ShardingSphere community has promoted the impressive development of Open Source products with new options for enterprise IT architecture," said Grace Guo, Sales Director of MySQL. "Congratulations on the graduation of Apache ShardingSphere! Looking forward to building collaboration between Apache ShardingSphere and MySQL communities to provide more diversified solutions for Open Source technologies and enterprises!"

"It's fantastic to see the work of the Apache ShardingSphere community being recognised," said Martin Woodward, Director of Developer Relations at GitHub, "We've been thrilled to see the community grow really well over the past two years with now over 120 direct contributors. This is thanks to the great work of the maintainers welcoming people to their project, with support from the Apache Software Foundation and their mentors. The team have also done a superb job on their documentation with easy-to-understand instructions available in both English and Chinese. Congratulations to everyone involved: a valuable addition for the whole Java community!"

"I'm very glad to hear that ShardingSphere graduated successfully," said Yanwei Zhou, Founder of ArkDB and Chairman of the Open Source database committee of China Computer Industry Association. "Another Open Source database project led by Chinese technology enthusiasts has officially become an Apache Project, which will further promote the development of Open Source database architecture, allowing more and more users to share the benefits of the technology ecosystem. I look forward to it continuing to get better and better."

"Congratulations to ShardingSphere for graduating as an Apache Top Level Project," said Yuchen Zhao, President of Tingyun, "In the past a few years, I've been very excited to see the progress that the ShardingSphere community has made, and expect the project will grow tremendously in the near future and make a deeper impact on database orchestration. As data becomes increasingly crucial for the digital world, Apache ShardingSphere provides an essential set of distributed database middleware solutions and implementations for making IT architecture easier, more robust, and secure. I recommend Apache ShardingSphere to anyone interested in building database solutions on massive and distributed data."

"Since entering the Apache Incubator, the ShardingSphere community has adopted The Apache Way of self-governance and has substantially increased the number of people using, developing, and supporting the project," said Craig Russell, Apache ShardingSphere Incubator Mentor. "The community has worked hard to make several releases under the Apache License and are expanding ShardingSphere’s functionality to meet the needs of the expanding number of Cloud-based enterprises that use the project."

"Apache ShardingSphere is on its way to becoming a standard distributed database solution," added Zhang. "When we were developing our initial architectural features and database dialects, it was clear that we needed contributions beyond those from our small group of dedicated individuals to accomplish the task. Thanks to our growing community, we are pleased to be graduating with our release goals completed. We welcome additional contributors to join the project to further diversify the Apache ShardingSphere community and develop a more flexible and lightweight platform together. It is a pleasure to collaborate with contributors in the open, and promote a fair and friendly atmosphere where we can enrich ShardingSphere and its community the Apache Way."

Catch Apache ShardingSphere in action at ShardingSphere Workshop (Beijing; 18 April 2020), DTCC (Beijing; 4-6 June 2020), and TiD (Beijing; 26-29 July 2020).

Availability and Oversight

Apache ShardingSphere software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache ShardingSphere, visit http://shardingsphere.apache.org/ and https://twitter.com/ShardingSphere

About the Apache Incubator

The Apache Incubator is the primary entry path for projects and codebases wishing to become part of the efforts at The Apache Software Foundation. Code donations and communities from external organizations and existing external projects enter the ASF through the Incubator to: 1) ensure all donations are in accordance with the ASF legal standards; and 2) develop new communities that adhere to our guiding principles. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. For more information, visit http://incubator.apache.org/

About The Apache Software Foundation (ASF)

Established in 1999, The Apache Software Foundation (ASF) is the world’s largest Open Source foundation, stewarding 200M+ lines of code and providing more than $20B+ worth of software to the public at 100% no cost. The ASF’s all-volunteer community grew from 21 original founders overseeing the Apache HTTP Server to 765 individual Members and 206 Project Management Committees who successfully lead 350+ Apache projects and initiatives in collaboration with 7,600 Committers through the ASF’s meritocratic process known as "The Apache Way". Apache software is integral to nearly every end user computing device, from laptops to tablets to mobile devices across enterprises and mission-critical applications. Apache projects power most of the Internet, manage exabytes of data, execute teraflops of operations, and store billions of objects in virtually every industry. The commercially-friendly and permissive Apache License v2 is an Open Source industry standard, helping launch billion dollar corporations and benefiting countless users worldwide. The ASF is a US 501(c)(3) not-for-profit charitable organization funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, Amazon Web Services, Anonymous, ARM, Baidu, Bloomberg, Budget Direct, Capital One, CarGurus, Cerner, Cloudera, Comcast, Facebook, Google, Handshake, Huawei, IBM, Indeed, Inspur, Leaseweb, Microsoft, ODPi, Pineapple Fund, Private Internet Access, Red Hat, Target, Tencent, Union Investment, Verizon Media, and Workday. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "ShardingSphere", "Apache ShardingSphere", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 03:38PM Apr 16, 2020 by Sally Khudairi in General | |

Tuesday September 24, 2019

The Apache Software Foundation Announces Apache® Rya® as a Top-Level Project

Scalable Open Source Big Data database processes queries in milliseconds; used in autonomous drones, federated situation-aware access control systems, and petabyte-scale graphs modeling, among many other applications.

Wakefield, MA —24 September 2019— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today Apache^® Rya^® as a Top-Level Project (TLP).

Apache Rya (pronounced "ree-uh") is a Cloud-based Big Data triple store (subject-predicate-object) database used to process queries in milliseconds. The project was originally developed at the Laboratory for Telecommunication Sciences, and was submitted to the Apache Incubator in September 2015.

"We are very excited to reach this important milestone showing the maturity of the project and of the community around it," said Dr. Adina Crainiceanu, Vice President of Apache Rya and Associate Professor of Computer Science at the U.S. Naval Academy. "RDF (Resource Description Framework) triple data format is simple and flexible, making it easy to express diverse datasets such as connections between users on social media, financial data and transactions, medical data, and many others. Rya provides a scalable solution to store and query such data. The publication of the first research article about Rya garnered interest from industry, academia, and several government agencies. Bringing the project to ASF allowed collaboration and increased pace of development."

With its ability to store billions of linked information sets and return answers to most computer-based questions in under a second, Rya's scalable RDF data management system is built on top of Apache Accumulo® to support SPARQL queries for RDF data. A MongoDB back-end is also implemented. Rya uses novel storage methods, indexing schemes, and query processing techniques that scale to billions of triples across multiple nodes.

Rya is in use at organizations such as Enlighten IT Consulting, Modus Operandi, Parsons Corporation, Semantic Arts, Semantic Web Company, Sierra Nevada Corporation, and U.S. Department of Defense agencies. Apache Rya is recognized as one of the most advanced database projects in the United States Department of the Navy, powering a new generation of drones, advanced tactical communications through manned-unmanned teaming, and supporting autonomous swarms of smaller robots, among numerous other applications. In addition, Apache Rya is being used for artificial intelligence projects involving semi-autonomous content production operations.

"I would like to thank our mentors for their guidance and recognize the Apache Rya founders for making their project available for all to use and further extend," said Jennifer Brown, Project Manager for Semantic Technologies at Parsons Corporation, and member of the Apache Rya Project Management Committee. "In 2012 the founders introduced an RDF store backed by Apache Accumulo that was capable of basic inferencing, scaling to billions of triples, and providing millisecond query times. Our Semantic Technologies team at Parsons Corporation has enjoyed the opportunity to collaborate with the Apache Rya community to contribute new indexing strategies, query planner optimizations, additional inference capabilities, alerting extensions, native support for popular graph processing frameworks, and more."

"It's great to see Apache Rya has matured into a Top-Level Project. Rya is a very innovative and Open Source RDF data management system based on Big Data technology," said Dr. Zhiyuan Chen, Associate Professor, Information Systems Department, University of Maryland Baltimore County. "We have used Apache Rya in a variety of research projects ranging from more efficient query processing techniques over geographically distributed RDF data to situation-aware access control in federated systems. We found Rya very easy to use, easy to extend, and extremely efficient."

"Apache Rya has the potential to become one of the most scalable RDF data management systems on the market," said Andreas Blumauer, Founder and CEO, Semantic Web Company GmbH and Director, PoolParty Software Ltd.

"Our technology helps organizations discover the rare and hidden patterns with applied semantics enhancements and AI/ML analytics, to develop Living Intelligence in a data domain," said Kim Ziehlke, Principal Software Engineer at Modus Operandi. "Patterns are used to predict potential opportunities and threats and as a result, our clients can take preventative action, or take leading-edge advantage in complex decisions. Modus Operandi has achieved 2+ BILLION triples, sub-second queries, thousands of unstructured docs processed per second all backed by the Apache Rya triple store."

"Apache Rya is a foundational piece of technology on our projects," said Roshan Punnose, Technical Director at Enlighten IT Consulting and member of the Apache Rya Project Management Committee. "We use Rya to model graphs and entities at petabyte scale. It is the only technology that we have found to scale this type of information with field level visibilities, which allow data protections required on our projects. We have worked with Rya for 7 years and have benefitted from the work the Apache Rya team has done to help increase performance. We would like to thank all the contributors for their diligence and hard work in making Rya a first class citizen of the Apache community."

"Apache Rya is a very exciting project at The Apache Software Foundation that combines the world of 'Semantic Data' with that of 'Big Data'," said Christopher Tubbs, ASF Member and Project Management Committee (PMC) member of Apache Accumulo and Apache Fluo. "Implementing anything at scale can pose a challenge, but making semantic data searchable using familiar standards, such as SPARQL, and optimizing it at scale is really quite an amazing feat. Yet, that's precisely what the Rya community has done. Building on highly scalable platforms such as Apache Accumulo, the Rya community has produced an impressive platform for storing and querying very large semantic data sets. Apache Rya is something that every data scientist should get to know, because it's pretty cool."

"Apache Rya is an amazing project that enables users to execute SPARQL against large RDF data sets," said Keith Turner, Principal Software Engineer at Peterson Technologies, Vice President of Apache Fluo, member of the Apache Accumulo Project Management Committee, and ASF Member. "I had the pleasure to work with the Rya community when they asked for advice on using Apache Fluo for pre-computed joins. During our discussions, I found the folks working on Rya didn't need much assistance because they already had a thorough understanding of the complex issues surrounding distributed consistency. When considering using a piece of software that solves a hard problem for you, it’s comforting to know great minds stand behind it. With great software and wonderful community, what are you waiting for? Give Rya a try. Also, as an extra bonus Rya is an Apache community and all are welcome to participate in shaping the future of Rya."

"We are grateful for the mentorship provided by the Apache Incubator in building a diverse and open community and learning the Apache Way," added Crainiceanu. "We are looking forward to continuing our journey as a Top-Level Project."

Availability and Oversight

Apache Rya software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache Rya, visit http://rya.apache.org/ and https://twitter.com/apacherya

About the Apache Incubator

The Apache Incubator is the entry path for projects and codebases wishing to become part of the efforts at The Apache Software Foundation. All code donations from external organizations and existing external projects enter the ASF through the Incubator to: 1) ensure all donations are in accordance with the ASF legal standards; and 2) develop new communities that adhere to our guiding principles. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. For more information, visit http://incubator.apache.org/

About The Apache Software Foundation (ASF)

Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server --the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 760 individual Members and 7,300 Committers across six continents successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official user conference, trainings, and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, Anonymous, ARM, Baidu, Bloomberg, Budget Direct, Capital One, Cerner, Cloudera, Comcast, Facebook, Google, Handshake, Huawei, IBM, Indeed, Inspur, Leaseweb, Microsoft, ODPi, Pineapple Fund, Pivotal, Private Internet Access, Red Hat, Target, Tencent, Union Investment, Workday, and Verizon Media. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Accumulo", "Apache Accumulo", "Fluo", "Apache Fluo","Rya", "Apache Rya", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 12:59PM Sep 24, 2019 by Sally Khudairi in General | |

Tuesday February 19, 2019

The Apache® Software Foundation Announces Apache Arrow™ Momentum

Open Source Big Data in-memory columnar layer adopted by dozens of Open Source and commercial technologies; exceeded 1,000,000 monthly downloads within first three years as an Apache Top-Level Project

Wakefield, MA —19 February 2019— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, today announced momentum with Apache® Arrow™, the Open Source Big Data in-memory columnar layer.

Since the founding of the project in January 2016, Apache Arrow has quickly become the defacto standard for representing and processing analytical data in memory, accelerating analytical processing and interchange by more than 100x.

"When we became a Top-Level Project, we projected that the majority of the world's data will be processed through Arrow within the next decade," said Jacques Nadeau, Vice President of Apache Arrow. "In just three years time, we are proud to see Arrow's substantial industry adoption and increased value across a wide range of analytical, machine learning, and artificial intelligence workloads."

Highlights of Apache Arrow's success include:

Industry Adoption —more than 20 major technologies adopted Arrow to accelerate in-memory analytics, including Apache Spark, NVIDIA RAPIDS, pandas, and Dremio, among others. A list of known Open Source and commercial implementations can be found at https://arrow.apache.org/powered_by/

Millions of Downloads —leveraging and integrating Apache Arrow into many other technologies has bolstered downloads to more than 1,000,000 each month.

New Language Support —as a cross-language development platform, supporting multiple programming languages is paramount. Apache Arrow has grown from supporting one language to eleven different languages today; they include C++, Java, Python, R, C#, Javascript, and Ruby, among others.

Seamless Data Format Support —Arrow supports different data types, both simple and nested, located in arbitrary memory such as regular system RAM, memory-mapped files or on-GPU memory. In addition, it can ingest data from popular storage formats such as Apache Parquet, CSV files, Apache ORC, JSON, and more.

Major Code Donations —Apache Arrow's new features and expanded functionality are due in part to code and component donations that include:

C# Library
Gandiva LLVM-based Expression Compiler
Go Library
Javascript Library
Plasma Shared Memory Object Store
Ruby Libraries (Apache Arrow and Apache Parquet)
Rust Libraries (Parquet and DataFusion Query Engine)

Community and Contributor Growth —over the past 12 months, nearly 300 individuals have submitted more than 3,000 contributions that have grown the Apache Arrow code base by 300,000 lines of code. The Arrow community is welcoming approximately 10 new contributors each month.

In January the project announced its most recent release, Apache Arrow 0.12.0, which reflects more than 600 enhancements developed during Q4 2018. The Apache Arrow community is actively working on a number of impactful new initiatives that include solving high performance analytical problems and allowing for more efficient data distribution across entire clusters.

"Apache Arrow's rapid industry adoption and developer community growth supports our original thesis of the importance of a language-independent open standard for columnar data," said Wes McKinney, member of the Apache Arrow Project Management Committee, and creator of Python's pandas project. "Additionally, we are seeing productive collaborations take place not only between programming languages but also between the database systems and data science worlds. We look forward to welcoming more data system developers into our community."

About Apache Arrow

Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy streaming messaging and interprocess communication. Languages currently supported include C, C++, C#, Go, Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.

Availability and Oversight

Apache Arrow software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache Arrow, visit http://arrow.apache.org/

About The Apache Software Foundation (ASF)

Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server --the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 730 individual Members and 7,000 Committers across six continents successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official global conference series. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, Anonymous, ARM, Baidu, Bloomberg, Budget Direct, Capital One, Cerner, Cloudera, Comcast, Facebook, Google, Handshake, Hortonworks, Huawei, IBM, Indeed, Inspur, LeaseWeb, Microsoft, Oath, ODPi, Pineapple Fund, Pivotal, Private Internet Access, Red Hat, Target, Tencent, Union Investment, and Workday. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Arrow", "Apache Arrow", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 11:00AM Feb 19, 2019 by Sally Khudairi in General | |

Wednesday January 23, 2019

The Apache Software Foundation Announces Apache® Hadoop® v3.2.0

Pioneering Open Source distributed enterprise framework powers US$166B Big Data ecosystem

Wakefield, MA —23 January 2019— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, today announced Apache® Hadoop® v3.2.0, the latest version of the Open Source software framework for reliable, scalable, distributed computing.

Now in its 11th year, Apache Hadoop is the foundation of the US$166B Big Data ecosystem (source: IDC) by enabling data applications to run and be managed on large hardware clusters in a distributed computing environment. "Apache Hadoop has been at the center of this big data transformation, providing an ecosystem with tools for businesses to store and process data on a scale that was unheard of several years ago," according to Accenture Technology Labs.

"This latest release unlocks the powerful feature set the Apache Hadoop community has been working on for more than nine months," said Vinod Kumar Vavilapalli, Vice President of Apache Hadoop. "It further diversifies the platform by building on the cloud connector enhancements from Apache Hadoop 3.0.0 and opening it up for deep learning use-cases and long-running apps."

Apache Hadoop 3.2.0 highlights include:

ABFS Filesystem connector —supports the latest Azure Datalake Gen2 Storage;
Enhanced S3A connector —including better resilience to throttled AWS S3 and DynamoDB IO;
Node Attributes Support in YARN —helps to tag multiple labels on the nodes based on its attributes and supports placing the containers based on expression of these labels;
Storage Policy Satisfier —supports HDFS (Hadoop Distributed File System) applications to move the blocks between storage types as they set the storage policies on files/directories;
Hadoop Submarine —enables data engineers to easily develop, train and deploy deep learning models (in TensorFlow) on very same Hadoop YARN cluster;
C++ HDFS client —helps to do async IO to HDFS which helps downstream projects such as Apache ORC;
Upgrades for long running services —supports in-place seamless upgrades of long running containers via YARN Native Service API (application program interface) and CLI (command-line interface).

"This is one of the biggest releases in Apache Hadoop 3.x line which brings many new features and over 1,000 changes," said Sunil Govindan, Apache Hadoop 3.2.0 release manager. "We are pleased to announce that Apache Hadoop 3.2.0 is available to take your data management requirements to the next level. Thanks to all our contributors who helped to make this release happen."

Apache Hadoop is widely deployed at numerous enterprises and institutions worldwide, such as Adobe, Alibaba, Amazon Web Services, AOL, Apple, Capital One, Cloudera, Cornell University, eBay, ESA Calvalus satellite mission, Facebook, foursquare, Google, Hortonworks, HP, Huawei, Hulu, IBM, Intel, LinkedIn, Microsoft, Netflix, The New York Times, Rackspace, Rakuten, SAP, Tencent, Teradata, Tesla Motors, Twitter, Uber, and Yahoo. The project maintains a list of educational and production users, as well as companies that offer Hadoop-related services at https://wiki.apache.org/hadoop/PoweredBy

Global Knowledge hails, "...the open-source Apache Hadoop platform changes the economics and dynamics of large-scale data analytics due to its scalability, cost effectiveness, flexibility, and built-in fault tolerance. It makes possible the massive parallel computing that today's data analysis requires."

Hadoop is proven at scale: Netflix captures 500+B daily events using Apache Hadoop. Twitter uses Apache Hadoop to handle 5B+ sessions a day in real time. Twitter’s 10,000+ node cluster processes and analyzes more than a zettabyte of raw data through 200B+ tweets per year. Facebook’s cluster of 4,000+ machines that store 300+ petabytes is augmented by 4 new petabytes of data generated each day. Microsoft uses Apache Hadoop YARN to run the internal Cosmos data lake, which operates over hundreds of thousands of nodes and manages billions of containers per day.

Transparency Market Research recently reported that the global Hadoop market is anticipated to rise at a staggering 29% CAGR with a market valuation of US$37.7B by the end of 2023.

Apache Hadoop remains one of the most active projects at the ASF: it ranks #1 for Apache project repositories by code commits, and is the #5 repository by size (3,881,797 lines of code).

"The Apache Hadoop community continues to go from strength to strength in further driving innovation in Big Data," added Vavilapalli. "We hope that developers, operators and users leverage our latest release in fulfilling their data management needs."

Catch Apache Hadoop in action at the Strata conference, 25-28 March 2019 in San Francisco, and dozens of Hadoop MeetUps held around the world, including on 30 January 2019 at LinkedIn in Sunnyvale, California.

Availability and Oversight

Apache Hadoop software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache Hadoop, visit http://hadoop.apache.org/ and https://twitter.com/hadoop

About The Apache Software Foundation (ASF)

Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server --the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 730 individual Members and 7,000 Committers across six continents successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official global conference series. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, Anonymous, ARM, Baidu, Bloomberg, Budget Direct, Capital One, Cerner, Cloudera, Comcast, Facebook, Google, Handshake, Hortonworks, Huawei, IBM, Indeed, Inspur, LeaseWeb, Microsoft, Oath, ODPi, Pineapple Fund, Pivotal, Private Internet Access, Red Hat, Target, Tencent, and Union Investment. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Hadoop", "Apache Hadoop", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 11:00AM Jan 23, 2019 by Sally Khudairi in General | |

Tuesday January 08, 2019

The Apache Software Foundation Announces Apache® Airflow™ as a Top-Level Project

Open Source Big Data workflow management system in use at Adobe, Airbnb, Etsy, Google, ING, Lyft, PayPal, Reddit, Square, Twitter, and United Airlines, among others.

Wakefield, MA —8 January 2019— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today Apache® Airflow™ as a Top-Level Project (TLP).

Apache Airflow is a flexible, scalable workflow automation and scheduling system for authoring and managing Big Data processing pipelines of hundreds of petabytes. Graduation from the Apache Incubator as a Top-Level Project signifies that the Apache Airflow community and products have been well-governed under the ASF's meritocratic process and principles.

"Since its inception, Apache Airflow has quickly become the de-facto standard for workflow orchestration," said Bolke de Bruin, Vice President of Apache Airflow. "Airflow has gained adoption among developers and data scientists alike thanks to its focus on configuration-as-code. That has gained us a community during incubation at the ASF that not only uses Apache Airflow but also contributes back. This reflects Airflow’s ease of use, scalability, and power of our diverse community; that it is embraced by enterprises and start-ups alike, allows us to now graduate to a Top-Level Project."

Apache Airflow is used to easily orchestrate complex computational workflows. Through smart scheduling, database and dependency management, error handling and logging, Airflow automates resource management, from single servers to large-scale clusters. Written in Python, the project is highly extensible and able to run tasks written in other languages, allowing integration with commonly used architectures and projects such as AWS S3, Docker, Apache Hadoop HDFS, Apache Hive, Kubernetes, MySQL, Postgres, Apache Zeppelin, and more. Airflow originated at Airbnb in 2014 and was submitted to the Apache Incubator March 2016.

Apache Airflow is in use at more than 200 organizations, including Adobe, Airbnb, Astronomer, Etsy, Google, ING, Lyft, NYC City Planning, Paypal, Polidea, Qubole, Quizlet, Reddit, Reply, Solita, Square, Twitter, and United Airlines, among others. A list of known users can be found at https://github.com/apache/incubator-airflow#who-uses-apache-airflow

"Adobe Experience Platform is built on cloud infrastructure leveraging open source technologies such as Apache Spark, Kafka, Hadoop, Storm, and more," said Hitesh Shah, Principal Architect of Adobe Experience Platform. "Apache Airflow is a great new addition to the ecosystem of orchestration engines for Big Data processing pipelines. We have been leveraging Airflow for various use cases in Adobe Experience Cloud and will soon be looking to share the results of our experiments of running Airflow on Kubernetes."

"Our clients just love Apache Airflow. Airflow has been a part of all our Data pipelines created in past 2 years acting as the ring-master and taming our Machine Learning and ETL Pipelines," said Kaxil Naik, Data Engineer at Data Reply. "It has helped us create a Single View for our client's entire data ecosystem. Airflow's Data-aware scheduling and error-handling helped automate entire report generation process reliably without any human-intervention. It easily integrates with Google Cloud (and other major cloud providers) as well and allows non-technical personnel to use it without a steep learning curve because of Airflow’s configuration-as-a-code paradigm."

"With over 250 PB of data under management, PayPal relies on workflow schedulers such as Apache Airflow to manage its data movement needs reliably," said Sid Anand, Chief Data Engineer at PayPal. "Additionally, Airflow is used for a range of system orchestration needs across many of our distributed systems: needs include self-healing, autoscaling, and reliable [re-]provisioning."

"Since our offering of Apache Airflow as a service in Sept 2016, a lot of big and small enterprises have successfully shifted all of their workflow needs to Airflow," said Sumit Maheshwari, Engineering Manager at Qubole. "At Qubole, not only are we a provider, but also a big consumer of Airflow as well. For example, our whole Insight and Recommendations platform is built around Airflow only, where we process billions of events every month from hundreds of enterprises and generate insights for them on big data solutions like Apache Hadoop, Apache Spark, and Presto. We are very impressed by the simplicity of Airflow and ease at which it can be integrated with other solutions like clouds, monitoring systems or various data sources."

"At ING, we use Apache Airflow to orchestrate our core processes, transforming billions of records from across the globe each day," said Rob Keevil, Data Analytics Platform Lead at ING WB Advanced Analytics. "Its feature set, Open Source heritage and extensibility make it well suited to coordinate the wide variety of batch processes we operate, including ETL workflows, model training, integration scripting, data integrity testing, and alerting. We have played an active role in Airflow development from the onset, having submitted hundreds of pull requests to ensure that the community benefits from the Airflow improvements created at ING. We are delighted to see Airflow graduate from the Apache Incubator, and look forward to see where this exciting project will be taken in future!"

"We saw immediately the value of Apache Airflow as an orchestrator when we started contributing and using it," said Jarek Potiuk, Principal Software Engineer at Polidea. "Being able to develop and maintain the whole workflow by engineers is usually a challenge when you have a huge configuration to maintain. Airflow allows your DevOps to have a lot of fun and still use the standard coding tools to evolve your infrastructure. This is 'infrastructure as a code' at its best."

"Workflow orchestration is essential to the (big) data era that we live in," added de Bruin. "The field is evolving quite fast and the new data thinking is just starting to make an impact. Apache Airflow is a child of the data era and therefore very well positioned, and is also young so a lot of development can still happen. Airflow can use bright minds from scientific computing, enterprises, and start-ups to further improve it. Join the community, it is easy to hop on!"

Availability and Oversight
Apache Airflow software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache Airflow, visit http://airflow.apache.org/ and https://twitter.com/ApacheAirflow

About The Apache Software Foundation (ASF)
Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server --the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 730 individual Members and 7,000 Committers across six continents successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official user conference, trainings, and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, Anonymous, ARM, Baidu, Bloomberg, Budget Direct, Capital One, Cerner, Cloudera, Comcast, Facebook, Google, Handshake, Hortonworks, Huawei, IBM, Indeed, Inspur, LeaseWeb, Microsoft, Oath, ODPi, Pineapple Fund, Pivotal, Private Internet Access, Red Hat, Target, Tencent, and Union Investment. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Airflow", "Apache Airflow", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 11:00AM Jan 08, 2019 by Sally Khudairi in General | |

Thursday August 23, 2018

The Apache Software Foundation Announces Apache® HAWQ® as a Top-Level Project

Advanced Big Data query engine and analytic database in use at Alibaba, Haier, VMWare, ZTESoft, and hundreds more.

Wakefield, MA —23 August 2018— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today Apache^® HAWQ^® as a Top-Level Project (TLP).

Apache HAWQ is an advanced enterprise SQL-on-Hadoop query engine and analytic database. It combines the key technological advantages of MPP database with the scalability and convenience of Apache Hadoop. HAWQ reads data from and writes data to HDFS natively, delivers industry-leading performance and linear scalability, and provides users with a complete, standards compliant SQL interface.

"We are very excited to see Apache HAWQ graduate as a Top-Level Project and we would like to thank our Incubation mentors for all their help," said Dr. Lei Chang, Vice President of Apache HAWQ. "This is a huge milestone that reflects the collective contributions from the growing global community to deliver a world-class SQL engine for analytics."

HAWQ operates natively in Apache Hadoop to provide users the tools to confidently and successfully interact with petabyte-range data sets. Features include:

Exceptional performance: parallel processing architecture delivers high performance throughput and low latency —potentially near real time— query responses that can scale to petabyte-sized datasets;
Robust ANSI SQL compliance: leverage familiar skills. Achieve higher levels of compatibility for SQL-based applications and BI/data visualization tools. Execute complex queries and joins, including roll-ups and nested queries; and
Apache Hadoop ecosystem integration: integrate and manage with Apache YARN. Provision with Apache Ambari. Interface with Apache HCatalog. Supports Apache Parquet, Apache HBase, and others. Easily scales nodes up or down to meet performance or capacity requirements.

Apache HAWQ is in use at Alibaba, Haier, VMware, ZTESoft, and hundreds of users around the world.

"We admire Apache HAWQ's flexible framework and ability to scale up in a Cloud ecosystem. HAWQ helps those seeking a heterogeneous computing system to handle ad-hoc queries and heavy batch workloads," said Kuien Liu, Computing Platform Architect at Alibaba. "Alibaba encourages more and more engineers to continue to embrace Open Source, and Apache HAWQ stands out as a star project. We are proud to have been collaborating with this community since 2015."

"Haier Group has deployed clusters of more than 30 nodes in the production environment from the very beginning of HAWQ," said Xiaoliang Wu, Big Data Architect at Haier. "We use HAWQ as an ad-hoc query and batch computation engine in areas such as social network services and IOT. Because of its superior scalability and stability, HAWQ greatly improves development efficiency and reduces operation and maintenance costs. We believe that Apache HAWQ is a very competitive product in the SQL-On-Hadoop field."

"We have been using Apache HAWQ at VMware for 4 years now," said Dominie Jacob, Lead Big Data Engineer at VMware Inc. "It is easy to manage and scale using Apache Ambari, and easy to provision and attach more nodes based on demand. Being virtualized, it is easy to provision and attach more nodes based on demand. In our BI Big Data world, HAWQ is the primary database for accessing the Hadoop datasets, building models, and executing predictive model workflows. HAWQ is working seamlessly with billions of records, thousands of Tables/Functions/Tableau-Reports, and hundreds of users. The demand for HAWQ is increasing. As VMware always encourages us to pick up and contribute back to Open Source technologies, we would love to collaborate with this community and see more enhancements. In our BI space, HAWQ is one of the top priorities."

"Apache HAWQ is an attractive technology for Big Data applications," said Zixu Zhao, Architect at ZTESoft. "HAWQ serves as the foundation of our Big Data platform and it has been used in a lot of applications, such as interactive analytics and BI on telecom data. We congratulate HAWQ on becoming an Apache Top-Level Project."

"Becoming an Apache Top-Level Project is an important milestone," added Chang. "There is much work ahead of us, and we look forward to growing the HAWQ community and codebase."

Catch Apache HAWQ in action at ApacheCon North America 24-27 September 2018 http://apachecon.com/acna18/ .

Availability and Oversight

Apache HAWQ software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache HAWQ, visit http://hawq.apache.org/ and https://twitter.com/ApacheHAWQ .

About The Apache Software Foundation (ASF)
Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server --the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 730 individual Members and 6,800 Committers across six continents successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official user conference, trainings, and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Aetna, Anonymous, ARM, Bloomberg, Budget Direct, Capital One, Cerner, Cloudera, Comcast, Facebook, Google, Hortonworks, Huawei, IBM, Indeed, Inspur, LeaseWeb, Microsoft, Oath, ODPi, Pineapple Fund, Pivotal, Private Internet Access, Red Hat, Target, and Union Investment. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "HAWQ", "Apache HAWQ", "Ambari", "Apache Ambari", "Hadoop", "Apache Hadoop", "HBase", "Apache HBase", "HCatalog", "Apache HCatalog", "Parquet", "Apache Parquet", "YARN", "Apache YARN", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 10:00AM Aug 23, 2018 by Sally Khudairi in General | |

Thursday May 17, 2018

The Apache® Software Foundation Announces Agenda, Keynotes, and Sponsors for ApacheCon™ North America 2018

Community-driven conference series to gather dozens of Apache projects and their communities in Montréal to share and learn about the latest Open Source innovations in Big Data, Cloud, Finance, IoT, Machine Learning, Search, Servers, and more in a collaborative, vendor-neutral environment

Wakefield, MA —17 May 2018— The Apache® Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today the program for its official conference series, ApacheCon™, taking place 24-27 September 2018 in Montréal, Canada.

ApacheCon showcases the latest breakthroughs from ubiquitous Apache projects and upcoming innovations in the Apache Incubator, as well as Open Source development and leading community-driven projects "The Apache Way".

"The Apache Software Foundation continues to lead in community-driven development that accelerates innovation across the Open Source ecosystem," said Rich Bowen, Vice President of Conferences at the ASF. "ApacheCon delivers state-of-the-art content, directly from Apache project developer and user communities."

Over the past 19 years, ApacheCon has drawn attendees from more than 60 countries to learn about core Open Source technologies independent of business interests, corporate biases, or sales pitches.

Topics include Big Data, Cloud, Community, Finance, Graphs, HTTP Server, IoT, Machine Learning, Messaging, NoSQL, Programming, Search, Security, Streaming, and more.

Dedicated Apache project tracks/co-located events include Apache Spark and Scala half-day workshop with IBM; CloudStack Collaboration Conference, TomcatCon, and Geospatial Track, organized by the Open Geospatial Consortium (OGC).

Sessions and Speakers

More than 100 presentations on dozens of popular Apache projects, including Avro, Beam, CouchDB, Drill, Fineract, Groovy, Hadoop, HAWQ, HBase, Hive, HTTP Server, Kafka, Karaf, Lucene, MyNewt, NetBeans, OpenNLP, ORC, Parquet, Phoenix, Ranger, RocketMQ, Solr, Spark, Struts, Unomi, YARN, and Zeppelin.

Featured projects undergoing development in the Apache Incubator include Druid, Dubbo, Gobblin, Hivemall, MXNet, OpenWhisk, PLC4X, Pulsar, S2Graph, and Traffic Control.

The complete schedule is available at https://apachecon.com/acna18/

Keynotes

Tuesday, 25 September

Cliff Schmidt, Apache Member, former ASF Board member, and Literacy Bridge founder on how Amplio uses technology to educate and improve the quality of life of people living in very difficult parts of the world.
Myrle Krantz, Apache Member and Vice President Apache Fineract, on how Open Source banking is helping the global fight against poverty.

Wednesday, 26 September

Bridget Kromhout, Principal Cloud Developer Advocate at Microsoft, on the really hard problem in software: the people.
Euan McLeod, ‎VP VIPER at ‎Comcast, on the many ways that Apache software delivers your favorite shows to your living room.

Events and Activities

ApacheCon will also feature popular sessions that include Lightning Talks, BarCampApache, Hackathon, PGP key signing, and ample networking opportunities with Apache projects and their communities.

Sponsors

ApacheCon is made possible by Platinum Sponsors Comcast and IBM; Gold Sponsor Amazon; Silver Sponsors Linode and Red Hat; and Bronze Sponsor ShapeBlue, who join Apache event sponsors CloudOps, Dito, GridGain, and Talener. Sponsorship opportunities are available for ApacheCon, as well as Apache Roadshow Berlin and Washington DC: contact help@apachecon.com for further details.

Venue

ApacheCon North America 2018 will be held at the Montréal Marriott Chateau Champlain in Montréal, Canada. ApacheCon has secured room blocks with special rates for sleeping rooms ($225 CAD per night, including WiFi) available through 24 August.

Registration

Registration for ApacheCon is open. Early registration incentives include savings of $225 off the general rate of $800 through 21 July, as well as special discounts for Apache Committers.

Members of the media and analyst community who would like to reserve a complimentary press pass to ApacheCon should contact Sally Khudairi at press@apache.org.

About ApacheCon

ApacheCon is the official global conference series of The Apache Software Foundation. Since 2000 ApacheCon has been drawing participants at all levels to explore ”Tomorrow’s Technology Today” across 300+ Apache projects and their diverse communities. ApacheCon showcases the latest developments in ubiquitous Apache projects and emerging innovations through hands-on sessions, keynotes, real-world case studies, trainings, hackathons, community events, and more. For more information, visit http://apachecon.com/ and https://twitter.com/ApacheCon .

About The Apache Software Foundation (ASF)

Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server —the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 680 individual Members and 6,500 Committers across six continents successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official user conference, trainings, and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, ARM, Baidu, Bloomberg, Budget Direct, Capital One, Cash Store, Cerner, Cloudera, Comcast, Facebook, Google, Hortonworks, Huawei, IBM, Indeed, Inspur, iSIGMA, ODPi, LeaseWeb, Microsoft, PhoenixNAP, Pivotal, Private Internet Access, Red Hat, Target, Union Investment, and Yahoo. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Avro", "Apache Avro", "Beam", "Apache Beam", "CouchDB", "Apache CouchDB", "Drill", "Apache Drill", "Fineract", "Apache Fineract", "Groovy", "Apache Groovy", "Hadoop", "Apache Hadoop", "HAWQ", "Apache HAWQ", "HBase", "Apache HBase", "Hive", "Apache Hive", "Apache HTTP Server", "Kafka", "Apache Kafka", "Karaf", "Apache Karaf", "Lucene", "Apache Lucene", "MyNewt", "Apache MyNewt", "NetBeans", "Apache NetBeans", "OpenNLP", "Apache OpenNLP", "ORC", "Apache ORC", "Parquet", "Apache Parquet", "Phoenix", "Apache Phoenix", "Ranger", "Apache Ranger", "RocketMQ", "Apache RocketMQ", "Solr", "Apache Solr", "Spark", "Apache Spark", "Struts", "Apache Struts", "Unomi", "Apache Unomi", "YARN", "Apache YARN", "Zeppelin", "Apache Zeppelin", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 10:50AM May 17, 2018 by Sally Khudairi in General | |

Wednesday April 18, 2018

The Apache Software Foundation Announces Apache® Oozie(TM) v5.0.0

Open Source workflow scheduler for Apache Hadoop used to build complex Big Data transformations.

Wakefield, MA —18 April 2018— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today Apache® Oozie^TM v5.0.0, the workflow scheduler for Apache Hadoop.

Apache Oozie is a scalable, reliable, and extensible Java Web application used for job workflow scheduling and operational services management within an Apache Hadoop cluster. Integrated with the Hadoop stack, Oozie supports jobs for Apache projects such as Spark, Hive, MapReduce, Pig, and Sqoop, and can also schedule system-specific jobs, such as Java programs and shell scripts. The project entered the Apache Incubator in 2011, and graduated as an Apache Top-Level Project in 2012.

"Apache Oozie 5's flagship feature, Oozie on YARN, started off as a 1 day hackathon project almost 4 years ago, and it's great to see that the Oozie community has taken it on and made it ready for everyone to use," said Robert Kanter, Vice President of Apache Oozie. "It's a big change to Oozie's architecture, and I think our users are going to be very happy with the benefits it brings."

Apache Oozie allows cluster administrators to build complex Big Data transformations out of multiple component tasks. This provides greater control over jobs and also makes it easier to repeat those jobs at predetermined intervals.

Oozie combines multiple jobs sequentially into one logical unit of work through 1) Oozie Workflow jobs -- Directed Acyclic Graphs (DAGs) of actions; and 2) Oozie Coordinator jobs -- recurrent Oozie Workflow jobs triggered by time (frequency) and data availability. Apache Oozie 5.0.0 includes new features, bug fixes and minor improvements that include:

moved launcher from MapReduce mapper to YARN ApplicationMaster;
switched from Tomcat 6 to embedded Jetty 9;
updated third party libraries;
completely rewritten workflow graph generator;
JDK 8 support;
deprecated Instrumentation in favor of Metrics;
added indexes to speed up DB queries; and
fixed CVE-2017-15712

The full list of new features can be found in the project release notes at https://oozie.apache.org/docs/5.0.0/release-log.txt

"Oozie 5 is a major milestone for the project," said Andras Piros, Apache Oozie committer and Apache Oozie v5.0 Release Manager. "We are proud to provide all the new functionality to big data administrators, data engineers, and data scientists who can leverage a faster, more streamlined, and more secure workflow orchestrator. Features like Oozie on YARN, Jetty 9 support, and ecosystem revamp enable Apache Hadoop users to create and schedule Hadoop jobs in an efficient and modern way not seen before."

"Oozie has long been a staple of a productive Apache Hadoop deployment, playing an important role in orchestrating the rest of the ecosystem. Oozie 5 represents the next step in where Oozie is headed," added Kanter. "The Apache Oozie community has already got some great features in the works for our next release. We welcome anyone who wants to contribute to join us in making Oozie the best it can be."

Availability and Oversight

Apache Oozie software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache Oozie, visit http://oozie.apache.org/

About The Apache Software Foundation (ASF)

Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server --the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 680 individual Members and 6,500 Committers across six continents successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official user conference, trainings, and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, ARM, Baidu, Bloomberg, Budget Direct, Capital One, Cash Store, Cerner, Cloudera, Comcast, Facebook, Google, Hortonworks, Huawei, IBM, Indeed, Inspur, iSIGMA, ODPi, LeaseWeb, Microsoft, PhoenixNAP, Pivotal, Private Internet Access, Red Hat, Target, Union Investment, and Yahoo. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Oozie", "Apache Oozie", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Posted at 02:06PM Apr 18, 2018 by Sally Khudairi in General | |

The Apache Software Foundation
Blogging in Action.

Entries tagged [bigdata]

Calendar

Search

Hot Blogs (today's hits)

Tag Cloud

Categories

Feeds

Links

Navigation

Sun	Mon	Tue	Wed	Thu	Fri	Sat
« October 2021
					1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28	29	30
31
Today

The Apache Software Foundation Blogging in Action.

Entries tagged [bigdata]

Calendar

Search

Hot Blogs (today's hits)

Tag Cloud

Categories

Feeds

Links

Navigation

The Apache Software Foundation
Blogging in Action.