Entries tagged [bigdata]

Tuesday September 14, 2021

Apache Ignite Momentum: Highlights from 2020-2021

When Apache Ignite entered the Apache Software Foundation (ASF) Incubator in 2014, it took less than a year for the project and its community to graduate from the Incubator and become a top-level project for the ASF. Since then, Ignite has experienced a significant and steady growth in popularity, and it has been used by thousands of application developers and architects to create high-performance and scalable applications used by millions of people daily. In this article, we’ll recap the achievements of Ignite in 2020-2021.


Ignite is Ranked as a Top 5 Project

The ASF has ranked Apache ignite as a Top 5 project in various categories since 2017. That year, Ignite was in the Top 5 of Apache Project Repositories by Commits and most active Apache mailing lists. Today, the momentum continues, and Ignite continues to be ranked as a Top 5 project in multiple categories: second on the Top 5 big data user lists, third on the Top 5 big data dev lists, second on the Top 5 of all user lists, third on the Top 5 repos by size.

Of greatest significance, the continued Top 5 ranking on the “dev list” reflects an active community of contributors who are committed to keeping the code base growing, while the Top 5 ranking on the “user list” means that more and more Ignite application developers come to the community to ask questions – indicating continued growth in adoption.

The Worldwide Ignite community is Engaged

This broad and growing interest in Apache Ignite has continued over the last year and a half. However, faced with the pandemic and shelter-in-place orders around the world, the community sought ways to stay in touch and continue sharing experiences. The community naturally turned to a virtual format and established two new successful programs.

The first was a series of Ignite Virtual Meetups, where Apache Ignite users, developers, committers, contributors and architects worldwide could share experiences on a wide range of topics, ask questions, and help drive the project forward. Since these virtual meetups began, the community has already held 17 events, which were attended by hundreds of community members and developers.

The second new program was launched this May with the virtual Ignite Summit, the first global conference designed for the entire Ignite community. Twenty-five speakers from industry-leading companies including finance, biotech, health & fitness, construction and cloud computing led 15 hours of discussion about how Apache Ignite delivers the performance and scale required to address the world’s most challenging computational and hybrid transactional/analytical processing requirements. The Summit had attendees from North America, Latin America, EMEA and APAC. Remarkably, attendees spent an average of nearly 5 hours at the event!

Innovation Continues at a Rapid Pace

Over the last year and a half, the community has released five new versions of Ignite 2.x. The releases introduce numerous improvements and optimizations, including major features, such as new monitoring and profiling frameworks, cluster snapshots, encoding keys rotation for transparent data encryption, and more.

The community also put significant effort into contributing and releasing new documentation, which is now hosted on the Ignite website. Since the new documentation was posted, it has become the most visited resource on the website – a clear indication that it is helping Ignite developers make faster, easier progress on their Ignite development and optimization tasks.

Further, Igniters have begun working on the next major release, Ignite 3.0, which introduces significant usability improvements, a new SQL engine based on Apache Calcite, a Raft-based consistency protocol, and many other improvements. Users can already try the first two Alpha versions:

The payoff – Ignite Downloads Continue to Soar

The inherent benefits of Apache Ignite, combined with all the effort of a dedicated community, has resulted in a popular project that continues to see increasing adoption. Ignite Maven monthly downloads are skyrocketing, and we have seen a 65% year-over-year growth in downloads so far in 2021, resulting in hundreds of thousands of downloads each month.

We eagerly look forward to the full release of Apache ignite 3.0 and fully expect downloads, adoption and community enthusiasm to continue to soar. Good luck to the Ignite community!

Thursday March 15, 2018

Apache Ignite 2.4 Brings Advanced Machine Learning and Spark DataFrames Capabilities


Usually, Ignite community rolls out a new version once in 3 months, but we had to make an exception for Apache Ignite 2.4 that consumed five months in total. We could easily blame Thanksgiving, Christmas and New Year holidays for the delay and would be forgiven, but, in fact, we were forging the release you can't simply pass by.



Let's dive in and search for a big fish.

Machine Learning General Availability

Eight months ago, at the time of Apache Ignite 2.0, we put out the first APIs that formed the foundation of the Ignite's machine learning component of today. Since that time, Ignite machine learning experts and enthusiasts have been moving the library to the general availability condition meticulously. And Ignite 2.4 became a milestone that let us consider the ML Grid to be production ready.


The component gained a variety of algorithms that can solve a myriad of regression and classification tasks, gave an ability to train models avoiding ETL from Ignite to other systems, paved a way to deep learning usage scenarios. All that now empowers Ignite users with the tools for dealing with fraud detection, predictive analytics, and for building recommendation systems...if you want. Note, ETL is optional, and the whole memory-centric cluster is at your service!


Moreover, Machine Learning Grid welcomed a software donation by NetMillennium, Inc. in the form of genetic algorithms that solve optimization problems by simulating the process of biological evolution. The algorithms haven't got to Ignite 2.4 and waiting for their time for a release in the master branch. Once you get them, you can apply the biological evolution simulation for real-world applications including automotive design, computer gaming, robotics, investments, traffic/shipment routing and more.

Spark DataFrames

It's not a joke or misprint. Spark users, the DataFrames are now officially supported for you! Many of you have been anticipating them for years and, thanks to Nikolay Izhikov, who was "promoted" to an Ignite committer for the contribution, now you can leverage from them.


No need to be wordy here. Just go ahead and start with DataFrames in Ignite.

Expanding Ignite ecosystem

It was unfair that only Java, C#, and C++ developers could utilize the breadth and depth of Ignite APIs in their applications. Ignite 2.4 solved the injustice with its new low-level binary client protocol. The protocol communicates with an existing Ignite cluster without starting a full-fledged Ignite node. An application can connect to the cluster through a raw TCP socket from any programming language you like.


The beauty of the protocol is that you can develop a so-called Ignite thin client that is a lightweight client connected to the cluster and interacts with it using key-value, SQL, and other APIs. .NET thin client is already at your service and Node.JS, Python, PHP, Java thin clients are in a forge and being developed for the next releases.

RPM repository and much more


So, now Apache Ignite can also be installed from the official RPM repository. Debian users, the packages for your operating systems to be assembled soon.


Overall, if to list all the features and benefits Ignite 2.4 brings, only 2 people will read the article till the end - me and my dear mom :) Thus, I'll let you discover the rest from the release notes.

Wednesday November 01, 2017

Apache Ignite 2.3 - More SQL and Persistence Capabilities

Putting aside the regular bug fixes and performance optimizations, the Apache Ignite 2.3 release brings new SQL capabilities and Ignite persistence improvements that are worth mentioning.

SQL

Let's start with SQL first.

Apache Ignite users have consistently told us that despite all of Ignite’s SQL capabilities, it’s been at times challenging trying to figure out how to start using Ignite as an SQL database.

This was mostly caused by scattered documentation pages, lack of “getting started” guides and tutorials. We’ve remedied this oversight! All related SQL knowledge has been curated in a single documentation domain.

Are you curious about the SQL scope? Go to the new SQL Reference Overview section!

Cannot wait to learn how the Ignite SQL engine runs internally? We’ve prepared an Architectural Overview section for you.

Simply need to know how to connect to an Ignite cluster from an SQL tool? Here is a tooling section for you.

Let’s take a look at some specific SQL features released in Ignite 2.3.

First, we’re proud to deliver support of ALTER TABLE command. Presently, the command allows adding new columns to an SQL schema in runtime -- avoiding any cluster restarts. Once a new column is added, it can be turned into an index. Again, in runtime. No restarts!

Another significant addition seen in Ignite 2.3 is the integration with SQLLine tool that is bundled with every Apache Ignite release and can be used as a default command line tool for SQL based interactions.

To prove that it's fairly simple to work with Ignite as with an SQL database using the tool, we recorded a short screencast for you:

screencast.png

Ignite Persistence

Ignite native persistence keeps getting more attention and installs -- which is why the community released a feature requested by at least a dozen users. The feature allows enabling the persistence for specific data sets. Before Ignite version 2.3, the persistence could be enabled globally only.

Now, it's up to you to decide which data to persist and which to store in RAM only. The persistence can be configured via data regions as shown below:

persistence_cfg.png

This data region will consume up to 500 MB of RAM and will store a superset of data on disk ensuring that no data loss happens in case of a crash or even if there is no more space left in RAM.

Anything else?

Flip through our release notes to see all the changes and improvements available in Apache Ignite 2.3 -- and, for sure, download and use this version in production.

Questions, comments? Let us know!

Wednesday August 30, 2017

Apache Ignite Community Update (August 2017 Issue)

by Tom Diederich

Igniters, here are some community highlights from the last couple week. If I missed anything, please share it here. Meetups! Did you know that Apache Ignite experts are available to speak at your meetup? And we also have spots open for YOU to speak at the following meetups that some of us co-organize:

Meanwhile, here’s where to catch some great talks about Apache Ignite! We have 19 newly scheduled meetup talks on the books since the last update. All upcoming Ignite events can be found here. Let’s take a closer look at some of them….

Scheduled speaking engagements

* Sept. 9: Big Data and Cloud Meetup (Santa Clara, Calif.)

Apache Ignite PMC chair Denis Magda will be speaking at the Big Data and Cloud Meetup September 9 from 10 a.m. to noon. His talk is titled: "Apache Spark and Apache Ignite: Where Fast Data Meets the IoT".

* Sept. 13: SF Big Analytics Meetup

Denis Magda will be the featured speaker at the SF Big Analytics Meetup on Sept. 13. Denis' talk is titled: "Apache Ignite: the in-memory hammer in your data science toolkit."

* Sept. 18: Meetup: Cambridge .NET User Group

Apache Ignite evangelist Akmal Chaudhri will speak at the Cambridge .NET User Group Sept. 17. The title of his talk: "Scale Out and Conquer: Apache Ignite for .NET Users."

* Sept. 21: Joint meetup! Bay Area In-Memory Computing Meetup & SF Spark and Friends

* Sept. 27: New York Kubernetes Meetup

Apache Ignite evangelist Akmal Chaudhri will focus on a DevOps perspective on the orchestration of distributed databases such as Apache Ignite. Akmal will speak on node auto-discovery, automated horizontal scalability, availability, and utilization of RAM and disk with Apache Ignite.

* Oct. 4: Openstack & Ceph User Group Amsterdam

Apache Ignite evangelist Akmal Chaudhri will show attendees how to build a Fast Data solution that will receive endless streams from the IoT side and will be capable of processing the streams in real-time using Apache Ignite's cluster resources.

* Oct. 13: Big Data Week London 2017: A Festival of Data (conference)

Akmal Chaudhri will be speaking at the Big Data Week conference Oct. 13 in London. His talk, titled "Powering up banks and financial institutions with distributed systems," will educate attendees about important Apache Ignite features for financial applications -- such as ACID compliance, SQL compatibility, persistence, replication, security, fault tolerance and more.

* Oct. 18: Silicon Valley Java User Group

Join Apache Ignite PMC Chair Denis Magda will introduce the many components of the open-source Apache Ignite. His talk, titled, “Catch an intro to Apache Ignite and skyrocket Java applications,” will teach attendees how to solve some of the most demanding scalability and performance challenges. He will also cover a few typical use cases and work through some code examples.

* Oct. 19: Eurostaff Big Data London

Apache Ignite evangelist Akmal Chaudhri will show attendees how to build a Fast Data solution that will receive endless streams from the IoT side and will be capable of processing the streams in real-time using Apache Ignite's cluster resources.

* Oct. 24: Spark Summit Europe 2017 (conference)

Akmal Chaudhri will be presenting at the Spark Summit Europe conference, Oct. 24-26 at the Convention Centre Dublin in Ireland.  His session is titled: "How to Share State Across Multiple Spark Jobs using Apache Ignite."

* Nov. 2: Byte-Academy-FinTech-Python-Blockchain-Education Meetup (London)

In his talk, titled, "Powering up banks and financial institutions with distributed systems,” Apache Ignite technical Akmal Chaudhri will explain important Apache Ignite features for financial applications -- such as ACID compliance, SQL compatibility, persistence, replication, security, fault tolerance and more. A customer case study will also be presented.

Blog posts


Webinars


Upcoming

Past webinars (recordings available!)

Deploy like a Boss: Using Kubernetes and Apache Ignite, with GridGain solution architect Dani Traphagen.

Calendar

Search

Hot Blogs (today's hits)

Tag Cloud

Categories

Feeds

Links

Navigation