Apache Ignite

Thursday May 31, 2018

Apache Ignite 2.5: Scaling to 1000s Nodes Clusters

Apache Ignite was always appreciated by its users for two primary things it delivers - scalability and performance. Throughout the lifetime many distributed systems tend to do performance optimizations from a release to release while making scalability related improvements just a couple of times. It's not because the scalability is of no interest. Usually, scalability requirements are set and solved once by a distributed system and don't require significant additional interventions by engineers.

However, Apache Ignite grew to the point when the community decided to revisit its discovery subsystem that influences how well and far Ignite scales out. The goal was pretty clear - Ignite has to scale to 1000s of nodes as good as it scales to 100s now.

It took many months to get the task implemented. So, please join me in welcoming Apache Ignite 2.5 that now can be scaled easily to 1000s of nodes and goes with other exciting capabilities. Let's check out the most prominent ones.


Massive Scalability


There are two components of Ignite that were modified in Ignite 2.5 to improve its scalability capabilities. The first one is related to 1000s nodes clusters while the other is related to the way we train machine learning (ML) models in Ignite. Let's start with the first.

Marrying Apache Ignite and ZooKeeper

Right, that 1000s nodes scalability goal was solved with the help of Apache ZooKeeper. Why did we turn to it?

Apache Ignite default TCP/IP Discovery organizes cluster nodes into a ring-topology form that has its advantages and disadvantages. For instance, on topologies with hundreds of cluster nodes, it can take many seconds why a system message traverse through all the nodes. As a result, necessary processing of events such as joining of new nodes or detecting of failed ones can take a while affecting overall cluster responsiveness and performance. That is a big deal if you'd like to run 1000s nodes clusters.

The new ZooKeeper Discovery uses ZooKeeper as a single point of synchronization where Ignite nodes are exchanging discovery events through it. It solved the issue with long-to-be-processed discovery messages and, as a result, allowed Ignite scaling to large cluster topologies.

As a rule of thumb, keep using default TCP/IP Discovery if it's unlikely that your Ignite cluster scales beyond 300s nodes and switch to ZooKeeper Discovery if that's the case.

Machine Learning: Partition-Based Datasets

That's the second prominent feature of Ignite 2.5 that improves the way of how far you can scale your Ignite clusters to train ML models over terabytes or petabytes of data. The partition-based datasets moved us closer to the implementation of Zero-ETL concept which implies that Ignite can be used as a single storage where ML models and algorithms are being improved iteratively and online without ETLing data back and forth between Ignite and another storage.

Read more about the datasets from this documentation page.

Genetic Algorithms


Ignite's ML component is ramping up and in the version 2.5 it accepted a contribution of genetic algorithms (GAs) which help to solve optimization problems by simulating the process of biological evolution. GAs are excellent for searching through large and complex data sets for an optimal solution. Real world applications of GAs include automotive design, computer gaming, robotics, investments, traffic/shipment routing and more.

Refer to excessive articles of my community-mates Turik Campbell and Akmal B. Chaudhri which cover main benefits of GAs:

Continuous Self-Healing and Consistency Checks


It's a known fact that many companies and businesses trusted Ignite its mission-critical deployments and solutions. As a result, sometimes Ignite doesn't even have a right to "misfire" and should be able to handle critical or unpredictable situations automatically or provide facilities to do deal with them manually.

With Ignite 2.5, we've kicked off the realization of continuous self-healing concept that implies that no matter what happens with Ignite in production it should be able to tolerate unexpected failures and stay up and running. The following was done in 2.5:

SQL: Security and Fast Data Loading

The community stays strong and determined in its goal of making Ignite SQL engine undistinguishable from SQL engines of famous and mature SQL database. What's the purpose? We want to make it easy for you to migrate from a relational database to Ignite, so that you can reuse all your skills gained before. Overall, this is what our SQL engine got in 2.5:

In-place Execution of Spark DataFrame Queries

Apache Spark users can applaud because the following ticket got merged in 2.5. In short, it means that from now on Ignite will be able to execute as many DataFrames SQL queries as it can in-place on Ignite servers side avoiding data movement from Ignite to Spark. The performance of your DataFrames queries should boost significantly. Enjoy!

DEB and RPM packages

Last but not least, if you're a Linux user, now you can install the latest Ignite versions directly from DEB and RPM repositories. Refer to how-to and share your feedback with us.

Finally, I have no more paper left to cover other optimizations and improvements. So, go ahead and check out our release notes.

Comments:

If you are know about bluetooth audio and wireless connection in windows operating system then comes to our tutorial page or follow there given few instructions and get all the knowledgeable things in this features.

Posted by Fixwindows10connections on June 09, 2018 at 07:19 AM UTC #

https://www.google.com dsafasdf

Posted by asdfasdfasdf on July 25, 2018 at 12:39 PM UTC #

Hello, Superb information about Marrying Apache Ignite and ZooKeeper wiyh latest algorithm & updates. Also shared SSL info.

Posted by James on September 03, 2018 at 03:46 PM UTC #

Yes, the role of the scaling is improved for the removal of the gaps of the society. The visits of the https://www.rushmyessays.org/ have been identified for the movement of the groups for all ideal paths and actors for the elimination of the bad things for the lives of the students.

Posted by Oliver Birch on September 11, 2018 at 07:44 AM UTC #

massive scalability is much important for scaling for more go with https://playstoredownloads.org/.

Posted by Play Store on October 23, 2018 at 07:35 AM UTC #

You provide awesome information. Most people don't know but you have clear it. Shilajit is a source of natural fulvic acid which provides bioactive compounds for the health. This is completely safe.

Posted by Sarah banks on November 03, 2018 at 12:16 PM UTC #

I love writing and I take help from https://www.10bestseo.com and I am doing it from many years and I am so much satisfied with it.

Posted by Jeremy McKivat on November 09, 2018 at 11:39 PM UTC #

http://uxoinfo.com/

Posted by Jeremy McKivat on November 23, 2018 at 09:41 PM UTC #

Thank you for posting this article, it was really helpfull. Great Article.

Posted by bubble shooter on January 04, 2019 at 07:52 AM UTC #

[url=googl.com]Google[/url] <a href="abc.com">abc</a> https://www.google.com

Posted by Jack William on January 05, 2019 at 04:46 AM UTC #

Is there any known implementation of client/server authentication with Ignite Cluster. How can I prevent unauthorized clients and servers to join my Cluster? thank you in advance.

Posted by Vitaly Selivanov on January 29, 2019 at 07:56 PM UTC #

Good one. Keep sharing, Looking forward in the future as well,.

Posted by Business setup in UAE on February 24, 2019 at 06:43 AM UTC #

How it goes with working of Apache.

Posted by rootmasterapk on February 27, 2019 at 09:40 PM UTC #

<a href="https://app.grammarly.com/ddocs/428003256">click here</a> https://www.google.com/ great work really interesting

Posted by dhiya on March 15, 2019 at 06:55 AM UTC #

The site is helping me a lot for successful creation of my project. This helps me to refer and understand deeply about apache. The available details are useful and understanding http://www.howtofixmyprinter.com . The way used to present the details is really good.

Posted by isabella dominic on March 15, 2019 at 11:23 AM UTC #

Great Article. Thank you for posting this article.

Posted by slotclub on March 26, 2019 at 11:33 AM UTC #

Thank you for this complete article! It is really interesting!

Posted by Bubble shooter on March 26, 2019 at 02:13 PM UTC #

Thank You fo this! You the Best!

Posted by wulkan777 on March 26, 2019 at 04:00 PM UTC #

Hello, information about Marrying Apache Ignite and ZooKeeper wiyh latest algorithm & updates.

Posted by Andrey on April 10, 2019 at 12:32 PM UTC #

The site is helping me a lot for successful creation of my project. This helps me to refer and understand deeply about apache.

Posted by Ideologs on April 10, 2019 at 12:33 PM UTC #

Amazing Article!

Posted by Topstore iOS on April 12, 2019 at 01:35 AM UTC #

I have been searching to find a comfort or effective procedure to complete this process and I think this is the most suitable way to do it effectively.

Posted by small girls cars on April 13, 2019 at 03:41 PM UTC #

nice post....watch latest movie Majili which super emotional cute dramatic real life love story

Posted by Latest Emus4u on April 15, 2019 at 11:12 AM UTC #

Internet commerce organization alternatives change determined by a new business's demands. These kind of alternatives is supplied in a new 'ready made' point out as well as have to have a tailor made designed option. A number of widespread internet commerce organization alternatives are generally presented down below: http://www.securebusinesssolution.com

Posted by hami manager on May 09, 2019 at 10:38 AM UTC #

business's demands. These kind of alternatives is supplied in a new 'ready made' point out as well as have to have a tailor made designed option. A number of widespread internet commerce organization alternatives are generally presented down below

Posted by Admin Manager on May 13, 2019 at 10:39 AM UTC #

Woah cool!

Posted by waromiya on May 14, 2019 at 05:08 AM UTC #

New version of the game java is now available in the best form of the HD to play. The company associated with the https://www.cleangrillsofsanantonio.com and work for the continuously expand the business. The company tackle the market by facilitates the user with best quality product. The best gaming experience enhance the market.

Posted by vurewah on June 12, 2019 at 10:55 AM UTC #

great post. http://www.subway-game.blogspot.com

Posted by zain on June 21, 2019 at 06:59 PM UTC #

http://www.gtagame100.com

Posted by zain on June 21, 2019 at 07:01 PM UTC #

http://www.gtagame100.com

Posted by zain on June 21, 2019 at 07:03 PM UTC #

Ohh, great! I think your opinion is correct, thank you, I will follow you regularly!

Posted by earn to die on June 24, 2019 at 06:59 AM UTC #

There are two components of Ignite that were modified in Ignite 2.5 to improve its scalability capabilities. The first one is related to 1000s nodes clusters while the other is related to the way we train machine learning (ML) models in Ignite. Let's start with the first.

Posted by film izle on June 29, 2019 at 08:26 PM UTC #

Posted by ảnh chế on July 01, 2019 at 05:42 PM UTC #

Nice article. I have been searching for this. Thank you for sharing this. It helps me to know more about Apache in deep. I have also bookmarked your website as I received good information from this website. Keep sharing.

Posted by sculptra los angeles on July 11, 2019 at 06:10 AM UTC #

http://www.zumagame100.blogspot.com

Posted by zain david on July 15, 2019 at 08:48 PM UTC #

Nice post

Posted by Top 10 Digital Currencies on July 19, 2019 at 04:44 AM UTC #

Good post

Posted by Top 10 Digital Currencies on July 19, 2019 at 04:45 AM UTC #

Great post <a href="https://xeroxservisim.com/">Top 10 Digital Currencies</a> i really like it

Posted by Willam on July 19, 2019 at 04:46 AM UTC #

Wonderful. Love the illustrations and animation. So much work.

Posted by vietmakrackca1971 on July 20, 2019 at 04:47 PM UTC #

Adore your shot, friend. NICE WORK

Posted by namculigo1984 on July 20, 2019 at 05:29 PM UTC #

very very grand !!!! just too good! Your colours are out of this world

Posted by inunomleb1986 on July 20, 2019 at 05:49 PM UTC #

All your projects have a fantastic concept, congratulations! belas ilustrações, parabéns

Posted by enfeedriami1988 on July 20, 2019 at 07:20 PM UTC #

@Tiago Antonio Antonio Thank you very much Great!

Posted by quewhifbover1975 on July 20, 2019 at 11:27 PM UTC #

one love I want to go, really amazing!!!

Posted by stoutinwoncy1982 on July 21, 2019 at 12:17 AM UTC #

So satisfying This is so touching, particularly if you think about the environmental politics in the city.

Posted by jamfmuzrido1979 on July 21, 2019 at 01:12 AM UTC #

good work:) good job love it

Posted by teoflataran1970 on July 21, 2019 at 02:51 AM UTC #

I realy enjoy that! Woww! Amazing

Posted by rasanphora1988 on July 21, 2019 at 04:47 AM UTC #

How u caught these birds, wow its very story telling art! Great work!

Posted by laficulphe1986 on July 21, 2019 at 05:19 AM UTC #

_Nice Work ! ! just wonderful

Posted by budhprotdenky1981 on July 21, 2019 at 06:00 AM UTC #

-Manny 很好的设计!

Posted by patedfefo1981 on July 21, 2019 at 06:33 AM UTC #

Extraordinary style!! Beautiful!! wow! superb

Posted by thumbralbowlgran1981 on July 21, 2019 at 07:06 AM UTC #

Aw, this is beaut. I realy like those simple packaging and labels

Posted by sepbekinmann1974 on July 21, 2019 at 07:38 AM UTC #

i remember watching these ending credits in the cinema, it was amazing! Cool! Icons is genius!!!

Posted by rfaratsicmey1970 on July 21, 2019 at 08:10 AM UTC #

u've made really usefull product for that business, great! love how your design dont took attract from main product, but all together its looks like something solid, great!! Thank you Abdul.@Abdul Aziz Susmoy

Posted by tabrarote1972 on July 21, 2019 at 09:15 AM UTC #

Wow... well done! These are brilliant! So amazing work!! :))

Posted by dthehalprotma1977 on July 21, 2019 at 09:47 AM UTC #

Awesome work! I love it :) Very awesome !!

Posted by kendfifilmti1988 on July 21, 2019 at 10:21 AM UTC #

TCS Webmail https://bit.ly/2K63HkR is an official email administration for Tata Consultancy Services Limited representatives around the world. TCS has disposed of direct access to the individual administrations and acquired to one stage for the email administrations to every one of the workers. http://bit.ly/2LLEZcC http://bit.ly/2GxMkZc http://bit.ly/311RiVT http://bit.ly/2GzUbW2

Posted by Ericagonzales on July 28, 2019 at 04:33 AM UTC #

Thanks, this is also a great resource: <a href="https://www.theacademicblueprint.com">https://www.theacademicblueprint.com</a>

Posted by Cola on July 28, 2019 at 03:25 PM UTC #

Thanks, this is also a great resource: https://www.theacademicblueprint.com

Posted by Cola on July 28, 2019 at 03:25 PM UTC #

This is great info, thanks! https://www.tucsonblockfence.com

Posted by Jason on August 01, 2019 at 03:41 PM UTC #

Useful information about Gonorrhea Disease here - http://www.honeybeenutrition.com/home-remedies-for-gonorrhea/ I get much useful information about this. Thanks

Posted by Honeybeenutrition.com on August 02, 2019 at 04:55 PM UTC #

There is no doubt in this that Russell Sage College Students has set the best example in the world of Theater Production. They are at their best every time. They set an example which is hardly possible to meet by anyone in this Theater indusctry. top seo companies in karachi are also willing to do such work but they can't match their level. https://www.webtrica.com

Posted by Lina Joe on August 09, 2019 at 07:57 AM UTC #

Post a Comment:
  • HTML Syntax: NOT allowed

Calendar

Search

Hot Blogs (today's hits)

Tag Cloud

Categories

Feeds

Links

Navigation