Entries tagged [in-memory]

Monday September 20, 2021

Apache Ignite 2.11: Stabilization First

The new Apache Ignite 2.11 was released on September 17, 2021. It can be considered to be a greater extent as a stabilization release that closed a number of technical debts of the internal architecture and bugs. Out of more than 200 completed tasks, 120 are bug fixes. However, some valuable improvements still exist, so let's take a quick look at them together.

Thin Clients

Partition awareness is enabled by default in the 2.11 release and allows thin clients to send query requests directly to the node that owns the queried data. Without partition awareness, an application executes all queries and operations via a single server node that acts as a proxy for the incoming requests.

The support of Continuous Queries added to the java thin client. For the other supported features, you can check - the List of Thin Client Features.

Cellular-clusters Deployment

The Apache Ignite internals has the so-called switch (a part of Partition Map Exchange) process that is used to perform atomic execution of cluster-wide operations and move a cluster from one consistent state to another, for example, a cache creation/destroy, a node JOIN/LEFT/FAIL operations, snapshot creation, etc. During the switching process, all user transactions are parked for a small period of time which in turn increases the average latency and throughput of the overall cluster.

Splitting the cluster into virtual cells containing 4-8 nodes may increase the total cluster performance and minimize the influence of one cell on another in case of node fail events. Such a technique also significantly increases the recovery speed of transactions on cells not affected by failing nodes. The time when transactions are parked also decreases on non-affected cells which in turn decreases the worst latency for the cluster operations overall.

From now on, you can use the RendezvousAffinityFunction affinity function with ClusterNodeAttributeColocatedBackupFilter to group nodes into virtual cells. Since the node baseline attributes are used as cell markers the corresponding BASELINE_NODE_ATTRIBUTES system view was added.

See benchmarks below that represent the worst (max) latency, which happens in case of node left/failure/timeout events on broken and alive cells.


New Page Replacement Policies

When Native Persistence is on and the amount of data, which Ignite stores on the disk, is bigger than the off-heap memory amount allocated for the data region, another page should be evicted from the off-heap to the disk to preload a page from the disk to the completely full off-heap memory. This process is called page replacement. Previously, Apache Ignite used the Random-LRU page replacement algorithm which has a low maintenance cost, but it has many disadvantages and greatly affects the performance when the page replacement is started. On some deployments, administrators even force a cluster restart periodically to avoid page replacement. There are a few new algorithms available from now on:

  • Segmented-LRU Algorithm
  • CLOCK Algorithm

Page replacement algorithm can be configured by the PageReplacementMode property of DataRegionConfiguration. By default, the CLOCK algorithm is now used. You can check the Replacement Policies in the documentation for more details.

Snapshot Restore And Check Commands


All snapshots are fully consistent in terms of concurrent cluster-wide operations as well as ongoing changes with Ignite. However, in some cases and for your own peace of mind, it may be necessary to check the snapshot for completeness and for data consistency. The Apache Ignite is now delivered with a built-in snapshot consistency check commands that enable you to verify internal data consistency, calculate data partitions hashes and pages checksums, and print out the result if a problem is found. The check command also compares hashes calculated by containing keys of primary partitions with corresponding backup partitions and reports any differences.

# This procedure does not require the cluster to be in the idle state.
control.(sh|bat) --snapshot check snapshot_name


Previously, only the manual snapshot restore procedure was available by fully copying persistence data files from the snapshot directory to the Apache Ignite work directory. The automatic restore procedure allows you to restore cache groups from a snapshot on an active cluster by using the Java API or command line script (using CLI is recommended). Currently, the restore procedure has several limitations, so please check the documentation pages for details.

Start restoring all user-created cache groups from the snapshot "snapshot_09062021".
control.(sh|bat) --snapshot restore snapshot_09062021 --start

# Start restoring only "cache-group1" and "cache-group2" from the snapshot "snapshot_09062021".
control.(sh|bat) --snapshot restore snapshot_09062021 --start cache-group1,cache-group2

# Get the status of the restore operation for "snapshot_09062021".
control.(sh|bat) --snapshot restore snapshot_09062021 --status

# Cancel the restore operation for "snapshot_09062021".
control.(sh|bat) --snapshot restore snapshot_09062021 --cancel

Tuesday September 14, 2021

Apache Ignite Momentum: Highlights from 2020-2021

When Apache Ignite entered the Apache Software Foundation (ASF) Incubator in 2014, it took less than a year for the project and its community to graduate from the Incubator and become a top-level project for the ASF. Since then, Ignite has experienced a significant and steady growth in popularity, and it has been used by thousands of application developers and architects to create high-performance and scalable applications used by millions of people daily. In this article, we’ll recap the achievements of Ignite in 2020-2021.

Ignite is Ranked as a Top 5 Project

The ASF has ranked Apache ignite as a Top 5 project in various categories since 2017. That year, Ignite was in the Top 5 of Apache Project Repositories by Commits and most active Apache mailing lists. Today, the momentum continues, and Ignite continues to be ranked as a Top 5 project in multiple categories: second on the Top 5 big data user lists, third on the Top 5 big data dev lists, second on the Top 5 of all user lists, third on the Top 5 repos by size.

Of greatest significance, the continued Top 5 ranking on the “dev list” reflects an active community of contributors who are committed to keeping the code base growing, while the Top 5 ranking on the “user list” means that more and more Ignite application developers come to the community to ask questions – indicating continued growth in adoption.

The Worldwide Ignite community is Engaged

This broad and growing interest in Apache Ignite has continued over the last year and a half. However, faced with the pandemic and shelter-in-place orders around the world, the community sought ways to stay in touch and continue sharing experiences. The community naturally turned to a virtual format and established two new successful programs.

The first was a series of Ignite Virtual Meetups, where Apache Ignite users, developers, committers, contributors and architects worldwide could share experiences on a wide range of topics, ask questions, and help drive the project forward. Since these virtual meetups began, the community has already held 17 events, which were attended by hundreds of community members and developers.

The second new program was launched this May with the virtual Ignite Summit, the first global conference designed for the entire Ignite community. Twenty-five speakers from industry-leading companies including finance, biotech, health & fitness, construction and cloud computing led 15 hours of discussion about how Apache Ignite delivers the performance and scale required to address the world’s most challenging computational and hybrid transactional/analytical processing requirements. The Summit had attendees from North America, Latin America, EMEA and APAC. Remarkably, attendees spent an average of nearly 5 hours at the event!

Innovation Continues at a Rapid Pace

Over the last year and a half, the community has released five new versions of Ignite 2.x. The releases introduce numerous improvements and optimizations, including major features, such as new monitoring and profiling frameworks, cluster snapshots, encoding keys rotation for transparent data encryption, and more.

The community also put significant effort into contributing and releasing new documentation, which is now hosted on the Ignite website. Since the new documentation was posted, it has become the most visited resource on the website – a clear indication that it is helping Ignite developers make faster, easier progress on their Ignite development and optimization tasks.

Further, Igniters have begun working on the next major release, Ignite 3.0, which introduces significant usability improvements, a new SQL engine based on Apache Calcite, a Raft-based consistency protocol, and many other improvements. Users can already try the first two Alpha versions:

The payoff – Ignite Downloads Continue to Soar

The inherent benefits of Apache Ignite, combined with all the effort of a dedicated community, has resulted in a popular project that continues to see increasing adoption. Ignite Maven monthly downloads are skyrocketing, and we have seen a 65% year-over-year growth in downloads so far in 2021, resulting in hundreds of thousands of downloads each month.

We eagerly look forward to the full release of Apache ignite 3.0 and fully expect downloads, adoption and community enthusiasm to continue to soar. Good luck to the Ignite community!

Wednesday March 11, 2020

Ignite 2.8 Released: Less Stress in Production and Advances in Machine Learning

With thousands of changes contributed to Apache Ignite 2.8 that enhanced almost all the components of the platform, it’s possible to overlook some of the improvements that can convince you to upgrade to this version sooner than later. While a quick check of the release notes will help to discover anticipated bug fixes, this article aims to guide through enhancements every Ignite developer should be aware of.

New Subsystem for Production Monitoring and Tracing

Several months of constant work on IEP-35: Monitoring & Profiling has resulted in the creation of a robust and elastic subsystem for production monitoring and diagnostic (aka. profiling). This was influenced by the needs of many developers who deployed Ignite in critical environments and were asking for a foundation that can be integrated with many external monitoring tools and be expanded easily.

The new subsystem consists of several registries that group individual metrics related to a specific Ignite component. For instance, you will find registries for cache, compute, or service grid APIs. Since the registries are designed to be generic, specific exporters can observe the state of Ignite via a myriad of tools supporting various protocols. By default, Ignite 2.8 introduces exporters for monitoring interfaces such as log files, JMX and SQL views, and contemporary ones such as OpenCensus.

Presently, this new subsystem is released in an experimental mode only to give Ignite users some time to check the new API and suggest any improvements. Since the developer community is already impatient to remove the experimental flag, don’t delay!

Advances in Ignite Machine Learning

Machine Learning (ML) capabilities of Ignite 2.8 are so drastically different from previous versions that if you’ve been waiting for the best moment to use the API, then the time has come. Let’s scratch the surface here and learn more details from the updated documentation pages.

A model training is usually a multi-step process that goes with preprocessing, training, and evaluation/valuation phases. A new pipelining API puts things in order by combining all the phases in a single workflow.

In addition to the pipelining APIs, Ignite 2.8 introduced ensemble methods, which allow combining several machine learning techniques into one predictive model to decrease variance (bagging) and bias (boosting), or improve predictions (stacking).

Furthermore, now you can import Apache Spark or XGBoost models to Ignite for further inference, pipelining other tasks. Feel free to keep training a model with your favorite framework and convert it to Ignite representation once the model needs to be deployed in production and executed at scale.

Beyond Java: Partition-Awareness and Other Changes

Even though Ignite is a Java middleware, it functions as a cross-platform database and compute platform that is used for applications developed in C#, C++, Python, and other programming languages.

Thin client protocol is a real enabler for other programming languages support, and with Ignite 2.8, it got a significant performance optimization by supporting partition-awareness. The latter allows thin clients to send query requests directly to nodes that own the queried data. Without partition awareness, an application that is connected to the cluster via a thin client executes all queries and operations via a single server node that acts as a proxy for the incoming requests.

Check the detailed blog post by Pavel Tupitsyn, Ignite committer and PMC, who elaborates on the partition-awareness feature and introduces other .NET-specific enhancements.

Less Stress in Production

This section lists top improvements that might not have striking or catchy names but can bring relief by automating and optimizing things, and by avoiding data inconsistencies when you are already in production.

The stop-the-world pauses triggered by Java garbage collectors impact performance, responsiveness, and throughput of our Java applications. Apache Ignite has a partition-map-exchange (PME) process that, as Java garbage collectors, has some phases that put on hold all running operations for the sake of cluster-wide consistency. For most of the Ignite usage scenarios, these phases complete promptly and are unnoticed. However, some low-latency or high-throughput use cases can detect a decline that might impact some business operations for a moment in time. This wiki page lists all the conditions that can trigger a distributed PME, and with Ignite 2.8, some of them were taken off the list -- the blocking PME no longer happens if a node belonging to the current baseline topology leaves the cluster or a thick client connects to it.

Next, we all know that things break, and what really matters is how a system handles failures. With Ignite 2.8, we revisited the way the cluster handles crash recoveries on restarts while replaying write-ahead-logs (check IGNITE-7196 and IGNITE-9420). Also, the read-repair feature was added to manage data inconsistencies between primary and backups copies of the cluster on-the-fly.

Furthermore, it’s worth mentioning that Ignite 2.8 became more prudent about disk space consumption by supporting the compaction of data files and write-ahead-logs of the native persistence. By sacrificing a bit more CPU cycles for the needs of compaction algorithms, you can save a lot on the storage end.

Last but not least, is an auto-baseline feature that changes a cluster topology for deployments with Ignite native persistence without the need for your intervention in many scenarios. Check this documentation page for more details.

Reach out to us on the community user list for more questions, details, and feedback.

Sincerely yours,
Ignite contributors and committers



Hot Blogs (today's hits)

Tag Cloud