Entries tagged [release]

Monday January 17, 2022

Apache Ignite 2.12.0: CDC, Index Query API, Vulnerabilities Fixes

As of January 14, 2022, Apache Ignite 2.12 has been released. You can directly check the full list of resolved Important JIRA tasks but here let’s briefly overview some valuable improvements.

Vulnerability Updates

The Apache Ignite versions lower than 2.11.1 are vulnerable to CVE-2021-44832 which is related to the ignite-log4j2 module usage.

The release also fixes 10+ CVE’s of various modules. See release notes for more details.

Change Data Capture

Change Data Capture (CDC) is a data processing pattern used to asynchronously receive entries that have been changed on the local node so that action can be taken using the changed entry.

Below are some of the CDC use cases:

  • Streaming changes in Warehouse;
  • Updating search index;
  • Calculating statistics (streaming queries);
  • Auditing logs;
  • Async interaction with external systems: Moderation, business process invocation, etc.

Ignite implements CDC with the ignite-cdc.sh application and Java API.

Below are the CDC application and the Ignite node integrated via WAL archive segments:

Apache Ignite CDC design

When CDC is enabled, the Ignite server node creates a hard link to each WAL archive segment in the special db/cdc/{consistency_id} directory. The ignite-cdc.sh application can be runruns on a different JVM and processes newly archived WAL segments. When the segment is fully processed by ignite-cdc.sh, it is removed. The actual disk space is free when both links (archive and CDC) are removed.

State of consumption is a pointer to the last processed event. A consumer can tell ignite-cdc.sh to save the consumption state. On startup event processing will be continued from the last saved state.

See implementation details here.

Index Query API

The Apache Ignite now provides Index Query API for existing indexes. Index queries work over distributed indexes and retrieve cache entries that match the specified query.

It will help in some cases, where:

  • SQL is not applicable by the design of user application;
  • IndexScan is preferable to ScanQuery for performance reasons.

Example of query:

// Find the persons who work in Organization 1 and have salary more than 1,000.
QueryCursor<Cache.Entry<Integer, Person>> cursor = cache.query(
    new IndexQuery<Integer, Person>(Person.class, "ORG_SALARY_IDX")
        .setCriteria(eq("orgId", 1), gt("salary", 1000))

See more details here.


Previous versions can perform snapshot restore in the same cluster topology only. The new version provides the ability to restore snapshots on different cluster topologies. Moreover, added support of encrypted caches.

Distributed Environment Tests

The new testing framework was implemented. The main goal is to have a large enough set of integration tests that cover most of the typical cluster usage scenarios.


  • Ignite nodes can be started/stopped on a Docker or a real cluster with any custom configuration;
  • Any Apache Ignite version is supported (released or compiled from sources);
  • Apache Ignite forks are also supported «out of the box»;
  • Any other application execution is also possible, e.g. we implemented starters for Spark and Zookeeper;
  • The cluster can be managed using the control.sh, we made this a part of the test API;
  • Custom Java applications can be executed remotely with/without a built-in Ignite node or a Thin client;
  • Any ssh command can be executed remotely, and the result will be available locally (at the python test);
  • A network can be broken by editing iptables to check communication issues;
  • Tests can be executed in parallel when the cluster size is bigger than tests requirements.

Framework based on Ducktape library from Kafka team, that's why we called it Ducktests.

Migration modules to the Apache Ignite Extensions

There is activity on the migration of frameworks to extensions:

  • GCE, AWS, Azure modules were migrated to gce, aws, azure extensions.
  • CacheSpringStoreSessionListener was migrated to the spring-tx extension.
  • TcpDiscoveryZookeeperIpFinder was migrated to the zookeeper-ip-finder extension.

The binaries archive now weighs 10 percent less.

Anything else?

See the release notes to learn about all of the new features and bug fixes.

Reach out to us on the community user list for more questions, details, and feedback.

Sincerely yours,

Ignite contributors and committers

Tuesday December 21, 2021

Apache Ignite 2.11.1: Emergency Log4j2 Update

The new Apache Ignite 2.11.1 is an emergency release that fixes CVE-2021-44228, CVE-2021-45046, CVE-2021-45105 related to the ignite-log4j2 module usage.

Apache Ignite with Log4j Vulnerability

All the following conditions must be met:

  • The Apache Ignite version lower than 2.11.0 is used (since these vulnerabilities are already fixed in 2.11.1, 2.12, and upper versions);
  • The ignite-logj42 is used by Apache Ignite and located in the libs directory (by default it is located in the libs/optional directory, so these deployments are not affected);
  • The Java version in use is older than the following versions: 8u191, 11.0.1. This is due to the fact that later versions set the JVM property com.sun.jndi.ldap.object.trustURLCodebase to false by default, which disables JNDI loading of classes from arbitrary URL code bases.

NOTE: Relying only on the Java version as a protection against these vulnerabilities is very risky and has not been tested.

Risk Mitigation Without Upgrading

Please note that all of these cases require a cluster downtime, but we still recommend to upgrade the Apache Ignite.

Method 1: Removing the Vulnerable Classes

When using an older Apache Ignite version, it is possible to remove the JndiLookup class from any Java application by executing this command:

find $IGNITE_HOME/ -type f -name "*log4j-core-*.jar" -exec zip -q -d "{}" org/apache/logging/log4j/core/lookup/JndiLookup.class \;

This will recursively find all log4j-core JAR files, starting from the IGNITE_HOME directory, and remove the vulnerable JndiLookup class from them.

Method 2: Disabling Message Lookups

This method can be used as an additional protection layer in case you suspect not all log4j dependencies have been properly updated. If you are using the Apache Ignite of an older version, we recommend to disable message lookups globally by setting the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true or, alternatively, run the Apache Ignite with the ‐Dlog4j2.formatMsgNoLookups=true command-line option.

Method 3: Replace log4j2 Dependency Manually

It is still possible to manually replace the Log4j of 2.x version in the Apache Ignite binary distribution to the 2.17.0 Log4j version if your log configuration does not imply to use the RoutingAppender. In case the RoutingAppender is used it may produce some error messages in a log file at the startup or empty lines during the execution, which are considered as a minor flow, however, we do not recommend this mitigation method in this case.

Monday September 20, 2021

Apache Ignite 2.11: Stabilization First

The new Apache Ignite 2.11 was released on September 17, 2021. It can be considered to be a greater extent as a stabilization release that closed a number of technical debts of the internal architecture and bugs. Out of more than 200 completed tasks, 120 are bug fixes. However, some valuable improvements still exist, so let's take a quick look at them together.

Thin Clients

Partition awareness is enabled by default in the 2.11 release and allows thin clients to send query requests directly to the node that owns the queried data. Without partition awareness, an application executes all queries and operations via a single server node that acts as a proxy for the incoming requests.

The support of Continuous Queries added to the java thin client. For the other supported features, you can check - the List of Thin Client Features.

Cellular-clusters Deployment

The Apache Ignite internals has the so-called switch (a part of Partition Map Exchange) process that is used to perform atomic execution of cluster-wide operations and move a cluster from one consistent state to another, for example, a cache creation/destroy, a node JOIN/LEFT/FAIL operations, snapshot creation, etc. During the switching process, all user transactions are parked for a small period of time which in turn increases the average latency and decreases throughput of the overall cluster.

Splitting the cluster into virtual cells containing 4-8 nodes may increase the total cluster performance and minimize the influence of one cell on another in case of node fail events. Such a technique also significantly increases the recovery speed of transactions on cells not affected by failing nodes. The time when transactions are parked also decreases on non-affected cells which in turn decreases the worst latency for the cluster operations overall.

From now on, you can use the RendezvousAffinityFunction affinity function with ClusterNodeAttributeColocatedBackupFilter to group nodes into virtual cells. Since the node baseline attributes are used as cell markers the corresponding BASELINE_NODE_ATTRIBUTES system view was added.

See benchmarks below that represent the worst (max) latency, which happens in case of node left/failure/timeout events on broken and alive cells.


New Page Replacement Policies

When Native Persistence is on and the amount of data, which Ignite stores on the disk, is bigger than the off-heap memory amount allocated for the data region, another page should be evicted from the off-heap to the disk to preload a page from the disk to the completely full off-heap memory. This process is called page replacement. Previously, Apache Ignite used the Random-LRU page replacement algorithm which has a low maintenance cost, but it has many disadvantages and greatly affects the performance when the page replacement is started. On some deployments, administrators even force a cluster restart periodically to avoid page replacement. There are a few new algorithms available from now on:

  • Segmented-LRU Algorithm
  • CLOCK Algorithm

Page replacement algorithm can be configured by the PageReplacementMode property of DataRegionConfiguration. By default, the CLOCK algorithm is now used. You can check the Replacement Policies in the documentation for more details.

Snapshot Restore And Check Commands


All snapshots are fully consistent in terms of concurrent cluster-wide operations as well as ongoing changes with Ignite. However, in some cases and for your own peace of mind, it may be necessary to check the snapshot for completeness and for data consistency. The Apache Ignite is now delivered with a built-in snapshot consistency check commands that enable you to verify internal data consistency, calculate data partitions hashes and pages checksums, and print out the result if a problem is found. The check command also compares hashes calculated by containing keys of primary partitions with corresponding backup partitions and reports any differences.

# This procedure does not require the cluster to be in the idle state.
control.(sh|bat) --snapshot check snapshot_name


Previously, only the manual snapshot restore procedure was available by fully copying persistence data files from the snapshot directory to the Apache Ignite work directory. The automatic restore procedure allows you to restore cache groups from a snapshot on an active cluster by using the Java API or command line script (using CLI is recommended). Currently, the restore procedure has several limitations, so please check the documentation pages for details.

Start restoring all user-created cache groups from the snapshot "snapshot_09062021".
control.(sh|bat) --snapshot restore snapshot_09062021 --start

# Start restoring only "cache-group1" and "cache-group2" from the snapshot "snapshot_09062021".
control.(sh|bat) --snapshot restore snapshot_09062021 --start cache-group1,cache-group2

# Get the status of the restore operation for "snapshot_09062021".
control.(sh|bat) --snapshot restore snapshot_09062021 --status

# Cancel the restore operation for "snapshot_09062021".
control.(sh|bat) --snapshot restore snapshot_09062021 --cancel



Hot Blogs (today's hits)

Tag Cloud