Protecting Apache Ignite from 'Meltdown' and 'Spectre' vulnerabilities
The world was rocked after the recent disclosure of the Meltdown and Spectre vulnerabilities that literally affect almost all software ever developed. Both issues are related to the way all modern CPUs are designed and this is why they have opened unprecedented security breaches -- making the software, including Apache Ignite, vulnerable to hacker attacks.
The vulnerabilities are registered in the National Vulnerability Database under the following CVEs:
How to protect Apache Ignite deployments?
First, the vulnerabilities can be fixed only on the operating system (OS) or hardware levels. All OS and hardware vendors are working on and releasing patches to fill-in the security breaches. Depending on the type of your Apache Ignite deployment, make sure to do the following:
- On-premise deployments - apply the patches prepared by your OS and hardware vendors. Consult with them to find out additional steps to act on. This page is a good place to start with.
- Cloud deployments - major cloud providers such as Amazon and Microsoft are in a process of patching their cloud computing services. Consider a cloud provider's security announcements and recommendations or follow up with a representative for suggestions.
Second, an Apache Ignite cluster becomes vulnerable to the attacks only if someone gets unauthorized access to cluster machines (both on-premise or cloud deployments) and executes a malicious shell script or connects to the cluster directly and executes a Java, .NET or C++ computation there.
Do the following to prevent this from happening:
- Make sure the cluster machines are secured with a hard-to-guess or hard-to-calculate password.
- Consider using 3rd party security components provided by enterprise vendors (such as this one) to strengthen a security shield of your deployments.
Finally, researchers who discovered Meltdown and Spectre have said that the first issue can be fixed with software patches while the second can be fully addressed only with hardware upgrades/replacement. Luckily, it's much more difficult for hackers to exploit Spectre. Thus, if the two recommendations given above are taken seriously, the chances that you will be impacted from Spectre are low.
What is the performance impact of security patches?
Many security patches are rolled out with a precaution that some of the applications can see up to a 30% performance degradation. Apache Ignite community is planning to measure the impact of general usage scenarios and will follow up with the results in a consequent post.
This general performance testing might not cover your use case. Therefore, it's highly recommended that you assess and test a possible performance drop of your Apache Ignite deployments before applying the patches in production. If the drop is significant, then contact us on the dev list.
Apache Ignite Essentials: 2-part Webinar Series for Architects and Java Developers
We finally made this happen! I’m happy to invite all of the software architects and engineers out there to a series of webinars that will introduce you to the fundamental capabilities of in-memory computing platforms such as Apache Ignite.
There will also be a mix of theory and practice. A lot of code examples are waiting to be shown so that you can apply the theory in practice right away.
The series consists of two parts.
- Cluster configuration and deployment.
- Distributed database internals (partitioning, replication).
- Data processing with key-value APIs.
- Affinity Collocation.
- Data processing with SQL.
- Collocated processing.
- Collocated processing for distributed computations.
- Collocated processing for SQL (distributed joins and more).
- Machine Learning.
- Memory Architecture.
Book your seat!
Apache Ignite 2.3 - More SQL and Persistence Capabilities
Putting aside the regular bug fixes and performance optimizations, the Apache Ignite 2.3 release brings new SQL capabilities and Ignite persistence improvements that are worth mentioning.
Let's start with SQL first.
Apache Ignite users have consistently told us that despite all of Ignite’s SQL capabilities, it’s been at times challenging trying to figure out how to start using Ignite as an SQL database.
This was mostly caused by scattered documentation pages, lack of “getting started” guides and tutorials. We’ve remedied this oversight! All related SQL knowledge has been curated in a single documentation domain.
Are you curious about the SQL scope? Go to the new SQL Reference Overview section!
Cannot wait to learn how the Ignite SQL engine runs internally? We’ve prepared an Architectural Overview section for you.
Simply need to know how to connect to an Ignite cluster from an SQL tool? Here is a tooling section for you.
Let’s take a look at some specific SQL features released in Ignite 2.3.
First, we’re proud to deliver support of ALTER TABLE command. Presently, the command allows adding new columns to an SQL schema in runtime -- avoiding any cluster restarts. Once a new column is added, it can be turned into an index. Again, in runtime. No restarts!
Another significant addition seen in Ignite 2.3 is the integration with SQLLine tool that is bundled with every Apache Ignite release and can be used as a default command line tool for SQL based interactions.
To prove that it's fairly simple to work with Ignite as with an SQL database using the tool, we recorded a short screencast for you:
Ignite native persistence keeps getting more attention and installs -- which is why the community released a feature requested by at least a dozen users. The feature allows enabling the persistence for specific data sets. Before Ignite version 2.3, the persistence could be enabled globally only.
Now, it's up to you to decide which data to persist and which to store in RAM only. The persistence can be configured via data regions as shown below:
This data region will consume up to 500 MB of RAM and will store a superset of data on disk ensuring that no data loss happens in case of a crash or even if there is no more space left in RAM.
Flip through our release notes to see all the changes and improvements available in Apache Ignite 2.3 -- and, for sure, download and use this version in production.
Questions, comments? Let us know!
Apache Ignite Community News (Issue 3)
by Tom Diederich
This is our third community update – there’s a lot going on, so let's get started.
Apache Ignite experts have already spoken at two meetups this month, both in Silicon Valley, but there are several more scheduled this month around the world.
On Sept. 9 Apache Ignite PMC chair Denis Magda was the featured presenter at the Big Data and Cloud Meetup in Santa Clara, Calif. His talk, titled "Apache Spark and Apache Ignite: Where Fast Data Meets the IoT," was highly rated and we’re planning a hands-on workshop with meetup organizers for November.
On Sept. 13 Denis also spoke at the SF Big Analytics Meetup in Mountain View, Calif. Again, to a packed room. The topic of his talk was "Better Machine Learning with Apache Ignite."
But Denis isn’t having all the fun – next Monday (Sept. 18), technology evangelist Akmal Chaudhri will speak at the Cambridge .NET User Group. The title of his talk: "Scale Out and Conquer: Apache Ignite for .NET Users."
We’re still looking for more meetups to speak at this month, so if you’re an organizer or would like us to speak at one you’re a member of, just let me know. In the meantime, here are the meetups planned for the remainder of September:
- Bay Area In-Memory Computing Meetup, Wednesday, Sept. 20 – Denis will present, "Apache Spark, Ignite and Flink: Where Fast Data Meets the IoT."
- Internet of Things (IoT) New York Meetup, Monday, Sept. 25 – Akmal will present, "Apache Spark and Apache Ignite: Where Fast Data Meets the IoT."
- NYC In-Memory Computing Meetup, Tuesday, Sept. 26 – Akmal will present, "Powering Up Banks and Financial Institutions with Distributed Systems."
- New York Kubernetes Meetup, Wednesday, Sept. 27 – Akmal will provide a DevOps perspective on the orchestration of distributed databases and Apache Ignite.
See? I told you Denis wasn’t having all the fun! Akmal is definitely on the road again. J
- Implementing In-Memory Computing for Financial Services Use Cases with Apache Ignite Sept 12 (recording available)
- Better Machine Learning with Apache Ignite, Wednesday, Sept. 27
On Sept. 5, Akmal published “Using Java and .NET apps to connect to an Apache Ignite cluster, that details how to create an Apache Ignite cluster that can support the reading and writing of user-defined objects in a common storage format. This is particularly useful in situations where applications need to work with objects but these objects will be accessed by different programming languages and frameworks.
On Sept. 7, Dmitriy Setrakyan published “Apache Ignite - In Memory Performance with Durability of Disk.”
Next up, on Sept. 12, was Akmal, who published “Kubernetes and Apache® Ignite™ Deployment on AWS.” That post walked through the steps required to get Kubernetes and Apache Ignite deployed on Amazon Web Services (AWS).
And then on Sept. 13 Dmitriy published “What is Apache Ignite.” I think the headline of that one is self-explanatory. J
In the news
Nikita Ivanov is also an InfoWorld contributor. Read the first in his series on in-memory computing, “Ensuring big data and fast data performance with in-memory computing.”
- Stack Overflow. Stack Overflow is a question and answer site for professional and enthusiast programmers.
- Habrahabr. Habrahabr (also "Habr") (Russian: Хабрахабр, Хабр) is a Russian collaborative blog with elements of social network about IT, Computer science and anything related to the Internet, owned by Thematic Media.
- In-Memory Computing Planet (blogs and events) Add you blog feed!
- “Meetup in a Box.” If you would like to speak at a meetup, start or support a meetup, or have questions about meetups in general – let me know! I can help get you up and running with everything you’ll need.
Please share any resources I've excluded in the comments section and I'll include them in the next edition.
Apache Ignite Community Update (August 2017 Issue)
by Tom Diederich
Igniters, here are some community highlights from the last couple week. If I missed anything, please share it here. Meetups! Did you know that Apache Ignite experts are available to speak at your meetup? And we also have spots open for YOU to speak at the following meetups that some of us co-organize:
- Apache Ignite London
- Bay Area In-Memory Computing Meetup
- NYC In-Memory Computing Meetup
- Moscow Apache Ignite Meetup
Meanwhile, here’s where to catch some great talks about Apache Ignite! We have 19 newly scheduled meetup talks on the books since the last update. All upcoming Ignite events can be found here. Let’s take a closer look at some of them….
Scheduled speaking engagements
* Sept. 9: Big Data and Cloud Meetup (Santa Clara, Calif.)
Apache Ignite PMC chair Denis Magda will be speaking at the Big Data and Cloud Meetup September 9 from 10 a.m. to noon. His talk is titled: "Apache Spark and Apache Ignite: Where Fast Data Meets the IoT".
* Sept. 13: SF Big Analytics Meetup
Denis Magda will be the featured speaker at the SF Big Analytics Meetup on Sept. 13. Denis' talk is titled: "Apache Ignite: the in-memory hammer in your data science toolkit."
* Sept. 18: Meetup: Cambridge .NET User Group
Apache Ignite evangelist Akmal Chaudhri will speak at the Cambridge .NET User Group Sept. 17. The title of his talk: "Scale Out and Conquer: Apache Ignite for .NET Users."
* Sept. 27: New York Kubernetes Meetup
Apache Ignite evangelist Akmal Chaudhri will focus on a DevOps perspective on the orchestration of distributed databases such as Apache Ignite. Akmal will speak on node auto-discovery, automated horizontal scalability, availability, and utilization of RAM and disk with Apache Ignite.
* Oct. 4: Openstack & Ceph User Group Amsterdam
Apache Ignite evangelist Akmal Chaudhri will show attendees how to build a Fast Data solution that will receive endless streams from the IoT side and will be capable of processing the streams in real-time using Apache Ignite's cluster resources.
* Oct. 13: Big Data Week London 2017: A Festival of Data (conference)
Akmal Chaudhri will be speaking at the Big Data Week conference Oct. 13 in London. His talk, titled "Powering up banks and financial institutions with distributed systems," will educate attendees about important Apache Ignite features for financial applications -- such as ACID compliance, SQL compatibility, persistence, replication, security, fault tolerance and more.
* Oct. 18: Silicon Valley Java User Group
Join Apache Ignite PMC Chair Denis Magda will introduce the many components of the open-source Apache Ignite. His talk, titled, “Catch an intro to Apache Ignite and skyrocket Java applications,” will teach attendees how to solve some of the most demanding scalability and performance challenges. He will also cover a few typical use cases and work through some code examples.
* Oct. 19: Eurostaff Big Data London
Apache Ignite evangelist Akmal Chaudhri will show attendees how to build a Fast Data solution that will receive endless streams from the IoT side and will be capable of processing the streams in real-time using Apache Ignite's cluster resources.
* Oct. 24: Spark Summit Europe 2017 (conference)
Akmal Chaudhri will be presenting at the Spark Summit Europe conference, Oct. 24-26 at the Convention Centre Dublin in Ireland. His session is titled: "How to Share State Across Multiple Spark Jobs using Apache Ignite."
* Nov. 2: Byte-Academy-FinTech-Python-Blockchain-Education Meetup (London)
In his talk, titled, "Powering up banks and financial institutions with distributed systems,” Apache Ignite technical Akmal Chaudhri will explain important Apache Ignite features for financial applications -- such as ACID compliance, SQL compatibility, persistence, replication, security, fault tolerance and more. A customer case study will also be presented.
- Scale-up vs. scale-out architectures
- Cloud Wars: Apache Ignite – Getting started with AWS for Beginners (Part I)
- Apache Ignite Tip: Peer Class Loading Deployment Magic
- The Future of In-Memory Computing
- Sept. 27: Better Machine Learning with Apache Ignite, with technical evangelist Akmal B. Chaudhri.
- Oct. 4: Postgres with Apache Ignite: Faster Transactions and Analytics, with GridGain senior solution architect Fotios Filacouris.
Past webinars (recordings available!)
Deploy like a Boss: Using Kubernetes and Apache Ignite, with GridGain solution architect Dani Traphagen.
Apache Ignite 2.1 - A Leap from In-Memory to Memory-Centric Architecture
The power and beauty of in-memory computing projects are that they truly do what they state -- deliver outstanding performance improvements by moving data closer to the CPU, using RAM as a storage and spreading the data sets out across a cluster of machines relying on horizontal scalability.
However, there is an unspoken side of the story. No matter how fast a platform is, we do not want to lose the data and encounter cluster restarts or other outages. To guarantee this we need to somehow make data persistent on the disk.
Most in-memory computing projects address the persistence dilemma by giving a way to sync data back to a relational database (RDBMS). That sounds reasonable and undoubtedly works pretty well in practice, but if we dig deeper, you’ll likely encounter the following limitations:
If you use either Apache Ignite 1.x or 2.0 along with the RDBMS for disk storage, then you will hit these limitations. It’s just the way in-memory architectures are integrated with the disk.
However, the limitations are no longer relevant for Apache Ignite 2.1! This version made a leap from in-memory to a memory-centric architecture that:
Curious about how Ignite achieved these huge advantages? Lifting the curtain….
Durable Memory Architecture
The Apache Ignite memory-centric platform is based on the durable memory architecture that allows storing and processing data and indexes both in-memory and on disk when the Ignite Persistent Store is enabled. The memory architecture helps to achieve in-memory performance with the durability of the disk using all of the resources available in the cluster.
The durable memory is built and operates in a way similar to the virtual memory of operating systems such as Linux. However, the one significant difference between these two types of architectures is that the durable memory one always keeps the whole data set and indexes on the disk -- if the Ignite Persistent Store is enabled -- while the virtual memory uses the disk for swapping purposes only.
Ignite Persistent Store
Persistent Store is a distributed ACID and SQL-compliant disk store that transparently integrates with the durable memory as an optional disk layer (SSD, Flash, 3D XPoint). Having the store enabled, you no longer need to keep all of the data in memory or warm-up the RAM after a whole cluster restart. The persistent store will keep the superset of data and all the SQL indexes on the disk making Ignite fully operational from the disk.
Tired of hooking up Ignite with an RDBMS? Go ahead and download Apache Ignite 2.1, enable Ignite Persistent Store, and launch your first durable Ignite cluster that distributes data sets and workloads relying on the performance of RAM and durability of the disk!
Finally, Apache Ignite 2.1 can boast about another achievements in .NET, C++, SQL and Machine Learning. Go ahead and discover them!
Apache Ignite 2.0: Redesigned Off-heap Memory, DDL and Machine Learning
We released the long-awaited Apache Ignite version 2.0 on May 5. The community spent almost a year incorporating tremendous changes to the legacy Apache Ignite 1.x architecture. And all of that effort paid off. Our collective blood, sweat (and perhaps even a few tears) opened up new and exciting opportunities for the Apache Ignite project.
Have I piqued your interest about this new release yet? Let's walk through some of the main new features that have appeared under the hood of Apache Ignite 2.0.
Reengineered Off-Heap Memory Architecture.
The platform’s entire memory architecture was reengineered from scratch. In a nutshell, all of the data and indexes are now stored in a completely new manageable off-heap memory that has no issues with memory fragmentation, accelerates SQL Grid significantly and helps your application easily tolerate Java GC pauses.
Take a peek at the illustration below and try to guess what’s changed. Afterward, please read this documentation to see if your eye caught everything that’s new.
Here’s something extremely noteworthy: the architecture now integrates seamlessly with disk drives. Why do we care about this? Stay tuned!
Data Definition Language.
This release introduces support for Data Definition Language (DDL) as a part of its SQL Grid functionality. Now you can define -- and, what’s more important, alter -- indexes in runtime without the need to restart your cluster. Apache Ignite users have long awaited this feature! Even more exciting news: users can leverage this with standard SQL commands like CREATE or DROP index. This is only the beginning! Go to this page to learn more about current DDL support.
Machine Learning Grid Beta - Distributed Algebra.
Apache Ignite is about more than in-memory storage. And it’s not just one more product for distributed computations or real-time streaming. It's much, much more than that. It's a hot blend of well-integrated distributed and highly concurrent modules that turned Apache Ignite into what is today: A robust data-fabric and framework with the goal of making your application thrive and outperform even the best of expectations.
But there was one thing missing until now. Drumroll, please: machine-learning support!
With Apache Ignite 2.0 you can check project’s own distributed algebra implementation. The distributed algebra is the foundation of the entire component. And soon you can expect to get distributed versions of widely used regression algorithms, decision trees and more.
Spring Data Integration.
Spring Data integration allows the interaction of an Apache Ignite cluster using the well-known and highly adopted Spring Data Framework. You can connect to the cluster by means of Spring Data repositories and start executing distributed SQL queries as well as simple CRUD operations.
Are you using Rocket MQ in your project and need to push data from the Rocket to Ignite? Here is an easy solution.
Hibernate L2 cache users have been anticipating support of Hibernate 5 on Apache Ignite for quite a long time. Apache Ignite 2.0 grants this desire. The integration now supports Hibernate 5 and contains a number of bug fixes and improvements.
Ignite.NET has been enhanced with an addition of a plugin system that allows the writing and embedding 3rd party .NET components into Ignite.NET.
The Ignite.C++ part of the community finally came up with a way to execute arbitrary C++ code on remote cluster machines.
This approach was initially tested for continuous queries. You can now register continuous queries' remote filters on any cluster node you like. Going forward you can expect support for the Ignite.C++ compute grid and more.
Want to learn more? Please join me June 7 for a webinar titled, “Apache® Ignite™: What’s New in Version 2.0.” I hope to see you there!
P.S. Just in case you can’t wait until June… here's a full list of the changes inside Apache Ignite 2.0.
Presenting Apache Ignite SQL Grid at Big Data Bootcamp
Apache Ignite community welcomes you to attend Big Data Bootcamp on March 27th, 28th and 29th 2017 in Santa Clara, USA.
The conference gathers experts and vendors from Big Data realm in sunny California who will be covering a variety of Big Data products and technologies, including, but not limited to, Hadoop, Spark, NoSQL, Data Science, Machine Learning, Artificial Intelligence & Deep Learning.
Apache Ignite will be introduced at the conference by its PMC chair and committer - Denis Magda.
As all we know, in-memory data grids bring exceptional performance and scalability gains to applications built on top of them. The applications truly achieve 10x more performance improvement and become easily scalable and fault-tolerant thanks to the unique data grids architecture. However, because of this particular architecture, a majority of data grids have to sacrifice traditional SQL support requiring application developers to completely rewrite their SQL-based code to support data grid specific APIs. This, however, is not true for Apache Ignite.
In this presentation, Denis will introduce Apache Ignite SQL Grid component that combines the best of two worlds - performance and scalability of data grids and traditional ANSI-99 SQL support of relational databases. Moreover, Denis will take an existing application that works with a relational database and will show how to run it on top of Apache Ignite with minimum efforts.
The talk is called "Apache Ignite SQL Grid: Hot Blend of Traditional SQL and Swift Data Grid" and takes place at 1:00 PM - 1:40PM on March 28. Refer to Big Data Bootcamp's agenda for more details.
Finally, use promotional code SPEAKER to receive $200 discount on or before March 15th by registering at the conference site.
See you at the conference!
Apache Ignite 1.9 Released
Apache Ignite community is pleased to announce Apache Ignite 1.9 - the next minor release of a well-known in-memory data fabric. The release, as usual, encompasses many bug fixes, performance improvements and fresh features. Below you can see a description of the most significant updates.
Apache Ignite was integrated with Kubernetes which is a modern open source container cluster manager. The integration helps to simplify a deployment of an Apache Ignite cluster in environments managed by Kubernetes and let the latter care of resources management, cluster's scalability and lifecycle.
For instance, you're no longer need to monitor a cluster state constantly to be sure that the number of cluster nodes doesn't go, let's say, below 4. If Kubernetes sees that one cluster node is disconnected and only 3 are left then it will start one more automatically to meet the deployment requirements.
Refer to Kubernetes Deployment Getting Started if this is the feature of interest for you.
Performance Optimizations and Benchmarks Automation
Apache Ignite 1.9 can boast of much better performance for core cache operations and SQL queries in compare to the previous Apache Ignite 1.8 release. In general, we observe up to 40% performance increase for particular operations.
It's no longer a challenge to reproduce the performance numbers. Starting with Apache Ignite 1.9 release all the benchmarks are delivered in every Apache Ignite distribution and can be easily executed in your own environment.
Data Modification Language and Queries Parallelism
The community keeps spending significant time improving SQL Grid component that empowers Apache Ignite users with in-memory database capabilities.
In this release, DML (Data Modification Language) support was expanded to the level of Ignite.NET and Ignite.C++ APIs. Plus, a streaming mode was introduced for DML allowing to execute DML operations even faster for specific scenarios like initial data preloading.
One more SQL Grid related optimization makes it possible to parallelize a query execution on every Ignite node where the query has been mapped. By default, a query is executed in a single thread on every participating node. However, for a variety of OLAP use cases it might be a bottleneck and this is where the query parallelism can help out.
Apache Ignite implemented .NET TransactionScope API allowing to work with distributed Apache Ignite transactions fully relaying on standard interfaces available in .NET Framework. Refer to this documentation page for more information.
Ignite.C++ introduced support for well-known continuous queries API. Now, you can listen to data modifications happened on Apache Ignite's distributed caches side from your C++ applications.
Ignite’s spark integration was upgraded to the latest Spark version. Presently, you can leverage from Ignite Shared RDDs in applications using latest Spark version.
Give a Try
Go and grab the latest 1.9 release from our main site. Looking forward to your feedback!