Apache Geode

Friday May 27, 2016

Active/Active WAN-based Replication in Apache Geode (incubating)

Guest post by Apache Geode (incubating) committer Bruce Schuchardt.

To learn more about the WAN Gateway feature in Apache Geode, join the next Geode Clubhouse virtual meeting on June 8 at 9AM Pacific.

This past February 2016 Pivotal contributed its Disaster Recovery (WAN) feature to Apache Geode. This feature has been incorporated into the Apache Geode 1.0.0-incubating.M2 release. This article describes how this feature benefits Geode users.

Ensuring your systems run smoothly even when your data center has a hiccup, or a real disaster strikes is critical for many companies to survive when hardships befall them. As we enter the age of the zettabyte, seamless disaster recovery has become even more critical and difficult. There is more data than we have ever handled before, and most of it is very, very big.

Most disaster recovery (DR) sites are in standby mode—assets sitting idle, waiting for their turn. The sites are either holding data copied through a storage area network (SAN) or using other data replication mechanisms to propagate information from a live site to a standby site. When disaster strikes, clients are redirected to the standby site where they’re greeted with a polite “please wait” while the site spins up.

At best, the DR site is a hot standby that is ready to go on short notice. DNS redirects clients to the DR site and they’re good to go.

What about all the machines at the DR site? With active/passive replication you can probably do queries on the slave site, but what if you want to make full use of all of that expensive gear and go active/active? The challenge is in the data replication technology. Most current data replication architectures are one-way. If it’s not one-way, it can come with restrictions—for example, you need to avoid opening files with exclusive access.

Comparing Replication for MySQL, Oracle and Geode

The open source database MySQL only supports one-way replication for DR. It is limited to active/passive replication to a standby server (for more on how this works see How does MySQL Replication really work?). Replication is single threaded and based on a binary log that is written by the database. While it’s a slave, you can query the data. When you fail over, the slave database needs to be made into master and web servers need to be brought online before the site is usable.

At least one Oracle replication service offers bi-directional active/active support but the cost is high as it involves a lot of database writes to disk back and forth and there are contention points that slow it down and can cause gaps in replication during a failover.

At the other end of the spectrum, Geode supports full, in-memory, bi-directional (or N-way) replication. A no-SAN, shared-nothing configuration like Geode’s WAN gateway lets data flow in both directions at high speed and scale. You can even hook all of your sites together in a ring to limit the amount of work a site has to do to replicate its changes. In this type of topology, changes are replicated to a neighboring site that does its own store-and-forward replication.

Parallel, Contention-free Replication

The replication service in Geode is transparent to applications and does not affect normal use of the distributed big data grid. Setup is easy too as there is nothing to change in your configuration other than enabling the WAN service and providing the WAN endpoint(s) to use for replication. Geode automatically sets up a parallel asynchronous replication system across your machines that reaches out to the remote site and efficiently batches changes asynchronously across the WAN. This is markedly simpler than what you get with most other systems, like http://gemfire.docs.pivotal.io/docs-gemfire/latest/managing/disk_storage/operation_logs.htmlEhcache, where you are responsible for setting up major components like a JMS messaging bus between sites.

The type of replication it uses is parallel replication. Parallel replication spreads the work across servers and eliminates contention, or communication bottlenecks between data stores. Data partitioning breaks the data and work apart across many servers, building in horizontal scalability. Each partition has a replication queue that has one or more redundant backups in case of failure and streams data to a disk store for recovery. To ensure data remains persistent across a shut-down or fail-over, Geode uses Oplogging, similar to Apache Kafka, for high performance instead of the random-access messaging of traditional databases that is so slow.

Parallel replication allows Geode’s solution to keep up with the blinding speed of Geode’s data grid. Prior to having this feature the WAN gateway had higher latency and could not keep up as easily with the busy data grid.

With the innovation of parallel replication, the replication mechanism itself is as elastic as the entire Geode data grid. If you add more capacity by adding more cache nodes, WAN replication capacity is also expanded at the same time. Communications between sites is so distributed that bottlenecks can be nearly eliminated and latency stays very low.

Dealing with Active/Active conflicts—A Detailed Example

Contention happens as the database prevents data from being updated when two or more processes try to modify the same data. Essentially, the first process to the data will lock the data down until it is done with it. In an active/active configuration, data flows back and forth and the times when the same data could be modified simultaneously increases. However, Geode allows the simultaneous updates and automatically detects the conflict and retains the latest data. You can also add your own GatewayConflictResolver to handle the problem. The conflict resolution approach is similar to how most active/active technologies work.

GatewayConflictResolver is handed both pieces of information along with details about the operation being performed and when the changes happened. It can choose to keep the incoming change, reject it, merge it, or do something completely different. The product comes with an example of active/active replication showing how to do it. For example, let’s say you have a collection of Items that were modified at the same time by two sites. One adds the item “Nike Fuel” to the collection and the other adds “Jawbone UP” to it. The GatewayConflictResolver would see this in its onEvent callback:

TimestampedEntryEvent (
Key = WishList103499
oldSystemId = 1
newSystemId = 2
oldTimestamp = 1363885242046 (milli clock)
newTimestamp = 1363888231022
oldValue = ( “Fit Bit” “Nike Fuel” )
newValue = ( “Fit Bit” “Jawbone UP” )

The resolver can then merge the two collections together into (“Fit Bit” “Nike Fuel” “Jawbone UP”) and use the “helper” that is passed to it via onEvent to change the event value. Both systems would see a conflict, and their respective conflict resolver plugins would be responsible for attaining a consistent result.

As mentioned, Geode comes with an active/active example–most of the code is already there and ready to be copied and pasted into your own resolver. It even shows you how to resolve a conflict on compact PDX serialized data to keep CPU cost low when resolving conflicts. Note, PDX is a high-speed serialization mechanism that comes with Geode.

Putting data into a Geode cache is easy. Geode allows you to carve up the datagrid into maps that it calls Regions. Each Region is a key/value store that can either be replicated on other nodes and/or partitioned across the grid. When you create a Region, you give it a name, and Geode uses this to reach out to other nodes in the grid and inform them of your participation in the grid. Then you can pull data out of the Region using Map, ConcurrentMap interfaces and the extensions added by Geode’s Region interface including queries and transactions.

Here are some good examples of using Geode with Spring Data—all you have to do is start up some Geode data nodes with a Gateway hub, give the hub the address of your other data centers, then add a Geode Repository to your app.

If you just need to replicate session state between sites for fast failover, Geode has plug-in modules for HTTP session management with tc Server and webservers similar to Apache Tomcat, cache for Hibernate, Memcached. Bi-directional WAN replication lets you use both sites and gives continuous availability of data. If one of the sites goes down, clients can be distributed among remaining sites. Data not yet transmitted by the failed site is persisted on disk on the down site—the data will be missing until one of the redundant WAN replication queues is brought back up. So this issue must be dealt with, but the issue is the same for any replication technology. Of course, there is always a lag in synchronization while the data is being carted from one system to the other. With Geode, the lag is very small, because the data is queued for replication as soon as it’s put in the cache.

About the Author:

Bruce Schuchardt is a senior engineer at The Pivotal Initiative and has worked on Geode since its inception (as GemFire) in 2002. He is responsible for the replication, consistency and membership systems. Bruce holds an MS in computer science from University of Massachusetts


I should say Interesting ! Are you looking at replicating hadoop data as well along with OLTP ? Typically my problem is dev ops issue of data replication between Hadoop (HIVE) prod and non prod environments. i m looking at possible replication mechanisms. Tx

Posted by Naveen Mansur on June 06, 2016 at 04:25 PM GMT #

hi its really awesome blog.thank you for sharing this awesome blog. <a href="https://www.gangboard.com/big-data-training/apache-spark-training">apachespark training</a>

Posted by logavanig on July 20, 2016 at 11:38 AM GMT #

http://advancefms.com/ | advancefms.com | http://www.advancefms.com/ | www.advancefms.com/ |

Posted by technicalonline on July 03, 2019 at 03:29 PM GMT #

http://www.housekeepingmumbai.in/ | http://housekeepingmumbai.in/ | www.housekeepingmumbai.in | housekeepingmumbai.in/ |

Posted by technicalonline on July 03, 2019 at 03:30 PM GMT #

love the cohesivity ❤️ Colour and character modelling, very good-looking! I always want to see it more!!!

Posted by health on July 19, 2019 at 10:40 PM GMT #

Great logo Absolutely in love with this work! Every detail is awesome, truly inspiring work, congratulations :)

Posted by discount on July 19, 2019 at 10:45 PM GMT #

I love the personality you give them. Great work! amazing, such a complete project.

Posted by medicine on July 19, 2019 at 11:28 PM GMT #

looking scary b e s t

Posted by on July 20, 2019 at 08:43 AM GMT #

Great work Andrey, the way you presented everything it's showing your creativity. Love the work amazing style. Keep it up! Hey @Aevo Studio

Posted by trudonhamsu1975 on July 20, 2019 at 03:50 PM GMT #

¡Genial! exquisite!

Posted by farkwurzpere1976 on July 20, 2019 at 04:07 PM GMT #

nice!so cool! Oh wow this gave me so many goosebumps. Scary facts but inspiring and creative way of visually representing them. I hope this motivates people to make changes!

Posted by nonetlimen1989 on July 20, 2019 at 05:47 PM GMT #

I'm curiously inspired by this. Great work! I mean how cute is this!

Posted by closlinapoc1970 on July 20, 2019 at 06:12 PM GMT #

really cool! I wish I can sit this on airbus one day :) Awesome work!

Posted by touchscurtusis1980 on July 20, 2019 at 11:12 PM GMT #

Beautiful gallery! Beautiful interactions and transitions! Lovely work.

Posted by dercmomopho1982 on July 21, 2019 at 12:05 AM GMT #

这个棒! But, to be honest, it gives me a stange feeling of our humanity and our environmental impact.

Posted by cockverguetie1982 on July 21, 2019 at 12:54 AM GMT #

what a great palette, well done Amazing design!

Posted by siconmaama1979 on July 21, 2019 at 02:28 AM GMT #

是用c4d弄的吗? very nice job

Posted by plemtisulgue1983 on July 21, 2019 at 04:06 AM GMT #

Really loved this style! THANKS TO ALL OF YOU GUYS! There is such amount of comments that I have to thank you all at once! hahaha I really REALLY appreciate you kind words about my work and positive comments about the color, textures and ideas. Have a great day buddies!

Posted by wavasastver1988 on July 21, 2019 at 06:26 AM GMT #

Безумие! Очень красивая сильная работа! Nice concepts!!! Very like it.

Posted by exaldefbert1985 on July 21, 2019 at 06:58 AM GMT #

оформлено красиво, но идея печальная @Zuco • @Pepper Studio

Posted by fcachincounsalz1974 on July 21, 2019 at 07:30 AM GMT #

The brain hole is big enough, stunning work! loved the concept and I find it visually super impactful and appealing

Posted by pepcontvacap1989 on July 21, 2019 at 08:02 AM GMT #

Excellent work, amazing 超有感觉耶

Posted by derglopdetab1975 on July 21, 2019 at 08:34 AM GMT #

Thank you! @Fábio Mansos @Lukas Ruoff @Виктория Шекштело Beautiful colors! ❤️❤️

Posted by suranksturzar1985 on July 21, 2019 at 09:07 AM GMT #

Very inspiring work ! And the gif is fantastic! :)

Posted by skinacosti1980 on July 21, 2019 at 09:40 AM GMT #

Classy! W O W!

Posted by sahechoosuc1981 on July 21, 2019 at 10:13 AM GMT #

It's beautiful, but one question. Where are their eyes? Now it's look like skulls. Sorry. this is amaaazing!!

Posted by rekuroodhpi1988 on July 21, 2019 at 10:46 AM GMT #

fantastic work amazing

Posted by cametalsve1981 on July 21, 2019 at 11:18 AM GMT #

Excellent work

Posted by hachtechnologies on October 22, 2019 at 03:10 PM GMT #

You must take the official lowes customer satisfaction survey on https://surveysaga.com/www-lowes-com-survey/ and win a $500 gift card for free. Lowes is the second biggest home improvement store in the united states.

Posted by Wills on November 28, 2019 at 07:27 AM GMT #

If you are looking for the best website to take the official customer satisfaction surveys, then visit this official survey saga website https://surveysaga.com/. You can all the latest information about the online surveys on this survey saga website.

Posted by Survey Saga on November 28, 2019 at 07:28 AM GMT #


Posted by rajesh k on December 17, 2019 at 04:51 AM GMT #

Post a Comment:
Comments are closed for this entry.



Hot Blogs (today's hits)

Tag Cloud