Active/Active WAN-based Replication in Apache Geode (incubating)

This past February 2016 Pivotal contributed its Disaster Recovery (WAN) feature to Apache Geode. This feature has been incorporated into the Apache Geode 1.0.0-incubating.M2 release. This article describes how this feature benefits Geode users. To learn more about the WAN Gateway feature in Apache Geode, join the next Geode Clubhouse virtual meeting on June 8 at 9AM Pacific.[Read More]

Posted at 03:50PM May 27, 2016 by gregchase in General  |  Comments [31]  | 

Announcing Apache Geode Milestone Releases 1.0.0-incubating M1 & M2 – Seeking Testers

Apache Geode has just made its second major release incorporating some popular features from the commercial technology for supporting higher-scale deployments with low-latency requirements, and analyzing performance. Now we're seeking testers for feedback!

Posted at 04:13AM Apr 23, 2016 by gregchase in General  |  Comments [4]  | 

Getting Started With Geode

1.0 - Getting Started With Geode

Welcome to Geode, this blog entry is a beginners guide and tutorial covering the usage and features of Apache Geode. Geode is a performance orientated in-memory datastore with features that can be used for data caching, a NoSQL database, lightweight distributed messaging or for distributed compute tasks. Each of these aspects of Geode will be explored in greater detail in future blog articles with today's focus on simply getting the Geode source, building and then validating with a very simple example using Geode demonstrating basic caching of a key-value pair.

Chances are if you are reading this blog then you have already been looking at the Apache Geode website and this has further sparked your interest in what is Geode and how you can use it. Currently Geode is an incubating project and as such hasn't yet met all the requirements to be a first level Apache project, as an incubating project this provides the opportunity for new open source contributions and their developers to address Apache organization requirements, build a community and get the source into the public. Geode itself is derived from the long running commercial GemFire product and today has a rich feature set that can be immediately be used for creating applications. Let's dive in...

One assumption we are going to make upfront is that all the operations are being done within a Linux terminal window and using the Bash shell.

Next validate that a compatible version of the Java JDK is installed. The Java version needs to be a version greater than JDK 1.7.75.

If you see an error or other issues using jps, you may not have a full JDK installed which will be important for later blogs as tools such as jconsole, jstack and jmap will used.

Then validate that git is installed and available which will be used to download the source code from the Apache servers.

Now we are ready to get the Geode source and build Geode. The next step is to use git to clone the Geode source repository to your host machine.

The source code is also mirrored from the Apache server onto GitHub and can be downloaded from there as well. In most cases downloading the source should only take a few minutes at most. Once completed there should be a directory called incubator-geode created which now contains a copy of the Geode source repository.

Change into the new incubator-geode directory and list the files and directory.

Because the Geode project uses git, the whole project along with many feature branches are available. The initial default starting branch is the master branch which should be considered stable and is only periodically updated. Check which branch you are using with the following command.

If on the master branch like above then change to the develop branch. The develop branch is currently very active with updates and is where new features are being added.

The file build.gradle is the build script which is used by the Gradle build tool to create the Geode binary. You can find out more about Gradle from http://gradle.org/ but for our simple build needs we are just going to use just one command. Don't worry if you don't have Gradle installed, the script will download the required files to run the build and create the binaries. So for just getting started quickly we will skip doing the tests at this time but you should run the full test suite as some future time (especially if making changes to the Geode source).

The build process will download Gradle if needed, then it will use the build script to create the binary files and normally will perform unit tests to validate the build. If the build took more than 5 minutes then you may have forgotten the skip.tests flag, the tests are fairly extensive and take many hours to complete. Once the build is complete, the Geode binary artifacts are placed in the directory gemfire-assembly/build/install/apache-geode and should now be ready to use. Time to change to this directory and do a few simple tests to validate the build for ourselves.

There should be four directories created, the bin directory contains scripts to run the gfsh command line tool, config contains some default configuration files, lib is all the jar files used by Geode and last is the tools directory which is where extensions to Geode live. Lets start a simple Geode distributed cache and put a little data into it to test it ourselves.

First we will start the Geode command shell and use this to start everything and enter some data. We will discuss further in a future blog using the gfsh shell, for now follow the basic commands.

Now that the gfsh shell is started lets start the first process which is the Locator and this provides a number of services to the cache with the most important being it helps servers find each other by providing a list of members started in the distributed system. The Locator doesn't store any application data but instead is used to manage and share metadata about the servers and is a critical component which needs to be started first.

Next lets start two servers that will manage and store our user created data.

The first server is started using a default port of 40404, so in starting a second server we will need to use a different server port than what the first server is using.

All the members of the distributed system can be listed, some of values below will be different based on the environment but you should at this point see three members ( locator, server1 and server2).

To store data in the servers a Region must be created. The Region is very similar to a Java hashmap in usage with many additional features with one of the most important being that the data stored within the region will be shared between the two servers.

Time to put some data into our servers and use the region that was created earlier. Again similar to a hashmap a key-value pair will be inserted or put into the region to be stored by the servers. In this case the key is a simple string as well as the value.

Once the data has been put in the servers the value can retrieved using the get command and the key.

Let's exit from gfsh.

From the command line we can see that gfsh has started three processes that continue to run after stopping the gfsh shell.

Back into gfsh and time to stop the servers.

Reconnect to the locator using the default port of 10334 after having exited and restarted gfsh.

Before stopping servers we should clean up a bit and delete the region that was created for testing the build.

Stop the servers and locator processes. Once both servers are stopped the data which was stored only in memory will no longer be available even if the servers are restarted.

If you inspect the build directory there have been three new directories created, one for the Locator and each Server created by gfsh and are named the same as the name parameter used in the gfsh commands to start each process. The log files for each process will be written to these directories and if you encounter any issues then those log files should be reviewed for exceptions.

At the end here we have completed getting Geode, building it from source and doing some manual testing (putting and getting a string from the cache). We have introduced the Locator and Servers along with using the GFSH shell to create and start these processes. In the next blog entry we will discuss the use cases for Geode and how it can be used in an application.

For additional details and information see the following URL´s:

Posted at 11:09PM Oct 14, 2015 by vford13 in General  |  Comments [4]  | 

Securing Apache Geode With Your Corporate LDAP

This is a guest blog post written by Apache Geode contributor Guillermo Tantachuco

To protect your data and to mitigate threats such as spoofing, information disclosure, replay attacks and elevation of privileges, Apache Geode (incubating) provides mature security features such as authentication, authorization and SSL communication that have consistently addressed customer’s security requirements for over a decade. Apache Geode provides a flexible security framework and lets you integrate it with your organization’s preferred authentication method. In this article, we are going to discuss system-level authentication with LDAP and a sample authorization method. For system-level security, you need a service account, which is an account that belongs to your application instead of to an individual end-user. This is similar to how most applications currently connect to a relational database.

Apache Geode provides two (2) types of authentication:

• Cache server authentication allows a new member to join a cluster. The membership coordinator is responsible for authenticating credentials of the new member. Geode assumes that the coordinator is authenticated and subsequent members authenticate against the coordinator. When using locators, the coordinator is the first locator to join the cluster. When using multicast, the coordinator is the first server to join the cluster
• Client authentication enables client apps to provide service account credentials to a server during connection initialization and for each operation request so that servers never process the same request twice

To protect the membership coordinator, it is important to point out that you should always run the Geode cluster in a secure environment.

To prevent replay attacks, the server generates a random unique identifier and returns it to the client to use in its next operation request (get, put, destroy, among others). Then the server verifies the client’s unique identifier, processes the client request, and responds with a new randomly generated unique identifier, for the client to include in its next request. Please see Apache Geode online documentation for more details about authentication

The authorization of operations performed by a client application on a cluster can be restricted or completely blocked according to your configurations and programmatic authorization callback. There are two (2) types of authorization callbacks:

• Pre-operation: you can program the authorization callback to do something before the operation is executed
• Post-operation: All client operations that return a result (like get and query) and all notifications can also be authorized where the callback can peek and even modify the result before it is sent out to the client

What about audit trails? Any authorization implementation can also create an audit trail given that the following information is available to you: authorized user, code of the operation performed, region name, remote client and operation data (e.g: data for PUT and PUTALL commands). All that information can be stored wherever you decide including but not limited to: database table, log file and Apache Geode region. Please see Apache Geode online documentation for more details about authorization

Sample security scenario

The best way to illustrate authentication and authorization is with an example. In this example, we use a local LDAP server. The picture below shows two (2) sample applications that provide bank account access to employees and customers, respectively. The LDAP service account used by the employee application can read and update multiple bank accounts (Put, PutAll, Get and Query). The LDAP Service account used by the customer-facing application can only read and update information of a single bank account (Put and Get)

In our example, the cluster hosts two (2) regions: The Account region is a partitioned region that stores bank account information such as account ID, account number, account type, balance and credit line. The PermissionPerRole region is a replicated region that stores the roles of each LDAP user for a given Apache Geode region

Running the sample code

You can find the source code of the sample security scenario on this Github repository . The README.MD file contains detailed instructions to build and execute our sample client application with a local Apache Geode cluster and a local LDAP server. Once you have the environment configured, you can execute the sample client application, change to the ‘sample-client-security’ folder and execute this command: java -jar target/sample-client-security-0.1.0.jar

To run the test scenarios using the Customer application’s service account, enter 1. To use the Employee application’s service account, enter 2. This screenshot shows the menu that the sample client application displays.

When we use the Customer application’s service account to execute a PUT on the Account region, the “server1/server1.log” file of the cluster contains the following ALLOWED log messages:

However, when we use the Customer application’s service account to execute a PUT-ALL on the Account region, the “server1/server1.log” file correctly indicates that said service account could not perform the aforementioned operation by displaying a DENIED log message:

Now let’s walk through our sample authentication and authorization implementations.

Implementing Authentication for Apache Geode servers

Apache Geode has two (2) authentication interfaces that you need to implement: AuthInitialize and Authenticator which are located in the com.gemstone.gemfire.security package. Our sample implementations of AuthInitialize and Authenticator are called UserPasswordAuthInit and LdapUserAuthenticator, respectively. Both classes are part of the org.mycompany.security.samples package.

Locators and servers use our sample gemfire-server.properties file to store any security-related properties. Here is an excerpt of our sample properties file, feel free to change any LDAP-related settings to run this example in your environment:

• security-ldap-url=ldap://localhost:10389/
• security-ldap-basedn=o=sevenSeas
• security-ldap-filter=(&(objectClass=inetOrgPerson)(uid={0}))
• security-username=geode-system
• security-password=pass

These properties represent the URL of your LDAP server; the search base is the location in the LDAP directory tree from which any LDAP user search begins; the search filter defines LDAP user search criteria. Last but not least, both username and password to be used to join the cluster.

The properties file also specifies the fully qualified names of our sample implementations.

• security-peer-auth-init=org.mycompany.security.samples.UserPasswordAuthInit.create
• security-peer-authenticator=org.mycompany.security.samples.LdapUserAuthenticator.create

The security-peer-auth-init property should be set to the name of a zero-argument static method that returns an AuthInitialize object on the members while the security-peer-authenticator property should be set to the name of a zero-argument static method that returns an Authenticator object on the members and locators. You can find the source code of UserPasswordAuthInit and LdapUserAuthenticator implementation classes on Github.

Implementing Authentication for Apache Geode clients

To enable client authentication, we need to configure both cluster members and the client application.

Cluster configuration: To configure the cluster, we need to add this entry to our properties file:

• security-client-authenticator=org.mycompany.security.samples.LdapUserAuthenticator.create

The security-client-authenticator property should be set to the name of a zero-argument static method that returns an Authenticator object on all the servers.

Client application configuration: the settings required for the client application are described below.

The security-client-auth-init property should be set to the name of a zero-argument static method that returns an AuthInitialize object on all the clients. We also set both username and password to be used by client applications to authenticate with the cluster.

It is important to note that, in our example, we are using the same UserPasswordAuthInit class to obtain service accounts of client applications and servers.

Implementing Authorization

There is an authorization interface that you need to implement: AccessControl, which is part of the com.gemstone.gemfire.security package. Our sample implementation of AccessControl is called PrePostAuthorization as described in below diagram. This class is part of the org.mycompany.security.samples package.

To enable cluster authorization, we need to add these two (2) entries to our cluster’s properties file:

• security-client-accessor=org.mycompany.security.samples.PrePostAuthorization.create
• security-client-accessor-pp=org.mycompany.security.samples.PrePostAuthorization.create

For pre-operation calls, set security-client-accessor to the fully qualified name of the static method you programmed to return an instance of the class. For post-operation calls, set security-client-accessor-pp to the fully qualified name of the static method you programmed to return an instance of the class. You can find the source code of the PrePostAuthorization class on Github.

Summary

Now you know the mechanisms to enable system-level authentication and authorization of a Apache Geode cluster. It is important to point out that Apache Geode is very flexible and allows you to configure system users and their corresponding roles and permissions based on your security policy and technology. We find that most enterprise customers typically have a common infrastructure such as single sign-on or centralized authentication systems they need to integrate with. You can join Apache Geode project right now and start reaping the benefits of using a powerful distributed in-memory database that provides applications ultra-low latency access to terabytes of operational data at a massive scale, in a secure manner.

Posted at 06:10PM Jul 29, 2015 by gregchase in General  |  Comments [5]  | 

Welcome!

It's already been three months since Apache Geode was announced as open source for Apache Software Foundation (ASF) incubation. A lot of really interesting work has been done in that time and I’d like to take this opportunity to share that with you in our first official ASF blog post.

To start, we had our first contribution from a non-committer (yet) with GEODE-38 (thanks Neil!). This served as a way to exercise the project infrastructure with JIRA, Wiki, ReviewBoard and GitHub integration.

The very first Incubation report with Geode status is out, tracking our progress towards becoming a top-level Apache Project (TLP) and providing an overall status of the project activities. The highlights from the report include:

• Social Media accounts and guidelines are in place along with other important Wiki pages. To keep up to date with project activity follow us on Twitter @ApacheGoede and read our mailing lists.

• Many community related events already scheduled and are listed on our community calendar. Click on "Look for more" in order to check what we have in the coming months. There are talks already scheduled up to October.

• For convenient testing, developers can use a docker image that is based on nightly builds of Geode available at DockerHub. Note that since Geode has not yet made an official releases of software you really shouldn't be using the Docker image for anything but testing.

Another interesting way to track the growing maturity of an ASF poddling is to evaluate the project using the Apache Project Maturity Model. This model describes "how an Apache project should operate in a concise and high-level way". To check what's current Apache Geode maturity level, read the Maturity Level wiki page, or read Community Development for a more general status. You may notice that the project, although new to ASF, Geode is actually a mature project given that Pivotal GemFire has been in development for more than 10 years (Geode was born from GemFire code donated to ASF). More details about Geode history are available in the Incubation proposal.

Last, but not least, our wiki already has a lot of content including many deep dive articles explaining Geode usage and internal architecture to help potential contributors get up to speed. These pages are organized under How to Contribute and Application Development including how to build the source code, which is essentially as simple as doing:

git clone https://github.com/apache/incubator-geode
./gradlew build -Dskip.tests=true

For those who would like to try and and test the most recent build of software, just download one of our nightly builds. Note that this is meant to be used only for testing purposes.

That's all I have for now. Stay tuned for more updates and Welcome to Apache Geode! Please join the conversation at dev@geode.incubator.apache.org or users@geode.incubator.apache.org.

Posted at 11:30PM Jun 19, 2015 by William Markito in General  |  Comments [3]  |