Entries tagged [esme]

Thursday May 26, 2011

LDAP authentication via Lift API in Apache ESME

This blog was written by our new committer Vladimir Ivanov who implemented a feature that users have been wanting for a long time. 

In the first two parts of this blog series, scenarios were discussed where the authentication process was performed by the container. While it often makes sense to delegate this task to the container in order to integrate with corporate services such as Single Sign-On, sometimes it might be better to take full control of the authentication process and perform this task directly in the source code. In this final part of the blog series, I'll show how to authenticate user in LDAP via the Lift API as well as introduce some changes that have been made in the authentication modules.

Changes in UserAuth.scala

In order to perform authentication via the Lift API, a new module LDAPAuthModule has been added. Basically, it uses the same operation sequence as the ContainerManagedAuthModule: it gets the user credentials, tries to authenticate and authorize the user, retrieves additional attributes from the LDAP for a new user and finally logs the user in. Since several operations are common to both authentication modules, it is worth placing them in a base trait LDAPBase:

trait LDAPBase {

LDAPBase is marked with a self-type to denote that any concrete class that mixes with this trait is a AuthModule instance

  this : AuthModule =>

To check whether the user has a specific role, a list of roles, separated by commas, is read from the property file, split and placed into a val.

  val rolesToCheck = Props.get("role_list") match {
    case Full(s) => s.split(',').toList
    case _ => Nil
A new variable was also added to hold the current role for the User.

  var currentRole : String = _

The object LDAPVendor and method that extracts attributes from LDAP are used by both modules and were left without changes

  object myLdapVendor extends LDAPVendor
  def myLdap : LDAPVendor = ...
  def getAttrs(dn : String) : Map[String, List[String]] = ...

The following two methods are used to construct distinguished name for the user/group pair. Since users and groups can have different bases and prefixes, specific properties are retrieved from the property file depending on the isGroup flag (The new feature, default parameters, introduced in Scala 2.8 is used to set default value for this flag).
  def constructDistinguishedName(who : String, isGroup : Boolean = false) = {
    val base = Props.get( if(isGroup) {"ldap.groupBase"} else {"ldap.userBase"} )  openOr ""
     val dn = "%s,%s".format(constructNameWithPrefix(who, isGroup), base)
  def constructNameWithPrefix(username: String, isGroup: Boolean = false) = {
    val prefix = if(isGroup) {"cn"} else {Props.get("ldap.uidPrefix") openOr ""}
    val nameWithPrefix = "%s=%s".format(prefix, username)

The method logInUser was modified to set the current role for the authenticated User.

  def logInUser(who: User) {
    S.notice(S.?("base_user_msg_welcome", who.niceName))

The singleton object ContainerManagedAuthModule now mixes with the LDAPBase trait and uses it's constructDistinguishedName method to get the LDAP attributes. In all other aspects, it has the same implementation as before.

object ContainerManagedAuthModule extends AuthModule with LDAPBase

Now let's review the new LDAPAuthModule.

object LDAPAuthModule extends AuthModule with LDAPBase {

At first, a set of standard methods for all authentication modules is defined:

  override def isDefault = false
  def loginPresentation: Box[NodeSeq] = ...
  def moduleName: String = "ldap"
  def createHolder() = ...

The method performInit makes most of the module's work:

  def performInit(): Unit = {

The new module is mapped to the /ldap/login URL.
    LiftRules.dispatch.append {
      case Req("ldap" :: "login" :: Nil, _, PostRequest) =>
        val from = S.referer openOr "/"

It is necessary to check whether LDAP is enabled based on the setting in the property file and if that's the case, the user credentials are read from the HTTP request and used for the subsequent authentication in LDAP with the LDAPVendor.bindUser method. It takes the distinguished name (composed of username that comes from the HTTP request and the prefix/user base taken from the property file) and password and returns a Boolean. The next step is to check if the user is authorized with the checkRoles method.

      val ldapEnabled = Props.getBool("ldap.enabled") openOr false
      if(ldapEnabled) {
        val name = S.param("username").map(_.trim.toLowerCase) openOr ""
        val pwd = S.param("password").map(_.trim) openOr ""
        if(myLdap.bindUser(constructNameWithPrefix(name), pwd) && checkRoles(constructDistinguishedName(name))) {

After successful authentication and authorization, an attempt is made to find the existing User. If found, the User is logged in into the application. Otherwise, a new instance of User class is created, populated with attributes from LDAP, saved into the database and then logged in.

          (for {
              user <- UserAuth.find(By(UserAuth.authKey, name),
                                    By(UserAuth.authType, moduleName)).flatMap(_.user.obj) or
              User.find(By(User.nickname, name))
            } yield user) match {
            case Full(user) =>
            case Empty =>
              val usr = User.createAndPopulate.nickname(name).saveMe
              val ldapAttrs = getAttrs(constructDistinguishedName(name))
              val firstName = ldapAttrs("givenName").head
              val lastName = ldapAttrs("sn").head
              val mail = ldapAttrs("mail").head
        } else {

The method checkRoles takes the username as a parameter and iterates through a list of roles defined in the property file to see whether the current role contains this name as a value of its uniqueMember attribute. If that's the case, it assign this role to the currentRole var of LDAPBase trait (it will be then saved in the HTTP session) and returns true, otherwise it returns false.
def checkRoles(who : String) : Boolean = {
    for (role <-rolesToCheck) {
      val ldapAttrs = getAttrs(constructDistinguishedName(role, true))
      val uniqueMember = ldapAttrs("uniqueMember").head
      if(who == uniqueMember) {
        currentRole = role
        return true
    return false;

The last thing I'm going to mention are the two new properties in the default.props file. The property ldap.groupBase specifies the path in LDAP under which groups are searched. The list of application roles is set in the role_list property as a string separated by commas.


;Group base DN to check whether user has specific role
;Allow access to application for following roles

Note that the lift-ldap module also has to be added as a dependency to the Maven pom.xml and/or the SBT project file to work with the Lift LDAP API.


In this article, we have discovered how to authenticate the User directly in LDAP using the Lift API as well as the corresponding changes in the UserAuth class.


Lift API: http://scala-tools.org/mvnsites/liftweb-2.3/

Wednesday April 13, 2011

LDAP-based Authentication with Apache ESME and Apache Directory Server

This blog was written by our new committer Vladimir Ivanov who implemented a feature that users have been wanting for a long time. 

In this blog, I'll discuss how container-managed authentication works with an LDAP server and how to connect and get additional information from it. I'll use Jetty and Apache Tomcat web servers with Apache Directory Server (ADS) 1.5.7 - a certified LDAPv3 compatible server. It is also possible to use different LDAP compatible server such as a 389 Directory Server or MS Active Directory.

An LDAP server is a directory service that contains objects organized in a hierarchical manner.

Apache Directory Server (ADS)

The ADS project page provides a detailed server installation and configuration guide, but here are the basic steps: during the installation, the default server instance will be created. The configuration settings for this instance are defined in the server.xml file as Spring bean definitions. Below is an excerpt from the configuration file that was used for this blog.


First of all, it is necessary to create a new partition:




<jdbmPartition id="lester" suffix="dc=lester,dc=org" />


Ports that LDAP server will use are specified as tcpTransport elements:

<ldapServer id="ldapServer"








<tcpTransport address="" port="10389" nbThreads="8" backLog="50" enableSSL="false"/>

<tcpTransport address="localhost" port="10636" enableSSL="true"/>



The local path to the directory containing LDIF files should also be specified:

<apacheDS id="apacheDS">




Other configuration settings were left unchanged.

Now It is time to create the directory structure. I used the popular open source tool JXplorer for this task.

It is possible to export the directory structure from the JXplorer tool to the LDIF file — directory content in text format. Let's review the generated file.


Below is a record list beginning from the top. Note that each record has specific set of object classes defined in this schema. Each object class defines specific set of attributes, for example, person class defines Surname (sn) and Given Name (givenName) attributes. One object class can extend another, for example organizationalPerson class extends person class. top is a superclass for all other classes.

Domain lester.org is the root of this hierarchy:

dn: dc=lester,dc=org

objectClass: extensibleObject

objectClass: domain

objectClass: top

dc: lester

The Organizational Unit esme is placed one level lower:

dn: ou=esme,dc=lester,dc=org

objectClass: organizationalUnit

objectClass: top

ou: esme

The Organizational Unit Groups resides under the esme Organizational Unit:

dn: ou=Groups,ou=esme,dc=lester,dc=org

objectClass: organizationalUnit

objectClass: top

ou: Groups

There is only one group esme-users on the lowest level of the hierarchy and it has the vivanov user (specified by a full path) as a unique member:

dn: cn=esme-users,ou=Groups,ou=esme,dc=lester,dc=org

objectClass: groupOfUniqueNames

objectClass: top

cn: esme-users

ou: Groups

uniqueMember: uid=vivanov,ou=Users,ou=esme,dc=lester,dc=org

The Organizational Unit Users resides under the esme organizational unit:

dn: ou=Users,ou=esme,dc=lester,dc=org

objectClass: organizationalUnit

objectClass: top

ou: Users

The user vivanov and its corresponding attributes are defined on the lowest level of the hierarchy:

dn: uid=vivanov,ou=Users,ou=esme,dc=lester,dc=org

objectClass: organizationalPerson

objectClass: person

objectClass: uidObject

objectClass: inetOrgPerson

objectClass: top

cn: vivanov

givenName: Vladimir

mail: vivanov@lester.org

ou: Users

sn: Ivanov

telephoneNumber: +7 111 222 33 44

uid: vivanov

userPassword:: cXdlcnR5

There is also a special user with administrative rights: uid=admin,ou=system (default password: secret) defined in system schema. It was used to connect to ADS from JXplorer.

Those were all basic steps necessary to configure ADS for purpose of this b log. Let's move to the configuration of the web servers.


Before digging into the configuration details specific for each web server, let's review the common properties used to connect to the LDAP server.

First of all, it is necessary to specify the hostname / ip address and the port of our LDAP server — localhost:10389, as well as the credentials for an account that has rights to perform search operation and get attributes for users and roles in a directory tree. The special admin user described in previous section was also used for this purpose in this blog. Sometimes anonymous access is also permitted.

The next set of properties, user base and group base, specify the base context with which to lookup users and groups. For our configuration web server will search users under ou=Users,ou=esme,dc=lester,dc=org and groups under ou=Groups,ou=esme,dc=lester,dc=org paths in the directory tree accordingly.

The user id and role name attributes specify the prefix for user/group search filter. In our example, it has the uid value for users and the cn value for groups.

The uniqueMember attribute is used to check whether the user belongs to the specified group.

Now it is time to review the configuration for each of the web servers.

Note: The required lift-ldap dependency has been already included in pom.xml.


In order to configure Jetty to use LDAP server, two additional Maven dependencies: jetty-plus and jetty-ldap-jaas should be added to the pom.xml file. Configuration of maven-jetty-plugin includes the following steps: set the JAASUserRealm as an user realm implementation and specify the ldaploginmodule as the login module name. It is also necessary to set the system property java.security.auth.login.config with the ldap-loginModule.conf value:





















<userRealm implementation="org.mortbay.jetty.plus.jaas.JAASUserRealm">













The file ldap-loginModule.conf is placed under the ESME_ROOT/server folder. It specifies login module implementation class - LdapLoginModule as well as LDAP-specific connection properties:


ldaploginmodule {

org.mortbay.jetty.plus.jaas.ldap.LdapLoginModule required




















Note that for some environments forceBindingLogin attribute must also be set to true.


The only required change in the Tomcat's server.xml configuration file (compared to the changes described in the last blog) is a different realm - JNDIRealm. This realm is used to connect to LDAP server and search users/groups:



<Realm className="org.apache.catalina.realm.JNDIRealm"



connectionURL="ldap://localhost:10389" debug="99"









LDAPVendor and ESMELdap.properties file

The web server is now configured to perform CMA. But the Servlet API makes only the user principal available for application. In order to fill ESME user's profile, additional attributes such as firstname, lastname and email are needed. We will use LDAP server to retrieve these attributes. Let's review changes in UserAuth.scala, specifically in ContainerManagedAuthModule object:


To connect to the LDAP server from application first of all, it is necessary to create aq subclass of net.lift.ldap.LDAPVendor class:

object myLdapVendor extends LDAPVendor

All LDAP-specific connection properties are placed into a resource bundle — plaintext property file with key-value pairs. It is possible to get property values by key with the S.? method:

def myLdap : LDAPVendor = {

val ldapSrvHost = S.?("ldap.server.host")

val ldapSrvPort = S.?("ldap.server.port")

val ldapSrvBase = S.?("ldap.server.base")

val ldapSrvUsrName = S.?("ldap.server.userName")

val ldapSrvPwd = S.?("ldap.server.password")

val ldapSrvAuthType = S.?("ldap.server.authType")

val ldapSrvReferral= S.?("ldap.server.referral")

val ldapSrvCtxFactory = S.?("ldap.server.initial_context_factory")

The next step is to configure the LDAPVendor subclass with these values:

myLdapVendor.configure(Map("ldap.url" -> "ldap://%s:%s".format(ldapSrvHost, ldapSrvPort),

"ldap.base" -> ldapSrvBase,

"ldap.userName" -> ldapSrvUsrName,

"ldap.password" -> ldapSrvPwd,

"ldap.authType" -> ldapSrvAuthType,

"referral" -> ldapSrvReferral,

"ldap.initial_context_factory" -> ldapSrvCtxFactory))



The method getAttrs takes the username as a parameter and returns a map of [attribute name / list of attribute values] pairs (attribute in LDAP might contain more than one value) for this user. Let's review the method definition. It is possible to get attributes for user with LDAPVendor.attributesFromDn() method. It takes the distinguished name as a parameter, so it is necessary to append the prefix and the user base from the property file to the username to construct it. Note that the attributesFromDn method returns javax.naming.directory.Attributes therefore the interfaces from javax.naming.directory package must be imported correctly:

import _root_.javax.naming.directory.{Attributes, Attribute => Attr}

The shorthand Attr is used for the javax.naming.directory.Attribute because the scala.xml.Attribute trait has already been imported and placed in scope.

Then attribute's id and values are used to populate the result map.

The getAttrs method definition is shown below:

def getAttrs(who : String) : Map[String, List[String]] = {


val uidPrefix = S.?("ldap.uidPrefix")

val userBase = S.?("ldap.userBase")


var attrsMap = Map.empty[String, List[String]]

val dn = "%s=%s,%s".format(uidPrefix, who, userBase)


val attrs : Attributes = myLdap.attributesFromDn(dn)

if (attrs != null) {

val allAttrs = attrs.getAll();

if (allAttrs != null) {

while(allAttrs.hasMore()) {

val attribute = allAttrs.next().asInstanceOf[Attr];

var attrValues = List.empty[String]

for(i <- 0 until attribute.size()) {

attrValues ::= attribute.get(i).toString


attrsMap += (attribute.getID() -> attrValues)






The last step is to modify the performInit method. First of all, it is necessary to check if LDAP is enabled as configured in the property file. Then the values for attributes givenName, sn and mail are extracted from the map, returned via the getAttrs method call and then used to populate User instance.

def performInit(): Unit = {


val usr = User.createAndPopulate.nickname(username).saveMe

//find and save additional attributes in LDAP if It is enabled

val ldapEnabled = S.?("ldap.enabled")

if(ldapEnabled.toBoolean) {

val ldapAttrs = getAttrs(username)

val firstName = ldapAttrs("givenName").head

val lastName = ldapAttrs("sn").head

val mail = ldapAttrs("mail").head





The ESMELdap property file is shown below. It essentially resembles connection properties in web server configuration files that we have seen previously.

ESMELdap.properties file

#This flag specifies whether LDAP should be used


# Hostname or IP of LDAP server


# Port of LDAP server


# Base DN from the LDAP Server


# User that has access to LDAP server to perform search operations


# Password for user above


# Authentication type


# Referral


# Initial context factory class


# Prefix for user to whom additional LDAP attributes belong, for example 'uid' or 'sAMAccountName'


# User base DN for user to whom additional LDAP attributes belong


The last thing that must be done is to tell Lift where to look for the ESMELdap.properties file. The list of resource file names is assigned to LiftRules.resourceNames var in Boot.scala:


LiftRules.resourceNames = "ESMELdap" :: "ESMECustom" :: "ESMEBase" :: "ESMEUI" :: Nil


We have just configured both web servers - Jetty and Tomcat - to perform authentication and authorization via LDAP. We have also improved ContainerManagedAuthModule to get additional attributes for authenticated user from LDAP with Lift LDAP API.


1. Apache Directory Server: http://directory.apache.org/apacheds/1.5/

2. Jxplorer: http://jxplorer.org/

3. Jetty login modules: http://docs.codehaus.org/display/JETTY/JAAS


4. Tomcat user realms: http://tomcat.apache.org/tomcat-6.0-doc/realm-howto.html

5. Lift API: http://scala-tools.org/mvnsites/liftweb-2.3/

Thursday April 07, 2011

Container-Managed Authentication with Apache ESME: Part 1

Part 1: Performing authentication with plaintext/xml user-role mapping

This blog was written by our new committer Vladimir Ivanov who implemented a feature that users have been wanting for a long time. 


Apache ESME currently supports two different authentication schemes: when user credentials are stored in the database and via OpenID.  Corporate users, however, might be interested in container-managed authentication (CMA) — because this scheme supports integration with enterprise services such as LDAP and Single Sign-On. In the first part of this blog,  I'll explain how ESME-based applications can use CMA and how to configure two popular web servers- Apache Tomcat and Jetty - with simple user-role mapping. In the second part, I'll describe LDAP integration to perform CMA and how get additional user attributes via the Lift LDAP API.

ContainerManagedAuthModule: The necessary code changes

A new authentication module ContainerManagedAuthModule was introduced to hook into the container-managed authentication process. First of all, it was registered along with the other authentication modules:






All authentication modules should extend the AuthModule trait:

object ContainerManagedAuthModule extends AuthModule

Currently, the list with security role (group) names is also defined in the source code:

  val rolesToCheck = List(



It is also possible to get the list of roles from some external source, for example, from a property file or a LDAP.

The method moduleName defines the name for the new auth module. This value acts as a discriminator and will be stored in the DB:

  def moduleName: String = "cm"

After the container finishes the authentication and authorization phases, it is  neccessary to hook into the normal user processing to save the user data. This task is performed in the performInit method:

def performInit(): Unit = {

CMA must be applied to a specific URL, for example /cm/login, so it is necessary to append a partial function to LiftRules.dispatch to perform the neccessary operations:

    LiftRules.dispatch.append {

      case Req("cm" :: "login" :: Nil, _, _) =>  {

        val from = "/"

Note: The majority of necessary steps to further utilize this new auth method have already been desribed in the Lift Wiki.

In short, it is neccessary to unwrap the javax.servlet.http.HttpServletRequest object to get the username and role names. If a user has one of the specified roles, the module should attempt to find an existing user with the same nickname which previously has logged in via this module. If such a user hasn't been found, a new User is created.  The last step is to save the userId in the HTTP session via User.logUserIn method call.

        S.request match {

          case Full(req) => {

            val httpRequest: HTTPRequest = req.request

            val hrs = httpRequest.asInstanceOf[HTTPRequestServlet]

            val hsr: HttpServletRequest = hrs.req

            val username : String = hsr.getRemoteUser


              val currentRoles = rolesToCheck.filter(hsr.isUserInRole(_))

              if(currentRoles.size == 0) {

                info("No roles have been found")


              } else {

                currentRoles.map(cr => {

                (for {

                    user <- UserAuth.find(By(UserAuth.authKey, username),

                                          By(UserAuth.authType, moduleName)).flatMap(_.user.obj) or

                    User.find(By(User.nickname, username))

                  } yield user) match {

                    case Full(user) => {



                    case _ => {

                      val usr = User.createAndPopulate.nickname(username).saveMe

                      //find and save additional attributes in LDAP if it's enabled











Now it's time to set-up the configuration for the CMA. All configuration settings for a Java EE web application (ESME is based on the Lift web framework, so it's packaged as a WAR file), including security settings, are defined in the web.xml file:


For this example, we will use form-based authentication:




Next, the login and error pages are specified.






Then, the security-role name, which any authenticated user must have for successful authorization, is defined:

    <!-- Security roles referenced by this web application -->


      <description>An authenticated ESME user</description>



And finally it is necessary to configure the mapping between the security role and the URL which is associated with the new authentication module:

  <!-- Secured resources -->




        <description>Secured page for forcing the container to request login</description>







The login page contains a form with specific action attributes and two input fields:






    <body id="cm_login">

        <form method="POST" action="j_security_check">

          Username: <input type="text" name="j_username"/><br>

          Password: <input type="password" name="j_password"/><br>

          <input type="submit"/>




Let's move on to the web server configuration for the next steps. We must define the users for our web application as well as mapping between these users and the security role that is specified in the web.xml file. I'll show how to configure simple user-role mapping for two popular web servers — Jetty and Tomcat.


The HashUserRealm implementation is used to specify the user-role mapping in the properties file for Jetty. The maven-jetty-plugin has been already included in the Maven project file pom.xml for the ESME application, so it is possible to configure Jetty in the plugin configuration section:









                        <userRealm implementation="org.mortbay.jetty.security.HashUserRealm">







The format for this property file has the following form: username: password [,rolename ...].

An example is shown below:


cmuser: cmuser, esme-users

That's it. Now Jetty is configured for CMA. Execute mvn clean jetty-run command to start Jetty and type http://localhost:8080/cm/login URL in your browser. You should see the form containing the username and password fields. Now try to log in with the user with the id cmuser.


The configuration of Tomcat web server is very similar to that of Jetty, except that the user-role mapping is specified in a XML file.  The MemoryUserDatabaseFactory implementation is used to define the mapping file.  The corresponding realm UserDatabaseRealm is also specified in the server.xml configuration file:



    <!-- Editable user database that can also be used by

         UserDatabaseRealm to authenticate users


    <Resource name="UserDatabase" auth="Container"


              description="User database that can be updated and saved"


              pathname="conf/tomcat-users.xml" />



      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"


Below is an example of the user-role mapping definition in the tomcat-users.xml file which is usually located in the  tomcat/conf directory.



  <role rolename="esme-users"/>

  <user name="cmuser" password="cmuser" roles="esme-users"/>


Now it's neccessary to package the WAR file with the mvn clean package command and deploy it to Tomcat either via maven plugin or in Tomcat's administrative console. Then proceed to the following URL:


The login form should be displayed.


In this part of the blog, I've covered the new authentication module, application and server configuration and simple user-role mapping. In the next part, I'll show how to configure Tomcat to use LDAP for CMA and get additional attributes for the authenticated user.


1. Lift Wiki - How to use Container Managed Security : http://www.assembla.com/wiki/show/liftweb/How_to_use_Container_Managed_Security

2. Jetty HashUserRealm: http://jetty.codehaus.org/jetty/jetty-6/apidocs/org/mortbay/jetty/security/HashUserRealm.html

3. Tomcat Realms: http://tomcat.apache.org/tomcat-6.0-doc/realm-howto.html

4. Java EE 5 Tutorial – Securing Web Applications: http://download.oracle.com/javaee/5/tutorial/doc/bncas.html

Thursday February 03, 2011

Using Scala Code Coverage Tool (SCCT) in ESME

Blog Author: Vladimir Ivanov


Scala Code Coverage Tool (SCCT) is a code coverage tool written in Scala and configured as SBT plugin (while integration with Maven is also possible). As for any large project, test are vital for ESME as they gurantee that application works as expected. SCCT gathers data (after source code instrumentation phase) while tests are being run and generates user-friendly report and statistics related to test coverage of source code.

Installation and configuration

Introducing SCCT support to ESME project is a matter of adding SBT plugin and consists of two steps: mixin project definition file with new trait as well as adding new repo and dependency to plugin definition file. Scala 2.8.1-compatible version of SCCT plugin is used in this article, but version for 2.7 is also available.

Project File

Project definition class for web project EsmeProject should mixin with ScctProject trait:

import reaktor.scct.ScctProject

class EsmeProject(info: ProjectInfo) extends DefaultWebProject(info) with ScctProject


SCCT related configuration in plugin definition file is shown below:

class Plugins(info : ProjectInfo) extends PluginDefinition(info) {
val scctRepo = "scct-repo" at "http://mtkopone.github.com/scct/maven-repo/"
lazy val scctPlugin = "reaktor" % "sbt-scct-for-2.8" % "0.1-SNAPSHOT"




To collect test coverage data with SBT its neccessary to perform following command in SBT console:

sbt> test-coverage

Report files will be produced in ESME_ROOT/server/target/scala_2.8.1/coverage-report/ directory after all tests have been run. Open index.html file in your browser to see the results*.

Here is a video explaining the test results.


    • Chrome browser is currently not supported.
    • Also note that SCCT tool is still in beta.


Mikko Koponen is the author of SCCT tool. Here is the project page on github. Here is the SCCT home page.

Tuesday January 18, 2011

Use Apache Ant to post to Apache ESME

Apache ESME has an integration with Apache Ant. It is now possible to send messages to Apache ESME that describe the progress of your build steps. This is especially useful when Apache ESME is being used in development projects where individual developers can be made aware of the status of various-build related actions. For example, a development team could be informed that a build is broken or if it is successful.  


  1. Install Apache Ant
  2. Install Apache Maven
  3. Download the Apache Commons packages "Codec" and Logger. We also need Apache "Httpclient" . Please note that we are currently using the legacy 3.1 version of the HTTP Client. 
  4. Check-out the ESME java client: https://svn.apache.org/repos/asf/esme/trunk/esme-java-client
  5. Perform a maven build in this directory with the following command "mvn compile package -Dmaven.test.skip=true" . This will create a jar file "esme-rest-api-0.1.jar" in the target directory. 
  6. Create lib directory and copy "rest-api-0.1-SNAPSHOT.jar", "commons-codec-1.2.jar", "commons-httpclient-3.1.jar" and "commons-logging-1.1.1.jar" into this directory. 
  7. Create src directory and copy the "EsmeAntTask.java" file to this directory
  8. Change the build.xml file to point to the appropriate directories
  9. Change the token in the "esme" target in the build.xml file to a correct token.
  10. Change the proxy in the "esme" target in the build.xml file to a correct proxy.
  11. call ant dist esme to test the application

Here is what the result looks like:

ant blog.jpg

Note: We wrote a version of this blog two years ago on our old blog. I've updated it and added an image plus more details. 


<project name="EsmeViaAnt" default="dist" basedir=".">
  <!-- set global properties for this build -->
  <property name="src" location="src"/>
  <property name="build" location="build"/>
  <property name="dist"  location="dist"/>

    <path id="project.class.path">
    <pathelement path="./dist/lib/EsmeViaAnt.jar:./lib/esme-rest-api-0.1.jar:./lib/commons-codec-1.4.jar:./lib/commons-logging-1.1.1.jar:./lib/commons-httpclient-3.1.jar"/>

  <target name="init">
    <!-- Create the time stamp -->
    <!-- Create the build directory structure used by compile -->
    <mkdir dir="${build}"/>

  <target name="compile" depends="init"
        description="compile the source " >
    <!-- Compile the java code from ${src} into ${build} -->
    <javac srcdir="${src}" destdir="${build}">
       <classpath refid="project.class.path"/>

  <target name="dist" depends="compile"
        description="generate the distribution" >
    <!-- Create the distribution directory -->
    <mkdir dir="${dist}/lib"/>

    <!-- Put everything in ${build} into the jar file -->
    <jar jarfile="${dist}/lib/EsmeViaAnt.jar" basedir="${build}"/>

  <target name="esme"
        description="Send Esme" >
        <java fork="true"
         <arg value="http://esmecloudserverapache.dickhirsch.staxapps.net/api"/>
         <arg value="31EL0R0M15NTD2LSOS0BKC5Y0P5JOVAZ6"/>
         <arg value="myproxy:81"/>
         <arg value="A message from the ant build process"/>
           <pathelement path="./dist/lib/EsmeViaAnt.jar:./lib/esme-rest-api-0.1.jar:./lib/commons-codec-1.4.jar:./lib/commons-logging-1.1.1.jar:./lib/commons-httpclient-3.1.jar"/>



EsmeAntTask java class

import org.apache.esme.api.EsmeRestApi;
import org.apache.esme.model.Message;

public class EsmeAntTask

public static void main (String[] args) {

 String apiUrl;
 String authToken;
 String localProxy;
 String message;
 EsmeRestApi esme;

 apiUrl = args[0];
 authToken = args[1];
 localProxy = args[2];
 message = args[3];

 try {

         esme = new EsmeRestApi(apiUrl);

         if ((localProxy != null) && !("".equals(localProxy)))
                // Split proxyHost:port into two parts
                String[] proxy = localProxy.split(":");
                esme.setProxy(proxy[0], Integer.parseInt(proxy[1]));


         Message esmeMsg = new Message();


         String[] tags = new String[1];
         tags[0] = "Ant Task";



 catch (Exception e) {}



Wednesday January 12, 2011

Using SBT in Apache ESME


While many projects have been using Apache Maven as widely adopted and recognized project management tool, Scala-based Simple Build Tool (SBT) became very popular dependency management tool recently. SBT has following advantages:

  • Buld file is written in Scala language which is much more concise comparing to verbose XML and provides full power of Scala platform and libraries

  • SBT intelligently tracks changes in source code to make accurate recompilation

  • SBT console mode keeps scalac in resident, which really improves compile times on subsequent runs. This is important for scalac, which is quite slow as compared to javac

  • SBT supports continious compilation and testing

  • While SBT is based on Apache Ivy, also very popular dependency management tool, it is possible to use both remote and local Maven repository with SBT

  • SBT has Custom Actions

That said SBT integration with popular IDEs and CI tools is not as good as Maven integration yet (I'll show an example of SBT and IDEA integration and give links to resources related to Hudson integration).

It is possible to build ESME project both with Maven and SBT. At the moment Maven is main build tool. SBT might be used locally by developers to improve performance. Subject of this article is building project with SBT.

Installation and configuration

1. Download SBT jar file and place it into local directory
2. Create script to launch SBT and specify path to SBT jar. For example, below is a script for Linux platform:

java -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m -Xmx512M -Xss2M -jar `dirname $0`/sbt-launch-0.7.4.jar "$@"

*JVM options above were added to launch script to avoid frequent OutOfMemory errors.

Now it is possible to launch SBT. But before proceeding to project structure, it would be useful to know how SBT manages its dependencies.

Dependency management

As noted before, SBT is based on Apache Ivy — popular dependency management tool. It means that all dependencies will be downloaded to local repository (located by default in $USER_HOME/.ivy2 folder) and orginized in a manner similar to Maven: organization/module/artifact/revision, where organization and revision are analogs to Maven's groupId and version accordingly. All automatically managed dependencies, specified in project definition file, are placed into PROJECT_ROOT/lib_managed directory.

Project structure

Overall project structure has following form:

--->ESME root
| |
| -----EsmeProject.scala
| |
| -----Plugins.scala
----- build.properties

SBT build artifacts have only been added to ESME server module, therefore they are located under server/project folder. These artifacts include:
ESMEProject.scala — project definition file containing build configuration
Plugins.scala — plugins for project definition are declared in this file
build.properties — contains project name, version, organization, Scala and SBT versions and other custom properties

The most important file among these is project file, so lets review it step-by-step.

Project File

Project definition class for web project EsmeProject should extend DefaultWebProject (which in turn implements base class for all types of projects sbt.Project):
class EsmeProject(info: ProjectInfo) extends DefaultWebProject(info)

Dependency versions are defined as vals:
val liftVersion = "2.2-RC6"
val compassVersion = "2.1.1"
val luceneVersion = "2.4.0"

It is possible to tell SBT to search dependencies in local Maven repository. To do that it's neccessary to define mavenLocal val and assign it to the path in the local filesystem where Maven repository resides:
val mavenLocal = "Local Maven Repository" at "file://"+Path.userHome+"/.m2/repository"

All additional repositories are also defined as vals. To include Scala Tools Snaphots repository predefined constant ScalaToolsSnapshots is used, for other repos explicit URL is specified:
val scalatoolsSnapshot = ScalaToolsSnapshots
val compassRepo = "Compass Repository" at "http://repo.compass-project.org"
val twitterRepo = "Twitter Repository" at "http://maven.twttr.com"

Project might contain additional files like licenses and notices. To include them in target jar it is neccessary to define method extraResources and override mainResources method, as shown below (assuming that both files are located in project's root directory) :
def extraResources = "LICENSE" +++ "NOTICE"
override def mainResources = super.mainResources +++ extraResources

It's possible to specify dependency definition with Ivy configuration file inline (for example to include or exclude dependent modules). To do that, it is neccessary to override ivyXML method:
override def ivyXML =
<dependency org="net.lag" name="configgy" rev="2.0.1">
<exclude org="org.scala-tools" module="vscaladoc"/>
<dependency org="com.twitter" name="ostrich" rev="2.3.2">
<exclude org="org.scala-tools" module="vscaladoc"/>

SBT manages dependencies based on dependency expressions in project's definition file. Dependency expressions have following form:
groupID % artifactID % revision % configuration

In case dependency was build with SBT, it is neccessary to change first % to %%:
groupID %% artifactID % revision % configuration

That way SBT knows how to download correct jar corresponding to Scala version which is used to build project (for example lift-webkit_2.8.1-2.2-RC6.jar).

It is possible to use range to specify version like in Maven. For example, value [6.1.6,) corresponds to all versions greater or equal to 6.1.6.

Configuration has form A->B, where configuration A use a dependencys configuration B. ESME project definition contains compile and test configurations. As an example, expression "junit" % "junit" % "4.4" % "test->default" says that ESME test configuration uses JUnit default configuration.

If no configuration mapping is specified explicitly, compile->default mapping is used by default.

One way to list all dependencies is to define libraryDependencies method containing Set of dependency expressions:
override def libraryDependencies = Set(
"net.liftweb" %% "lift-webkit" % liftVersion % "compile->default",
"net.liftweb" %% "lift-mapper" % liftVersion % "compile->default",
"org.compass-project" % "compass" % compassVersion % "compile->default",
"org.apache.lucene" % "lucene-core" % luceneVersion % "compile->default",
"org.mortbay.jetty" % "jetty" % "[6.1.6,)" % "test->default",
"junit" % "junit" % "4.4" % "test->default",
) ++ super.libraryDependencies

Build properties

SBT loader reads the versions of Scala and sbt to use to build the project from the project/build.properties file. If this is the first time the project has been built with those versions, the loader downloads the appropriate versions to project/boot directory. The sbt loader will then load the right version of the main sbt builder using the right version of Scala. Other user-defined properties might also be set in this file. Below is an example of build properties file for ESME:

project.organization=Apache Software Foundation
project.name=Apache Enterprise Social Messaging Environment (ESME)


All plugins for SBT are specified in plugins definition file Plugins.scala. It's structure is very similar to project definition file. Lets review plugins configuration via example of sbt-idea plugin which is used to generate project artifacts for IntelliJ IDEA.

Plugins class should extend PluginDefinition base class:
class Plugins(info : ProjectInfo) extends PluginDefinition(info)

Additional repositories which are used by plugins are declared as vals:
val sbtIdeaRepo = "sbt-idea-repo" at "http://mpeltonen.github.com/maven/"

Like in a project definition file, dependencies for plugins are specified via dependency expressions:
val sbtIdea = "com.github.mpeltonen" % "sbt-idea-plugin" % "0.1.0"

With sbt-idea plugin configured, it is possible to issue commands:
sbt update
sbt idea

and IDEA project file will be generated in project's root directory.

Main commands

All interaction with SBT is performed via commands, which are usually executed in SBT console. To enter into SBT console, it's necessary to run sbt in a project directory.

First group of commands is used to clean generated and downloaded artifacts.
clean command deletes target directory where all generated files are located.
clean-cache command deletes local Ivy repository where downloaded artifacts and metadata for automatically managed dependencies for this user are resides.
clean-lib command deletes lib_managed - managed library directory for this project.

update command is used to resolve and download all external dependencies into local Ivy repository.

compile command preforms compilation of all Scala source files located in src/main/scala folder. It's possible to specify additional options for compile task by overriding compileOptions method.

test command executes all tests that have been found during compilation. Runs test-compile command first (similar to compile task it's possible to specify additional options for test-compile task by overriding method testCompileOptions).

jetty-run command starts the Jetty server and serves this project as a web application on http://localhost:8080 by default. This variant of starting Jetty is intended to be run from the interactive prompt.
jetty-stop command stops the Jetty server that was started with the jetty-run action.

Dependencies list

Sometimes it's necessary to analyze all tree of used dependencies (to prevent conflicts between them for example). Maven has specific plugin dependency:tree to do just that. It's also possible to perform similar task with SBT via Project Console.

console-project command starts the Scala interpreter with access to project and to sbt.

For example, to get list of dependencies in compile classpath, current.publicClasspath command should be executed. Similar commands current.testClasspath and current.optionalClasspath exist for test classpath and optional classpath accordingly.

Integration with Hudson

While integration of SBT with Hudson, popular Continious Integration tool, hasn't been used in ESME project yet, there are some resources on this available:
Hudson SBT plugin: https://github.com/hudson/sbt-plugin
Christoph Henkelmann’s Blog entry - “How to build sbt projects with Hudson and integrate test results” : http://henkelmann.eu/2010/11/14/sbt_hudson_with_test_integration


Simple Build Tool home: http://code.google.com/p/simple-build-tool/
Apache Ivy home: http://ant.apache.org/ivy/
Lift Wiki page related to SBT: http://www.assembla.com/wiki/show/liftweb/Using_SBT
Mikko Peltonen's sbt-idea plugin home: https://github.com/mpeltonen/sbt-idea

A big thank you goes out to Vladimir Ivanov for the SBT implementation and for writing up this blog post explaining how it works!



Hot Blogs (today's hits)

Tag Cloud