Apache ESME [Retired]

Tuesday November 15, 2011

Revisiting Pools

Recently, we've had a few questions about the concept of pools in ESME. I spent some time reading old threads from our mailing lists to collect the motivations behind our design decisions. This blog is a collection of tidbits from these mail threads. 

First of all, “pools" are not interchangeable with "groups". They mean different things. A pool is about the messages. A group is about the people.

Groups are personal things where I assign different people into different groups and the meaning of a group is individual to me and it's all about my view of the world.  This keeps to the "opt in" mechanism that we absolutely must preserve in ESME.  If we do this type of group in the future, that's way cool, but once again, it's a personal thing that has nothing to do with access control or "sending".

Using the term "group" might lead people to think they are sending a message to a group of people, whereas they will actually be making it *available* to a group of people, should anyone in that group choose to look in the pool.

Pools are collections of messages that can only be read by people who have been granted access to that pool. A person who has access to a pool is able to see messages put into that pool that otherwise meet the person's criteria (who they are following, what their filter rules are.)  There is no "send to a pool" concept.  It's "place a message in a pool" and all messages are placed in one and only one pool and by default, that pool is the server-local public pool.  ESME is opt-in.

A user has a relationship with a pool.  That relationship is read/read-write/administer (which implies read-write).  

So, how do you get a message into a pool?  You will define your default pool.  This is the pool that your messages get put into unless you specify otherwise.  This means that the CEO can choose to put things in the "c-level" pool.  Most people will post to the public pool by default.  

If a pool is deleted, the messages in the users’ timeline stay, but it is as if all the users were deleted from the pool.

A message may only be in one pool.  There is no way for a message to escape the pool (eg. resend cannot change the pool) and any replies are in the pool of the original message (this is for performance and security purposes.)

We are using groups and pools to mean something different than people are used to.  ESME is a different medium than people are used to.  That gives ESME its power.  ESME is powerful because it is a dynamic, opt-in, social medium rather than a point-to-point communications medium. There are different concepts in ESME than in point-to-point mediums.  Let's do the extra work now to make sure we understand those differences and celebrate those differences and get others excited about those differences so that ESME can thrive for what it is... a social tool for social animals.

Tuesday January 18, 2011

Use Apache Ant to post to Apache ESME

Apache ESME has an integration with Apache Ant. It is now possible to send messages to Apache ESME that describe the progress of your build steps. This is especially useful when Apache ESME is being used in development projects where individual developers can be made aware of the status of various-build related actions. For example, a development team could be informed that a build is broken or if it is successful.  


  1. Install Apache Ant
  2. Install Apache Maven
  3. Download the Apache Commons packages "Codec" and Logger. We also need Apache "Httpclient" . Please note that we are currently using the legacy 3.1 version of the HTTP Client. 
  4. Check-out the ESME java client: https://svn.apache.org/repos/asf/esme/trunk/esme-java-client
  5. Perform a maven build in this directory with the following command "mvn compile package -Dmaven.test.skip=true" . This will create a jar file "esme-rest-api-0.1.jar" in the target directory. 
  6. Create lib directory and copy "rest-api-0.1-SNAPSHOT.jar", "commons-codec-1.2.jar", "commons-httpclient-3.1.jar" and "commons-logging-1.1.1.jar" into this directory. 
  7. Create src directory and copy the "EsmeAntTask.java" file to this directory
  8. Change the build.xml file to point to the appropriate directories
  9. Change the token in the "esme" target in the build.xml file to a correct token.
  10. Change the proxy in the "esme" target in the build.xml file to a correct proxy.
  11. call ant dist esme to test the application

Here is what the result looks like:

ant blog.jpg

Note: We wrote a version of this blog two years ago on our old blog. I've updated it and added an image plus more details. 


<project name="EsmeViaAnt" default="dist" basedir=".">
  <!-- set global properties for this build -->
  <property name="src" location="src"/>
  <property name="build" location="build"/>
  <property name="dist"  location="dist"/>

    <path id="project.class.path">
    <pathelement path="./dist/lib/EsmeViaAnt.jar:./lib/esme-rest-api-0.1.jar:./lib/commons-codec-1.4.jar:./lib/commons-logging-1.1.1.jar:./lib/commons-httpclient-3.1.jar"/>

  <target name="init">
    <!-- Create the time stamp -->
    <!-- Create the build directory structure used by compile -->
    <mkdir dir="${build}"/>

  <target name="compile" depends="init"
        description="compile the source " >
    <!-- Compile the java code from ${src} into ${build} -->
    <javac srcdir="${src}" destdir="${build}">
       <classpath refid="project.class.path"/>

  <target name="dist" depends="compile"
        description="generate the distribution" >
    <!-- Create the distribution directory -->
    <mkdir dir="${dist}/lib"/>

    <!-- Put everything in ${build} into the jar file -->
    <jar jarfile="${dist}/lib/EsmeViaAnt.jar" basedir="${build}"/>

  <target name="esme"
        description="Send Esme" >
        <java fork="true"
         <arg value="http://esmecloudserverapache.dickhirsch.staxapps.net/api"/>
         <arg value="31EL0R0M15NTD2LSOS0BKC5Y0P5JOVAZ6"/>
         <arg value="myproxy:81"/>
         <arg value="A message from the ant build process"/>
           <pathelement path="./dist/lib/EsmeViaAnt.jar:./lib/esme-rest-api-0.1.jar:./lib/commons-codec-1.4.jar:./lib/commons-logging-1.1.1.jar:./lib/commons-httpclient-3.1.jar"/>



EsmeAntTask java class

import org.apache.esme.api.EsmeRestApi;
import org.apache.esme.model.Message;

public class EsmeAntTask

public static void main (String[] args) {

 String apiUrl;
 String authToken;
 String localProxy;
 String message;
 EsmeRestApi esme;

 apiUrl = args[0];
 authToken = args[1];
 localProxy = args[2];
 message = args[3];

 try {

         esme = new EsmeRestApi(apiUrl);

         if ((localProxy != null) && !("".equals(localProxy)))
                // Split proxyHost:port into two parts
                String[] proxy = localProxy.split(":");
                esme.setProxy(proxy[0], Integer.parseInt(proxy[1]));


         Message esmeMsg = new Message();


         String[] tags = new String[1];
         tags[0] = "Ant Task";



 catch (Exception e) {}



Wednesday January 12, 2011

Using SBT in Apache ESME


While many projects have been using Apache Maven as widely adopted and recognized project management tool, Scala-based Simple Build Tool (SBT) became very popular dependency management tool recently. SBT has following advantages:

  • Buld file is written in Scala language which is much more concise comparing to verbose XML and provides full power of Scala platform and libraries

  • SBT intelligently tracks changes in source code to make accurate recompilation

  • SBT console mode keeps scalac in resident, which really improves compile times on subsequent runs. This is important for scalac, which is quite slow as compared to javac

  • SBT supports continious compilation and testing

  • While SBT is based on Apache Ivy, also very popular dependency management tool, it is possible to use both remote and local Maven repository with SBT

  • SBT has Custom Actions

That said SBT integration with popular IDEs and CI tools is not as good as Maven integration yet (I'll show an example of SBT and IDEA integration and give links to resources related to Hudson integration).

It is possible to build ESME project both with Maven and SBT. At the moment Maven is main build tool. SBT might be used locally by developers to improve performance. Subject of this article is building project with SBT.

Installation and configuration

1. Download SBT jar file and place it into local directory
2. Create script to launch SBT and specify path to SBT jar. For example, below is a script for Linux platform:

java -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m -Xmx512M -Xss2M -jar `dirname $0`/sbt-launch-0.7.4.jar "$@"

*JVM options above were added to launch script to avoid frequent OutOfMemory errors.

Now it is possible to launch SBT. But before proceeding to project structure, it would be useful to know how SBT manages its dependencies.

Dependency management

As noted before, SBT is based on Apache Ivy — popular dependency management tool. It means that all dependencies will be downloaded to local repository (located by default in $USER_HOME/.ivy2 folder) and orginized in a manner similar to Maven: organization/module/artifact/revision, where organization and revision are analogs to Maven's groupId and version accordingly. All automatically managed dependencies, specified in project definition file, are placed into PROJECT_ROOT/lib_managed directory.

Project structure

Overall project structure has following form:

--->ESME root
| |
| -----EsmeProject.scala
| |
| -----Plugins.scala
----- build.properties

SBT build artifacts have only been added to ESME server module, therefore they are located under server/project folder. These artifacts include:
ESMEProject.scala — project definition file containing build configuration
Plugins.scala — plugins for project definition are declared in this file
build.properties — contains project name, version, organization, Scala and SBT versions and other custom properties

The most important file among these is project file, so lets review it step-by-step.

Project File

Project definition class for web project EsmeProject should extend DefaultWebProject (which in turn implements base class for all types of projects sbt.Project):
class EsmeProject(info: ProjectInfo) extends DefaultWebProject(info)

Dependency versions are defined as vals:
val liftVersion = "2.2-RC6"
val compassVersion = "2.1.1"
val luceneVersion = "2.4.0"

It is possible to tell SBT to search dependencies in local Maven repository. To do that it's neccessary to define mavenLocal val and assign it to the path in the local filesystem where Maven repository resides:
val mavenLocal = "Local Maven Repository" at "file://"+Path.userHome+"/.m2/repository"

All additional repositories are also defined as vals. To include Scala Tools Snaphots repository predefined constant ScalaToolsSnapshots is used, for other repos explicit URL is specified:
val scalatoolsSnapshot = ScalaToolsSnapshots
val compassRepo = "Compass Repository" at "http://repo.compass-project.org"
val twitterRepo = "Twitter Repository" at "http://maven.twttr.com"

Project might contain additional files like licenses and notices. To include them in target jar it is neccessary to define method extraResources and override mainResources method, as shown below (assuming that both files are located in project's root directory) :
def extraResources = "LICENSE" +++ "NOTICE"
override def mainResources = super.mainResources +++ extraResources

It's possible to specify dependency definition with Ivy configuration file inline (for example to include or exclude dependent modules). To do that, it is neccessary to override ivyXML method:
override def ivyXML =
<dependency org="net.lag" name="configgy" rev="2.0.1">
<exclude org="org.scala-tools" module="vscaladoc"/>
<dependency org="com.twitter" name="ostrich" rev="2.3.2">
<exclude org="org.scala-tools" module="vscaladoc"/>

SBT manages dependencies based on dependency expressions in project's definition file. Dependency expressions have following form:
groupID % artifactID % revision % configuration

In case dependency was build with SBT, it is neccessary to change first % to %%:
groupID %% artifactID % revision % configuration

That way SBT knows how to download correct jar corresponding to Scala version which is used to build project (for example lift-webkit_2.8.1-2.2-RC6.jar).

It is possible to use range to specify version like in Maven. For example, value [6.1.6,) corresponds to all versions greater or equal to 6.1.6.

Configuration has form A->B, where configuration A use a dependencys configuration B. ESME project definition contains compile and test configurations. As an example, expression "junit" % "junit" % "4.4" % "test->default" says that ESME test configuration uses JUnit default configuration.

If no configuration mapping is specified explicitly, compile->default mapping is used by default.

One way to list all dependencies is to define libraryDependencies method containing Set of dependency expressions:
override def libraryDependencies = Set(
"net.liftweb" %% "lift-webkit" % liftVersion % "compile->default",
"net.liftweb" %% "lift-mapper" % liftVersion % "compile->default",
"org.compass-project" % "compass" % compassVersion % "compile->default",
"org.apache.lucene" % "lucene-core" % luceneVersion % "compile->default",
"org.mortbay.jetty" % "jetty" % "[6.1.6,)" % "test->default",
"junit" % "junit" % "4.4" % "test->default",
) ++ super.libraryDependencies

Build properties

SBT loader reads the versions of Scala and sbt to use to build the project from the project/build.properties file. If this is the first time the project has been built with those versions, the loader downloads the appropriate versions to project/boot directory. The sbt loader will then load the right version of the main sbt builder using the right version of Scala. Other user-defined properties might also be set in this file. Below is an example of build properties file for ESME:

project.organization=Apache Software Foundation
project.name=Apache Enterprise Social Messaging Environment (ESME)


All plugins for SBT are specified in plugins definition file Plugins.scala. It's structure is very similar to project definition file. Lets review plugins configuration via example of sbt-idea plugin which is used to generate project artifacts for IntelliJ IDEA.

Plugins class should extend PluginDefinition base class:
class Plugins(info : ProjectInfo) extends PluginDefinition(info)

Additional repositories which are used by plugins are declared as vals:
val sbtIdeaRepo = "sbt-idea-repo" at "http://mpeltonen.github.com/maven/"

Like in a project definition file, dependencies for plugins are specified via dependency expressions:
val sbtIdea = "com.github.mpeltonen" % "sbt-idea-plugin" % "0.1.0"

With sbt-idea plugin configured, it is possible to issue commands:
sbt update
sbt idea

and IDEA project file will be generated in project's root directory.

Main commands

All interaction with SBT is performed via commands, which are usually executed in SBT console. To enter into SBT console, it's necessary to run sbt in a project directory.

First group of commands is used to clean generated and downloaded artifacts.
clean command deletes target directory where all generated files are located.
clean-cache command deletes local Ivy repository where downloaded artifacts and metadata for automatically managed dependencies for this user are resides.
clean-lib command deletes lib_managed - managed library directory for this project.

update command is used to resolve and download all external dependencies into local Ivy repository.

compile command preforms compilation of all Scala source files located in src/main/scala folder. It's possible to specify additional options for compile task by overriding compileOptions method.

test command executes all tests that have been found during compilation. Runs test-compile command first (similar to compile task it's possible to specify additional options for test-compile task by overriding method testCompileOptions).

jetty-run command starts the Jetty server and serves this project as a web application on http://localhost:8080 by default. This variant of starting Jetty is intended to be run from the interactive prompt.
jetty-stop command stops the Jetty server that was started with the jetty-run action.

Dependencies list

Sometimes it's necessary to analyze all tree of used dependencies (to prevent conflicts between them for example). Maven has specific plugin dependency:tree to do just that. It's also possible to perform similar task with SBT via Project Console.

console-project command starts the Scala interpreter with access to project and to sbt.

For example, to get list of dependencies in compile classpath, current.publicClasspath command should be executed. Similar commands current.testClasspath and current.optionalClasspath exist for test classpath and optional classpath accordingly.

Integration with Hudson

While integration of SBT with Hudson, popular Continious Integration tool, hasn't been used in ESME project yet, there are some resources on this available:
Hudson SBT plugin: https://github.com/hudson/sbt-plugin
Christoph Henkelmann’s Blog entry - “How to build sbt projects with Hudson and integrate test results” : http://henkelmann.eu/2010/11/14/sbt_hudson_with_test_integration


Simple Build Tool home: http://code.google.com/p/simple-build-tool/
Apache Ivy home: http://ant.apache.org/ivy/
Lift Wiki page related to SBT: http://www.assembla.com/wiki/show/liftweb/Using_SBT
Mikko Peltonen's sbt-idea plugin home: https://github.com/mpeltonen/sbt-idea

A big thank you goes out to Vladimir Ivanov for the SBT implementation and for writing up this blog post explaining how it works!



Hot Blogs (today's hits)

Tag Cloud