Using Terracotta for Configuration Management

Java Development News:

Using Terracotta for Configuration Management

By JR Boyens

01 May 2008 | TheServerSide.com

Configuration management, for lack of a better term, is a serious problem for a lot of growing businesses. Having only one server and, therefore, one configuration is not really something that requires a lot of management. Once you start to get past 5 – 10 servers with different configurations for each one, it gets to be a little much. Continue on and very quickly configuration gets to be a real problem. Making sure that the right host/app runs the right configuration at the right time in a dynamic environments is tricky at best. After evaluating numerous possible solutions we have finally settled on using Terracotta. We can easily share object graphs for managing our configurations across all of our servers.  This simplifies having each node running many different applications with unique configurations. The purpose of this article is to let you know about our issues, what we initially thought of doing, what we ultimately decided upon, and how we implemented it.

Terracotta, if you're not familiar, is a sort of network shared memory system. It shares object graphs across the cluster, through something called a "shared root." You declare a Java field as a shared root and then Terracotta handles updating all the nodes in the “cluster” with the information as it changes. It avoids excessive overhead by only sending deltas to keep the objects in sync. The Terracotta folks do a great job explaining their technology in their Introduction to Terracotta - and you can see the object graph being declared as a shared root in the configuration later in this article.

Speaking of what this article is about: basically, the Java Properties system is based on maps. You feed the Properties object a key, and get a value out. What we needed to be able to do is specify a "global properties" object that would allow us to update the properties after the application had been deployed, and alter values based on hostnames and server roles.

Getting a value out of the configuration looks something like this:

String myProperty = Config.getInstance().get("hostname", "propertyname"); 

However, this was too restrictive. In a lot of cases, we had JVMs that served specific roles, and the hostname wasn't important - or we had JVMs that needed specific tuning by hostname and the role didn't matter. So we decided to use regex for the "primary" index lookup, so that we could specify "hostname.*" or ".*rolename", or a full, exact match, to pull out the property we were interested in.

Terracotta makes most of the implementation very easy. All we had to do was design the Config class such that we could get properties out and store them, then we tuned Terracotta such that the Terracotta runtime knew that the Config object's map was to be shared across other machines. Then, at runtime, whenever the JVMs accessed the Config object's internal map, Terracotta would keep track of changes and propagate them automatically. Network traffic was very low, as not only are the values very small, but only the changes are sent across the wire. To be the "most current," all the client JVMs would have to do is keep asking the Config object for values, instead of caching them. (They can cache the returned property values, of course... but if you wanted the values to refresh, you'd have to hit the Config object again eventually, so you'd have the cache expire every so often.)

Here's an example Config object for us:

package config;

import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

public class Config {
    private Map<String, Map<String, String>> configuration = new ConcurrentHashMap<String, Map<String, String>>();

    private static Config instance=new Config();

    public static Config getInstance() {
        return instance;
    }

    public String getMatchingPrimaryKey(String regex) {
        if(getInstance().configuration.containsKey(regex)) {
            return regex;
        }
        for(String s : getInstance().configuration.keySet()) {
            if(s.matches(regex)) {
                return s;
            }
        }
        return null;
    }

    public String get(String primary, String secondary) {
        String v = null;
        Map<String, String> m = getInstance().configuration.get(primary);
        if(m == null) {
            // okay, we need to try to fetch based on a regex. Let's iterate through, looking
            // for the first match
            String s = getMatchingPrimaryKey(primary);
            if(s != null) {
                m = getInstance().configuration.get(s);
            }
        }
        if(m != null) {
            v = m.get(secondary);
        }
        return v;
    }

    public void set(String primaryKey, String secondary, String value) {
        Map<String, String> m=getInstance().configuration.get(primaryKey);
        if(m == null) {
            m = new ConcurrentHashMap<String, String>();
            getInstance().configuration.put(primaryKey, m);
        }
        m.put(secondary, value);
    }
}

Note how vanilla the code is. All of the hard work will be managed by the Terracotta configuration file. (Note, also, the getMatchingPrimaryKey() - if you are using a regex, you're far better off calling this early and storing the resulting key.) The <dso> part of the tc-config.xml should contain these sections:

      <instrumented-classes>
         <include>
            <class-expression>config.Config</class-expression>
         </include>
      </instrumented-classes>

      <roots>
      <root>
         <field-name>config.Config.instance</field-name>
      </root>
   </roots>

This means that the config.Config class is to be processed as being distributed by the DSO engine. The roots section explains what instance variables to share across the DSO network, and the locks tells DSO how to synchronize access (and what kind of access to synchronize.) As long as we start the JVM with the DSO bootjars in place, this is all that's necessary to give us our distributed configuration.

Remember the code we used to access the Config? Here're some classes to show you the tests, as you run them along with each other:

package executables;

import config.Config;

public class SetValue {
    public static void main(String[] s) {
        System.out.println(s[0]+","+s[1]+","+s[2]);
        Config.getInstance().set(s[0], s[1], s[2]);
    }
}

package executables;

import config.Config;

public class ReadValue {
    public static void main(String[] s) {
        System.out.println(Config.getInstance().get(s[0], s[1]));
        try {
            Thread.sleep(10000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        System.out.println(Config.getInstance().get(s[0], s[1]));
    }
}

To see these in action, start up three console windows. In the first, crank up the DSO server with "start-tc-server", then in the second, run the ReadValue class with keys you'd like to use:

dso-java -cp . executables.ReadValue myhost myprop

Immediately after starting the ReadValue class, run the SetValue class in the third console:

dso-java -cp . executables.SetValue myhost myprop value1

The ReadValue class will give you a null report immediately (because the value hasn't been set); then, after ten seconds, the "value1" will be dumped. If you re-run the ReadValue, you'll get "value1" still - showing that the configuration is persistent across client JVM runs - and if you rerun SetValue with a different third argument, you'll see that ReadValue updates properly. Easy stuff, really, and it's very convenient.

Biography

JR Boyens is a Senior Developer for Interactions Corporation. Interactions optimizes customer service by integrating human intent recognition into standards based voice platforms delivering significant cost savings and unparalleled caller experience. His previous work included being a contributor to the RIFE web framework (http://www.rifers.org ). He enjoys short walks around indoor pools and talking about himself in the third person. In his free time he solves complex addition problems and mentors mallards on self-actualization with his 1 year old daughter. He can be contacted via email at jboyens@interactions.net.