Using StarMX to Build Self-Managing Systems

Java Development News:

Using StarMX to Build Self-Managing Systems

By Reza Asadollahi

01 Mar 2009 | TheServerSide.com

New computing systems are becoming more and more complex to be able to address the new business requirements. The ever-increasing level of complexity in these systems poses several challenges in managing them at runtime. To deal with such challenges, applications should be able to manage themselves in accordance with the guidance provided by humans. Self-managing or self-adaptive systems refer to the group of applications that are capable of taking appropriate actions dynamically in response to the changes in their runtime environment.

StarMX is an architectural framework which facilitates self-management characteristics in Java-based enterprise applications. It allows defining management logic programmatically or descriptively, and automates the execution of that logic. Generally, management logic deals with non-functional properties and quality factors like performance, security and availability, and similar to other crosscutting concerns, it must be maintained out of the business logic. It basically defines the conditions to be observed, the actions to be taken, and the relationship between the conditions and actions.

The StarMX framework enables utilizing well-established approaches to build self-managing systems. JMX presents a standard technique for managing different resources in the system and is widely supported by application servers today. There are also many JMX console applications to manage different system's resources. While such console applications allow managing resources manually, StarMX enables different features of the JMX technology for the purpose of automated resource management through combining the JMX facilities with the management logic. Another promising technique is the use of policies or rules to describe management logic. There are several policy/rule engines available, for instance JBoss Drools or Apache Imperius, which allow describing the logic in form of condition-action rules. StarMX provides an easy integration mechanism with these engines via an adapter interface to benefit from their features in policy definition and evaluation. Due to the limitations of policy languages, it is not always possible to describe complicated logic. Hence, to avoid this constraint, StarMX permits the developer to define the management logic programmatically as well.

self-managed application using StarMX

For the framework's detailed documentation and download, refer to the StarMX website at sourceforge.net

Applications

The general application of StarMX is to address self-management concerns from different aspects, which are often called self-* properties, and to make the systems self-managed. Some of the most important properties are:

  • Self-configuring: the ability of a system to adapt itself with the runtime conditions through changing the configuration. It allows adding or removing components or resources while system is working. A good example of this property is hot-deployment facility in application servers.
  • Self-optimizing: the ability of a system to monitor and tune its performance. Auto-updating can also be considered an example of self-optimizing, which aims to improve system behavior.
  • Self-healing: the ability of a system to detect and diagnose the failures and repair them. This property is mainly concerned with service availability.
  • Self-protecting: the ability of a system to detect malicious attacks and to protect the system against them, in order to improve system's security.

Some realistic examples of these applications include:

  • Automate mundane administration/management activities
  • Check availability of different resources or services (like DB, Mail Server, etc.), and take appropriate actions in case of failure (like restarting)
  • Maintain performance under heavy load through disabling some fancy services, and enabling them when the load is normal
  • Detect and apply configuration changes automatically without restarting the system
  • Raise event in certain circumstances (e.g. buffer full or some special exceptions) and take responsive actions
  • Perform cleanup tasks periodically or when is needed

StarMX provides a flexible architecture to construct more complex management models which correlate the monitoring inputs to the corrective actions, instead of simple if-condition-then-action policies. For example a layered model can be created to deal with the runtime issues at different levels, such that the lower layer tries to resolve the problem and if it is not successful it requests the next layer to handle the situation, and so on.

How it works

The following picture shows the runtime model of the framework.

runtime model of the adaptation processes

Management logic is defined as set of entities called managing processes. Each process can be developed as a policy file containing the policy script, or as a Java class by implementing the org.starmx.core.Process interface. In order for each process to perform its job, a set of objects are needed, called anchor objects. Each anchor object can be either an MBean or a POJO. These objects are used to communicate with the underlying resources for monitoring or effecting purposes. In addition to resource instrumentation, they can also provide any other services to the process as a helper object. If the anchor object is an MBean, a proxy object is created by the framework to represent the MBean as a simple Java object, and the process works with the proxy. If it is a POJO, an instance of its class is created by either the framework or a factory method.

The main executable module is called execution chain which is composed of one or more processes, and it has an activation mechanism, as shown in the picture. The activation mechanism defines how the execution chain should be activated for execution. It can be a timer-based mechanism, which defines how often it should be executed, or a JMX Notification-based mechanism, which describes the information of the notification that activates the chain. When the execution chain is activated, its processes are executed in order, provided each process with its required set of anchor objects. If the process is specified as a policy file, the related policy engine will be invoked through its adapter class to execute the policy.

Several execution chains can be defined to address different self-management properties in the target application. All the information of processes, anchor objects, execution chains, and their activation mechanisms are defined in the framework configuration file, starmx.xml.

Key features

StarMX key features include:

  • Enabling different JMX features for automated application management via a simple configuration (no JMX code is required)
    • Using MBeans and MXBeans as anchor objects
    • Using Monitor MBeans to observe some properties and emit notification at certain conditions
    • Supporting variety of mechanisms to access MBeanServers
    • Supporting JMX notification as an execution activator
  • Supporting programmatic and descriptive specification of management logic
  • Enabling integration with different policy/rule engines
  • Supporting timer-based and event-based management logic execution
  • Using POJOs as anchor objects besides MBeans
  • Creating chain of policies or managing processes to be executed in order
  • Using working memories with different scopes for creating stateful policies or exchanging information among policies or managing processes
  • Injecting anchor objects to managing processes using annotations
  • Using a configurable logging facility based on Log4J

Example

The following shows a HelloWorld example to demonstrate some basic features of StarMX, and how it can be used in practice. Assume that in a web-based system, we want to print a "hello world!" message when the load is normal, and to send an alert email to administrator when the load is high.

Anchor objects

We create two classes, one as an MBean and the other as a simple Java object, to print the message, check whether the load is high, and send email. The following shows the code for these two anchor objects.

package test;
public interface HelloWorldMBean {
        
   public void helloWorld();
   public boolean isLoadHigh();
}

----------------------------------------------------
package test;
public class HelloWorld implements HelloWorldMBean {

   public void helloWorld(){
      System.out.println("Hello world!");
   }
        
   public boolean isLoadHigh(){
      // returns true if the load is more than what expected, 
      //         false, otherwise
   }
}
package test;
public class MailUtil {
   public void sendEmail(String to, String subject, String body){
      // send an email
          ...
   }
}
Policy

We develop the management logic in form of two policies using the anchor objects. The policies are described in the SPL language format and evaluated by the Imperius policy engine. StarMX has built-in support for Imperius, and no extra coding effort is required. If a different engine is used, an adapter class must be developed by implementing the org.starmx.policy.PolicyAdapter interface. The code for these policies resides in "helloworld.spl" file.

Import Class test.HelloWorldMBean : helloWorldMB ;
Import Class test.MailUtil : mailSender;

Strategy Execute_All_Applicable;
Policy {
   Condition {  
      ! helloWorldMB.isLoadHigh()
   }
   Decision {   
      helloWorldMB.helloWorld()
   }
}:1;

Policy {
   Condition {  
      helloWorldMB.isLoadHigh()
   }
   Decision {   
      mailSender.sendEmail("admin@website.com", "High low problem", "...")
   }
}:1;
Configuration

The framework configuration is defined in the starmx.xml file. We assume that both the application and StarMX are deployed on the JBoss application server, and HelloWorld MBean is registered to the JBoss' default MBeanServer with "sample:name=HelloWorld" object-name. In the configuration, one execution chain is created including only one policy-based process to execute our management logic. It is also configured to be executed every minute.

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE starmx PUBLIC "-//STAR Lab//StarMX Configuration DTD 1.0//EN" "starmx-1.0.dtd" >
<starmx>
   <!-- The mbeanserver information is needed to create MBeanServerConnection and get access to the MBeans.
       In this example it is created using jndi lookup -->
   <mbeanserver id="ms" lookup-type="jndi" >
      <jndi-param jndi-name="jmx/invoker/RMIAdaptor">
         <property name="java.naming.factory.initial">org.jnp.interfaces.NamingContextFactory</property>
         <property name="java.naming.provider.url">jnp://localhost:1099</property>
         <property name="java.naming.factory.url.pkgs">
           org.jboss.naming:org.jnp.interfaces:org.jboss.naming.client</property>
      </jndi-param>
   </mbeanserver>
   
   <!-- Defining anchor objects. Each mbean has an mbeanserver attribute which references the id 
      of an mbeanserver tag -->
   <mbean id="hwmb" object-name="sample:name=HelloWorld" interface="test.HelloWorldMBean" mbeanserver="ms" />
   <bean id="mailUtil" class="test.MailUtil" />

   <!-- This is the execution chain, which is activated every minute and evaluates the policy -->
   <execute>
      <timer-info interval="1" unit="minute" />

      <!-- The policy is also defined by process tag and contains the list of required anchor objects -->
      <process id="hello_world_policy" policy-type="spl" policy-file="helloworld.spl">
          
         <!-- The id of each anchor object is mapped to its local name in the policy -->
         <object name="helloWorldMB" ref="hwmb" />
         <object name="mailSender" ref="mailUtil" />
      </process>
   </execute>
     
</starmx>

If we want to deploy StarMX on a separate machine and remotely manage the application, we just need to change the mbeanserver configuration properties to use JMX Remote API for creating MBeanServerConnection to JBoss' default MBeanServer.

   <mbeanserver id="ms" lookup-type="jmx" >
      <jmx-param service-url="service:jmx:rmi://<host>/jndi/rmi://<host>:1090/jmxconnector" />
   </mbeanserver>
Implementing the policy as a Java class

An alternative approach is to develop the management logic in Java classes. In this case, we need to implement the org.starmx.core.Process interface. The process instance is instantiated and initialized once and executed for several times. The following implementation shows two different approaches to access anchor objects. The first one is based on dependency injection using @AnchorObject annotation, and the second one is a manual lookup using the getAnchorObject method in ProcessConfig interface.

package sample;

import org.starmx.core.AnchorObject;
import org.starmx.core.ExecutionContext;
import org.starmx.core.Process;
import org.starmx.core.ProcessConfig;
import org.starmx.naming.LookupException;

import test.HelloWorldMBean;
import test.MailUtil;

public class HelloWorldProcess implements Process {

   @AnchorObject
   private HelloWorldMBean helloWorldMB;
   
   private MailUtil mailSender;
   
   public void init(ProcessConfig config) {
      try {
         mailSender = (MailUtil)config.getAnchorObject("mailSender");
      } catch (LookupException e) {
         throw new RuntimeException(e);
      }
   }

   public void execute(ExecutionContext context) {
      if (! helloWorldMB.isLoadHigh()){
         helloWorldMB.helloWorld();
      } else {
         mailSender.sendEmail("admin@website.com", "High low problem", "...");
      }
   }

   public void destroy() {
   }
}

The modified version of starmx.xml to use the Java class looks like:

   ...
   <execute>
      <timer-info interval="1" unit="minute" />

      <process id="hello_world_policy" javaclass="sample.HelloWorldProcess">
         <object name="helloWorldMB" ref="hwmb" />
         <object name="mailSender" ref="mailUtil" />
      </process>
   </execute>
   ...

Running the example

To run this example the following JAR files must be available in classpath:

  • starmx.jar
  • log4j
  • A StAX xml parser, like stax-api.jar and wstx.jar
  • Imperius (or the used policy engine) jar files
  • other classes referenced in starmx.xml

The framework looks for starmx.xml and the policy files in the classpath unless another path is specified by starmx.config.path system property. Moreover, if it is deployed with the target application on the same application server, there should be some code in the application to start and stop the framework properly, as illustrated in the following sample code. It can be done at deploy/undeploy time of the application. To execute StarMX separately, you can use run.bat or run.sh executables.

// starting StarMX
try {
   starmx = StarMXFramework.createInstance();
} catch (StarMXException e) {
   e.printStackTrace();
}
---------------------------------
// stopping StarMX
try {
   starmx.shutdown();
} catch (StarMXException e) {
   e.printStackTrace();
}

Acknowledgement

I would like to thank my colleagues in Software Technologies Applied Research Group at the University of Waterloo who helped me in developing this framework. Particularly, thank to Mazeiar Salehie for his great contribution and to Dr. Tahvildari for her advice and continuous support.

References

  1. StarMX framework, http://sourceforge.net/projects/starmx
  2. Java Management Extension (JMX), http://java.sun.com/jmx
  3. An architectural blueprint for autonomic computing, http://www-01.ibm.com/software/tivoli/autonomic/pdfs/AC_Blueprint_White_Paper_4th.pdf
  4. Apache Imperius, http://incubator.apache.org/imperius
  5. JBoss Drools, http://www.jboss.org/drools