In this interview Dean Jacobs, Architect at BEA Systems, looks at stateless and stateful services in a clustered environment. He discusses entity bean caching in Weblogic, clustering techniques using JMS, and failover with cluster-aware EJBHome stubs and EJBObjects. He examines in-memory replication, singleton services, and different types of clustering topologies.
Watch Dean Jabob's Interview
I've just had a recent experience with BEA clustering on 7.0.2 and did a few performance tests. I have one question. How many boxes does it take for a cluster to be more performant (in terms of throughput, simultaneous users and transactions/second) than any of the boxes that makes the cluster ? I ask this question because on the tests I conducted with someone from BEA over a few days, performance on a cluster of 2 servers was significantly lower than using one WebLogic instance on one server. With session replication on, it was even worse and very unstable (crashed after a few minutes). I understand 2 servers might not have been enough. Yet I was extremely surprised as WebLogic clustering has been known to be very good. That was my impression until I actually tested it.
Another thing that I found disappointing is the time it takes to start a cluster. It is extremely slow (> 5 minutes per instance) which makes tuning very difficult. And tuning a cluster is much harder than tuning a single instance... Perhaps this issue has been fixed in 8.1 (as I noticed that deployment on 8.1 is now very fast compared to 7.0.x) but I did not have time to try 8.1 clusters yet.
My opinion at the moment is that using a cluster for performance is a significant investment, and needs to be thoroughly planned and double-checked with a performance tool, especially if you want high availability and failover. For EJB applications using entity beans, I imagine the overhead can be even bigger. Software load-balancing should also not be used (apart from round-robin) as it does not seem to work as expected (in my own experience).
Finally, the biggest surprise came from the overhead in terms of performance entailed by the Apache 2.0-WebLogic plug-in. We hit a low performance ceiling that could not be exceeded...
This is my own experience with WebLogic 7.0 clusters in a proper testing environment (perhaps we did some things that were wrong, or lacked time to do further tuning)... but it really makes we wonder whether the whole vertical approach recommended by Dean Jacobs (and BEA in general) with multiple clusters for one application is genuinely useful, reasonable and reliable. Has anyone ever used this configuration in production ? If so, what were your impressions with it ?
I'm not bashing, just wondering...
I haven't worked with WL clusters but JBoss.
And two JBoss instances are definitely faster than one (I can send you test results) as it should be. I used SLSB as they can be clustered and scaled very well. Currently I am working on neuron network based algorithm which learns on the fly which instance is faster and sends to it more requests.
If two WL's appears to be slower than one, than I think there is something screwed up very much.
I would be very interested in your test results if you have such a document (caroffy at hotmail dot com).
It would be normal for a cluster of 2 sitting on one server to be slower with the clustering overhead but on 2 different machines, I find it a bit odd. Session replication uses a lot of network resources it seems but I fail to understand how it can be slower without it... Bad tuning perhaps...
How long does it take on JBoss to start a cluster ? Can you add and remove instances on the fly ?
How long does it take on JBoss to start a cluster ?
I have a cluster with two machines called november and legion. JBoss with cluster service starts in 50 seconds. Also you dont need to spend a lot of time setting up a cluster. Just start JBoss with full set of services (run.bat -c all). It joins cluster partition with default name.
>Can you add and
>remove instances on the fly ?
Yes, easy. I just have to copy jar to /farm directory . If I delete a jar then it is undeployed on all cluster nodes. Oh, and server nodes can be added and removed on the fly, too.
Regarding my results... My SLSB calculates Carmichael numbers. It can find all numbers in range 0-3000000 in:
150 seconds (one node only, november )
107 seconds (one node only, legion )
65 seconds (clustered on both machines)
As you see clustered EJB runs a lot faster than non-clustered.
JBoss with cluster service starts in 50 seconds.
Reduced startup time comes at the expense of required features. For example: initial pool size > 0 should load Beans during startup. A feature which most applications need and JBoss doesn't provide.
but I fail to understand how it can be slower without it...
One fast node can be faster standalone, then clustered with slow nodes.
In my example I added to cluster my P4 computer (EJB client runs on it)
and it can find all those numbers in 48 seconds. While clustered with legion and novemeber the same job can be done in 53 seconds !
It's because other nodes are very old computers and my P4 has to wait them to finish calculations. In RoundRobin mode (JBoss default) all invocations are distributed across nodes evenly. It's not very optimal in this case.
I have written an algorithm that can solve this problem. Haven't tested in real cluster yet. Now I'm digging JBoss source code.
A lot of enhancements have gone into 8.1, so please give it a try. I find it hard to believe that clusters took so long to startup in 7.0! This has not been my experience.
Haven't tried 8.1 yet, but planning to soon.
My experience with WLS clustering goes back to 5.1, including 7.0, and it has been the most positive one. I have put several high load critical customers to production with it, and have never seen it crash with no apparent reason, or be slow with sufficient resources.
And yes, I did try and have done some work with some other ones. Specifically in the area of clustering and failover, WLS is the best. Forget about jboss, it's a toy. You don't run an airline on jboss :-)
However, WLS does have lots of things to configure and tune, and you got to know what you'r doin'. It can be screwed up, in which case you start writing complaining messages all over... :-)
I should be able to test WebLogic clusters again soon and this time, I will use 8.1.
My experience so far with WebLogic clusters has been pretty good (since 5.1) but to be honest, I never did nor took part to any such thorough benchmark before. Hence my surprise. On the other hand, earlier clusters had less functionality.
Also, we mostly did stress tests (many simultaneous users with no think time) and session replication just killed everything past a certain threshold due to an error which looks like a bug (it has something to do with the replicated session id not being unique, I can't remember precisely).
The time it takes to start a cluster was about 5 minutes per instance (time to detect and synchronise with other instances) on entry and middle-range servers.
I will definitely give 8.1 clusters a try. Hopefully, that should be this month.
I have designed/extended the jboss load balancing platform. I designed and implemented a framework, over JBOSS Clustering framework, that can be used for writing load balancing politicies that select the better Jboss instance for redirecting a request (remote invocation). The criteria for determinating which instance is better is based on run-time information of the instance (virtual machine onto the Jboss is running), such as: Numbers of threads in a wait state, running state, available memory and so on.
These policies can be used in a special applications contexts where round-robin, first available and random aren't better.
for more information about this project please contact me to jo-arias at javeriana dot edu dot co
BEA is not aware of any scalability problems in a cluster. Did you guys approach BEA Support to resolve the problem? During our internal tests we have close to linear scalability with clusters. Also, your claims about session replication are quite surprising since we have customers in production who are using in-memory replication and haven't experienced the same issues. It would be great if could provide us with some details about your application. What version of WLS are you using (there's no such 7.0.2 version), JDK, OS and so forth?
Again it will be useful if you could approach Support with more details about your deployment. For example how many EJBs, JDBC connections, and other services. We did make significant improvements to improve startup time in single servers as well as clusters.
As for Entity beans, have you tried CMP beans and the new features we have added in 7.0?
In regards to the plugin, keep alive is off by default. Have you tried with keep alive turned on?
We have made many improvements in startup time in 8.1 along with new features and load balancing functionality for java clients.
As Dean correctly points out, the concept of a "distributed" singleton is
needed. At least we would need it in our J2EE applications.
I just wonder why it is not supported by the J2EE standard.
Also I could not find a direct support for it in BEA's public API.
Does someone as information how to do it in BEA?
PS: A "distributed" singleton is an object, that exists only once in a J2EE cluster. This is different from a "normal"/"Gang-of-four" singleton that only exists once in a JVM.
we are trying to find out if and roughly how weblogic implements clustering in a web service application.
To be more exact, assume that we have a weblogic web service application developed according to the weblogic documentation and deployed in a weblogic cluster. If this application is using sessions we are trying to
find out if and how are this session information is propagated to the other members of the cluster, so as if one member of the cluster fails,
the other members of the cluster to be able to answer the requests arriving from the requesting application.