In case you haven't noticed, distributed computing is hard.

The problem is that it is becoming increasingly important in the world of enterprise application development. Today, developers continuously need to address questions like: How do you enhance scalability by scaling the application beyond a single node? How can you guarantee high-availability, eliminate single points of failure, and make sure that you meet your customer SLAs? All questions that, in one way or the other, imply distributed computing.

In this article, Jonas Bonér walks through a fairly generic but common distributed computing problem and shows how it can be simplified using clustering at the JVM level.

The main benefit of using clustering is a simplified programming model. The way I see it, clustering, and distribution in general, is something that should be transparent to the application developer. It is clearly a cross-cutting concern that should be orthogonal to and layered upon the application, a service that belongs in the runtime. In other words, what we ultimately need is clustering at the JVM level.

Read Distributed Computing Made Easy.

This do-it-yourself approach to essentially grid computing scenario is fine for a small example and trivial use cases. I however wonder by how many hundreds of lines of code this example would grow when you account for at least some of the following considerations: - failure of remote nodes (system or application level) and failover of the original task - security (node-level authentication, firewalls, global vs. local distribution) - matching each available resource (node and its characteristics) with a specific split of the original task to get the optimal distribution, and thus having a “intelligent” split - collision resolution (more than one task competing for the same node) - deployment questions (new code is one node, old code is on other nodes) The complexity of grid computing is not in a simple split/aggregate scenario (which can be mimicked pretty easily indeed) but in all surrounding functionality that is ultimately needed. And the real art is to make this surrounding stuff as simple as possible as to not take away from overall simplicity of the basic idea behind grid computing. I mean, just look at Globus to see what happens when simplicity is regarded as a foul word...

I have pointed out before that lack of interfaces for you to latch on to doesn't mean TC has no API. Annotations and zero-code and declarative programming != no API. It does == no source code changes. While I see that this sort of thing can sound magical, it is not. It can be highly advantageous in that pretty much your entire list of questions are actually runtime problems that do not change the application logic, so why code differently? In fact,several of these things are features that exist for us or are on our roadmap. And to further the point Tangosol (if you don't know it is the leading API-based clustered cache) doesn't ask you to code to a different API in order to get at said features--its the same Tangosol from a developer perspective in or out of the presence of WAN-clustering.

- failure of remote nodes (system or application level) and failover of the original task
- security (node-level authentication, firewalls, global vs. local distribution)
- matching each available resource (node and its characteristics) with a specific split of the original task to get the optimal distribution, and thus having a “intelligent” split
- collision resolution (more than one task competing for the same node)
- deployment questions (new code is one node, old code is on other nodes)

1. 1. Failure of remote nodes - we have a feature that gracefully handles object / memory rollback and reasignment of workload, today
2. 2. Security, firewalls, WAN /LAN should arguably be handled in the box, not in the app. I would like to say, for example that changes to "session" objects are not synchronously updated on the WAN while changes to "customer" are.
3. 3. Matching each available resource with a specific split - this is a feature of your app, more so than the cluster. You would have to ID a node in a system.property and then ask it to only work on work of class ID, for example. This works with or without an API, no?
4. 4. Actually, versioning is a very hard problem that is best left for a separate topic--it has to do with serialization and since we use none, we actually allow versions to co-exist. Someone will have to do something in source code only when fields change type, and someone will have to tell us the default value for new fields on existing objects, but we gracefully handle added or deleted fields today. Same story as Oracle versioning which has existed since 9i.

This is just terracotta all over again. TC is cool. I could definitely use it successfully in many applications. But it has two main problems: 1. It falls into the old of trap of convincing developers that code can run anywhere and it's not going to make any difference. 2. The lack of an API means that the concurrency control is restricted to JVM mutexes. A mutex is not a read-write lock, for example. If you want to build a distributed jvm, to me it only really makes sense to do the communications over RDMA - provided by infiniband among others. Then I can start making an argument that my code really can run anywhere, because the latency is microseconds and the bandwidth is huge. For example the Pathscale adapter was measure to have 1.3us latency. But short of that I don't find a jvm distributed over kernel-space communications particularly exciting. I can easily come up with workload which it can only run inefficiently. So short of using RDMA I think the solution which resonates well with most shops is a transactional api like JTA and distributed caching and concurrency control. Guglielmo Enjoy the Fastest Known Reliable Multicast Protocol with Total Ordering .. or the World's First Pure-Java Terminal Driver

This is just terracotta all over again.

TC is cool. I could definitely use it successfully in many applications. But it has two main problems:

1. It falls into the old of trap of convincing developers that code can run anywhere and it's not going to make any difference.

Right, the same distributed computing fallacy that was pointed by SUN fellows like several decades ago…

2. The lack of an API means that the concurrency control is restricted to JVM mutexes. A mutex is not a read-write lock, for example.

If you want to build a distributed jvm, to me it only really makes sense to do the communications over RDMA - provided by infiniband among others.

… or something like Azul. Regards, Nikita Ivnaov.

The lack of an API means that the concurrency control is restricted to JVM mutexes. A mutex is not a read-write lock, for example.

If you want to build a distributed jvm, to me it only really makes sense to do the communications over RDMA - provided by infiniband among others.

… or something like Azul. I don't know anything about their architecture, except that it's designed to run jvms. Guglielmo

Guglielmo, do you know of any Java API for Infiniband ? Would be nice to support this in JGroups, as a transport. I actually visited Pathscale, their latency is incredibly low, they're speaking nanosecs/microsecs, and not millisecs... :-) Bela

I've stared at IB also but Java serialization costs would seem to me to outweigh the improvements. I don't think you'd see much difference over gig-E. You'd need a very lightweight layer between IB and the application to reap the benefits. Pathscale was around the 7us wasn't it?

I've stared at IB also but Java serialization costs would seem to me to outweigh the improvements. I don't think you'd see much difference over gig-E. You'd need a very lightweight layer between IB and the application to reap the benefits.

Pathscale was around the 7us wasn't it?

I saw latency numbers at around 1.5-3 microsecs, depending on the distance between boxes. Yes, Java serialization would definitely be a factor, so you'd essentially have to get a buffer from the user, and avoid copying all the way down into the network card's buffer. At one point, the buffer has to cross over into the C heap (typically with JNI, but I understand there are other, faster options), so there's a performance penalty, too. I don't know how approaches such as R/DMA can solve that problem, and I haven't seen numbers on how much faster they'd be, compared to regular JNI. Googling this topic, I understand SUN once looked at providing RDMA (or SDP?) directly in the VM, but dropped it because of lack of interest. Damn, they didn't ask *me* ! :-)

I've stared at IB also but Java serialization costs would seem to me to outweigh the improvements. I don't think you'd see much difference over gig-E. You'd need a very lightweight layer between IB and the application to reap the benefits.

Pathscale was around the 7us wasn't it?

RDMA to me is one of those technologies that requires a different application design, namely NUMA as opposed to message-passing. If I were TC I would re-design their implementation by assuming an abstract NUMA model, and then get into the business of re-selling IB-enabled blades with the jvm already on them, because people don't have the expertise to do this themselves usually. Many vendors do not advertise their support for IB. The only circle where IB is very well known and there is expertise is compute clusters. There is a business opportunity here for an integrator to market IB-based clusters for OLTP. Oracle supports it, but again it's hard to find DBAs that know this stuff. Guglielmo Enjoy the Fastest Known Reliable Multicast Protocol with Total Ordering .. or the World's First Pure-Java Terminal Driver

Guglielmo,

do you know of any Java API for Infiniband ?

No. Take a look at UDAPL. Guglielmo Enjoy the Fastest Known Reliable Multicast Protocol with Total Ordering .. or the World's First Pure-Java Terminal Driver

As I understand it (and seeing as I am partnered with Azul, I would hope I know at least a little about them), Azul is internally fault tolerant and would run this workload on 1 JVM, but it would need you to change Jonas's app to leverage 1 JVM via multi-threading. What Jonas achieved in the sample in his article is to have built a clustered manager / worker pattern w/o hand-coded threading, etc. This is the BEAUTY OF SPRING! Very powerful stuff when you consider that in this example they not only delivered context to the running app via dependency injection but, in conjunction with VM-level clustering, delivered a scalable way to get more work done on commodity infrastructure and with simple code. That's most likely the key take away here. --Ari

The articles is "A Note on Distributed Computing" at: http://research.sun.com/technical-reports/1994/abstract-29.html Highly recommended... -- Andre

A mutex is not a read-write lock, for example.

If you actually download the docs, you will find that TC has MANY locking semantics that can be configured outside code. At least grab the docs and page through those. --Ari

A mutex is not a read-write lock, for example.

If you actually download the docs, you will find that TC has MANY locking semantics that can be configured outside code.

At least grab the docs and page through those.

--Ari

You mean like a deployment descriptor? Do I have to register to read the docs? Guglielmo Enjoy the Fastest Known Reliable Multicast Protocol with Total Ordering .. or the World's First Pure-Java Terminal Driver

Do I have to register to read the docs?

http://terracottatech.com/documentation.shtml

Thanks. Just to make it easier for people: http://terracottatech.com/product-docs/product-guide/ProductGuide-1.5-1_Revised_HTMLFormat.1.53.html http://terracottatech.com/product-docs/product-guide/ProductGuide-1.5-1_Revised_HTMLFormat.1.54.html http://terracottatech.com/product-docs/product-guide/ProductGuide-1.5-1_Revised_HTMLFormat.1.67.html#10001003 It's good that TC does indeed have support for several different kinds of locks. But I think that these are not sufficiently advertised. They only talk about synchronized, which is a mutex. Also, it appears (third link) that the locking behavior has to be placed in a configuration file (because there is "no api" there is no place to put it.) Can you use annotations instead? Guglielmo

Thanks. Just to make it easier for people:

http://terracottatech.com/product-docs/product-guide/ProductGuide-1.5-1_Revised_HTMLFormat.1.53.html

http://terracottatech.com/product-docs/product-guide/ProductGuide-1.5-1_Revised_HTMLFormat.1.54.html

http://terracottatech.com/product-docs/product-guide/ProductGuide-1.5-1_Revised_HTMLFormat.1.67.html#10001003

It's good that TC does indeed have support for several different kinds of locks. But I think that these are not sufficiently advertised. They only talk about synchronized, which is a mutex.

Also, it appears (third link) that the locking behavior has to be placed in a configuration file (because there is "no api" there is no place to put it.) Can you use annotations instead?

Guglielmo

That is something that we have discussed. Annotations for defining shared roots (fields that holds the object graph you want to share) - @Clusterable or similar, as well as locks - @Lock(Policy.WRITE) or similar. Would not be transparent, but would indeed be very useful in certain situations (could also help in providing more generic tool and IDE support etc.).

That is something that we have discussed. Annotations for defining shared roots (fields that holds the object graph you want to share) - @Clusterable or similar, as well as locks - @Lock(Policy.WRITE) or similar. Would not be transparent, but would indeed be very useful in certain situations (could also help in providing more generic tool and IDE support etc.).

The problem here is that your promise of "no api" is not feasible when you go outside the semantics of the language. If you are only going to support the synchronized semantics (exclusive lock) then you can have no api. But if you go beyond that (and my point was that people in general want to do that) then you are forced into such solutions as: 1. Configuration file 2. Annotations 3. API I think either 2 or 3 are much better than 1, even if only because separating concurrency control characteristics into a separate file makes the code harder to read. Guglielmo Enjoy the Fastest Known Reliable Multicast Protocol with Total Ordering .. or the World's First Pure-Java Terminal Driver

.... The way I see it, clustering, and distribution in general, is something that should be transparent to the application developer. It is clearly a cross-cutting concern that should be orthogonal to and layered upon the application, a service that belongs in the runtime.

As others in this thread have mentioned, I think this is a fallacy. Some applications are simpler not designed to be clustered. You can't just add some aspects, throw it on a cluster, and hey presto you have a nice scaleable application. Amdahl's law shows that only the parallelizable portion of a workload can be scaled, any sequential work won't. For embarrassingly parallel problems this may work, but in the general case you can't avoid redesigning your application to scale on a cluster.

Excellent article, Jonas - I obviously appreciate the JavaSpaces/Tuple-spaces focus here (being as I am with GigaSpaces). Hey - if you are going to talk about JavaSpaces/TupleSpaces, Master-Worker and Spring/POJO integration, you should probably also let folks know that such a product exists today and is being used successfully by hundreds of customers: GigaSpaces EE (www.gigaspaces.com). In addition to the technical aspects, the beauty of JavaSpaces is that it is a standard and is one of the earliest technologies to come out of "JavaSoft" (and is almost celebrating its 10 year anniversary) - so it's a very mature platform that have proven itself consistently. Gad Barnea GigaSpaces Technologies, Inc. http://www.gigaspaces.com

Over the last years i designed 2 distinct distributed systems. I presented both at JavaOne (2001 / 2005). One systems does heavy XML stuff on distributed processors (cluster size ranges from 1 to 20 cores). The other one is a Cluster of >250 machines doing risk-calculations on data warehouse databases. The problems described in this article are just the most obvious problems which come to the surface just before you dive into coding. As soon as you scale, it gets really interesting. - how to control memory usage on each VM - what happens if every VM suddenly starts initialising data from Databases - what happens if one VM-Worker Thread just enters an endless loop or waits for some resource which it never gets - what happens if a master VM dies - for whatever reason ? - .... this list could go on forever... I think it is rather useless to state "product X is suitable for distributing (any) workload transparently". Even when most Grid/Cluster-software vendors state exactly that in their whitepapers. Even with Jini, JGroups, .... it is rather trivial to perform quasi-automatic workload distribution. The real problems come later - failure recovery, stateful against stateless workers, keeping the clusters resources in sync (local repositories against one or more central repositories), how to recover from a real catastrophe (like a reboot of the cluster) without loosing jobs ... and so on Terracotta seems to be a pretty sexy approach - but this article misses to much detail in what it can do about the infrastructure around a cluster - or how terracotta can help in some of those topics. I am pretty sure that terracotta might help in sharing resources across VMs - but even then: what about memory, garbage collection,...

All excellent points. I would suggest, especially given your background, that you should download Terracotta Enterprise Edition and try it out. Some of what you question is addressed reasonably well (distributed GC, cluster reboot, some failures are gracefully handled, etc.). It is, at least, a large step forward and I would personally welcome you to open a dialog with me directly [ARI at TERRACOTTATECH.COM], or our product mgmt team to discuss any insights you may have after building a [sample] app. But, without citing proof, I realize, I still want to state that Terracotta is not just about sharing memory and leaving "hard problems" around failure and resource management to API-based solutions. It is quite the opposite. You might be surprised when you download it and roll up your sleeves. --Ari

Is it possible to get the sample code for the Terracotta for Spring article? Thanks, Ted

Is it possible to get the sample code for the Terracotta for Spring article?

Thanks,

Ted

Hi Ted. All code needed is actually already in the article. I can wrap it up as an "official" demo and make it part of the next release. There are some similar demos in the current release, please sign up for the beta program if you are interested in trying it out. Thanks for your interest. /Jonas