Breakthrough BigMemory™ for Enterprise Ehcache

BigMemory. That's what Terracotta is calling their solution to the garbage collection (gc) problem.

Terracotta's customers, like any other customer running sizable Java based applications, have been dealing with the fact that when the JVM decides to take out the trash, applications can grind to a hault, unable to respond to incoming requests in an acceptable amount of time. Add on the fact that one can't always predict when garbage collection might take place, the issue of gc can be a serious one.

But Terracotta says they have solved the problem. Well, at least, they've solved the problem for anyone using Terracotta's Ehcache. By simply adding in a few extra lines of XML to the configuration files, the snap-in Ehcache module provides off-heap memory storage, completely bypassing the the standard JVM memory management mechanism. Curiously, the solution is actually 100% Java based, so it's actually using a Java based solution to solve one of Java's biggest problems. 

With BigMemory, the garbage collection process now uses a predictable amount of resources, and does not cause any related performance spikes. As a result, applications can dedicate hundreds of gigs of memory to individual JVMs without having to worry about the gc delays that are commonly associated with huge heap sizes. It's some good news.

So, will it be open sourced? Don't hold your breath. If you want to use BigMemory, you've got to be using Terracotta's Ehcache. But if you are an Ehcache customer, you can start using it right away. And you'll finally be able to start taking advantage of all of that hardware and memory you purchased that until now, was sliced up into a ridiculous number of smaller JVMs as you tried to avoid the gc problem.

Breakthrough BigMemory™ for Enterprise Ehcache

Why You'll Love Terracotta

The Performance Paradox of the JVM: Why More Hardware Means More Failures

Any Terracotta techie could share with us some implementation details so that is easier for us to determine in which contexts this is applicable?

My best blind guess at this point is that since it is pure Java solution (I am assuming RAM as storage), secondary JVM is started with, for example, large number of large byte[] arrays (64MB per array for example), all allocated at JVM startup so there is no need to GC these arrays, and from primary JVM they are able to translate get(ID), put(ID, val) into reads/writes from specific array on secondary JVM followed by serialization/deserialization routines.

This approach seems actually quite interesting since now cache can operate independently from primary JVM, and primary JVM does no pay GC penalty for cache data.

When I sat on the call with Terracotta, I told them that TSS readers would be interested in this, but they'd want the technical details, not just the PR stuff. Clearly I wasn't wrong.

At the time I spoke with them, they hadn't even solidified the actual name BigMemory, and I'm not even sure if they had a policy regarding which parts of their implementation they'd be discussing in public, and which parts they'd be keeping as their special sauce. I'll shake their tree again on this topic and see if they'll be forthcoming on any of the implementation details for us.

"Please shoot me an email and let me know when you'll be blogging on the topic and what you might be covering..."

Here is a link to my first one...

http://dsoguy.blogspot.com/2010/09/little-bit-about-bigmemory-for-ehcache.html

 

Cheers,

Steve

java.nio.ByteBuffer#allocateDirect ?

 

Not sure if direct flag is relevant for this case since in secondary you are allocating in RAM, while direct flag is mostly used if you are doing some kind of IO with high performance guarantees.

In secondary case it is sufficient to allocate in RAM and just keep references to those big regions and GC will not have big problems since you dont have fragmented heap (no huge number of small objects).

Actually you may be right, direct flag seems better in this case the more I think about it.

If direct flag has same predictable behaviour on all supported JVMs and consistent behaviour on every single JVM between runs, then you could probably use single JVM with fixed heap for you application, plus use direct flag to consume remaining RAM on machine since direct allocations do not allocate from JVMs heap.

In case direct is not predictable and consistent, you could fallback to primary/secondary approach without direct flag.

 

With out spilling too much of the beans I can say that:

1) it is 100% pure java tiered cache (onHeap, off Heap, onDisk) and runs on any 1.5 and up JVM (64bit preferred, jrocket, ibm, sun tested) on any operating system that supports those JVMs (also 64 bit preferred).

2) It has been tested to caches over 350G in size with almost no degradation in performance due to size of the off-heap in memory portion of the cache.

3) The off heap portion is pause-less, fully concurrent and scales with CPU.

4) No JVM Spawning

I'm going to blog a bunch more details in the next week or so.

Stay tuned...

 

 

I can't wait to read more about this, so looking forward to read your future blog posts.

In any case this looks like really interesting solution, and it is clear that is solves a problem in a need of a solution.

Please shoot me an email and let me know when you'll be blogging on the topic and what you might be covering...

cmckenzie at techtarget dot com

No :) While I'm fairly sure that for flat caches managing memory off the heap will help a great deal, it won't help for the generic object graph. So while a great idea and a definite performance improvements for large flat caches it will certainly not solve the GC problem. Zing VM is more promising in that respect. 

BTW my first thought would be a byte array + some kind of distributed reference counting for the flat caches. Since Teracotta already does object lazy materialization and distributed GC, it should be quite a logical step for them. Kudos for pulling this off, though I'll save the real congratulations until I see benchmarks and implementation details.

I have seen a very large number of enterprise and other use cases fit precisely the Key,Value based storage i.e. a cache. A large chunk of application data (Book of Record, Authentication, Reference data, User data) maps really well to a <K,V> data store. It is very valuable to not incur the overhead of reference based object store i.e. java heap for such data.

 

In the caching environment; complex task of recognizing the garbage is already taken care by the user, its just the matter of allocating memory when things are put into the cache and releasing them when destroy is called from the cache....It looks like the good old malloc/free calls are made used in underneath the hood, rather than letting the complex GC algos to kick-in...

 

 

In the caching environment; complex task of recognizing the garbage is already taken care by the user, its just the matter of allocating memory when things are put into the cache and releasing them when destroy is called from the cache....

Indeed, that was the same thing I was thinking.

This is not really solving the GC problem is it? It's basically managing memory yourself again, which often was a problem itself and GC was the solution to it.

What happens with items in the cache that nobody has the key for? Won't these simply leak? If there is some sort of algorithm that purges items from the cache, isn't this then GC again?

I definitely see value in some areas of memory that are GC'ed and some areas that are manually managed, but Java RT actually does this too. It's maybe a shame that this mechanism isn't brought over into the mainstream Java language/JDK. Java RT remains a rather obscure implementation really.

Right now I'm not exactly sure how this Terracotta thing works. If it's just a cache, what's all the hype about? Caches have been with us for quite some time. But if it's a transparent cache, where memory allocations done via the new operator end up in the cache, then who's ever going to free this memory???

 

 

 

 

 

 

I just finished reimplementing this in Java for a project here. It's a specific use case, implementing, essentially, Map<Integer, int[]>.

The base line was how this would work with 25M entries. On a 64-Bit VM, even a simple Map<Integer, Integer> is 2Gb with 25M entries.

So, I have redone all that using arrays. Interestingly, I originally used a malloc/free approach for this, but ended up writing a crude GC as it was much faster than my crude malloc/free. My scheme uses a long[] and int[], 200M for the keys, 100M(ish) for the data.

Now, my only complaint is that this structure is not "hidden" from the GC. If a Full GC ever happens, then the large, underlying arrays will be moved to the new heap. It's definitely cheaper to move a couple huge arrays than 50M objects, but if I can actually isolate them somehow to keep them from even being moved, that would be an interesting bonus.

I might look in to that nio direct buffer concept, if that actually isolates the memory in the heap. Are those really pinned in the heap (or perhaps alongside the heap)? Do they affect Xmx size? i.e. if I have a 2G heap and use a direct buffer for 500MB, does that buffer consume 500MB of the 2G heap?

 

I just finished reimplementing this in Java for a project here. It's a specific use case, implementing, essentially, Map.

The base line was how this would work with 25M entries. On a 64-Bit VM, even a simple Map is 2Gb with 25M entries.

So, I have redone all that using arrays. Interestingly, I originally used a malloc/free approach for this, but ended up writing a crude GC as it was much faster than my crude malloc/free. My scheme uses a long[] and int[], 200M for the keys, 100M(ish) for the data.

Now, my only complaint is that this structure is not "hidden" from the GC. If a Full GC ever happens, then the large, underlying arrays will be moved to the new heap. It's definitely cheaper to move a couple huge arrays than 50M objects, but if I can actually isolate them somehow to keep them from even being moved, that would be an interesting bonus.

I might look in to that nio direct buffer concept, if that actually isolates the memory in the heap. Are those really pinned in the heap (or perhaps alongside the heap)? Do they affect Xmx size? i.e. if I have a 2G heap and use a direct buffer for 500MB, does that buffer consume 500MB of the 2G heap?

 

Obviously this BigMemory thing is a win for specific use cases, but in addition to the leak problem you mention above, fragmentation is another problem that has to be addressed.

It's kind of a semantic distinction: BigMemory won't be garbage collected, but it will certainly have to be managed in some way.

Not to mention that not all of an application's data fits the paradigm of Key-Value. This TSS article IMHO way overstates the case. Which is too bad cause it definitely serves a need, it just doesn't serve every need. Otherwise, Java probably would have been implemented that way in the first place.

BTW, it's pretty apparent that they are using the the NIO allocateDirect API from the way the FAQ and docs mention proper setting of -XX:MaxDirectMemorySize.

How is this different from what rest of the industry had for ~ 5 years (starting with Coherence, and then GigaSpaces, Infinispan and GridGain)? More specifically, how is this different from any distributed partitioned cache when data is kept on remote nodes and there's obviously no GC induced by cached data and most of the GC happens on young generations anyways. 

Furthermore, it's all bolted on Ehcache which is a generation behind on features comparing to something like Coherence or even newcomers like Infinispan and GridGain.

Unless I'm missing something - this all smells like a very awkward informercial of repackaging an existing technology (that Terracotta finally developed too) under a new, extremely confusing name.

I would really like to see a real technical article explaining pros and cons - and not a BS on Ari's blog or "secrecy". I mean, c'mon - this is not a high-school anymore :)

Nikita Ivanov

<a href="http://www.gridgain.com">GridGain = Compute + Data + Cloud</a>

"How is this different from what rest of the industry had for ~ 5 years (starting with Coherence, and then GigaSpaces, Infinispan and GridGain)?"

I will give you one explanation how it is different: Coherence & GigaSpaces are proprietary, Infinispan is far far from being production ready (I haven't heard of anyone using it in production), GridGain is, last time I checked, nice little wrapper/API on top ox existing solutions so you can't qualify as cache solution (you had binding for JMS transport, Coherence etc), and you indeed have nice API, no question about it, but you still have no underlying infrastructure that qualifies you as cache, and you have Terracotta as open source commercial grade cache solution.

 

"how is this different from any distributed partitioned cache"

Well if you had read what they had said thus far, it seems they are not using additional JVMs, but do some other tricks. 

 

"which is a generation behind on features comparing to something like Coherence or even newcomers like Infinispan and GridGain"

Coherence is without a doubt the leader, but placing Infinispan and GridGain before Terracotta, when they clearly are not, is baseless and false claim, bordering with silly. I have analyzed all three of those both at source code level and user level (TC, GG, IS), and superiority is on TC's side. If I was tasked to choose best (features and price) Java caching solution for certain business problem, choice would be between TC, Coherence and GS. Majority of people haven't even heard of GG or Infinispan. 

"this all smells like a very awkward informercial of repackaging an existing technology"

Funny that this comes from company which last time I checked had no group and membership protocol of its own, and is selling APIs and wrappers for existing solution.

 

Infinispan is far far from being production ready (I haven't heard of anyone using it in production)

Not true at all.  Some impressively large systems already run on Infinispan and more keep coming online all the time.  We're kept pretty busy assisting with tuning, architectural reviews, migrations, etc.  We're just not as "marketing-focused" as others.  :)

Dude, at least others on this forum have balls to post their real name...

 

Forget about GridGain 3.0 - it's just a bunch of wrappers :) 

 

But to compare Infinispan or even GigaSpaces to EHCache - you really got be bending backwards... I know this product through and through. We've looked at it - and dropped it like a stone after careful analysis (no transactions back then, no cache queries of an kind, no async operations, simplistic expiration only, no eventual consistency, etc, etc.). It was picked up by Terracotta (good for them). We've decided to build our own data grid in GridGain 3.0 - and I'll let others be the judge. 

 

Regards,

Nikita.

If you feel entitled to post in very dismissive tone like

"Furthermore, it's all bolted on Ehcache which is a generation behind on features comparing to something like Coherence or even newcomers like Infinispan and GridGain. Unless I'm missing something - this all smells like a very awkward informercial of repackaging an existing technology (that Terracotta finally developed too) under a new, extremely confusing name."

I feel entitled by same principle to counterargue and present my opinion.

Moreover, you mentioned competing products first because up to that point noone was mentioning other alternatives.

"We've decided to build our own data grid in GridGain 3.0 - and I'll let others be the judge."

Yes, let others be the judge and that's what we have been doing in this thread before you came. We were having nice talk about how it could potentially work, and they said they would post more details so we can judge and form opinions on our own, and then you came sir and flagged all the work they been doing as bull#$@$.

Those who want respect give respect to others, and you sir showed no respect to Terracotta who in this thread did not bash any other product, just said what they have to say about their own product.

Mr. Thrall,

I will give you one explanation how it is different: Coherence & GigaSpaces are proprietary ..

As is this Terracotta solution. What's your point?

For reference:

The Ehcache Enterprise download kit contains a trial license key which must be added to activate the off-heap store.

There's nothing wrong with owning something that you build. There's also nothing wrong with choosing to give it away. Terracotta does some of each.

BTW Greg, this looks like a nice solution you've built. If you're at JavaOne, let's try to meet up.

Peace,

Cameron Purdy | Oracle Coherence

http://coherence.oracle.com/

"As is this Terracotta solution. What's your point?...trial licence key"

I assumed that they are doing some kind of dual mode licensing (something like GPL and commercial license), and if this is not the case, then it is error on my part.

 

newcomers like Infinispan and GridGain.

Infinispan isn't a newcomer at all. It's the new name of JBoss Cache ;)

We have just released the technical documentation for Ehcache BigMemory which explains the details: ehcache.org/documentation/offheap_store.html

I repost my reply from InfoQ: if you guys think that you are alone (or even first) doing this GC optimization - you are into big surprise :)

Best,

Nikita.

GridGain Systems

 

 

DirectByteBuffer has existed since Java 1.4. That part is not new. The BigMemory implementation does a lot of clever stuff to provide excellent performance and scalability characteristics within the context of Ehcache. We have tested this on Cisco UCS boxes up to 350GB with excellent results. We have also done extensive competitor comparative testing. We believe, based on our testing, that our BigMemory delivers on the promise of a scalable in-process but off-heap cache storage, whereas the competitors we have tested do not.

So GG has "scalable in-process but off-heap cache storage"?

 

This technique was used for years by many products (for example, that's how man IDEs avoid crashing when OOM happens). GridGain 3.0 has this feature (actually, more so). I believe other products too. Don't know who came up first with it. All this "off heap and in-process" simply means creating large enough byte array and, in case of data grids, keeping data in serialized form there instead of many smaller arrays that get GC. Technical aspects are pretty clear.

My original post was that the whole "marketing" seems rather misplaced and awkward. Great feature, great to see EHCache maturing, nice internal optimization (one of many dozens that can be done in a data grid product)... If every product would announce their internal optimizations in similar matter - we'd be seeing BigFlop "products" every week. 

For anyone who's well versed in the data grid marketplace it is pretty clear that this optimization is very important for Terracotta due to the limitation of their original architecture (EHcache gets distribution via Terracotta). This is IMHO, of course. But the whole technique is noteworthy for sure anyways.  

I'm pretty sure that you are getting similar if not bigger performance boost from serious serialization/deserialization optimization, something like Cohererence's protocol or GridGain's pluggable marshallers.

 

Regards,

Nikita.

GridGain Systems.

We have just released the technical documentation for Ehcache BigMemory which explains the details: ehcache.org/documentation/offheap_store.html

This seems to be similar to coherence and gigaspaces. Congrats on adding big memory to Ehcache.

This seems to be similar to coherence and gigaspaces. Congrats on adding big memory to Ehcache.

 

Peter,

I explained a bunch of what we are talking about and what we have achieved and how we tested over here:
blog.terracottatech.com/2010/09/bigmemory_expla...

Needless to say, I disagree with the other vendors. They have a vested interest in making claims and refuting ours, but the benchmark and product will speak for themselves. I am 100% confident in that (because I spoke to over 100 Java shops in the last 2 weeks from EBay and Facebook to EC2-based web games and more) and all were super excited.

 

--Ari

This seems to be similar to coherence and gigaspaces. Congrats on adding big memory to Ehcache.

 

Peter,

I explained a bunch of what we are talking about and what we have achieved and how we tested over here:
blog.terracottatech.com/2010/09/bigmemory_expla...

Needless to say, I disagree with the other vendors. They have a vested interest in making claims and refuting ours, but the benchmark and product will speak for themselves. I am 100% confident in that (because I spoke to over 100 Java shops in the last 2 weeks from EBay and Facebook to EC2-based web games and more) and all were super excited.

 

--Ari

Thanks for providing additional details. Looking at the blog entry big memory does look different, even if using byte array to hide it from the GC isn't new.

The only downside I see is buying a server with more than 16Gb of RAM so that I can use a 200GB in-memory cache. From a capacity planning perspective, buying a bunch of cheap servers with 16GB of ram "might" be more practical than buying 1 big machine with 350Gb of ram. The last time I had to price out a high end server full loaded, it was quite expensive and took a long time to deliver. Buying a bunch of cheap servers "can" be more cost effective.

On the otherside, if a fortune 1000 company already has big machines with hundreds of GB of ram, it would be foolish to not consider it as an option.

This seems to be similar to coherence and gigaspaces. Congrats on adding big memory to Ehcache.

 

Peter,

I explained a bunch of what we are talking about and what we have achieved and how we tested over here:
blog.terracottatech.com/2010/09/bigmemory_expla...

Needless to say, I disagree with the other vendors. They have a vested interest in making claims and refuting ours, but the benchmark and product will speak for themselves. I am 100% confident in that (because I spoke to over 100 Java shops in the last 2 weeks from EBay and Facebook to EC2-based web games and more) and all were super excited.

 

--Ari

I've thinking about the pros/cons of having really large cache on a single JVM and it's quite intriguing. On one hand, being able to load up fewer nodes to handle large caches sounds appealing from a Sys Admin perspective. Even though other products can supports hundreds and thousands of nodes, I have to ask myself "do I really want to manage that many servers?" Even with good management tools, having fewer nodes "feels" like a no brainer.

From a capacity planning perspective, going to the accountants to ask for several million dollars has it's own pain points. Back in 2002, our project spec-ed out the systems we wanted, which was several million dollars worth. It took 12 months for those systems to get delivered. Each of the app servers were fully loaded with RAM and darn expensive due to the ECC requirement. Even with cheap RAM today, getting 200GB+ of ECC ram isn't cheap. On the plus side, there's a lot of shops with mainframes that do have hundreds of GB of ram.

For me, the decision to use large cache versus distributed cache is largely dependent non-technical factors. Some shops simply don't have the floor space or rack space for hundreds of cheap servers.

In any case, it's good to see Ehcache move forward and add big memory.

imho, BigMemory illustrates/shows JVM lacks of an off-heap (but, in-memory) high-speed (explicit) passivation mechanism.

BigMemory is one proof such a mechanism is doable (and shows the path for the inclusion of such mechanism into the JDK ?).

 

Dominique

http://www.jroller.com/dmdevito

Doesn't Java RT already had immortal memory that is never collected? just bring over java rt inovations into java se!

Doesn't Java RT already had immortal memory that is never collected? just bring over java rt inovations into java se!

 

I agree.

Well, I think that, beyond that point, as Java SE is going to be more and more into pressure (due to more and more data-intensive computations and limitations, like GC), developers are going to need more control, they need to get back more control/opportunities to drive VM way of life.  And guess what, they have already more control within constrainted environments, and then, within Java RT for example. So, indeed, I think one (interesting) possibility to let evolve/improve Java SE is to look into the direction of Java RT...

 

Dominique

http://www.jroller.com/dmdevito

See http://www.theserverside.com/discussions/thread.tss?thread_id=60933 where this discussion is being continued.