Was trying to tune my application for better performance and started getting deeper and deeper into Garbage Collections..My main concern currently is to minimize big pauses for a major collection..I know we can do this to an extent using a Throughput collector/ Incremental GC, but this would be at the cost of worser performance.
My application is redundant in the sense that it is clustered across 6 different nodes...Is there any way that a servant cluster node can send a signal to the master saying that a major collection is going on and the master in turn stops sending traffic to this node till it gets a signal from the slave saying that the GC is over...(in the meantime, it distributes requests only to the 5 other nodes)...I feel if one can do this, we would have better throughput and almost non-existent pause times for a redundant/clustered application. Does anybody know if this is possible?
I can suggest you some ideas,
1. can use wait/notify mechanism.
2. set high priority to servant nodes.
3. Keep global object (like session) , at any point of time it should hold servant nodes status. Master has to check this object before distributing request to the rest.
Have you considered using real time JVM? BEA provides real-time JVM (JRockit), which gaurantees GC pause time. Currenly, they can guarantee that GC puases won't be longer then 10mc. To the end of this year they promise to go to submillisecond pause time. This might be easier.
See desctription of the BEA WebLogic Real Time (http://www.bea.com/framework.jsp?CNT=index.htm&FP=/content/products/weblogic/realtime/
I've only heard a suggestion to use Sun's real-time no-pause VM. I was also told its license is hefty. If JRockit can really do the same (or close) free of charge, hey, why not try it out? That is, if throughput degradation is not too severe. I barely can live (dramatization) with the thought that we foregone a 20% throughput increase of Java 6 after much testing (long story, due to one bug we had to roll back onto Java 5). Oh, wait, I'm getting ahead of myself, real-time VM from BEA is also not available for free. Heck, they will not even present a price quote on it without contacting sales!
As was explained to me, once GC has started, there's no way to do anything in that VM. I started asking, what if HotSpot were to notify (via JMX or something) that it is about to stop, before it stopped, and the suggestion was, that HotSpot could publish a value between 0 and 1 that would indicate probability of GC being needed. One could set a threshold and instruct load balancer to exclude node that is about to have Full GC, and re-introduce this node after GC completes (or even force a Full GC proactively).
Bug about this issue folks over at
Instead of expecting or finding out a mechanism to detect whether GC is ON on a node, would you prefer to initiate garbage collection at regular intervals from the master to the slave. Thus, the master will know which node is notified to do gc, and keep a 'x' seconds time limit,and then assume that the gc in that node is finished. similarly do this in a round robin fashion to all the nodes. thus you are at control of the gc.
anyway, if gc takes so much time, then there is a need to think about how the memory is consumed by your application. do you use caching efficiently. in case you are not using any third party api to do caching and you are willing to implement a custom api for caching then i would suggest you can use the java.lang.ref.SoftReference to wrap the object that needs to be cached. This is very interesting feature I have used in my application(s).
We've found after much pain that caching, if it is any efficient (that is, if cached object is to be reused at least several times), guarantees that a cached object will be promoted to tenured/old gen space. And under decent load, collecting 1GB of heap takes over 5 seconds no matter what. It's not easy to fix by just thinking "about how the memory is consumed by your application".
Write a JVMPI client, that way not only can you monitor and signal to the other VM's that a particular VM is undergoing GC, but you can also signal upon end of GC as well.