|
Sponsored Links
Resources
Enterprise Java
Research Library
Get Java white papers, product information, case studies and webcasts
|
News
News
News
|
Messages: 21
Messages: 21
Messages: 21
Printer friendly
Printer friendly
Printer friendly
Post reply
Post reply
Post reply
XML
XML
XML
|
 |
How We Solved our Garbage Collection Pausing Problem
Greg Luck was suffering a nine second GC pause every fifty seconds, when Sun suggested some garbage collection tunings that solved the problem (down to two major GCs every day, he says, in "How We Solved our Garbage Collection Pausing Problem.")
The tunings were pretty simple: -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:NewSize=1200m -XX:SurvivorRatio=16 As Greg says:-XX:+DisableExplicitGC - some libs call System.gc(). This is usually a bad idea and could explain some of what we saw.
-XX:+UseConcMarkSweepGC - use the low pause collector
-XX:NewSize=1200m -XX:SurvivorRatio=16 - the black magic part. Tuning these requires emprical observation of your GC log, either from verbose gc or jstat (a JDK 1.5 tool). In particular the 1200m new size is 1/4 of our heap size of 4800MB. This is all documented in "Tuning Garbage Collection with the 1.4.2 Java Virtual Machine," which has a 1.3 version and 1.5 version. Mr. Luck also didn't quite mention which specific JDK he was running, although his reference to jstat indicates it might have been 1.5. If it wasn't 1.5, his use of -server or -client mode might have factored in as well, because -server changes the allocation of memory to Eden and the survivor spaces as well.
It's very common to need tuning of this sort in long-running Java processes, in particular in J2EE applications. It's so common that it should be part of the tuning checklist of every application, along with a few other gems like "don't use reflection unless you really have to, and if you have to, cache the method references!" That said, are you aware of how to measure GC times, and tune it?
Mr. Luck refers to Sun engineers looking at the verbose garbage collection logs, and offering the settings based on those. Is there a deterministic (or mostly deterministic) way to determine good settings for these? Could a lack of garbage collection tuning be part of J2EE's undeserved reputation for slow runtime performance?
Message was edited by: joeo@enigmastation.com to add the 1.5 GC tuning page, pointed out by Hack Kampbjørn
|
|
|
Message #213169
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: How We Solved our Garbage Collection Pausing Problem
Could a lack of garbage collection tuning be part of J2EE's undeserved reputation for slow runtime performance?
It's rarely the problem. Usually it's bad code and bad design ideas that have been pervasive in the Java world.
|
|
|
Message #213178
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: I solved it with -XX:MaxGCPauseMillis=5000
Ah, thank you for the 1.5 GC link! I looked but couldn't find it - so much for omniscience, eh. :)
|
|
|
Message #213181
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: How We Solved our Garbage Collection Pausing Problem
I don't know that I'd say it's "rarely" the problem - because in my experience, tuning memory is the lowest-hanging fruit in almost every performance-tuning exercise. To be sure, fixing awful code (i.e., references, often reading system resources that change rarely, poor algorithms) is also a huge win, but saying GC is rarely the problem is not my experience.
|
|
|
Message #213185
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
OT: reflection
It's so common that it should be part of the tuning checklist of every application, along with a few other gems like "don't use reflection unless you really have to, and if you have to, cache the method references!"
Sorry, completely OT, but my belief is that reflection is alot less of a problem in JRE 1.4+ than it was up to JRE 1.3. OK, for serious tuning, yes I agree. But I'm not sure it is the performance issue it once was.
I am prepared to be proved wrong though.
Kit
|
|
|
Message #213189
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: How We Solved our Garbage Collection Pausing Problem
I don't know that I'd say it's "rarely" the problem - because in my experience, tuning memory is the lowest-hanging fruit in almost every performance-tuning exercise. To be sure, fixing awful code (i.e., references, often reading system resources that change rarely, poor algorithms) is also a huge win, but saying GC is rarely the problem is not my experience.
What I am saying is that a lot of Java programs produce way more long-lived garbage that what is reasonable. Looking at the article I have several questions:
why was a major GC running every 50 seconds? That's not normal. It was probably caused by either something calling System.gc() or by not giving the VM enough 'breathing room' to grow without running a GC. Actually in the latter case you still need to be careful with the concurrent GC because it can give false positives for OOM Errors (there's another GC setting to cope with that but it's probably just easier to make the max heap a good bit larger than what is strictly needed.
Did the author try out the settings separately? Perhaps the disableExplicitGC setting really fixed the issue. I can't tell from the article.
Having said that, I'm a big fan of the concurrent GC for aything that needs consistent response time (I use it with Eclipse, for example) and sometimes these issues are out of your hands because they are in the App server, not your code. But the point is that GC problems are *usually* a symptom, not the root cause. That's my experience anyway.
|
|
|
Message #213202
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: OT: reflection
Well, compared to 1.3, yes, 1.4+ is far improved. However, looking at the context switch involved in using native code - which reflection does - it's very expensive, no matter HOW improved it is, compared to caching method references. Even those can go through the security manager, which isn't a good idea, performance-wise.
|
|
|
Message #213210
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Treating symptoms vs root causes
But the point is that GC problems are *usually* a symptom, not the root cause. That's my experience anyway.
Agile developers are supposed to take the shortest path to meet today's requirements.
How often does the finding and correcting the root cause meet that criteria?
While the engineer in me despises the idea of not correcting the root cause, there is a subtle beauty to the agile logic.
|
|
|
Message #213214
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Treating symptoms vs root causes
But the point is that GC problems are *usually* a symptom, not the root cause. That's my experience anyway.
Agile developers are supposed to take the shortest path to meet today's requirements.
How often does the finding and correcting the root cause meet that criteria?
While the engineer in me despises the idea of not correcting the root cause, there is a subtle beauty to the agile logic.
That's really neither here nor there. The question was "Could a lack of garbage collection tuning be part of J2EE's undeserved reputation for slow runtime performance?" and I say no, because the lack of tuning is not usually the cause of the poor performance. Tuning is a way to resolve the symptoms. Saying that the lack of tuning is the cause is like saying that that my car's axle is damaged because the mechanics have not fixed it. The mechanics' lack of action didn't damage the axle. It was my running over a dead deer that did it.
|
|
|
Message #213233
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Treating symptoms vs root causes
While the engineer in me despises the idea of not correcting the root cause, there is a subtle beauty to the agile logic.
That usually comes in the form of a contract extension to fix the hidden bugs, right? :)
|
|
|
Message #213261
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Treating symptoms vs root causes
But the point is that GC problems are *usually* a symptom, not the root cause. That's my experience anyway.
Agile developers are supposed to take the shortest path to meet today's requirements.
How often does the finding and correcting the root cause meet that criteria?
While the engineer in me despises the idea of not correcting the root cause, there is a subtle beauty to the agile logic.
I'm going to disagree that tuning the GC to alleviate the symptoms of some other problem would qualify as part of the Agile development process. In the case of a product going live, if this is necessary, then so be it. This is purely a pragmatic solution however, and does not address the problem in an acceptable manner. Sometimes pragmatic solutions are necessary, but I'd expect any decent engineering effort to eventually address issues that cause the VM to hang due to long periods of garbage collection.
Revisiting code in order to fix this issue would, in my opinion, qualify as agile.
|
|
|
Message #213262
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: OT: reflection
Re: reflection speed: depends on what "very expensive" means to you. Reflection method calls are slow compared to direct method calls. If I do a method lookup and call I can do only about a million calls per second on my laptop. If I cache the "method pointer" I can do 10,000,000 reflection calls per second.
Test program somewhere in here: http://forum.java.sun.com/thread.jspa?threadID=669560&messageID=3915259 -- be sure to use "java -server", there is a 6x difference between -server and -client.
|
|
|
Message #213448
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
how about profiling your app
Any app that is GCing that much is poorly conceieved, written and deployed
Given that most java apps are written by people who have mediocre concepts of object lifetime, session mgmt and threading and the affect of newbies "newing" objects all the time, I really think its disingeniuous to post an article about your GC probs without discussing the actual code.
It sounds like you are asking for everyone elses help to make your app not suck. Is that what TSS is about now?
The articles on TSS seem to be getting get worse and worse in favor of Java newbies on a daily basis.
|
|
|
Message #213449
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
is that how YOU do reflection?
I guess if you are committed to apaches bean.getProperty
you wont be caching.
Any model seriously dependent on reflection would cache the method references.
But honestly, your reference to a "context switch" and "native code" is highly suspect.
Why does java need to call native code to look up a java method even if its not cached in its internal object representation of the instance? "Native code" just to bypass type checking I think not.
A context switch is the term for when a CPU has to switch threads or processes and save off the registers to cache memory to switch tasks. Since the VM does no native IPC I cant see that that would ever really happen in a Java app.
Do you know the VM well enough to say thats actually true?
Given the amount you post on here I hope you know what youre talking about man. Cuz that sounds like FUD to me.
|
|
|
Message #213488
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: how about profiling your app
Any app that is GCing that much is poorly conceieved, written and deployed
Given that most java apps are written by people who have mediocre concepts of object lifetime, session mgmt and threading and the affect of newbies "newing" objects all the time, I really think its disingeniuous to post an article about your GC probs without discussing the actual code.
I've been repeatedly surprised by what shows up in tenured space in the Sun JVM .. some of our shortest lived objects were showing up there, for example. Understanding why (in this case because exactly one tenured object had a short term reference to the short-term object) allowed us to make minor code changes that increased throughput on some of our benchmarks by 10%.
Just to be clear, I'm not bragging .. quite the opposite in fact: It's [unfortunately] very easy to end up with GC behavior that is explainable with an effort of research, optimizable [fortunately] with little effort, but generally the behavior is unknown because of the lack of such methodical inquisition. This particular "GC issue" of ours had been in the software from day one, and that despite the fact that we do regular code reviews, profiling, GC testing and tuning, etc.
I guess what I'm saying is that I wouldn't be so harsh, having seen how easy it is to have such issues ;-)
Peace,
Cameron Purdy
Tangosol Coherence: Clustered Shared Memory for Java
|
|
|
Message #213497
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: how about profiling your app
Any app that is GCing that much is poorly conceieved, written and deployed
I find this to be a rather disingenuous statement. For example, the aforementioned System.gc() could be happening, and a simple code change together with proper gc tuning can eliminate the problem forever. Does this speak to the rest of the app's overall quality? No way.
Good GC Tuning is, at any rate, a process that must be managed by advanced developers with fairly broad knowledge of the many factors involved. I suspect this thread is in-fact quite useful to a large number or our readers. Thank you, TSS!
Cheers,
Gideon
|
|
|
Message #213498
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: how about profiling your app
I guess what I'm saying is that I wouldn't be so harsh, having seen how easy it is to have such issues ;-)
I just want to clarify that I agree with the two previous posts and that my point was not that GC tuning isn't ever strictly necessary but that for most Java applications the default setup is more than sufficient and the well-know 'preceived Java performance' problem is related to the performance of old JVM technology and some particularly poor design philosophies that have been (less so lately) popular in the Java world along with some individual algorithmic ignorance.
|
|
|
Message #213514
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: how about profiling your app
I guess what I'm saying is that I wouldn't be so harsh, having seen how easy it is to have such issues ;-)
I just want to clarify that I agree with the two previous posts and that my point was not that GC tuning isn't ever strictly necessary but that for most Java applications the default setup is more than sufficient and the well-know 'preceived Java performance' problem is related to the performance of old JVM technology and some particularly poor design philosophies that have been (less so lately) popular in the Java world along with some individual algorithmic ignorance.
Yeah, no kidding! I still get to read (on slashdot) how Java is slow because it is interpreted ;-)
Peace,
Cameron Purdy
Tangosol Coherence: Clustered Shared Memory for Java
|
|
|
Message #214018
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: how about profiling your app
system.gc should never be used in an app.
Everyone knows that. System.gc == figure out your problem and stop micro managing the VM.
This many years into java your still calling system.gc?
Cmon man!
|
|
|
Message #214019
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: how about profiling your app
I agree its easy to make mistakes and thats all they are simple mistakes.
I dont agree that this is the forum for people
who make those easy mistakes to post FUD about the garbage collector without posting the code that caused the problem once they figure it out.
Im not a JVM snob. But I know that when you find a problem in your code its your business.
When you find a problem in SUNs code then post it AND the code
we can actually appreciate solving it and move on.
|
|
|
| New content on TheServerSide.comNew content on TheServerSide.comNew content on TheServerSide.com |
|
|
Reza Rahman continues to explore the features of the proposed JSR 299, Contexts and Dependency Injection for Java EE (CDI). When approved, it promises to be a key feature of Java EE 6.
(January 21, Article)
Ted Neward is an independent consultant specializing in high-scale enterprise systems, and an authority in Java and .NET technologies. He is the author and co-author of several books, including Effective Enterprise Java. At TheServerSide Java Symposium in March, he will be presenting sessions on pragmatic architecture, ECMAScript and Scala.
(January 15, Article)
Now that Oracle is absorbing Sun Microsystems, there mixed views on what should come of the Java Community Process (JCP). While some say Oracle should become the new steward of Java and keep the JCP much as it was, others argue that it may be time to open-source this widespread language.
(November 24, Article)
Reza Rahman explores the features of the proposed JSR 299, Contexts and Dependency Injection for Java EE (CDI). When approved, it promises to be a key feature of Java EE 6.
(November 2, Article)
SAML is an XML-based standard for exchanging authentication and authorization data between security domains. The single most important problem that SAML was created to solve is the Web browser Single Sign-On problem. Many organizations are debating whether to stay with version 1.1 or move to 2.0. This article makes observations about both options.
(September 28, Article)
Joe Ottinger takes a look at how people learn, and applies it to the practice of programming. He notes that understanding how people learn is an essential part of working in a programming team.
(September 22, Article)
Stephen Maryka gave us an article about the Asynchronous Web and posed a number of questions that get examined like an approach to delivering Asynchronous Web capabilities through extensions to existing Java EE technologies.
(July 14, Article)
JavaServer Faces Flex goal is to provide users capability in creating standard Flex components, part of flexSDK which is open sourced through MPL license, as normal JSF components. This article by Ji Hoon Kim will provide an overview of creating a simple multilingual JSF page consisting of JSF Flex tags.
(June 29, Article)
In this session Jeff explores the key characteristics of successful SOA projects. He covers some of the patterns, and anti-patterns, tool sets, and strategies that he himself learned the hard way. Last, he provides a strategy and blueprint for achieving a high likelihood of success in your SOA project.
(June 23, Tech Talk)
Ari Zilka, CTO of Terracotta, Inc., talks about the new features in Terracotta 3.1, announced during JavaOne and available now.
(June 15, Tech Talk)
In this Tech Talk, Josh Long explores an integration challenge using Spring Integration and walks through the implementation, employing and expanding on the basic patterns of Enterprise Application Integration to tie together components into a function integration solution, and then demonstrates how Spring Integration helps address the integration requirements.
(June 15, Tech Talk)
In this Tech Talk, David Geary teaches you: The basics of Google Web Toolkit; How to implement Ajax-enabled applications in Java; Internationalization; Hooking into the browser history mechanism; Remote procedure calls.
(June 4, Tech Talk)
Jon Kern discusses the best architecture/technical solutions and ensure that they are repeated by all developers. By tackling the architecture up-front in a serial manner, subsequent parallel development will be much more manageable and predictable.
(May 28, Tech Talk)
This keynote describes the frustrations of modern knowledge workers in their quest to actually get some work done, and solutions for how to guard yourself against all those distractions. Neal Ford talks about environments, coding, acceleration, automation, and avoiding repetition as ways to defeat the misguided attempts to sap your ability to produce good work.
(May 26, Tech Talk)
Gil demonstrates how new, aggressive uses of already abundant compute capacity by common applications offer competitive value for application designers.
(May 21, Tech Talk)
Chris Keene introduces WaveMaker as a new way to automate the ability to generate Hibernate classes in order to more quickly bring OR mapping into an application.
(May 19, Article)
Download the entire book of Jakarta-Struts Live and learn about Struts MVC, Tiles, the Validator, DynaActionForms, plug-ins, internationalization, and more.
(Book PDF Download)
The Application Server Matrix is a detailed listing of J2EE vendors and their application server products, with information on latest version numbers, J2EE spec support and licensing, pricing, platform support, and links to product downloads and reviews.
(Application Server Comparison Matrix)
|
|
