Discussions

News: Sure, but what if we like twiddling knobs?

  1. Sure, but what if we like twiddling knobs? (19 messages)

    TSS' associate editor Kirk Pepperdine has posted a blog entry, called "My advice on JVM heap tuning, keep your fingers off the knobs!" There's a lot here, but the title is marvelously to-the-point: diddling JVM settings at random isn't normally a good way to make a JVM perform well. Kirk is also speaking at TheServerSide Java Symposium (should that be "The TheServerSide Java Symposium" instead?) as the track chair on performance - it's always good to see that your speakers have a clue about what they're speaking on. (Don't look down, Kirk - you won't enjoy it as much as we will.) Some relevant snippets in a fairly dense and informative blog entry:
    Just in the last few months I've been seeing an increase in the number of people seeking advice in the forums on how to tune garbage collection and/or size Java heap. While it is encouraging to see that more people are becoming more aware that JVM configuration is important to good performance, I got the sense that people are still struggling to sort out how all of the automated memory management actually works. They get the idea that things are kept in heap and that heap is subdivided. They get the idea that the garbage collector works to free up memory that is no longer being "used". They are out looking at list that go on and on... and on.. .... and on, and on with every possible switch configuration setting and then are all of a sudden overwhelmed with the array of choice. Having no time to build a complete understanding, which is an understandable situation, they go off to a forum and ask a question which of course draws dozens of responses all of which are mostly different thus crystallizing the only thing we know about GC and heap sizing, your guess is a good as mine. And so, we guess. When it comes to performance and tuning, it has been my experience that guessing is not a very good way of ensuring desired results in a consistent manner which has resulted in ... the mantra, Measure, don't guess. The question is, how do we take the guessing out of heap sizing and collector selection? It is my expressed opinion that the primary goal of all performance tuning exercises should be maximize the end user experience given the resource constraints we are required to work under. A less formal expression of this would be; minimize response times to the end user (though low latency maybe a more important target). So it falls from this that we need to be measuring user response times. If the user response times are within tolerance then we are done! There is no need to touch anything. ... GC efficiency is defined as the % of time it is running in exclusion to your application over the run time of the application. You may find it useful to limit this definition from the amount of time the JVM has been running to the amount of time your application has been active. No matter the definition, the best way to calculation this value is to collect the logs produced by -verbose:gc switch (-Xloggc:logfile.name preferred for Sun JVM) and feed it through a tool such as HPJTune (free download). If the value produced by the tool is greater than 10%, you have a case for proceeding with the tuning process. If it is less than 10% but greater than 5%, tuning may help but it might not give you the boost you were hoping for. Anything less then 5% and you're most likely wasting your time.
    Good stuff. One final point that may have slipped through the cracks in that last paragraph: Kirk specified the Sun JVM. This point is more important than it might have appeared - don't forget that individual JVMs have very different performance characteristics! Kirk pointed out recently (and privately) that JRockit tends to have a fairly long warmup time, because it's been designed to run well for weeks - and thus, JRockit tuning that focuses on a process that only runs for minutes isn't likely to pay off much.

    Threaded Messages (19)

  2. A good and lengthy posting by Kirk. One comment I do have with regard to the proposed approach of tweaking one or two JVM arguments is that even this is not scalable with frequent deployments of new versions of a web application. It is workable for ad hoc performance consultancy engagements but not necessarily on-going performance engineering and management were one really needs to have a good working knowledge of the application execution behavior (use cases/ workflows) and be able to track, via snapshot repositories, typical resource usage (i.e. object allocation rates) across releases and down to a sufficiently fine granularity such as a particular use case. Without such knowledge acquisition activities (continuos education) one is merely acting like a witch doctor. Detecting changes in resource patterns during the testing (and not necessarily performance testing) of a new release can alert one to the possibility that current production VM parameters (knobs) might be invalidated. One could try to tune JVM arguments in test and then copy these into production but I think this is in general a hit and miss which is the basis of Kirks posting and rarely does one have the exact deployment topology and can truly simulate production workloads and transaction mixes. Kind regards, William Louth JXInsight Product Architect JINSPIRED
  3. reads like it was written for 1.3.1, ever heard of cms, pargc?
    What every GC problems I've faced, I've found that I've rarely needed to set anything other than max memory and the survivor ratio
    ever tried tuning a softswitch app? or even any large app? just look at the jvm flags vendors use for spec benchmarks
  4. Hi, Vendors performing a benchmark have much more time to twiddle around with settings, have a fixed application code base to work with, can accurately simulate the real-world load (it is a benchmark!) and have a much greater reward than merely saving a few seconds of resource consumption. In the real-world and not having direct access to the development team responsible for the JVM runtime itself the cost benefit analysis does not look good unless of course you have a serious (in your face) issue with an application that you cannot change and need to mask the underlying allocation issues which is when I mostly see concurrent collectors being used (for normal business web applications). William
  5. Hi,

    Vendors performing a benchmark have much more time to twiddle around with settings, have a fixed application code base to work with, can accurately simulate the real-world load (it is a benchmark!) and have a much greater reward than merely saving a few seconds of resource consumption.
    Well put! Benchmarks are artificial to the point of almost being pathological. I addressed pathological cases in the blog. Ok, I gave it a passing thought. Point is, general purpose application tuning has to take into account that the forces that drive the application have ebb and flow and consequently the demands on the system and the setting for optimal performance would need to change in harmony. Say, hey, isn't that what ergonomics are suppose to do??? ;-) Kirk
  6. reads like it was written for 1.3.1, ever heard of cms, pargc?
    What every GC problems I've faced, I've found that I've rarely needed to set anything other than max memory and the survivor ratio
    ever tried tuning a softswitch app? or even any large app? just look at the jvm flags vendors use for spec benchmarks
    Mr. Taye, while these are fair questions, I think it's also fair to point out that Kirk is one of the industry's leading lights on JVM performance, along with Jack Shirazi, Stephen Haines, Brian Goetz, and a very few others. I have a feeling the answer would be "yes," on all of these questions, considering that Kirk's list of clients include some of the heaviest hitters in the Java performance space - and here, I mean even vendors, not just people who happen to need end-game runtime performance. And when I say "I have a feeling the answer would be 'yes'," what I mean is yes, I know for a fact that Kirk's done these things, having worked with him and seen him in action for years now. Come on, dude. Give TSS just a little credit. We don't bring associate editors on board just because we think their names sound cool or something.
  7. I am mortally wounded Joseph. I thought I was alone but now I realized I am not even on the list or within the click. Maybe I should talk about applying database wait based approaches to Java performance management or how cheap object allocation is, ;-).
  8. Maybe I should talk about applying database wait based approaches to Java performance management
    This one I'd like to hear about. But beware, you did slam me pretty hard some time ago for suggesting that one could use GC to moderate load and improve performance. ;-) - Kirk
  9. Maybe I should talk about applying database wait based approaches to Java performance management
    In fact Joe, can you free up budget and get this one scheduled? Kirk
  10. Maybe I should talk about applying database wait based approaches to Java performance management
    In fact Joe, can you free up budget and get this one scheduled?
    You betcha. William, email, dude.
  11. reads like it was written for 1.3.1, ever heard of cms, pargc?
    What every GC problems I've faced, I've found that I've rarely needed to set anything other than max memory and the survivor ratio
    ever tried tuning a softswitch app? or even any large app? just look at the jvm flags vendors use for spec benchmarks
    Mr. Taye, while these are fair questions
    Yeah, fair enough. I've tuned one or two large apps. Since you mention the 1.3.1 then you must know that GC wasn't all that wonderful back then and you didn't have all that many options and the ones you did have you needed to work hard to set them to reasonable values. But times have changed and so must our ideas of how to tune. The 1.4.1 started giving us some real options. The 1.4.2 got even better and that trend has continued in the 1.5 and even accelerated in the 1.6. I expect new types of GC to show up in the upcoming releases maybe starting with the 1.7. Some of the ideas that may work or seem to work on boutique hardware such as Azul should trickle into the general purpose hardware space. All in all I should be looking at much less work tuning heap which is fine by me. --Kirk
  12. reads like it was written for 1.3.1, ever heard of cms, pargc?
    What every GC problems I've faced, I've found that I've rarely needed to set anything other than max memory and the survivor ratio
    ever tried tuning a softswitch app? or even any large app? just look at the jvm flags vendors use for spec benchmarks
    Mr. Taye, while these are fair questions


    Yeah, fair enough. I've tuned one or two large apps.

    Since you mention the 1.3.1 then you must know that GC wasn't all that wonderful back then and you didn't have all that many options and the ones you did have you needed to work hard to set them to reasonable values. But times have changed and so must our ideas of how to tune. The 1.4.1 started giving us some real options. The 1.4.2 got even better and that trend has continued in the 1.5 and even accelerated in the 1.6. I expect new types of GC to show up in the upcoming releases maybe starting with the 1.7. Some of the ideas that may work or seem to work on boutique hardware such as Azul should trickle into the general purpose hardware space. All in all I should be looking at much less work tuning heap which is fine by me.

    --Kirk
    Not necessarily, there are many arguments against hardware assisted gc, and it doesn't look like the major processor vendors are getting into this. You're right, most apps don't need gc tuning, all that self-tuning/ergonomics work is finally paying off, even if the target apps were just desktop clients and enterprisey. But there will still be lots of tuning work for 1.7 ... between G1, closures and the planned support for dynamic langs, it won't be your dad's Oldsmobile.
  13. reads like it was written for 1.3.1, ever heard of cms, pargc?
    What every GC problems I've faced, I've found that I've rarely needed to set anything other than max memory and the survivor ratio
    ever tried tuning a softswitch app? or even any large app? just look at the jvm flags vendors use for spec benchmarks
    Mr. Taye, while these are fair questions, I think it's also fair to point out that Kirk is one of the industry's leading lights on JVM performance, along with Jack Shirazi, Stephen Haines, Brian Goetz, and a very few others. I have a feeling the answer would be "yes," on all of these questions, considering that Kirk's list of clients include some of the heaviest hitters in the Java performance space - and here, I mean even vendors, not just people who happen to need end-game runtime performance.

    And when I say "I have a feeling the answer would be 'yes'," what I mean is yes, I know for a fact that Kirk's done these things, having worked with him and seen him in action for years now.

    Come on, dude. Give TSS just a little credit. We don't bring associate editors on board just because we think their names sound cool or something.
    Mr. Ottinger, dude, you're taking this personally. For the record, I have no problem with Kirk, I like his javaperformancetuning.com website, and I don't mean to attack him. But dude, my problem is how you've been running this site, why has everyone moved to infoq?
  14. Mr. Ottinger, dude, you're taking this personally.
    No, I'm not. You presented a query; I answered it. That's not "taking it personally," that's "justifying the decision."
    For the record, I have no problem with Kirk, I like his javaperformancetuning.com website, and I don't mean to attack him. But dude, my problem is how you've been running this site, why has everyone moved to infoq?
    Dunno. Maybe it's like you said: it's how I'm "running this site." Care to share more details about how this works? I mean, I'm not good at inferring specifics from "how you're running this site" - can you share what I'm doing wrong, exactly? And while you're at it - and I'm eyeing traffic data here - how do you decide everyone's moving to InfoQ again? And even further - when did the web become a zero-sum game?
  15. TheServerSide vs. InfoQ[ Go to top ]

    Just butting in with my $0.02 CDN... I read both on a daily basis. I like TheServerSide because it focuses on Java issues on, um, the server side. I like InfoQ because it has a lot more information about issues peripheral to Java development. To me, the aren't competitors but rather quite complementary. Dave Rooney Mayford Technologies
  16. Mr. Ottinger, dude, you're taking this personally. For the record, I have no problem with Kirk, I like his javaperformancetuning.com website, and I don't mean to attack him. But dude, my problem is how you've been running this site, why has everyone moved to infoq?
    Gotta admit that I'm a little bit confused. Sometimes it seems to me as if I'm just being used... Pardon me, but the more I read this, the less I understand. Your initial post fired off at Kirk's post, saying:
    ever tried tuning a softswitch app? or even any large app? just look at the jvm flags vendors use for spec benchmarks
    My response was to point out that, um, chances are that yes, Kirk has probably tried tuning those kinds of apps - including "any large app." You even say you knew this, because you "like his javaperformancetuning.com website." If you like his site on performance tuning, why are you asking if he's done these things? The question only makes sense if you have no idea who Kirk is... but you do! Perhaps it's me, and my sense of etiquette, but generally if I go up to a recognized expert in a field and start questioning his knowledge of his field of expertise... I think that's a little personal. The only reason I replied was to point out Kirk's qualifications. My request for a "little credit" was possibly questionable, but... I'm human too. I mean, sheesh, I don't position someone as a recognizable figure in the industry on a lark. But.. did you mean to attack me? I mean, I didn't take it as such at first, but... your statement about how I'm running this site, how people are fleeing to InfoQ... I'm a little lost at this point. For one thing, I don't see how any of this has anything to do with the original post. Nor do I see what it has to do with Kirk, or his qualifications. It looks to me like a direct shot at TSS, and the "how you're running it" aspect makes it look like a direct shot at me - like TSS deserves better. I'm not going to argue that point - hey, if you're offerin', I'll be glad to forward your name to my employers. I'm sure you - or perhaps drunken lab rats - could do better. I just wonder about the motivation and reasoning. I'm certainly not averse to tuning how I'm "running the site" based on consumer need, but at the same time, I don't like being random about it. What's more, wouldn't it be nice if you weren't being personal when you say "it's not personal?"
  17. Keep up the good job[ Go to top ]

    I like TSS the way it is, InfoQ aint competition. Just look at the list of InfoQ Java topics, very low quality, with almost zero discussion. I am even starting to like flames like GigaSpaces open letter to BEA customers, its educating to see comments from more experienced, wiser and mature people regarding the Java and its direction, with a little touch of personal passion.
  18. I'm not going to argue that point - hey, if you're offerin', I'll be glad to forward your name to my employers. I'm sure you - or perhaps drunken lab rats - could do better.

    I just wonder about the motivation and reasoning. I'm certainly not averse to tuning how I'm "running the site" based on consumer need, but at the same time, I don't like being random about it.

    What's more, wouldn't it be nice if you weren't being personal when you say "it's not personal?"
    I'll ignore your flamebait, I visit this site for tech news/discussions, I have zero interest in getting in a personal shouting match. So keep this technical, don't chase away your readers.
  19. Joseph, for what it's worth, you have my full support. Given your experience, I'm sure you already know that the nay-sayers are always the loudest ... What really matters is the vast (and silent) majority of folks like me, who really like TSS and generally don't say anything because there's nothing to complain about. For one, I'm happy to thank you for all the work you've done, and the success of TSS is well deserved. -Laurent
  20. I tend to agree with the basic premise of the blog post. Keeping things simply is good where JVM tuning in concerned, and I have seen many options used, and most of the time they make things worse. One thing that I have learned recently, is that a lot can be gained from using large page memory support on 64-bit OS'es and the 64-bit JVM. If anyone wants to understand how to set this up, you can see my blog post on the subject at: http://andrigoss.blogspot.com/2008/02/jvm-performance-tuning.html With this one change, you can pretty much throw all the other twiddling away.