Optimizing your Java Applications for Performance

Discussions

News: Optimizing your Java Applications for Performance

  1. A new white paper by The Middleware Company, outlines two critical investments to tune your application for performance and speed up development. Automating performance testing will let you focus on developing code, improving productivity. Investments in tools will allow you to attack the most critical performance problems more efficiently.

    Read 'Optimizing Java Applications' By The Middleware Company

    Threaded Messages (27)

  2. I wouldn't want people to neglect performance tuning because of the hurdle of buy or setting up a bunch of tools. If your application is exhibiting poor performance (and you can judge this simply by using it), you can examine the code and architecture and come to a pretty good conclusion where your problems are. They say that 20% of the code is running 80% of the time, so when you narrow down a particularly slow section of code, you will probably be able to figure out the problem.

    I'd recommend this book for some easy and FREE ways to improve your application performance.
  3. First, the defacto book is as follows:

    Title: Performance Analysis for Java Web Sites
    ISBN: 0-201-84454-0
    Publisher: AddisonWesley

    The short of it is this. This isn't really a discussion on tooling, in as much as its a mindset. Sure, your application may run crappy, but if you weren't knowlegeable to write good code in the first place, exactly how do you go back to "fix" it ?

    As to tooling (which can get into yet another religious war), the real trick is to understand the entire scope of what your application does, and the environment it operates in. You then need to determine the tools you need to assist you in that process. This is a learned process, as it really brings many skills to the table, and it takes time to get into that mindset.

    My advice - Read this book first.
  4. Recommendations[ Go to top ]

    Is this whitepaper a sponsored paper? We are all aware of the Precise link but the coverage of OptimizeIt is somewhat suprising. Some sentences within the whitepaper appear familiar to a recent Borland STrace whitepaper.

    It would be nice to know when TMC is recommending a product because they use it themselves or because they are paid to do so.


    I am very interested in hearing experiences with code profilers because I personally have found it near impossible to get real world memory/cpu snapshots from them. Has TMC used JProbe or JProfiler before?


    Here is my recommended book list:
    - Performance Solutions (The SPE Bible) C. Smith
    - Java Performance Tuning (The best of the Java related ones) Jack Shiraz
    - Capacity Planning for Web Services (Patrick Killea)
    - Performance Analysis for Java Web Sites
    - The Practical Performance Analyst



    William Louth
    Product Architect
    Inspired
    www.jinspired.com
  5. But tools are easy to justify.[ Go to top ]

    I definitely agree that the ideas expressed in the paper are valid with or without performance tools: automating your performance test cases, just as you would automate any other JUnit test case; treating performance testing like any other integration exercise; expanding performance management to cover the entire cycle.

    But you buy performance tools because they are worth the price. Contention in threaded systems and memory leaks are simply hard problems to solve. If you're one of the rare Java developers who can do this type of task without help, then you shouldn't buy a tool. But most of us will need help, and the price of a tool should be as easy to justify as a pipe wrench for a plumber: it's just a critical tool in the box for a programmer.

    In this area, Borland gets it. Another company that embraces this philosophy? IronGrid. They have a model that embraces cheap tools to solve tactical performance problems (like how do you begin to manage the thousands of SQL statements that flow through an application server?http://irongrid.com/catalog/promotion_email.php?track_id=MWC022403A1) It will be interesting to see if they make a splash. I'm betting that they will. It's just too expensive to leave performance until the end of the cycle.
  6. Tool vendors that get it[ Go to top ]

    Bruce,

    Have you checked out JDBInsight.

    After working for Borland for the last 3 years I feel the opposite - "Borland does not get it" - unless your talking about your money. The recent product releases from Borland have somewhat depressed me. Now I look to other software vendors to provide the tools that I use daily. Take for example IntelliJ. This is a great tool and one that has really brought back found memories of software development. For the last half year I worked with Borland I routinely used IntelliJ because it was far superior to our offering. JBuilder at this stage looks like Rocky 10 all battered, bruised and somewhat dated.


    Don't get me wrong. I loved JBuilder 3.5, 4 and 5. At that time it was king and rightly so. I moved from Delphi to JBuilder when I realized that it was not going to win and that the new kid on the block, Java, had lots of potential. This is the same reason I have moved to IntelliJ.

    The great thing about working with great tools is that they make you want to create great things - JDBInsight.

    Regards,

    William Louth
    Product Architect
    Inspired
    www.jinspired.com
  7. Tool vendors that get it[ Go to top ]

    I Think performance analysis of an application depends upon the application requirements, hardware on which it deployed, network on which it is running and software platform on which it developed. As a developer, it will be nice to have a good knowledge on performance tuning tool and books. Most important fact on successful completion of a project with a good performance chart sheet is awareness of application requirements in a development point of view. There will be 100s of way to do same requirement. Selection of best way depends upon the knowledge and practical experience of the development team in that particular business. Even though there are 100s of books and tools in market none of them seems like meeting the exact performance approach in a particular application.
       I came across one case where my manager asked me to come up with an architect solution where they want 3 seconds average response time with an average load of 300 users per second. The applications in production at that point were miserable for such a load even though they had the best hardware in place. I decided to look into different part of the application such as over all architect level, hardware level, jvm level, cluster level, and finally code level. There were places I could make some compromise for better performance. There is no hard coded law to follow in order to achieve the expected performance. In theory or in some books, they might have specified one particular way of doing application is the best for performance. In practical experience, a complete analysis of all parts of the application is the best way to achieve the expected performance.

    Thanks
    T.Q
  8. <Dave C>
    I wouldn't want people to neglect performance tuning because of the hurdle of buy or setting up a bunch of tools.
    C>

    I agree that you can get a surprising amount of information simply by examining your architecture and your source code - you can deduce quite alot without even seeing the code at runtime.

    But that only gets you so far. In every major project I've worked on, there have been one or more "surprises" in performance areas. Not necessarily major surprises, but nonetheless the code would bottleneck in some areas that I would never have thought of looking at on my own.

    What helped me ferret these cases out were simple profiling tools. Big expensive tools are cool to work with and give you an awful lot of power (like using Mercury Interactive Loadrunner to drive your loads), but you can get a surprising amount information out of free or nearly free tools.

    For example, in a JMS server I'm working on there was both an unknown performance bottleneck and a mysterious memory leak. I used Java 1.4's profiling switches to flip on profiling, ran the system underload, and then ran HPJUnit (web site here) on the resulting output. I was productively hunting down my problems in a couple of hours (timed from the initial google search).

    My point here isn't really to hype HPJunit (but it is cool and free), but to point out that there is no true "hurdle" that developers have to get over when it comes to performance, load, and testing tools in the Java space. There's alot of great stuff for free, even more for moderate prices, and then big hulking howitzers for 10's of thousands of dollars like Loadrunner. Developers who don't check these tools out (at whatever price level suits them) are cheating themselves and shooting in the dark unnecessarily.

    The above aside, I agree completely with the article that automated performance tests should be setup early in a project lifecycle for any medium or large side project, and run on a regular basis along side integration tests. It's always better for everyone involved to catch performance problems early when they're most easily fixed, and an automated system/script can do this for regularly, easily, and cheaply.

         -Mike
  9. and then ran HPJUnit


    But that's profiling from a file. I mean, everybody should have some kind of profiler (OptimizeIt, JProbe) these days. A very good one is JProfiler. It's only $399.

    -- Andreas
  10. <quote>
    But that's profiling from a file. I mean, everybody should have some kind of profiler (OptimizeIt, JProbe) these days. A very good one is JProfiler. It's only $399.

    -- Andreas
    <quote>

    The only serious performance problem I have suffered was solved by logging method call durations with System.out.println() and System.currentTimeMillis(), after fruitless attempts at profiling WebSphere JVMs.
    The very long redundant operation repeated for every transaction was found on the first test run and eliminated with caching; most timing printouts were left in the code for reassurance.
  11. < Quote>
    The only serious performance problem I have suffered was solved by logging method call durations with System.out.println() and System.currentTimeMillis(), after fruitless attempts at profiling WebSphere JVMs.
    /Quote>

    Yep - which is why I like the idea of applying common sense as well as utilizing whatever tools you have at your disposal (and _knowing_ what tools are out there!). As you pointed out, sometimes the tools fail and you have to fall back on old tried and true methods.

         -Mike
  12. <Mueller>
    But that's profiling from a file. I mean, everybody should have some kind of profiler (OptimizeIt, JProbe) these days. A very good one is JProfiler. It's only $399.
    /Mueller>

    They're all certainly good tools. But my point was that you can get an enormous amount of useful profiling information using nothing more than JVM switches and free tools. Memory usage, leaks, call times (CPU & Wall clock time), lock contention, etc. Even if the application hasn't been setup with any sort of profiling in mind, and you have a practically bare system, you can turn on a few JVM switches and download a profiler analysis tool that takes up a few megabytes at most.

    What that means to me is that there is no extraordinary hurldes that developers face to profile their code. They don't need to invest days or weeks learning baroque tools, they don't need to beg their manager to cut a purchase order, they don't have to worry about overly intrusive instrumentation tools (unless they want to).

    For me, this sort of baseline profiling support is very important, becuase it means I can diagnose problems in context without having to try to replicate someone else's environment. If a peculiar configuration of my code on someone's server appears to be leaking memory, or running slowly, or eating too much CPU, I can slap on a couple of JVM switches, copy the profiling file to my own workstation and analyze it at my leisure. In fact I've done precisely this in many situations.

         -Mike
  13. For me, this sort of baseline profiling support is very important, becuase it means I can diagnose problems in context without having to try to replicate someone else's environment. If a peculiar configuration of my code on someone's server appears to be leaking memory, or running slowly, or eating too much CPU, I can slap on a couple of JVM switches, copy the profiling file to my own workstation and analyze it at my leisure. In fact I've done precisely this in many situations.

    That's certainly a good idea and an area where such an offline analyzer makes sense.

    -- Andreas
  14. Offline Analyser[ Go to top ]

    Hi Andreas,

    Borland OptimizeIt does have support for offline snapshot generation. It allows you to specify the generation of snapshots at fixed intervals. I have never used this as I think that this feature needs to go hand in hand with another feature.....

    By the way I have downloaded an evaluation of JProfiler based on one day usage it looks a well designed Java Code profiler. Good value at a very reasonable price.


    Will
  15. Offline Analyser[ Go to top ]

    Borland OptimizeIt does have support for offline snapshot generation. It allows you to specify the generation of snapshots at fixed intervals. I have never used this as I think that this feature needs to go hand in hand with another feature.....

    I used OptimizeIt quite a long time. However, after Borland boughts it, I had trouble to upgrade old licenses and looked for alternatives. JProfiler is one. It's very good stuff. Typical German engineering! ;-)

    -- Andreas
  16. Hi Mike,

    I have witnessed developers spend enormous amount of time trying to parse and understand text file outputs. One developer spent 2 weeks writing a parser and then importing the contents in Excel and analysing the information from there. Of course his little solution only worked on his machine and nobody else was able to figure out what his excel spreadsheet was in fact presenting. The turn around for each examination was not fast (AGILE).

    Does it take 2 weeks effort to get a relatively small P.O.? Maybe you are talking wall clock time and not development time, ;-). Are developers really paid so badly these days that 2 weeks of development effort is less than 100-1000 USD? Thoughts?

    If reading text files is state of art information visualization then we (performance tool vendors) should be ashamed. I would hope that in the future that profiling tools do a better job than they currently do. A good profiler should within a matter of minutes/hours provide a powerful insight into the execution nature of your Java/J2EE application.


    Regards,


    William Louth
    Product Architect - JDBInsight
    "Tune with Insight"
    Inspired
  17. <Louth>
    I have witnessed developers spend enormous amount of time trying to parse and understand text file outputs. One developer spent 2 weeks writing a parser and then importing the contents in Excel and analysing the information from there. Of course his little solution only worked on his machine and nobody else was able to figure out what his excel spreadsheet was in fact presenting. The turn around for each examination was not fast (AGILE).
    /Louth>

    Yes, I've seen people waste time that way. As I said, developers should know what their options are and apply them intelligently. Sometime a tool is the right solution - sometimes even a very, very expensive tool in the tens of thousands if you have a really sticky problem. But sometimes the old printf() equivalent is the only thing that will do the job.

    <Louth>
    Does it take 2 weeks effort to get a relatively small P.O.? Maybe you are talking wall clock time and not development time, ;-). Are developers really paid so badly these days that 2 weeks of development effort is less than 100-1000 USD? Thoughts?

    If reading text files is state of art information visualization then we (performance tool vendors) should be ashamed. I would hope that in the future that profiling tools do a better job than they currently do. A good profiler should within a matter of minutes/hours provide a powerful insight into the execution nature of your Java/J2EE application.
    /Louth>

    I think perhaps you've misunderstood me. I don't pour over the profiling output from the JVM. Instead, I use something like HPJmeter which organizes the profiling data in a manner similar to JProbe et al. It's a swing app that provides various charts, graphs, call trees and other visualizations of the profiling data.

    It's not the best in the world and it has its limitations (as does the default JVM profiling output), but it's no where near as bad as you're portraying it to be. Most of the time it's more than sufficient for me.

        -Mike
  18. HPJmeter[ Go to top ]

    Mike,

    I thought the hp tool simply processed the text file and presented another nice formatted "text file". Did not realize the tool had some visual presentation. I will download it and have a look.

    I hate to say this but the one performance issue that is consistent across most J2EE projects (light or heavy) is the old printf (well really the println unless you write alot of native code ;-)). This performance problem is prevalent in first-time J2EE projects where the developers do not understand what is exactly going on in the container.

    Since early 1999 I have occasionally performed J2EE training classes for some VIP customers and in every single class I have asked the question:

    "Has anybody read one or more of the J2EE specifications?"
    100% - NO.
    0% - YES.

    "Has anybody started or nearly completed a J2EE project?"
    75% - YES
    25% - NO.

    What does that say about the quality of J2EE projects?


    Will
  19. HPJmeter[ Go to top ]

    <Louth>
    I thought the hp tool simply processed the text file and presented another nice formatted "text file". Did not realize the tool had some visual presentation. I will download it and have a look.
    /Louth>

    Quite all right, my initial description was pretty shallow. But it really does a nice job with things like call graph trees and correlating information.

    <Louth>
    I hate to say this but the one performance issue that is consistent across most J2EE projects (light or heavy) is the old printf (well really the println unless you write alot of native code ;-)). This performance problem is prevalent in first-time J2EE projects where the developers do not understand what is exactly going on in the container.
    /Louth>

    Unfortunately I agree with your assessment. There's an appalling number of developers out there who do not understand the technologies they use very well - at the Java level, the J2EE level, the app server level, and with various tools. To this day I see people using Vector/ArrayList contains() on lists with thousands of entries (they seem to not know the performance difference between list scanning and hastable lookups). Or calling getters() on remote entity beans (not understanding that the EJB is remote). Or manually copying arrays instead of using System.arrayCopy(). The list goes on and on....

    Of course not everyone is like that. There are competent people out there, and the various forums available show that. However, the forums also mask the reality somewhat. For every competent person posting here there are probably 10's of developers who don't really know what they're doing and are hanging on by cutting and pasting code from tutorial sites.

    Many developers simply do not take the time to get to know their technologies. And by that I mean really _know_ them, not memorizing lists of facts to regurgitate on certification exams. And areas like performance especially highlight this, since there are relatively few good references on performance testing & tuning in complex environments (relatively compared to the EJB tutorials, the Java turorials, the JSP tutorials, etc ad nauseum).

    And of course we all pay the price of this, since chances are you'll have to work with these people on many projects. This is one of the reasons I'm so leery of the more "out there" agile techniques like XP, because the underlying assumption of competent professionals that it relies on doesn't match reality.

         -Mike
  20. Couldn't agree more[ Go to top ]

    HPJMeter,

    I couldn't have said it better myself. If you just consider the size of the J2EE specification itself, and the time and effort it would take to truly learn the specification - and then ferret out all of the best practices for how to best develop a J2EE interface .. you see the result

    And the real shame, is the number of "Learn Java in 21 days" books. I keep trying to find the "10 easy steps to Brain Surgery" line of books, but I can't seem to locate any. And the real crime, is that I don't see this getting any better anytime soon.

    What I didn't mention in my previous post, is that you have 2 ways of doing your monitoring - Proactive and Reactive - Now while you would expect most developers to proactively check their code, you may need to use a tool like Wily Introscope to check the program after the fact. Which one you decide to use is going to depend on your budget, resources and approach, but in this world of outsourcing, you may need to consider using tools like Introscope as a guide/measure of how well your code was written and possibly establish SLA's based on that.
  21. Free chapters from my book[ Go to top ]

    Cool pick of a book. The Java Performance and Idoms books reads well and has good advice.

    For system-level performance and scalability testing I'll plug my own book: Achieving Reliability. Chapters are available for free download at http://www.pushtotest.com/ptt/thebook.html.

    -Frank
  22. Don't neglect the database[ Go to top ]

    A badly designed database schema and lack of appropriate indexes can make a huge difference in system performance.

    Also, you should isolate use cases very early that have critical performance requirements. Make sure these use cases are tackled fairly early in the development life cycle.
  23. Tuning and Testing[ Go to top ]

    I would also like to draw attention to the fact that testing and tuning should go hand in hand. Especially when we are talking about complex transaction logic.

    The majority of the times I am called into solve J2EE performance issues I find errors in deployment descriptor settings and component-to-component interactions.

    "I did not realize it was talking to the database like that"
    "That update should not be happening"
    "Why are we writing to a field that we do not read from"
    "Why are there so many resource transactions executed for a single business transaction"


    Some performance tools claim to support XP/Unit Testing but the interpretation of the word 'support' is really open to abuse and is. Being free/open source does not automatically qualify a product for the XP/Agile tag.


    William Louth
    Product Architect - JDBInsight
    Inspired
  24. Oracle includes a basic profiler with their free IDE (JDeveloper), which can be easily run on anything running within the IDE. I wish Eclipse came with this...
  25. Yet another JProfiler fan[ Go to top ]

    I also recommend JProfiler. I've used JProbe to trace memory issues but I find Jprofiler a nicer tool overall. It is simpler than Jprobe but that's not necessarily a bad thing. There are still some things that JProfiler can't do that JProbe can but JProfiler is quickly getting there and as someone alredy mentioned the quality of JProfiler in terms of robustness and stability is unmatched.
  26. What's wrong[ Go to top ]

    You want to build the application of the stars. You buy the tools, you do requirements, you do design , then you implement. When you test you find out that the application is too slow.

    What is wrong?

    I think that the development lifecycle is wrong. You need to do an architectural proof of concept as soon as possible (at the beginning of the project?). By proof of concept I mean a very , very minimal application (1 JSP + 1EJB or JDBC that will do database operations) that will use the architecture you want to have in production. Then you test the small application to the combinations (no of users, no of operations with database, etc.) you expect. Examples:
    Scenario 1 : 10000 users, 3 database operations
    Scenario 2: 1 user 100 000 database operations, etc...

    After you validate the architecture (several modifications may be required until the performance is good), going forward you will still have some minor performance problems but those problems can be identified quickly with profilling tools and usually solved without architectural changes.

    Also, if the lifecycle development would be iterative (X most important use cases handled from begining to the end), the eventual problems will be spotted much earlier in the process and this will save you from a lot of trouble.

    Beside profiling tools that do a good job in analyzing the code, you may need some tools (or people with good SQL skills) to optimize your database queries. The database side can have a great impact on your performance.

    I think that the ideas in the paper are good, however I don't know what to say about so much testing since I didn't try this.

    In other industries when you get a product you get also a list of its qualities (Like what is the bigest force it will resist at). Now, in software industry, when you get a product you don't know how many users will handle and how fast will it be if lots of database operations are reuqired. This is why a architectural proof of concept is necessary.
  27. Great post. It is the life cycle.[ Go to top ]

    A proof of concept is critical. It's not the same as a prototype: it takes the critical risk elements and trys them under fire.

    In addition to doing a proof-of-concept, you've got to have a smart way to detect when performance breaks. When you think about it, a performance test should be every bit as important as any other type of JUnit test. When you can automate, you should.
  28. What's wrong[ Go to top ]

    To support Adrian's message. Of the 25 or so J2EE performance exercises I have been involved with in the last 12 months, over half of the problems were with the database. Interesting that half of the database problems came from sub-optimal SQL and the other half due to logic errors in the applications. Tracing the SQL calls can be just as important since more often then not, you find the application unnecessarily making the same SQL call over and over and over.... The performance problem can be traced to logic errors.

    Also, to support stress tests. The most complex and difficult performance rear their ugly when the app server is under stress. It is the competition for scarce resources - like what happens in production - that cause the really tough problems
    Chuck