â€śThe application is slow and we do not have the access to source code. What are our options to improve the situationâ€?. Oh god, please not again, was the first reaction. More often than not the stories starting like this have a sad ending. This time it was different and I thought it to be worthwhile of sharing.
A closer look to the application at hand revealed it consisting of several batch jobs bundled together. Drilling down through the â€śperformanceâ€? criteria revealed that the time it takes to run a specific job was taking a bit too long. Some more scrutinising later I was given a measurable target. I needed to shave off two minutes from a particular job runtime to fit in a pre-allocated time window.
The troubled application was a pretty innocent-looking small JAR file. Which, to my luck also bundled the load tests.
Running the application with the GC logging turned on (-XX:+PrintGCTimeStamps -Xloggc:/path-to/gc.log -XX:+PrintGCDetails) and visualizing the logs quickly revealed first target for the optimization. The accumulated GC pause times added up to three and a half minutes, hinting I might stand a chance.
In a situation like this one has several tools at his disposal, some of which are simple and straightforward:
- Modify the heap/permgen size
- Change GC algorithm
- Configure the ratios between the memory regions
I took the path of altering the heap size. Besides just a lucky guess, it was based on the recently learned lesson about the correlation between live data set size and recommended heap size. From the GC logs I also noted that the live data set of the application was approximately 240m. So according to my recently acquired knowledge, the sweet spot for this application heap was somewhere in between 720 and 960m.
And indeed it was. Increasing heap to 720m reduced the accumulated GC pauses to 20 seconds, down from the original three and a half minute. Or increased the throughput from 92.25% to 99.55% if you prefer to publish results the other way around. Full test results are available in the original post in Plumbr Java performance tuning blog.
The example is a textbook case from a performance tuning textbook. You are set to success if you have a measurable goal and you can measure the results instead of guessing. If I had been forced to jump in without a clear goal or load testing capability I would still be tweaking random bits of the configuration.