I need to put a document together about performance testing tools and charateristics. I found quite a few of them, but used only a few of commercial ones (AstraLoadTest etc). I would like to get some info from user's perspectivem, not from sales side.

What is your experience with performance testing tools for J2EE applications? Any suggestions for commercial ones? Astra-LoadTest? Load Runner? Other?

What about free ones? Any recommendation and experiences?

Regards,
Marcel BAN
Toronto
www.TheBans.com

I worked with Astra and empirix-eTester. I feel like empirix is lighter than Astra and also easy to use.

Thanks,
Senthil.

I used Emprix's BeanTester a while back. There were some things I liked...mostly code-generation of the Java clients to access EJBs....and the ability to create mock objects intrigued me quite a bit. The ability to run tests and record and report results was interesting...but I suppose the whole creation of Java clients and running of tests could be done with JUnit/Cactus as well. The mock objects....saw it...intrigued by it....but didn't get into it too deep.

The performance tools come in two main flavours:

1. Tools designed specifically for generating load: In this category comes tools like LoadRunner, SilkRunner, AstraLoadTest and Microsoft web stress tool. Amongst these, Microsoft web stress tool is the only freeware. LoadRunner is easy to use and probably the best in terms of producing almost realtime load. SilkRunner is better when you have too much of personalization involved in the site you are going to perform load testing on.

2. The second category includes the tool that give you memory and resource consumption of various modules in the application. You dont need to run them under heavy load. Their main usage is to highlight the potential problematic areas in the application by identifying possible memory leaks, improper usage of transaction,incorrect handling of available resources etc etc. You can say, you need this set of tools to optimize a running application further.
Some tools that fall in this category are PerformaSure, Jprove suite, Wiley, OptimizeIt, IntroScope etc..

PerformaSure -- It gives you information on latency, memory and cpu data from begining to the end of any user transaction (request --> response)

Jprobe -- Jprobe is more of a development enviornment specific tool due to its high resource reqmt. It has the capability of identifying the problematic areas to a very deep level (pin pointing exact LOC)
Since both of these are Quest products, the general trend is to use PerformaSure in Production/QA enviornment to determine the broad areas that need improvement and then run Jprobe on DEV to further narrow it down.


Wiley is another such tool that combines the functionalities offered by PerformaSure and Jprobe.

http://www.javaperformancetuning.com/resources.shtml

We use Load Runner where I work and it works well. It is great for scripting tests and reporting on response times but honestly it is nothing that couldnt have been coded in house using a performance extension to JUnit such as JUnitPerf (http://www.clarkware.com/software/JUnitPerf.html). The ability to use some of the same test cases between automated regression testing and performance testing makes me think that would be a more efficient way to test.

Don't forget JMeter. It integrates very nicely into Ant targets and produces some nice reports.

Regarding the LoadRunner vs. Junitperf comparison.... JUnitPerf is a great way to decorate your existing unit tests into timed tests and load tests. Whitebox performance testing of functional tests. LoadRunner is blackbox systems and integration testing...the scripting and execution of end-user use-cases. LR also has very nice reporting, graphing, test execution, and monitoring of tests (monitoring of DBs, CPUs, memory, EJBs, etc...) that make it very worthwhile.

I don't see LR and JUnitPerf as an either or choice. Sometimes it is very late in the development cycle before end-user scenarios can be scripted with LR...and then it might be too late to realize you have an issue. So the ability to performance test earlier in the development cycle with JunitPerf...and catch performance regressions with timed tests is useful too.

Also...LR costs $$...and JUnitPerf is free. So that is a significant factor too. But if your company has the $ for it, it is a great tool.

Mike

Check this list on my site, it contains over 90 load testing tools.
http://www.performancetester.com/mambo/component/option,com_weblinks/Itemid,4/catid,68/

A lot of the tools are actually open source tools and some are getting pretty advanced.

Roland Stens
rstens at performancetester dot com
http://www.performancetester.com

I work on PerformaSure, a J2EE performance product, at Quest Software. PerfomaSure allows you to track your app performance across the different tiers in your architecture. It works across HTTP, RMI, etc.

Check it out at java.quest.com.

Yarek

We use two different tools:

. Mercury Interactive LoadRunner.
. Quest PerformaSure.

Both of then are very good tools, and works pretty well. We use LR to write scripts (if finally generates C code) and to simulate load. If has a very good reporting tool, and has a lot of facilities to design scenaries. It also supports a lot of protocols appart from HTTP/HTTPs (you can write scripts to test 3270 applications, fat-client Oracle applications, etc.).


PerformaSure is a less mature product than LoadRunner (it is not very stable) but it is absolutely useful to find out slow components. We use this tool to record sessions while we run load tests with LoadRunner, and when we finish the testing, we can detect with PerformaSure slow methods and SQL sentences but also misconfiguraction problems (data sources with less resources than needed, problems with threads of the web container, etc.).

Using both LoadRunner and PerformaSure, you can assume that you are going to be able to turn a poor-performing application into a brilliant software.


Jose Ramon Huerga
http://www.terra.es/personal/jrhuerga/

I work on PerformaSure, a J2EE performance product, at Quest Software. PerfomaSure allows you to track your app performance across the different tiers in your architecture. It works across HTTP, RMI, etc.Check it out at java.quest.com.Yarek

A couple more tools...JProbe and OptimizeIt...that I have used. Like them both. Not for generating load....but if you iterate a use-case with another tool such as LR you get a very detailed analysis of where the peformance issue is.

Maybe in conjunction...using a more macro tool such as LR to run use-cases and identify which pages are slow. And then switching to a more micro tool such as JProbe or OptimizeIt to isolate from there.

I have Load tested our web based application with OpenSTA tool. It is free toll but quite comprihensive and the result analysis it provides is good. The documentation available for this over internet may be is small but the help provided in the toll is good.

Pls feel free to contact me for any clarification about OpenSTA and its doubts.