Discussions

News: The ideal load test tool

  1. The ideal load test tool (10 messages)

    In his latest blog entry, Matt Albrecht lists what he wants to see in a load test tool and which tools meet his requirements. Among his list of must-haves, Albrecht requires that the tool is distributed, and that it is scriptable and has support for scripting in more than one language. As "features he'd like to see," Albrecht includes Remote Client Machine Control and System Performance Metrics.
    Really good tools would allow for controlling other things on the clients, like re-ghosting a machine (to allow "quick" changes in configurations). AutoT has some of this with its multi-boot support.
    Albrecht formulated this list after his last entry, in which he described what he dislikes about LoadRunner ("The user interface to it is horrible"). The list is based mainly on Albrecht's experience with LoadRunner, JMeter, Grinder, AutoT and a "home-brew" tool. Based on your experiences, what would your ideal load test tool include? What areas need improvement in the tools you've worked with?
  2. While we (Autoriginate) don't yet support performance testing, a lot of Matt's requirements around Remote Client Machine Control are definitely something that comes up over and over in any type of testing. Amazingly, no one from the big test tool vendors (Mercury, IBM, CompuWare, or Borland) addresses this issue. We have a product called HostedQA that specifically tries to solve that very issue for a selection of application platforms. Right now, if you are testing a Java-based web app, HostedQA can automatically deploy and reset the environment for you on every test. You can pick which operating system, application server, and database using a web-based interface. We're close to adding generic Apache support, so PHP, Perl, and Ruby (rails) can be tested too. Of course, for performance testing, the requirements are a little different. For example, in HostedQA we use virtualization rather than actual machines. I know for performance you would prefer a real machine (hence Matt's comment about ghosting the server, rather than putting it in a VM). I think the same concepts apply though, and we're looking closely at adding performance testing to HostedQA based on the same principles. Overall, I think that the general concept of resetting the testing environment to a known good state is a feature that none of the big test tool providers do currently, and is a large reason why HostedQA exists today. If anyone is interested, you can sign up for a free trial to check it out.
  3. Sounds like novice QA gripes[ Go to top ]

    Hey, I've used LoadRunner and plenty of other load testing tools, most of them just trying them out looking for a better one or one my current employer can afford. I'm glad to see someone getting some air time complaining about LoadRunner. LoadRunner's UI does suck and Mercury has and will continue to do nothing about it. I believe the reason why is that they have a monopoly and they are not motivated to improve the product for what they deem aesthetic reasons. The thing Matt seems to have missed about LoadRunner is that it is a solid, powerful tool with tons of features no one else has. Sure, anyone can throw another Grinder or Microsoft Web Stress Tool or JMeter out there, and generate a lot of virtual user load on a simple Web app. It's been a while since I used it, but here are some things I remember about LoadRunner that you might find interesting: - automated correlation of performance metrics (from a slew of sources, for which you must pay to enable) with resources involved in the load test. When you see your Weblogic database and thread pool stats with your Oracle d.b. stats, laid against the OS and hardware stats (these are free, fortunately), all splaid out in a multi-colored chart and associated with times and actions of the load test . . . well, it's pretty easy to see (and demonstrate) what's going on. This is the Correlation tool - metrics are expensive but LoadRunner can get you almost anything. For free the tool will gather performance metrics of the OS and CPU type you bought it for, and for HTTP metrics if you're doing Web testing. But Mercury charges customers for everything and this is one of their gotchas -- they have an incredibly cool correlation tool and they have metrics for everything under the sun (specific app servers, database vendors, Web servers, etc., etc.), but they license them separately so you have to pay to enjoy the power. - Load generation. LoadRunner can scale to generate a lot of load. I know of one license at a well-known app server company for 1 million virtual users. - Test protocols. Load Runner licenses test clients/generators separately too, but they have tons of them for load testing everything from mainframes to databases to plain old HTTP.
  4. LoadRunner and Diagnostics[ Go to top ]

    LoadRunner's UI does suck and Mercury has and will continue to do nothing about it. I believe the reason why is that they have a monopoly and they are not motivated to improve the product for what they deem aesthetic reasons.
    I have to agree that the LoadRunner has some clunkyness about it and to some extent the core has been in maintenance mode for quite some time. However, there are more recent editions to it's capture and analysis, including the Diagnostics module which allows you to capture very detailed Java or DotNet performance data using bytecode instrumentation. LoadRunner does continue to dominate the testing market (something like 70%), but they are making improvements to those areas that matter to server-side Java or Dot Net developers.

    The thing Matt seems to have missed about LoadRunner is that it is a solid, powerful tool with tons of features no one else has. Sure, anyone can throw another Grinder or Microsoft Web Stress Tool or JMeter out there, and generate a lot of virtual user load on a simple Web app.

    It's been a while since I used it, but here are some things I remember about LoadRunner that you might find interesting:
    - automated correlation of performance metrics (from a slew of sources, for which you must pay to enable) with resources involved in the load test. When you see your Weblogic database and thread pool stats with your Oracle d.b. stats, laid against the OS and hardware stats (these are free, fortunately), all splaid out in a multi-colored chart and associated with times and actions of the load test . . . well, it's pretty easy to see (and demonstrate) what's going on. This is the Correlation tool

    - metrics are expensive but LoadRunner can get you almost anything. For free the tool will gather performance metrics of the OS and CPU type you bought it for, and for HTTP metrics if you're doing Web testing. But Mercury charges customers for everything and this is one of their gotchas -- they have an incredibly cool correlation tool and they have metrics for everything under the sun (specific app servers, database vendors, Web servers, etc., etc.), but they license them separately so you have to pay to enjoy the power.

    - Load generation. LoadRunner can scale to generate a lot of load. I know of one license at a well-known app server company for 1 million virtual users.

    - Test protocols. Load Runner licenses test clients/generators separately too, but they have tons of them for load testing everything from mainframes to databases to plain old HTTP.
    A few corrections here - Mercury now bundles all protocols, monitors, and metrics within a base LoadRunner package that is more agressively priced to fit into the market. The additions which will cost you extra are the deeper diagnostics available in capturing class and method level data within a JVM. There are also additional features with the Diagnostics which allow you to diagnose memory leaks, slow sql, network/cross-jvm transactions, and thread/synchronization issues. Licensing still primarily operates on the number of virtual users that you want to run against your site, and now the pricing is simplified so that every little protocol or metric is bundled in the base product. I'm a bit biased towards the Mercury tools since I use them frequently to reproduce and diagnose production and pre-production performance issues. However, since I also participate in developing applications, I appreciate how the Mercury tools make it easier to pinpoint critical issues without having to invest additional development time into programming scripts. Also, the same diagnostics are available for monitoring production applcations, so you can gain the same insight into your apps whether they are in test or production. Clay J2EE 911
  5. Re: The ideal load test tool[ Go to top ]

    Why not try my free tool : JUnitScenarion :) http://junitscenario.sourceforge.net/ few options... but easy to setup :)
  6. As we evaluated most of the open source tools , like Jmeter , Grinder etc But we very much impressed by the OpenSTA. Within short periord of time you can start your load testing activity. Also it has many good features with scripting .. More deatils you can : http://portal.opensta.org
  7. I second that. I really like opensta.
  8. I second that. I really like opensta.
    I liked OpenSTA too. Unfortunately, it's Windows only. I found JMeter to be a good alternative, though not as intuitive to use.
  9. best of breed tools[ Go to top ]

    I think the reason you don't see this particular set of features combined in a single tool is that both load testing and PC configuration management are two very complex and very different problems. Any tool that does both is likely to not be very good at either. Instead I would recommend choosing the "best" tool in each category that can be automated via an external interface. This allows you to pick the individual tools that best match your requirements and then you can control/direct them with an automated test/build framework such as JUnit, ANT, Maven, etc. FWIW, I work for a load-testing company (http://webperformance.com/). I've never heard a request for this particular combination of features. However, we are currently working on several automation features to allow the product to integrate into automated test/build environments.
  10. Re: The ideal load test tool[ Go to top ]

    We tried the usual load testing tools like LoadRunner, AutoT and OpenSTA. However, we found that an appliance-based approach was better and more cost-effective. Most people don’t realise that you can have an appliance-based load testing tool at the higher end – they’re not just for network-level testing. With a hardware-based testing solution, it’s easier to trust the results and there’s no need for vast arrays of client machines, which could have a detrimental effect on any test. And because all hardware test kit is the same, you can easily benchmark the tests. I found details of Spirent's Avalanche system whilst looking for an alternative load test tool on http://www.softwareqatest.com/qatweb1.html#LOAD
  11. TestMaker and some additional ideas[ Go to top ]

    Matt's blog is a good starting point. Here's how my TestMaker open-source framework and utility stacks up: Distributed: No, however, yes with the commercial TestMaker Platform Scriptable: Jython is the primary scripting language Scripting In More Than One Language: Yes, Java too CSV Reports: Yes File Distribution: No, however, yes with the commercial TestMaker Platform Central Statistics Source: No, however, yes with the commercial TestMaker Platform Central Time: No System Performance Reports: No, however, yes with the commercial TestMaker Platform Remote Client Machine Control: No, however, yes with the commercial TestMaker Platform System Performance Metrics: No Things on my list that did not appear in Matt's blog: 1) A testing methodology to go along with the tool. Read chapter 5 of my new book titled FastSOA to understand the SOA and Web Service test methodology supported by TestMaker. 2) Multiple protocol support: HTTP, SOAP, REST, AJAX, email, etc. 3) Good documentation 4) Client generated load and server daemon capabilities. Both are important. For instance, testing the user sign-in feature to see if the service actually did send the email with a password reminder. 5) A class library of helpers to create load and to parse responses. -Frank Cohen http://www.pushtotest.com http://www.xquerynow.com