I'm in the planning stage for a project to migrate our data services to a web service. I need some suggestions on doing an effective performance test on the new services. Can I use a tool (like jmeter) for this or would it be better to create a client that would make make the necessary calls into the web service? Thanks for any suggestions you can provide.

There are a number of options available to you in this regard, most of which will, however, likely require you to do some coding on the client side.

The open source options include primarily JMeter and the Grinder. These are general-purpose load drivers that you can use to develop web service clients for performance tests, but the operative word is "develop". They both come with support for HTTP, and you may find some samples of simple web service tests, but you will actually have to do some non-trivial development to truly exercise and validate your web services.

Also check out TestMaker. It's open source, and it's geared primarily toward testing web services. The drawback is that it's just a tool for recording/creating test scripts (agents), and the load generation mechanism must either be hand-coded into the scripts with custom threads, or purchased (in the form of their commercial tool/service, TestNetwork).

The commercial load generator vendors Mercury Interactive, Empirix, Segue Silk Performer, and others have all pretty much developed frameworks to drive web service performance tests, but obviously the price tag is pretty high (and you're still not completely immune from having to develop the logic to populate the request and validate the response).

Check out this article from developerWorks for some additional insight.

The bottom line is that, whichever way you go, you will probably have to do some development work to generate the request (populate the data, etc.) and then check the response. Most of these tools will free you in varying degrees from having to worry about writing the load-generation code, as well as the XML/SOAP/WSDL/UDDI infrastructure.

Hope that helps!

cramer

All these tools seem to work with HTTP only.
Out of curiousity, are there any other tools for testing any other protocol or directly testing the EJB layer ?

Actually, I believe most, if not all, of them support virtually all imaginable protocols. Both JMeter and the Grinder are basically generic load generation frameworks, requiring specific client protocols to be developed by the user (using their respective API's). In the case of the more common protocols, such as HTTP, there are pre-developed packages included. (Additionally, JMeter comes with JDBC support, and the Grinder has example scripts for testing EJB's, JMS servers, web services, JDBC, and others.)

With the commercial vendors, support is a bit more varied. Mercury Interactive's LoadRunner, for example, sells different test client packages for the various protocols and transports. For instance, web application testing requires the web client. EJB testing would require the Java client. Empirix targets primarily Java, and they've built they're platform from the ground up with EJB testing in mind. I imagine most commercial vendors have some mechanism to support a wide array of protocols.

Even TestMaker, with its Jython-based scripts, can test virtually any imaginable interface, HTTP, EJB, or otherwise.

The disadvantage of protocols other than HTTP, however, is that script creation requires more effort. Most of the tools I've tried come with some mechanism to "record" HTTP scripts by sniffing an example conversation (usually with the use of a custom proxy).

Hope that helps!

cramer

I'd add also OpenSTA.org - very easy in use and intuitive distributed load test tool. For functional test (with SOAP support, btw) here is an excellent Ant-based framework - anteater (http://aft.sourceforge.net)

The only complaint I have about OpenSTA is that its scripting language supports little more than a basic declarative syntax. There is no support for complex logic, including looping, branching, etc. (At least, this was the case the last time I checked it out, approximately four months ago.) For HTTP applications, this can be essential in order to promote maintainability (e.g., allow script modularity for reuse of common functions, reduce script size, etc.).

But as far as a distributed load generation architecture goes, it's right up there with the commercial tools I've used. It would be nice to see some additional monitoring capabilities, too....

cramer

Thanks Steve for including TestMaker in your comments. I just posted XSTest to the PushToTest Web site. It is a framework that turns TestMaker recorded functional tests into load tests. Details are at http://www.pushtotest.com/Docs/howto/xstest.html. Hope this helps.

-Frank Cohen
www.pushtotest.com
Author of Java Testing and Design: From Unit Tests to Automated Web Tests from Prentice Hall, details at http://thebook.pushtotest.com