It frequently surprises me how few enterprises, institutions and organizations have a method to test services for scalability and performance. One fortune 50 company asked a summer intern to run a few performance tests when he had time between other assignments to check and identify scalability problems in an SOA-based information system! That was their entire approach to scalability and performance testing.
I'm an advocate for testing XML-based Service-Oriented Architecture (SOA), Enterprise Service Bus (ESB), Web services and Business Integration (BI) based services. Even risky ad-hoc testing is good. (It's risky because most of the time results from ad-hoc tests are misleading). In my experience the business value of running scalability and performance tests comes once an organization formalizes a test method that includes the following:
- The organization chooses an appropriate set of test cases. For instance, testing a multiple-interface and high volume service is different than testing a service that handles periodic requests with huge message sizes. The test needs to be oriented to address the end-user goals in using the service and deliver actionable knowledge.
- The organization runs enough tests and accurately. Understanding the scalability and performance of a service requires dozens to hundreds of test case runs. Ad-hoc recording of few test results is unsatisfactory.
- The organization makes the right conclusions when analyzing the results. Understanding the scalability and performance of a service requires understanding how the throughput - measured as Transactions Per Second (TPS) at the service consumer - changes with increased message size and complexity and increasing concurrent requests.
All of this requires much more than an ad-hoc approach to reach useful and actionable knowledge. I usually begin a test project with the following question: What do I need to learn from a performance test and how will it benefit my company, business, or institution?
This question gets directly to the heart of the reason why we test services built with XML, Web Service, ESB, and BI tools. It may be more difficult to answer than it first appears. In some test projects the answer took longer to find than the time it took to run the actual test.
The question challenges us to understand what parts of a service will be optimized to contribute a business benefit for the testing effort. For instance, most services are built on tools that provide the following components:
- Application and Service Components: Workflow, Business Process Execution, Orchestration
- Service Buses: Connectors, Caching, Policies, Scripting, Messaging
- Infrastructure Components: SOAP bindings, XML parsers, Message service bindings, clustering, replication and high availability
The components build on each other in three tiers: At the bottom are the fundamental components for SOAP bindings, XML parsing, Java inter-application messaging services and clustering. Building on these are service bus components for services to interoperate at a message level. The top tier provides interoperability at an application level exchanging messages at a business process level. Given this threefold layer, where would you start testing to understand the scalability and performance of an SOA implementation?
For instance, would it make sense to test only the connectors and caching objects at the service bus level exclusively? If you did so you may miss key performance bottlenecks in the SOAP bindings and message service. Each component impacts the scalability and performance of the resulting service.
Service architectures today depend on a consumer, service and broker architecture. I find most performance tests normally check, at most, two of the three. For instance, one performance test may check a consumer and service, and in another example, test a consumer and a broker. The following should help you identify which test will most benefit your system for scalability and performance.
- XML Parsing - The benefit is to decrease the time it takes to route a message to a service-handling object to lower network bandwidth and server hardware costs. This is a stateless test. The parameters to understand the test results include schema complexity (depth and element count), document size, concurrent request levels.
- Persistence - The benefit is to decrease the time it takes to store and retrieve messages to lower network bandwidth, server hardware, and disk costs. This is a stateful test. The parameters to understand the test results include schema complexity (depth and element count,) document size, and concurrent request levels.
- Transformation - The benefit is to decrease the time it takes to transform a message into a given XML schema to lower network bandwidth and server hardware costs. This is a stateless test. The parameters to understand the test results include source and destination schema complexity (depth and element count,) request and output document size, and concurrent request levels.
- Aggregation and Federation - The benefit is to decrease the time it takes to respond to service requests that require inputs from several up-stream data sources and to reduce network bandwidth and server hardware costs. This is a stateful test. The parameters to understand the test results include schema complexity (depth and element count) for each up-stream service, data persistence quantities, and time-to-live (TTL) values for each message.
- Data Mitigation - The benefit is to improve service availability and user satisfaction by reducing the times when a service is unavailable at times of peak usage. This is a stateful test. The parameters to understand the test results includes schema complexity (depth and element count) for each request.
The above list covers the service test goals I most often encounter.
However, there are many more possibilities and the pace of innovation within SOA building tools is very high. The above list describes stateful, stateless, and test parameters. The following explains these.
- Stateless testing checks the impact of concurrent requests and message payload size on a service. The service responds to each request independently from all the other requests. It is therefore stateless.
- Stateful testing is similar to stateless testing; however, the service provides data persistence, workflow transaction processing, message queuing, sessions, or data indexing. These required shared resources that are often interdependent. Sometime even with extra CPU, network and memory resources available the throughput does not increase.
- The parameters to understand the scalability index define the inputs to a test that will show us the scalability index - the usage pattern that shows us what we can expect of our service's performance in production.
By taking these steps, considering these parameters, and adopting a test methodology appropriate to your service your business benefits by developing actionable knowledge. Improved performance and scalability come as a result.