In "Determine Your Java Software's Performance Capacity During Development," author Raghu Donepudi discusses how you can analyze how much load your software can take when you're best able to alter it: during development, rather than during deployment or quality assurance phases.

... It is not cost effective to do an exact production simulation load test on a pre-production or test environment, which has created a market for numerous software tools that perform simulated load tests. All the testing tools basically run automated tests, during which you can collect other vital operating system information to calculate your software's performance capacity.

Because development teams seldom address load test issues during development, initial load test runs are prone to crashing. Lots of load testing time can be saved if problems such as open cursors and overly used CPU-intensive queries are addressed during the development phase itself. This article walks through how to analyze load test data and do capacity planning for your enterprise Java applications while they are in development.

However, the article focuses on only a few areas: watching for open cursors on your databases, analyzing expensive SQL queries, commenting out System.out references, and watching CPU load during expensive API calls, completely ignoring other potential performance sinks such as inefficient garbage collection, memory analysis, or even how to generate enough load such that the system can be stressed enough to show load issues.

What techniques do you use to verify load capacity?

I'm using my own tool (LGPL) : JUnitScenario (http://junitscenario.sourceforge.net/). it's not very accurate but it allows us to reuse our test to simulate the application usage. In few minutes, we can tests if the application will manage with the load. Just 10 minutes is required

Oh my... Commenting out the "System.out.println()" statements in the code before running the load tests ? I'd never have thought of that one. Great idea!

"Say your server has 1 GHz processor and 1 GB of RAM, and the capacity of each server is 20,000 users. The capacity of your enterprise software is 20,000 concurrent users on a server with a 1 GHz processor with 1 GB of memory. If your application has to handle 100,000 users during peak loads, you would request five such servers..."

This a pretty facile assumption. What about the DB? You can't just make the above assumption. Wss this article really valuable enough to post here?

"Oh my... Commenting out the "System.out.println()" statements in the code before running the load tests ? I'd never have thought of that one. Great idea!"

And hopefully you would not have raw debugs like this in the first place. If you do have them, if you are going to go through and comment them all out, you should resolve the issue and use proper logging.

"Oh my... Commenting out the "System.out.println()" statements in the code before running the load tests ? I'd never have thought of that one. Great idea!"And hopefully you would not have raw debugs like this in the first place. If you do have them, if you are going to go through and comment them all out, you should resolve the issue and use proper logging.

Hint: there was some degree of sarcasm in my post

"Oh my... Commenting out the "System.out.println()" statements in the code before running the load tests ? I'd never have thought of that one. Great idea!"And hopefully you would not have raw debugs like this in the first place. If you do have them, if you are going to go through and comment them all out, you should resolve the issue and use proper logging.

Hint: there was some degree of sarcasm in my post

I've worked on apps that had extensive logging for SLA purposes. By extensive I mean logging db query times, appserver loads, webserver queries against appserver and page generation. All of these logs are then fed to a reporting tool at night, which produces daily reports of the load. About the only way to write apps that can handle major load is to test performance as you develop.

It's also important to measure the expected capacity of the POC (proof of concept) so that management can start the paper work to get the necessary hardware. Depending on how capital items are ordered, it may take 12 months to get the systems. The article doesn't seem all that useful or insightful.

peter

"The article doesn't seem all that useful or insightful."

Well, if you were wondering how many times 20 goes into 100 it might be very enlightening.

"The article doesn't seem all that useful or insightful."

Well, if you were wondering how many times 20 goes into 100 it might be very enlightening.

Are you going to give us a hint at least?

Peace,

Cameron Purdy
Tangosol Coherence: Clustered Shared Memory for Java

"The article doesn't seem all that useful or insightful."

Well, if you were wondering how many times 20 goes into 100 it might be very enlightening.

4 is the correct number right :)

peter

"The article doesn't seem all that useful or insightful."Well, if you were wondering how many times 20 goes into 100 it might be very enlightening.

4 is the correct number right :)peter

I am still trying to find 4 others to take their socks off so I can do the math - or as we say 'round these parts - cipherin'.

"Hint: there was some degree of sarcasm in my post"

Hint: Nothing could be more obvious. What made you think this needed to be clarified exactly?

And hopefully you would not have raw debugs like this in the first place. If you do have them, if you are going to go through and comment them all out, you should resolve the issue and use proper logging.

You can also use sed ;-)

find . -name \*java; do cat $f | sed -e "s/System\.out\.println/\/\/System.out.println/g" > $f.new; mv $f.new $f; done

(untested)

Enjoy the Fastest Known Reliable Multicast Protocol with Total Ordering

.. or the World's First Pure-Java Terminal Driver

What's wrong with sed -i? :)

"Oh my... Commenting out the "System.out.println()" statements in the code before running the load tests ? I'd never have thought of that one. Great idea!"And hopefully you would not have raw debugs like this in the first place. If you do have them, if you are going to go through and comment them all out, you should resolve the issue and use proper logging.

After read the "comment the System.out before load test" technique and the "100000/20000 = 5" math I think their clients should review all load test done before.

I use a lot of Oracle provided sqlj code where stored procedures are written in java.
sqlj does not allow us to use any logger hence we dont have any option but to use System out.
Same sqlj code is used in my J2EE applications as well.
What's your say on this????

You might want to check out the Oracle FAQ at:

http://www.oracle.com/technology/tech/java/sqlj_jdbc/htdocs/jdbc_faq.htm#36_00

My main complaint about "comment out system.out" is that hardcore service applications have to log performance, so disabling logging is counter productive. If the application can't meet the scalability and performance requirements with required logging on, then testing it without logging doesn't do anything useful.

For financial applications, extensive logging of transations are needed for auditing purposes, so simplistic models of capacity planning are rather useless. But like someone else said, if you wanted to know the answer to 100,000 / 20,000, the article is useful.

peter

My main complaint about "comment out system.out" is that hardcore service applications have to log performance, so disabling logging is counter productive. If the application can't meet the scalability and performance requirements with required logging on, then testing it without logging doesn't do anything useful.For financial applications, extensive logging of transations are needed for auditing purposes, so simplistic models of capacity planning are rather useless. But like someone else said, if you wanted to know the answer to 100,000 / 20,000, the article is useful.peter

I think the point is that the use of very common components like Log4J, all you have to do to "turn off System.out()" is to kick up the log threshold a notch (or 4). So, instead of "DEBUG", only output on "ERROR".

Anybody using direct sysout statements is either very inexperienced, or has a very peculiar (and unusual) circumstance. If you need to log those INFO statements for performance logging in production, then switch your output from a blocking type (text file) to a concurrent type (SQL table) with a quick change of configuration.

Either way, the logging "issue" is (or should be) trivial.

--
David Bicking

I've already done that a few years back when I wrote a couple documents on tomcat performance. I followed that up with more specific tips on how one goes about load and stress testing. they're all available on either tomcat website or jmeter website.

peter

I took the sample code to see the effect of CPU utilization on System outs.
I am surprised. The time and CPU utilization increased many folds.
Many of us dont use System outs. But the point is the effect of unnecessary overheads.
Instead of complaining about what is 100000/20000 why dont you propose other technical issues to look for during load test.

Looks like these guys developed only single user applications and never worked on apps to handle thousands of users.

"why dont you propose other technical issues to look for during load test."

I did. I pointed out that you can't just assume that if one server can handle 20K concurrent users that 5 will handle 100K. I pointed out that the database in most such situations is shared by all app servers and becomes a potential bottleneck.

Other issues:

What is the comparative muscle of the dev/test servers and the production servers? Where I work the prod servers have much more memory and has more cpus.

What other processes will be running in the environment in prod that are not in testing?

What resources, other than the DB, are shared (LDAP, JMS, etc.). Do they present possible bottlenecks. Are they able to handle the required load?

Are all the components (e.g. VMs) the same in both environments?

Are your tests varied in size, complexity and duration. Runnning a ton of a single test case doesn't necessarily prove anything.

SQLJ simply generates JDBC code making calls on Statement, PreparedStatement and CallableStatement. This means that JInspired's JDBInsight could provide you with profile and timeline analysis of the data access code. If the datasource is accessed from a container managed datasource reference via JNDI then you can also enable JXInsight JDBC tracing (by default turned off) which unlike the JDBInsight proxy traces and profiles every single JDBC API call.

Customers sometimes use both because the traces from the JXInsight instrumented datasource represent direct calls from the application code which sometimes do not reach the vendor or JDBInsight datasource because of the containers connection pool optimizations such as prepared statement reuse. I should really write a new Insight article on this as we have detected by a worrying performance issue in a leading J2EE container vendors mnaged datasource implementation.

Please check out: http://www.jinspired.com/products/jxinsight/index.html


Regards,

William Louth
JXInsight Product Architect
CTO, JInspired

"J2EE tuning, tracing and testing with JXInsight"
http://www.jinspired.com

Regards,

William

Hopefully the following articles might prove more insightful when looking at load and capacity issues in the real-world for J2EE applications.

Database Lock Contention
Performance Problems in mixing OLAP and OLTP Workloads
http://www.jinspired.com/products/jxinsight/olapvsoltp.html

Network Latency Overhead - Excessive Client-to-Server-to-Database Roundtrips
http://www.jinspired.com/products/jxinsight/networklatency.html

Database Concurrency - Timeline Analysis and Execution Behavioral Patterns
http://www.jinspired.com/products/jxinsight/concurrency.html

OutOfMemoryError's Are Not Always Caused By Memory Leaks - Capacity Issues in Production Environments
http://www.jinspired.com/products/jxinsight/outofmemoryexceptions.html

User Experience Testing - JXInsight Trace Injection
http://www.jinspired.com/products/jxinsight/usertesting.html

Benchmarking and Garbage Collection - JXInsight Tracer Counters and Thread Call Stacks
http://www.jinspired.com/products/jxinsight/callstackbenchmark.html


Regards,

William Louth
JXInsight Product Architect
CTO, JInspired

"Tuning, testing and tracing with JXInsight"
http://www.jinspired.com

Hopefully the following articles might prove more insightful when looking at load and capacity issues in the real-world for J2EE applications.

Definitely! Thanks. BTW, your product is wonderful and reflects profound understanding of the problem space; I have to give the latest version a try.

Hi Alain, thanks for the product endorsement. I look forward to recieving additional feedback once you have downloaded our latest release.

------------------------------------------------------

The comments related to System.out and Logging has inspired me to write a new Insight article discussing the cost benefit analysis of current instrumentation (tracing) efforts by most development teams around the world.

Here is the link to the draft version of the article we will be publishing on our website once we have corrected grammar and spelling mistakes.

http://www.jinspired.com/products/jxinsight/tracing.html


If you have any comments on the article then please send me an email at: william dot louth at jinspired.com.


Kind regards,

William Louth
JXInsight Product Architect
JInspired

"J2EE tuning, testing and tracing with JXInsight"
http://www.jinspired.com

It is those typical article with attractive titile. It looks like it is going to talk about something based on the title, but actually it talks about nothing inside the article.

So what is the theory to determining load capacity during development? what is the procedure to determining load capacity during development?

If your organization has licenses for Mercury's LoadRunner, then conducting low-load diagnostics tests earlier in the development cycle is very effective. There are additional plug-ins to LoadRunner which capture detailed performance data for Java or .Net based applications. Mercury also provides a *free* version of the Java diagnostics probe used in LoadRunner which can be used standalone as a profiler (and takes only a few minutes to install and configure). Mercury Profiler
Often, I see dual effort in organizations to conduct load testing - developers use free tools to conduct low-load tests while QA uses more precise load-testing tools and end up duplicating the scripts that development created. What I've been advocating for customers who already own LoadRunner is to have development obtain a few licenses (a 5-10k investment) for the Virtual User Generator which allows you to easily record user actions on your application and play them back on your desktop. If you run with the profiler, you can tune the probe's instrumentation so that you can start handing off scripts and instrumentation "points" to your QA department for more thorough load testing earlier in the development lifecycle. QA still has their work cut out for them in performing the detailed analysis and developing test plans that ensure coverage, etc., but this method gives the developer a means of establishing a common process and vocabulary with QA.
I've seen this method work well in develoment shops that are moving more towards Agile, Scrum, or any of the other iterative development methodologies, as well as those employing more traditional methods. Agile groups still need to retain the load and pre-production testing that QA typically provides (see Scott Ambler's comments on this subject in this article), but also facilitiates a tighter integration with QA testing.
If you want to take this model even further, you can use the same Mercury scripts and instrumentation for development (VuGen + Profiler), QA load testing (LoadRunner + J2EE/.Net Diagnostics), and production monitoring (Diagnostics standalone). Basically, the scripts and instrumentation you create in development can be used throughout the application lifecycle.

Clay Roach
J2EE 911