I am a J2EE newbie, wondered if anyone could please help me get some things into perspective re: application server benchmarking? I'm a student researching this for a firm intending to upgrade to J2EE environment from an old Netscape app server version 4.
What is the purpose of benchmarking standards like ECperf and TPC-W... is it mainly for "benchmarketing" by the big vendors?
Are there benefits to implementing ECperf as the workload in a benchmark to aid purchasing decisions (as opposed to using our own J2EE application + a load injector)?
Does anyone have any general thoughts about key procedures/measurements/problems for benchmarking J2EE application servers? Any comments gratefully received!
Thanks in advance
I think the purpose of benchmarking standards like ECperf is not solely for the vendors to show impressive numbers in their ads, but also for admins to see how effective their tuning attempts are, and to have compareable and reproducable results over a prolonged period of time.
The nice thing of standards is that they are objective in a way - they treat every contestant in the same manner. You may optimize for a benchmark, but as everyone does it, the procedure is very fair, because the benchmark is similar in every run. So if you really want an objective comparison, regardless of the usecase you may have finally in mind, you should use some standardize benchmark like ECperf.
On the other hand, if you have a very specialized app and know that this will be the only job the appserver will be doing, perhaps the results of the benchmark will not reflect the performance of your app...
Just my $0.02 :)
Our app servers host the business logic for a web portal (500K + registered users) and it seemed to me that the ECperf workload is more ERP focused. TPC-W might be a more relevant workload but doesn't test J2EE app server functionality, while Sun's Pet Store is a design blueprint more than a realistic workload.
I did wonder how comparable ECperf results are given the wide variation in SUT configurations, CMP vs BMP etc...?
Well, then why not just take a look at ECperf? I mean, you get the full spec and sources (for free), so it should be no problem to look at the benchmark, compare its workload and the way it is created to your usecase, and then decide if it fits the goal you have in mind.
Comparing CMP with BMP performance is fairly easy with ECperf, as all entity beans are distributed with source code for either variant. Just flicking a compiler switch, in a way :)
You can read up everything about the different SUT configs in the spec. But we have a figure of speech in German that goes "Probieren geht ueber studieren", which means something along the lines of "Trying it out is better than reading about it"...