Discussions

News: New IBM ECperf Results Using a 7-Node Cluster

  1. New IBM ECperf Results Using a 7-Node Cluster (11 messages)

    IBM has posted new ECperf results in which similar price and price/performance metrics were achieved compared to their last post; however, this time with a lighter 7-node cluster configuration. The latest results yielded a performance figure of 25658.13 BBops/min@Std and a price/performance figure of $12/BBops.


    Q & A with Matt Hogstrom (Added May 1, 2002)
    -----------------------------------------------

    Q: This recent submission doesn't show as high a number as the one from a week ago. Why did IBM provide this result?

    A: We wanted to show a few things. First, this submission shows a seven node result which can be compared to our previous nine node submission. Comparing the 25658.13/BBOPs in this submission to the 32581.47/BBOPs of the previous submisison we see that by increasing from seven nodes to nine nodes a corresponding increase of about 27% is achieved. This increase in throughput shows a near linear increase in BBOPs as mentioned in the nine-node submission.

    The second item to observe is that the increase in throughput for the 32581.47 result was accomplished by increasing the capacity of the database server as well as adding two additional application server nodes. This submission used an xSeries 360 as the database server while the previous submission used an xSeries 440. Both servers utilize four Intel Xeon MP processors and were configured with the same amount of RAM.

    The other item to note is that the cost per BBOP remained about the same (it actually decreased with the larger database server). This demonstrates that increasing hardware to accomodate larger workloads does not necessarily translate into a higher cost per transaction.


    Check out IBM's latest ECperf Results
  2. Oh No.. not again :-)

    Seems they are hell bent on running the ECperf tests weekly/daily and i'am sure, if they runs the same test again on the same setup and configuration, they will come up with different results everytime :-)
  3. Yep, I think this is the last one for a while ;)

    The thing with this one is to show scalability from 7 to 9 nodes (ratio .77) and with the work increasing from 25K to 32K (.78), thats all. Linear scale up.

    So, we scale horizontally and we shows already that we can run a single JVM on an SMP and scale vertically too.

    Thats it from me. See ya.
    Billy (IBM)
  4. Who cares! ECperf is fast going the way of TPC-C with results that are so far fetched (anybody for 350k tpmC or whatever gobbleygook they call these numbers).

    Why doesn't ECperf have a clear distinction between results that use a SMP box versus those that use clusters? At present, it is an unfair (and meaningless comparison) to compare a 9 node (18 CPU) result with a single node result!

    In any case, given all the shady pricing deals/benchmark specials, these ECperf results aren't the least bit surprising. ECperf has lost all credibility as far as we are concerned (I work for a Fortune 50 company and let me tell you, we don't base our decisions based on these absurd ECperf scores).
  5. <snip>
    In any case, given all the shady pricing deals/benchmark specials, these ECperf results aren't the least bit surprising. ECperf has lost all credibility as far as we are concerned.
    </snip>
    I dont beleive there was any 'shady' pricing deals in IBM's or BEA's submissions (I dont work for either of the two). All the pricing used is available to ALL customers regardless of prior purchases or future commitments. Which was the intent of the spec (that the pricing shd be what any customer wouldpay to get the same configuration).

    As for benchmark specials, there were no specials. Be it CMPOpt or any other such- they were all legit features available to all apps. (Save the issue of data-stomping, which also is now fully fixed in Ecperf1.1- now called SPECjAppServer2001).

    Being closely involved with ECperf specification, I want to understand your concerns (so we can try and fix it). The benchmark has complete backing from all the leading vendors in the J2EE space (with fullsupport from leading Database and hardware companies). There is genuine interest in ECperf serving its purpose (of comparing AppServers). As you think otherwise, I (as wld the whole Ecperf spec team) would really like to understand specific issues you may have.

    Cheers,
    Ramesh
    - Pramati



  6. Cary Bloom said:
       Why doesn't ECperf have a clear distinction between
       results that use a SMP box versus those that use
       clusters? At present, it is an unfair (and
       meaningless comparison) to compare a 9 node (18 CPU)
       result with a single node result!

    What makes it unfair or meaningless? It is perfectly fair since all vendors get to choose the architecture on which they test and they can choose to publish on the results that they feel are representative.

    The performance comparison seems to be perfectly valid to me. Any way I construct a system is fine, if I care most about performance then which ever architecture is the fastest is the one I want. If I care about fault tollerance and high availability, then a clustered solution is probably more important to me even if it were possible to show that a single node would offer the best performance. If I believe that total cost of ownership is important, then I'd want to fact costs associated with system maintenance and support into the equation rather than using the raw ECPerf cost comparison.

    What I would like to see is a measure of the effectiveness and efficiency of session replication within a clustered solution. ECPerf does not measure that.

    Chuck
  7. ECperf is fast going the way of TPC-C with results that are so far fetched


    Have you looked at most of the other benchmarks out there? People very rarely submit benchmarks for lower end (i.e. cheaper) systems.

    > an unfair (and meaningless comparison) to compare a 9 node (18 CPU) result with a single node result!

    Not if you include the costs of both systems.

    > ECperf has lost all credibility as far as we are concerned

    So ECperf had lots of credibility until people started releasing results for it? It's been published for a long time (and the 1.1 specs have just been approved) - did you feed back your concerns then?

    > (I work for a Fortune 50 company and let me tell you, we don't base our decisions based on these absurd ECperf scores)

    Yes, my experience of large corporate decision making is that it is based on the quality of the meals bought by the supplier - why replace that with an 'absurd' price-performance measurement?

    Bottom line - performance on enterprise systems is a difficult and complex thing to measure. Thus (destructive) criticism is easy and finding (constructive) suggestions harder. You can obviously do the former, but can you do the latter?

    /david
  8. Matt Hogstrom, from IBM Websphere Performance has shed some light on the reasoning behind this new ECperf posting. I've added a brief Q & A to the news announcement at the top.

    Perhaps this clears up some doubts?

    -Nitin
  9. Cary: "Why doesn't ECperf have a clear distinction between results that use a SMP box versus those that use clusters?"

    Each submission includes a full disclosure on hardware and software used.

    Cary: "At present, it is an unfair (and meaningless comparison) to compare a 9 node (18 CPU) result with a single node result!"

    It is fair, it is expected, and most importantly, it is useful. Clustering is a fact of life now for reliabilitly and scalable-performance reasons, and people should have general information to help them choose between 40 1-CPU Linux on commodity x86 servers vs. 40 1-CPU Sun Netra T1s vs. 20 2-CPU HP 24x0s vs. 10 Sun e420s vs. 1 32-CPU IBM pSeries 690 vs. 1 32-CPU Sun e15k vs. ....

    Cary: "In any case, given all the shady pricing deals/benchmark specials, these ECperf results aren't the least bit surprising."

    Those are all available to the average Joe. The annoying thing with IBM is the discounts are bundles of other IBM products which may or may not fit your corporate "profile" (e.g. if you standardized on Oracle already).

    Cary: "ECperf has lost all credibility as far as we are concerned (I work for a Fortune 50 company and let me tell you, we don't base our decisions based on these absurd ECperf scores)."

    Lots of IS/IT employees at Fortune 50 companies -- probably including your own -- do use this information, although hopefully they analyze it with a critical eye towards the issues that you are describing.

    One other side-effect of ecPerf is that it has forced both IBM and BEA to really improve their performance and their support for the J2EE standard. Try running ecPerf on Weblogic 6.x vs. 7.0 to see what I mean w.r.t. performance. IMHO, IBM has greater market leverage (hardware + OS + app server + database server + global services + CICS + those indestructible IBM keyboards) while BEA has (thus far) maintained a sizable lead technologically (remember, it's only my opinion, and we do an extensive amount of work with both ;-).

    It's also provide companies like Pramati a forum to highlight their unique capabilities (e.g. the data stomping issue) plus a way to make a clear economic argument (assuming they can show a lower $$$/BBOP at specific BBOP levels).

    The real question is, "When are all the other vendors going to start submitting?", not "When are IBM and BEA going to stop submitting?"

    Peace,

    Cameron Purdy
    Tangosol, Inc.
  10. Looking at the dates, this is the only test from IBM within the same week. BEA ran two tests back to back to show the difference in JDBC Driver. I thought the point was to come up with different configurations.

  11. ECperf submissions till date, other than Pramati's, have been made without any Concurrency Control in place. Had they chosen to run with concurrency control, the options available are very expensive. Most vendors rely on DB Isolation Levels to effect Concurrency Control. This could be very expensive (as all the rows accessed, even in finders or non-transactional methods, will be locked. Thus impacting concurrency and thruput). Some vendors do have Pessimistic locking solutions that work in non-clustered and Exclusive Database alone (and not in a cluster nor when database is not exclusive, as required in ECperf).

    This (above) was in reference to data-stomping. What could be the impact of running with concurrency control inplace? Can anyone comment on this?
  12. the reference in my previous message is to a posting made on this thread-
    http://www.theserverside.com/home/thread.jsp?thread_id=13179