Discussions

News: SPEC Announces Development of J2EE/.NET Web Services Benchmark

  1. SPEC has announced that they are developing a benchmark to compare performance of the J2EE and .NET platforms running web services.

    SPEC representatives told us:

    "Unlike specification-only benchmarks that do not prescribe an implementation, this benchmark will provide implementations for both J2EE and .NET that will follow a rigorous design. We believe this will result in a benchmark that provides customers a fair comparision of the platforms involved."

    When will the results be posted?

    No dates have yet been mentioned, so we will all have to stay tuned.

    Visit the SPECappPlatform home page:
    http://www.spec.org/appPlatform

    Visit SPEC itself (Standard Performance Evaluation Corporation):
    http://www.spec.org

    Threaded Messages (8)

  2. Will they ever learn?[ Go to top ]

    will people ever learn how utterly useless all these benchmarks are?
    I couldnt care less if J2EE or .Net is 7% faster than the other.
    Unless someone completely neutral shows me proof beyond a shadow of a doubt over a series of different designs, implementations and applications that one or the other is 10 times faster than the other I will never care about this!
    And no Rolf, I dont want to see your "neutral" home-made benchmarks that prove just that...
    ("Look, .Net is 10 times faster looping thru for(int i = 0;i < 10;i++) than Java is looping thru for(int i = 0;i < 100000000000000000000000;i++)").

    The main difference in performance is not in the chosen language, its in the quality of the design, architecture and code, which means that people are more important than anything else.

    I can already hear the flaming further down this thread.. Oh well, it makes for comical reading at least... :)
  3. Will they ever learn?[ Go to top ]

    will people ever learn how utterly useless all these benchmarks are?


    Truly neutral benchmarks can be very good tools when you must decide between different platforms and speed is an issue.

    But there's cheating almost in every of them. Hardware manufacturers cheat to get better scores. Motherboards have higher frequencies than specified to give that critical 5% lead over competitor. Recently NVIDIA was found quilty of cheating in Futuremark's graphics benchmark application where their 'optimized' drivers gave 25% speed advantage over real life results. Some apps (like NVIDIA's drivers) do actually detect that a benchmark program is being run and adjust the behaviour accordingly.

    Now, if I can get the sources for both platforms and find them comparable, I'll believe. It would be nice to see stress testing also.
  4. Will they ever learn?[ Go to top ]

    Also, sometimes benchmarks, even invalid benchmarks, can help improve current implementations.

    A few years ago, Mindcraft did some benchmarking on Windows and Linux. The benchmarks supposedly compared the throughput of Linux and Windows but there were several flaws (see http://lwn.net/1999/features/MindCraft1.0.php3 ) where the Linux implementations were deoptmized and the Windows implementations were optimized. Even when you take those things into account, they didn't measure a real-world situation because in the real world bandwidth is limitted by the network card speed and both Windows' and Linux's throughput for the benchmark well exceeded the fasted network card at the time.

    Despite all of this, it did appear that the Mindcraft results were measuring something. Careful inspection of the code brought to life several missed opportunities at optimization and once this was brought to light, several fixes were created. This resulted in several key improvements in the performance of the networking code which had noticeable effects in several real-world apps. (see http://www.kegel.com/mindcraft_redux.html ).
  5. Will they ever learn?[ Go to top ]

    Of course benchmarking and performance monitoring has its place during a development project or comparable, my remarks were first and foremost aimed at all these commercial benchmarks trying to prove one products superiority over the other..
  6. Will they ever learn?[ Go to top ]

    Of course benchmarking and performance monitoring has its place during a development project or comparable, my remarks were first and foremost aimed at all these commercial benchmarks trying to prove one products superiority over the other..


    You should also realize that commercial benchmarks are used extensively by all the vendors to optimize their products. Many of these optimizations do benefit a wide variety of customers. I agree with you that many times, result publication claiming superiority of one product to another is pure marketing ... but do realize that behind the marketing, a lot of engineering work is likely to have taken place (which is not simple cheating).

    Shanti
  7. Will they ever learn?[ Go to top ]

    You should also realize that commercial benchmarks are used extensively by all >the vendors to optimize their products. Many of these optimizations do benefit >a wide variety of customers.


    I would categorize these benchmarks as "development". :)

    I´m talking about the uselesness of the "My product is much better than yours, look at my benchmark!"-benchmark.

    I am in no way putting down the "development"-benchmark (the one that you actually use to validate and/or further optimize your product).
    Almost every project I have been involved in has used benchmarks and extensive performance monitoring and testing, its worth the extra effort to get some kind of feel for wether or not it will all go down the dumps as soon as there is more than 2 concurrent users. :)
  8. Will they ever learn?[ Go to top ]

    IMHO, There have really only been two principal benefits of the benchmarks offered by SPEC and TPC (especially the TPC-C benchmark):

    1) The requirement for full disclosure often yields very interesting and valuable tips for configuring and optimizing a system (if you can manage to find posted results that resemble your configuration closely enough, that is--an increasingly challenging task if you're on a Unix platform). Though the posted configurations widely ignore high-availability and resiliance requirements, this information can still help system architects to squeeze out the most from their production systems.

    2) The competition itself led to a valuable focus on performance. Even if the focus geared itself towards optimizing for the benchmark, the consumer would still benefit from the platform improvements realized as a result.


    You're right that any attempt to discern the best or most performant based upon the benchmark results was always at best a shaky proposition, but up until about 5 or 6 years ago (in the TPC-C case), you could try to gauge a vendor's commitment by their participation in the competition (as well as harvest the fruit of the full disclosure mentioned in point #1 above). These days (since 1997), MS-related configurations average about 37 results per year, compared to the approximately 21 results averaged by all non-Microsoft configurations (down about 50% from the number of results posted between 1995 and 1997).

    This is not to suggest any bias or lack of objectivity on the part of the TPC or the benchmark itself; on the contrary, I applaud these vendors for their active participation. But the dearth of recent results from *nixes robs us of whatever benefits these benchmarks offered, and makes it that much more difficult to compare enterprise platforms.

    After all, even though the benchmarks cannot yield a true objective comparison between platforms, there's nothing else out there that comes closer.

    cramer
  9. Will they ever learn?[ Go to top ]

    My benchmark is easy, does it work and how much does it cost to develop and maintain?

    Time to market (or production) is more important than the time to run through a loop.