Discussions

News: PushToTest publishes SOA Evaluation Toolkit for app servers

Bookmark and Share Get thread feed

  1. PushToTest publishes SOA Evaluation Toolkit for app servers (11 messages)

    • Posted by: John Doppke
    • Posted on: July 21 2005 15:46 EDT
    PushToTest - maker of free open-source test software, performance kits, and global services solutions - has created an SOA benchmark toolkit to help implementers complete a timely evaluation of application servers and make an optimal purchase decision. The resulting tools provide a comprehensive approach that enables customers to select the best server to meet all of their business objectives. PushToTest then used this toolkit to evaluate BEA WebLogic Server 8.1, BEA WebLogic Server 9.0, JBoss Application Server 4.0, Apache Geronimo Milestone release 3, Oracle Application Server 10g (OAS), and IBM WebSphere Application Server V6.0.

    As we all know, evaluating application servers is a difficult task. The SPEC benchmarks provide performance information, but often architects need to know how app servers stack up for their own needs and in their own environment. This toolkit allows users to generate relevant performance results in their own environment and to evaluate app servers based on their own needs. The toolkit also provides a thorough feature comparison of the app servers, as well as a "developer's journal" containing notes kept by the developers as they built the test suite, which speaks volumes about what level of developer productivity to expect with each app server.

    Resources:
    Reply to this

    Threaded Messages (11)

  2. Test results[ Go to top ]

    You might want to check out the results of their testing, which is available here.
    Reply to this Reply to original

  3. Test results[ Go to top ]

    Saw a PDF - full of XML processing results? Are app servers not used for anything else? Oh! no, gotta go stop using app servers for all this business stuff...

    yawn
    Reply to this Reply to original

  4. Test results[ Go to top ]

    If you've ever been involved in SOA performance testing, then you'll know that one of the top bottlenecks is going to be the web services stack, and in particular XML parsing. I happen to find these results very informative, and I think a lot of other people would share that thought. When you wake up, you might care to download the test kit and run the test scenarios for yourself.

    Sweet dreams.
    Reply to this Reply to original

  5. Dream sweet dream[ Go to top ]

    When finally we break the association SOA - SOAP -XML?
    Exposed CORBA server is a service, so RMI etc.
    Those numbers are horrible - all of them!!!
    Worst CORBA services will outperform XML based solutions many times over and will provide much friendlier environment for everybody...
    Reply to this Reply to original

  6. Dream sweet dream[ Go to top ]

    When finally we break the association SOA - SOAP -XML?Exposed CORBA server is a service, so RMI etc.Those numbers are horrible - all of them!!! Worst CORBA services will outperform XML based solutions many times over and will provide much friendlier environment for everybody...

    I would beg to differ when it comes to exposing services to third party partners..

    Although I agree with your other point: SOA != SOAP, but SOAP can in many cases be a useful part of an SOA.
    Reply to this Reply to original

  7. Dream sweet dream[ Go to top ]

    Clarification:
    I would beg to differ with regards to the point about "friendlier environment".
    A good document-based SOAP service is very easy for outside people to get to grips with in no-time, regardless of their platform of choice.
    Reply to this Reply to original

  8. Dream sweet dream[ Go to top ]

    A good document-based SOAP service is very easy for outside people to get to grips with in no-time, regardless of their platform of choice.
    1st - in case of Document oriented style SOAP is unnecessary and harmful - any protocol would deliver the payload to way faster than SOAP (simple XML over HTTP would do the trick).
    2nd - there is no guarantee whatsoever that XML document can be easily understood by humans;
    3rd – 99.9999% of the time humans do not look at those documents as communications happens between systems;
    Simply put: those documents are NOT for human consumption and therefore should not be very human oriented;
         there is an empirical proof:
          - widespread adoption and use of WSDL2Java and alike code generators, which allow humans NOT to look at XML documents and work with programmatic interfaces;
         - attempts to standardize ‘binary’ XML: http://www.w3.org/XML/Binary/ - do not you think that this is collective deja’vu and wheel reinvention? CORBA does exactly that for decades and does it very efficiently;
         - XML related stack of specs is much bigger that CORBA and still lacks many CORBA features and way less efficient in implementations;

    Wake up guys, learn from the past and build on achievements!
    Reply to this Reply to original

  9. CORBA Discussion?[ Go to top ]

    I see the kit and think of it as a comparative work of the SOAP part of SOA. It doesn’t say anything against nor in favor of other technologies. It doesn’t discuss if it is better or not. It simply defines the actual types of applications you actually see in the real world made over SOAP, that actual developers are working on now, and test the development issues and server performance.

    The issue with the XML is representation. Document over SOAP could mean binary payloads, but conceptually it means packed information far more complex that an integer, a string or even an array. XML was not thought as a transmission artifact, but a representation one, a flexible format that anyone can understand (I mean no humans, but platforms), but the result verbosity is very high. So, we have info we call a document in XML format which I use locally and want to send it to a service so it can process it. That means we have a transmission problem and a general solution is SOAP. So, which server offers me the best tools and best performance and lower learning curve? That is what the kit wants to answer.

    Is this kit of use for CORBA users? I think not. But I think it may help several developers that do have to work with SOAP due to market standards. I would guess they need to improve its use. Personally, from what I read, I didn’t care too much about the numbers, but there is other info there that I found that goes beyond simple comparisons. I found a deep investigation of actual state of the matter and the journal gives you info that you may need if you have to work with any of those servers. And they do it with the latest server versions! So it is helpful if you work with Oracle, Jboss or Weblogic alike (for this last one, 8.1 and 9.0 flavors are included).

    Bottom-Line: I think it is a good job, despite it doesn’t mention CORBA…

    Will.
    Reply to this Reply to original

  10. Test results[ Go to top ]

    Saw a PDF - full of XML processing results? Are app servers not used for anything else?

    Did we read the same document? There's a lot more in this study than just XML processing. The kit was built to test overall performance of web services across app servers. Naturally, a big aspect of web service performance is XML processing, so for completeness PushToTest covered the different options for how to parse XML. But the point here is the comparison of web services overall across app servers.

    Also, what I find interesting about this approach is that you can repeat the tests yourself in your own environment. If you download the kit, you can run the tests on your app servers of choice and see what the results look like.

    Finally, beyond the question of performance comparisons, the other artifacts that PushToTest created also discuss feature-level comparisons as well as a journal kept by developers detailing how they built the kit on different app servers. It's really interesting to see what the developers did to get things working on each platform.

    John Doppke
    BEA Systems
    Reply to this Reply to original

  11. Sun's evaluation ?[ Go to top ]

    Sun has posted a document on their website based on this bechmark. Interesting read.

    Document can be found at
    http://java.sun.com/performance/

    http://java.sun.com/performance/reference/whitepapers/Java_FastInfoset.pdf
    Reply to this Reply to original

  12. Fast Infoset: Web Services now 4 times faster[ Go to top ]

    Sun Java System Application Server 8.1 focuses on developer productivity by providing a full-featured, high-performance, small-footprint container that is free for development, deployment, and redistribution. In addition to the out of the box technologies (NetBeans, Java Studio Creator, and Java Studio Enterprise), Java System Application Server provides Java Web Services Developer Pack. This is a free toolkit to build, test, and deploy XML applications, Web services, and Web applications with the latest Web services technologies and standard implementations. The new Java WSDP 1.6 release contains Fast Infoset technology that can increase Web services performance upto four times by using ANS.1-based binary encoding that decreases transmission and processing times for messages,compared to XML and ASCII messages with zero code changes (from PushToTest results). For more information visit http://java.sun.com/performance
    Reply to this Reply to original