Discussions

News: ECperf 1.1 released, ECperf to become SpecJAppServer2001

  1. In a major news item, the ECperf 1.1 public draft has been released. The major changes in this draft include the requirement that transactions be consistent (many vendors were running without transactions as a way to get better performance), as well as the announcement that ECperf 1.1 will be repackaged as SPECjAppServer2001, and administered by SPEC.

    Official SPECjAppServer2001 results will be posted on the www.spec.org site, however TheServerSide will also host results on our ecperf.theserverside.com site, and continue to operate the site as a community portal surrounding ECperf, as well as all the news coverage we provide here on TheServerSide news forum.

    Download ECperf 1.1.

    Shanti Subramanyam (ECperf Spec. Lead) provided the following information about the changes:

    ECperf 1.1 has been released for Public Review on Feb 28. The major differences between 1.1 and 1.0 are :

    . ECperf transactions are now required to be consistent. In CMP deployments, transaction consistency can be achieved by any pessimistic or optimistic concurrency method supported by the container. ECperf 1.0 did not have a consistency requirement as at the time it was designed, the only common mechanism available to achieve this was by using database isolation levels. Today, many appservers provide sophisticated concurrency control mechanisms that result in little or no performance degradation. See Clause 4.11.6 in the specification for more details. Because of this requirement, ECperf 1.1 results should not be compared with those obtained running ECperf 1.0.

    . Although ECperf 1.1 can be used internally for performance testing and tuning, no results can be officially published. ECperf 1.1 will be repackaged as SPECjAppServer2001 by the Standards Performance Evaluation Corporation (http://www.spec.org). All result publications must be on SPECjAppServer2001. See Clause 8 for more details.

    Contacts from the various J2EE Vendors will also be submitting comments in the next few days. Pramati had the following to say:

    Ecperf 1.0 was silent about the consistency requirements. With EJB spec also not clear about mandating any concurrency control, the vendors were free to choose any mode. So submissions have happened with absolutely no concurrency control in place (resulting in rampant data-stomping)!

    Among other things, Ecperf 1.1 addresses this by clarifying, from the benchmark standpoint, the EJB reqts around concurrency control. With the expert group winding down its activities, it is the right time to transfer the benchmark to an industry group. SPEC is a natural choice as SPEC is already in the performance benchmarking space, and most of the app server vendors are part of SPEC.
  2. Cool. Does this mean that BEA result is a 'rampant data-stomping' ?
  3. lol...

    Guess it doesn't matter to BEA if the data is valid... just fast.

    Funny the $$/transaction doesn't seem to include the cost of recovering/cleaning the data.

    Glad to see ECperf getting more "real world"...

       -=<>=-
  4. ALL THE ECPERF RESULTS RELEASED TO DATE HAVE THIS ERROR, INCLUDING IBM's ...

    SO GET YOUR FACTS RIGHT!
  5. If the ecperf spec didnt say anything about concurrency control, I am curious to know how BEA & IBM chose relaxed locking to get their results?

    pat
  6. A while ago, i saw a reference to ecperf submissions with data-stomping. Am
    new to the EJBs; can someone tell me if this data-stomping will happen in
    all applications or is it only to do with Ecperf?
  7. Hello,
       I have been checking into both the BEA and IBM results to find any "data stomping" As much as I hoped to find something interesting, I must say that both companies are playing well within the rules. I have seen no cheating whatsoever.
       For those of you who do not understand what all the hype is about, here is my version:
    1. There are 4 DB Isolation Modes, ranging in their level of isolation of the data during a transaction. Most DB applications, including production applications in many companies run in Isolation Mode 2, as does ECPerf, otherwise known as read committed. This allows for data corruption with concurrent transactions, lost updates. Here is an example:
    Trans 1: Read A;
            Trans 2: Read A;
    Trans 1: A=A+1, write A, Commit;
            Trans 2: A=A+1, write A, Commit;
        Transaction 1's update is lost, ie, A=(old A) +1;
       These issues were around long before AppServers, or the WWW for that matter. So to accuse the AppServer Vendors of Data Stomping is naive. Level 4 is supposed to run as if all transactions are serialized, for utmost care of the data. For many years now "a leading DB vendor's" level 4 Isolation mode has had a data corruption bug in it, and no one seems to discuss it.
                  GS

  8. Sure. Databases havent had the best of isolation level implementations. They are written for performance in just one iso level (In Informix- DirtyRead, & in Oracle-CommittedRead.. and so on). But applications rarely relied on iso levels for concurrency control. And we have had extremely mission critical apps that cannot afford any level of data-stomping. They achieved this bu using SLEECT.. FOR UPDTAE locking to control concurrent access and prevent data-stomping!

    In app-servers, this responsibility is on app server (for CMP) as the DB access code is beyond CMP bean writer. And this is where the responsibility is reneged (by not locking the rows that are modified). And data stomping allowed!

    Cheers,
    Ramesh
  9. Hello Ramesh,
        I agree with most of what you said, but where are the AppServers failing with ECPerf? There are configurations that allow for looser (more risky) database consistency, but they are not being used by IBM or BEA. I have carefully checked for this and doubt that I have missed anything (we have all heard that before, please guide me if I have missed something). It seems to me the CMP code calls the DB and does not use the cache at all (except IBM, which I believe is avoiding many DB calls for read only methods). Of course the increased latency increases the chance of a lost update, but with the given architecture and age old DB locking bottlenecks, there is little to be done with the current levels of technology for handling data.
         In summary, the AppServer vendors can not solve the problems that are inherent in RDBMSes. Like any application, they can make them worse: this they must be on guard for.
         I encourage more to discuss this as for large applications this is one of the huge issues.
                  Regards, GS
  10. "I agree with most of what you said, but where are the AppServers failing with ECPerf? There are configurations that allow for looser (more risky) database consistency, but they are not being used by IBM or BEA. I have carefully checked for this and doubt that I have missed anything (we have all heard that before, please guide me if I have missed something). It seems to me the CMP code calls the DB and does not use the cache at all"

    Gary,

    The database isolationlevel used is READ COMITTED. In this mode, database has no responsibility to lock any row being read. Now in the container, the data is read. And the business ops are performed. If any of the fields are updated, the containers dont update the DB until the end of the transaction (when the ejbStores get called). Take an example invooving inventory updates: assuming the business op takes 200 ms. In the beginning the inventory entity bean is loaded. Say at about 50th ms, the inventory count is updated- based on order quantity (reduced from, say 200, to 150). The business op continues to perfrom othertasks. And the tx ends at about 180ms. When the ejbStores and DB commit is performed.

    During this time (50th ms to 180th ms) this particular entity bean instance is not locked in the DB nor is it locked by the container. Any other transaction in the ejb container (or stillworse, some other legacy app operating on the same database) can come and modify theinventory count, assuming current quantity of 200. But when the above tx ends, th edata gets overwritten to 150! Thus data-stomping.

    In ecperf this is gaurenteed to happen. As some of the entity beans (such as inventory) are extremely concurrently accessed. (Only due to this, is ecperf1.1 needing additional concurrency control requirements- which at this point is extremely performance degrading in most vendors as theonly option available is to use higher DB isolation levels that can kill performance. Just an FYI that our submission was with absolute assurrance of concurrency control. No data-stomping at all!).

    Cheers,
    Ramesh
    - Pramati
  11. Hello Ramesh,
         I agree with all you wrote: in fact it looks similar to my previous post but with more detail. However, you mentioned something new that I was not aware of: your published result maintained full data consistency, right? That is worth telling everyone about.
         What muddied the story was the beating up of other AppServer vendors for not doing this. While I agree that it is a very GOOD thing if Pramati can do this better than others (I am very curious to find out how), it is an age old problem that AppServers simply inherited from DBs. Before EJBs existed there were several companies, which I was aware of, that had this same problem with 3-tiered Apps. Using distributed transactions and 2 phase commits (Tuxedo could help, but not radically improve performance) was how this was done in the old days. Since the advent of AppServers, ~6 years ago, this problem has started to appear. "Data Stomping" is a loaded word. If someone realizes that any isolation level below Serializable is to some degree a sacrifice of ACIDity, this problem has long preceded the WWW. What makes it more prounounced is the increased latencies of 3 tiered Apps, and even more prounounced the AppServers, which are much slower than Tuxedo or CICS.
        Ramesh, it is a pleasure chatting with you. Here is my suggestion for your headline (if my assumptions are right): Pramati AppServer tops performance under high data integrity scenario! Now please, on to the rest of the story....