Discussions

News: Persistence Reloaded: JDO or CMP?

  1. Persistence Reloaded: JDO or CMP? (99 messages)

    JDO experts Craig Russell (JDO spec lead) and David Jordan (JDO consultant) have published a concise summary of the differences and similarities between JDO and CMP, based on their new book entitled "Java Data Objects". A handy table summarizing the programming, metadata, modeling, and environmental differences is also provided.

    Read JDO or CMP?.

    Also, see the now 3 year old (and still running) thread Java Data Objects vs. Entity Beans.

    Threaded Messages (99)

  2. ...[ Go to top ]

    Hibernate
  3. Hibernate[ Go to top ]

    Hibernate is not a sufficient solution to the full problem at hand, which is how to persist business entities in a J2EE application. (The same comment goes for JDO.)

    (1) Are you advocating the use of Hibernate to implement Data Access Objects to realize BMP Entity Beans?

    (2) Are you advocating the use of Hibernate to implement business entities that are persisted directly by an EJB Session Facade without the use of Entity Beans?

    (3) Are you advocating the use of Hibernate directly from a Webapp, and eliminating the use of EJBs altogether?

    I use approach (2) today with JDO. The PersistenceCapable objects behave as DAOs. When made transient, they behave as DTOs through the EJB Session Facade. This approach seems to work pretty well, especially when fully code-generated.
  4. Hibernate is good, EJB is good too.[ Go to top ]

    As my opinion, Hibernate(JDO) or EJB is good too, the truth is where to use them. If only require persist object to RDBMS, no cluster, no state replication, Hibernate is good. EJB is "Enterprise" level technology, many non-enterprise apps will not need them. EJB must be deployed in a EJB container, but Tomcat/Resin has no one.

    Which is better, decide on where and when to use them and the requirement.

    Hunte
  5. I have seen so many people specially last 1-2 years, using EJBs for a simple MVC type application. Spending time & money, breaking their heads as to how will they fit EJBs in their. Its more of a fashion than to make the application robust. To some extent we deserve the "DOT GONE" era. ..anyway
       Its good to see books coiming out of EJBs, specially when not to use them and comparing them with other technology. Some chapters of the Bitter EJB book are really good.
       To both the authors of this book and all other good authors of good books - Thanks you for giving us great insight & knowledge to make good decisions in this world full of new technology emerging every day.
  6. In my opinion, JDO & hibernate are also perfect to use in a cluster on enterprise level.

    I am changing the persistency of my open source project from CMP-CMR-EJB2.0 to hibernate because I found that EJB forces me to make my application app-server-specific. The main reason is that EJB-QL is not powerfull enough. Because all app-servers provide non-standard extentensions, I'm forced to use them.

    By using hibernate I have a powerfull query-language and my application only needs to use JDBC from the application server. I found out that my (rather large) app-server-specific deployment descriptors could be removed and I got a more portable application.

    As a programmer, I found it easier and more natural to write the persistency using hibernate. I don't agree with people that say that XDoclet (which is a very usefull project) or expensive IDE's can simplify the work which is made complicated by cmp-ejb's.

    Tom Baeyens
  7. Hibernate and SSB[ Go to top ]

    Hi Tom,

    I agree with you 100%! I also want to replace many of my CMPs with Hibernate and iBatis. I think, EntityBean, JDO, OR/Mapping (Hibernate), SQL/Mapping (iBatis) have their own purposes. Just use the right tool on the right place ;-)

    Lofi Dewanto.
    http://openuss.sourceforge.net
    http://ejosa.sourceforge.net
  8. Hibernate[ Go to top ]

    Interesting that the CMP v JDO article is written by two guys that are writing a JDO book. I wonder which they prefer :)
  9. Interesting?[ Go to top ]

    Interesting that the CMP v JDO article is written by two guys that

    > are writing a JDO book. I wonder which they prefer :)

    More interesting that one of them (Craig Russell) is responsible for the SunONE Application Server's CMP/CMR implementation which, by the way, is implemented under-the-covers using JDO! I believe this makes them extremely well qualified to discuss the topic.

    Kind regards, Robin.
  10. Interesting?[ Go to top ]

    Why would one wants to use JDO to implement the CMP container services? Are't you duplicating most of the services and ending up incurring twice the overhead?

    Instead of Struts Action->SLSB->Hibernate, imagine you do Struts Action->SLSB->CMP->JDO...
  11. Interesting?[ Go to top ]

    Why would one wants to use JDO to implement the CMP container services? Are't you duplicating most of the services and ending up incurring twice the overhead?


    application servers can use JDO.

    >
    > Instead of Struts Action->SLSB->Hibernate, imagine you do Struts Action->SLSB->CMP->JDO...

    Why not ?
    Struts Action -> Business Delegate -> Business using JDO
  12. Interesting?[ Go to top ]

    Why would one wants to use JDO to implement the CMP container services? Are't you duplicating most of the services and ending up incurring twice the overhead?

    No. JDO and CMP are specifications and APIs. There is nothing inherent to them other than that. JDO as a CMP engine makes good sense to me from a high level POV. Don't be too surprised to see a "major" app server vendor do just that.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Easily share live data across a cluster!
  13. Interesting?[ Go to top ]

    OK, so what is the benefit of using JDO vs. JDBC by a particular CMP container?If it is such a great idea, how come WebLogic/WebSphere and especially JBoss has not done it? Purely because of political reasons?
  14. Interesting?[ Go to top ]

    OK, so what is the benefit of using JDO vs. JDBC by a particular

    > CMP container?If it is such a great idea, how come WebLogic/WebSphere
    > and especially JBoss has not done it?

    JBoss 4.x will supposedly use AOP Persistence (which is basically the same as JDO, but may or may not support the JDO spec) for CMP.
  15. JDO & CMP[ Go to top ]

    Cameron,

    > No. JDO and CMP are specifications and APIs. There is nothing inherent
    > to them other than that. JDO as a CMP engine makes good sense to me from
    > a high level POV. Don't be too surprised to see a "major" app server
    > vendor do just that.

    Hehe, I thought Sun already did that? But talking about CMP engines - a good CMP engine for instance separates the persistence aspect from the transactional aspect of things.

    Our persistence engine (CMP) does not manage transactions - we have our transaction manager for that; but the CMP engine is aware of the start (and end) of transactions. In that sense, you could quite easily rip off the persistence manager and replace it with JDO.

    But then you'd have to ask yourself - why would you do that? For one, you could argue that your APIs are standardized; but that would only be useful if you want your persistence manager to be used outside of an AppServer environment. And the other reason (my personal opinion) would be marketing hype.

    My guess would be that the use of JDO as an API would be just that - an additional layer. How the implementation is - would be pretty much the same underneath - as it always used to be. Or the vendor would have to rewrite/redesign major portions of their CMP engines; and I'm not too sure if there would be any 'value add' (apart from another buzzword you can claim to support ;)

    -krish
  16. JDO & CMP[ Go to top ]

    As far as I know SunOne Application Server will use JDO as the underlying strategy for their CMP implementation. However I am not aware of any plans to expose the JDO layer to clients. In this regard they must be using JDO to underpin CMP because it makes sense technically to do so, since there is no direct "we support JDO" marketing benefit. That is they must perceive concrete benefits, perhaps time to market, perhaps performance, perhaps flexibility of this implementation and its ability to support value-add for users of CMP (e.g. additional vendor-specific features).

    Of course I might be wrong. Perhaps Sun does plan to expose the JDO API to non-CMP clients. Perhaps they do hope to win marketing points by doing so. But it is well demonstrated that JDO can be bundled with an app server without the developers of that app server having to use it for their CMP implementation. So once again it must be being done for its technical merit.

    Kind regards, Robin.
  17. JDO & CMP[ Go to top ]

    Cameron: JDO as a CMP engine makes good sense to me from a high level POV. Don't be too surprised to see a "major" app server vendor do just that.

    Krishnan: Hehe, I thought Sun already did that?

    Hehe, just wait 'til a "major" app server vendor does it. ;-)

    Krishnan: But then you'd have to ask yourself - why would you do that? For one, you could argue that your APIs are standardized; but that would only be useful if you want your persistence manager to be used outside of an AppServer environment. And the other reason (my personal opinion) would be marketing hype.

    Or we could ask ourselves why certain companies have avoided supporting JDO, even though they have two representatives in the expert group! ;-)

    To answer your question though, for one, so you could use the same O/R mapping in your applications whether you were using CMP Entity Beans or just Servlets accessing a data store. Makes sense to me.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Easily share live data across a cluster!
  18. Accuracy[ Go to top ]

    Hi Aaron

    The acknowledgements section of the book lists Linda DeMichiel, Sun Microsystem's EJB specification lead as a reviewer. She and Craig Russel are co-workers at Sun. So it is not likely that the JDO and CMP comparison is anything other than fair.

    Cheers
    David
  19. But is Hibernate JDO[ Go to top ]

    But from what I can tell, Hibernate isnt JDO. Its an OR mapping tool, which
    is quite different from using JDO or CMP (ie the context of this thread)

    /s
  20. No, but so what?[ Go to top ]

    Hibernate is neither JDO or CMP.

    However, in my opinion it provides a better solution for object persistence and querying than either JDO or CMP. It is also easy to use Hibernate with an application server.
  21. 1. CMP can make use of JDO implementations to persiste data.
    2. JDO is not J2EE. It can be used inside or outside J2EE app servers
    3. JDO is faster than CMP.
    4. CMP can be used remotely.

    Both are to persist data, but for different needs.
  22. CMP remotely????[ Go to top ]

    "...4. CMP can be used remotely. "

    Is somebody using EB remotely without SB ???

    IMHO, it is against a general OOP idea:
    Ask for service, do not ask for data.

    Is there any practical use cases where
    EB interface can be wide-open public for
    remote usage?

    Thanks in advance.

    Alex V.
  23. CMP remotely????[ Go to top ]

    I dont want flame, but to me CMP is useless after JDO.
  24. CMP remotely????[ Go to top ]

    I disagree. Accessing Entity Beans remotely is a nice feature,
    the only problem is that it doesn't perform.
  25. Remote JDO[ Go to top ]

    Hi All

    As David and Craig point out the JDO spec does not define any kind of remoting. It sticks to persistence. This is great. However it is possible to implement remoteness under the covers within the JDO API.

    JDO Genie (a JDO implementation) supports PersistenceManager's running in different VMs to the JDO Genie server. The remote PMs have access to the full JDO API. The same business code can run in a client VM (e.g. Swing GUI app) or on the server (web or ejb) without changes. JDO Genie hides all the communication with the server. This makes it possible to build simple multitier applications quickly and efficiently as you do not have to do value object transfers or worry about defining remote interfaces or whatever.

    Cheers
    David
    JDO Genie - High Performance JDO for JDBC
  26. Remote JDO[ Go to top ]

    But why reinvent remoteness when the stateless sesion beans do all this for you in a tried and tested manner?
  27. Remote JDO[ Go to top ]

    Hi Aaron

    JDO instances lose their association with the PM when they are Serialized. So your client app can only get transient instances (value objects) from the session beans. This may force you to send data that is not used by the view or otherwise construct data transfer objects. You may also need to send the JDO object identity of any instances you want to update later (if not using application identity).

    To do updates the client has to send the value objects and OIDs or DTO back with the changed data. The session bean has to retrieve the original JDO instance(s) and copy in the new data. This is great for decoupling but makes a lot of work for simple apps. Again it might not be easy for the client and session bean to figure out exactly what changed and extra data may be sent and/or updated.

    With remote PMs the client uses the JDO API. Only data actually touched is fetched from the server. JDO automatically tracks changes in the transaction and only sends the changes back to the server to persist. This makes it much easier to write simple apps and may be more efficient.

    Cheers
    David
  28. Remote JDO[ Go to top ]

    JDO instances lose their association with the PM when they are Serialized. So your client app can only get transient instances (value objects) from the session beans. This may force you to send data that is not used by the view or otherwise construct data transfer objects. You may also need to send the JDO object identity of any instances you want to update later (if not using application identity).


    David. This is very observant of you. This is one of the many reasons we chose not to implement JDO in CocoBase, and instead produce our own shared-source Transparent Persistence architecture and implementation. Not only does CocoBase TP handle this distributed functionality AUTOMATICALLY, it does it with a either a local or distributed Generic Session bean implementation, and it even supports transparent dynamic lazy loading (locally or even distributed with the session bean implementation), and it automatically integrates with the JTS and Connection pools of standard application servers. We had intended to support JDO, but 1.0 was too ODBMS derived, and many of the design choices make a lot of sense with ODBMS/C++ non-distributed applications written in the early 1990s, but they seem totally unrelated to Java or distributed architectures. But that would be a different thread wouldn't it? :)

    CocoBase TP also works out of the box with EJBQL (both with Entity Beans and objects that are standard java objects and not ejbs), and Query By Example. And because CocoBase isn't encumbered by many of the other design flaws of either JDO or CMP, our implementation actually often out-performs raw JDBC issuing identical SQL across the board functionality wise(select/ insert/ update and delete)! I'm not aware of any other O/R tool that even comes to that level of efficiency or performance. This is a result of our implementation architecture and JIT/JVM optimizations and having the maturity of being the first and oldest O/R tool in java. That experience means our customers get to benefit from a mature product, and not just a 1.0 (or even 2.0) specification produced largely from less experienced sources...

    >
    > To do updates the client has to send the value objects and OIDs or DTO back with the changed data. The session bean has to retrieve the original JDO instance(s) and copy in the new data. This is great for decoupling but makes a lot of work for simple apps. Again it might not be easy for the client and session bean to figure out exactly what changed and extra data may be sent and/or updated.

    With CocoBase this isn't necessarily. Because we generally deploy our Distributed TP through a 'stateful' session bean, we automatically manage and reconcile data with optimistic locking based on a validated starting context that doesn't require this burdensome, error prone and slow re-read of data. The developer simply calls cbfacade.save(obj) with their root node (or any node in their graph) and CocoBase does the magic. It detects only changed relationships and changed fields, and producing optimistic locking and a very high performance including automatic cascade & graph management. And it also integrates fault tolerant caching that produces performance increases up to 400+ times (that's times and not percent, it's 40,000 percent!) faster than raw jdbc access.

    Oh and for customers that have to have Entity Beans, CocoBase supports those (BMP & CMP for most servers), but CocoBase adds dependent object support and table spanning instances, stored procedure support, distributed caching and a variety of other features as well.

    >
    > With remote PMs the client uses the JDO API. Only data actually touched is fetched from the server. JDO automatically tracks changes in the transaction and only sends the changes back to the server to persist. This makes it much easier to write simple apps and may be more efficient.

    It's even easier with CocoBase, because you can get the benefit of both. Most customers think of java development being an either/or with EJBs or Transparent persistence. If the developer has chosen to use JDO 1.0 (a premature, outdated ODMG derived, non-distributed and ODBMS centric specification), this is probably going to be true. If however a mature Transparent Persistence implementation such as CocoBase is used, it allows developers to go from 2-tier to 3-tier 'transparently' through a connection parameter change. This means having to choose standalone vs. j2ee is a runtime decision, and it also means reduced code duplication, shared mapping, etc. etc. etc.. With CocoBase you also have more advanced TP functionality such as automatic cascade management, EJBQL querying, query by example, user manageable caching, stored procedure support, 100% tuneable SQL, JTS transaction integration, J2EE standard connection pool integration, REAL optimistic locking - not just the mis-labeled pessimistic-re-read in the JDO Spec - and an O/R layer that runs FASTER than hand coded jdbc.

    We're going to be releasing our new Service Release 3 of CocoBase 4.5 in the next few days for Java One, and we are including a new performance test suite (as source code so developers can see for themselves the jdbc & SQL code is valid) and the numbers we're getting are that for a data sets of size 1000, we're almost twice as fast as JDBC once the system is up and running and startup overhead and JIT compilation occurs. This is without any caching enabled, just identical SQL and side by side issuance.

    So if you want to debate specs, and philosophies of APIs & designs this is a great forum for doing it. If your back is against the wall, and you need a product that works well and performs out of the box - then just go download CocoBase. We provide a free 30 day evaluation, and you can prove it to yourself in your own environment.

    Good luck to all of you, and may we all hope that JDO 2.0 and EJB 3.0 makes this all an irrelevant discussion :)

    Ward Mullins
    CTO
    http://www.thoughtinc.com

    >
    > Cheers
    > David
  29. Cocobase[ Go to top ]

    Ward wrote:

    If the developer has chosen to use JDO 1.0 (a premature, outdated ODMG derived, non-distributed and ODBMS centric specification),

    ----

    Wow - talk about tired, ranting and raving. I used to think I might consider Cocobase as a persistence solution, but this attitude does nothing but steer me away. I'm currently going through an evaluation of several JDO implementations, and I'm very happy with what I've seen so far. I'm finding that, in general, any issues that JDO may have are being addressed by all the vendors in one way or another - for example, multiple runtime fetch groups. I won't be surprised to see these features make it into the next JDO spec.

    Ward, I'll be frank with you. I think you and Cocobase are going to have to change your tune or the market is simply going to let you die. If you've got beefs with JDO, then you need to take it up with Sun. I think everybody is pretty much sick and tired of hearing your complaining here.

    God bless,
    -Toby Reyelts
  30. Remote JDO[ Go to top ]

    Hi Ward

    "Not only does CocoBase TP handle this distributed functionality AUTOMATICALLY, it does it with a either a local or distributed Generic Session bean implementation, and it even supports transparent dynamic lazy loading"

    JDO Genie's support for remote persistence managers does all of this within the standard JDO API and without any session beans. The technology can be used in any sort of application not just EJB. It is easy to add a Swing GUI client to an existing web application or to make a simple 2 tier application 3 tier with no code changes and no application server or session beans.

    "We had intended to support JDO ... many of the design choices make a lot of sense with ODBMS/C++ non-distributed applications ... seem totally unrelated to Java or distributed architectures"

    There is nothing in the JDO spec that excludes support for remote persistence managers. Most available JDO implementations map to relational databases so JDO is certainly not just for ODBMS. JDOQL is designed so it can be translated into efficient SQL.

    "Because we generally deploy our Distributed TP through a 'stateful' session bean, we automatically manage and reconcile data with optimistic locking based on a validated starting context that doesn't require this burdensome, error prone and slow re-read of data."
    "If however a mature Transparent Persistence implementation such as CocoBase is used, it allows developers to go from 2-tier to 3-tier 'transparently' through a connection parameter change."

    Again this is exactly what you get with remote persistence managers. And you get to use the standard non-proprietary JDO API without having to deploy any session beans. All you have to do is specify the name of the server and your 2-tier app is now 3-tier.

    "REAL optimistic locking - not just the mis-labeled pessimistic-re-read in the JDO Spec"

    I don't understand what you mean here. Optimistic transactions in JDO certainly do not require re-reading of data on commit.

    Cheers
    David
    JDO Genie - High Performance JDO for JDBC
  31. JDO is so easy to use[ Go to top ]

    My eagerness to try JDO was motivated primarily by my CMP 2.0 experience. JDO has been so much easier to use. No more hacking around Enity Bean's lack of inheritence.

    One of the BOF's at Java One will cover the implementation of the RUBiS benchmark using JDO. Two previous versions of the RUBiS benchmark (published on TSS and elsewhere), covered the performance of JDBC and CMP. The comparison withy JDO will be interesting, especially the lines of code count, and the performance.
  32. What is the best JDO implementation?[ Go to top ]

    Open Source or not.
  33. Beset JDO Implementation[ Go to top ]

    I would like to re-phrase my question. I never spent time with JDO beyond reading articles such as this and by now, I am persuaded that I must check it out.

    If anyone will be so kind and point out a good JDO implementation (open source or not) that I can use to kick off my learning experience.

    Appreciated.
  34. Beset JDO Implementation[ Go to top ]

    I have pretty pleasant experience with KODO http://www.solarmetric.com/ .
    I changed Castor based persistence layer in my application to Kodo-JDO based in less than 8 hours and I never used any JDO before.
  35. Best JDO Implementation[ Go to top ]

    Hi

    You can have a look at JDO Central. JDO Central is a vendor independent JDO portal sponsored by several JDO vendors. The discussion forums cover newbie and advanced topics.

    Cheers
    David
  36. Great comparison. I love big pictures. To me the big difference is the development time and learning curve of a persistence layer. Before remoting *was* the problem but performance rants seem to have quieted down since local interfaces. To me the learning curve argument has lessoned with the use of XDoclet which does CMP/JDO/Hiberant. XDoclet gets CMP back to being as close to a POJO as it is ever going to be. Switching costs (mental and monetary) are reasonable.

    Maybe the better way to approach CMP/JDO/Hibernate is to compare them against a common set of scenarios covering complexity/performance/development speed/budget/politics/scaling/HA/yadda yadda.

    cheers
  37. IMO: JDO is better than CMP, but it does not mean that EntityBeans do not make sense.
    My conceptual classification:
    - 'Hello World' type persistence – JDO, CMP, Castor, etc.
    - Object-RDBMS mapping tools: Hibernate, CocoBase, TopLink
    - BMP – EntityBeans – enterprise level persistence, which may use different persistence techniques under the hood.

    CMP != EntityBeans != EJB
  38. It is not news that EJB should not be used for persistance, but we can do better than JDO and better than JDBC.
    It should say O/R or E/R is the big question.
    Some people with simple small applictions can do O/R such as JDO.

    E/R (Entity Relationship)or SQL based DAO's include iBaits, Scaffolding, CommonsSQL, etc. Look, before E/R - SQL DAO's major applications used JDBC so that they can write complex/fast applications with SQL.

    I think anyone who thinks that you can write complex/fast applications without masterring SQL is not realistic, so my 2 cents is that any O/R based, such as JDO/Hibrenate will have to be replaced by a SQL E/R DAO.

    If you used to do EQL, or HQL, no reason to do OQL, when you can do ANSI SQL once and not have to swtich arround.

    .V
  39. I think anyone who thinks that you can write complex/fast applications without masterring SQL is not realistic, so my 2 cents is that any O/R based, such as JDO/Hibrenate will have to be replaced by a SQL E/R DAO.


    you can use SQL 92 with some JDO implementations. Advantage ? When you want to persist data you have only one API, doesnt matter if your datastore is a file system, RDMBS, ODMBS, ERP, and whatever you want.
  40. Of course, if anyone who is realy looking to read and write DB data from a file for a web app, they should use JDO, so they have OQL.

    But.... there are no such theoretical people, since everyone imports the file to db and uses it from there, so that they can have full power of SQL.

    If you are saying that there should be a DAO above JDO so that you can use say MQ or other feeds, that is right. A DAO could hide if the Collection came from a DB or from MQ, or other source.

    Of course, JDO provides nothing for using MQ, or anything but a DB, but we should not use it for a db. Only a file by theoretical people. Look, ADO.net is taking market share away from J2EE, and we have to show cheaper/faster here, or learn C#.

    So to keep it real, JDO brings little to the table, and OQL should be avoided, in favor or SQL. Since SQL is not a stadard part of JDO and never will be (OQL), using non standard JDO is OK, but... . what's the point then. It helps if people preaching actualy have experience deploying a large comercial site.
    The only thing that is a plus for a JDO is that EJB is so bad, that JDO is better than EJB.

    Also, there is a JDO mail list, the only one I know of:
    http://www.basebeans.net:8080/mailman/listinfo/jdo

    I use E/R SQL DAO from iBatis called dblayer happily, before that I was forced to code a JDBC layer to get SQL power.
    I urge architects to leverage the SQL E/R power for larger, more complex sites. Certanliyt move of the expenseive, slow, complex EJB, no need for it. But instead of going O/R (similar to EQL) go E/R.

    Cheers,
    .V
  41. SQL vs. JDOQL[ Go to top ]


    I use E/R SQL DAO from iBatis called dblayer happily, before that I was forced to code a JDBC layer to get SQL power.
    I urge architects to leverage the SQL E/R power for larger, more complex sites. Certanliyt move of the expenseive, slow, complex EJB, no need for it. But instead of going O/R (similar to EQL) go E/R.
    <<<<<<<<<<<<<

    Vic,
    Yes. Why learn a new query language JDOQL when you have SQL. In effect JDOQL is going to make you treat the RDBMS as a black box. No enterprise application can scale with treating the database as a black-box. JDO may be good for some smaller applications. But when it comes to the true enterprise class aplications SQL is the way to go. And IBatis approach may be a good one.

    If portability is a real question, we have a product called SwisSQL which converts SQL from one databse dialect to another. A combination of IBatis approach with SwisSQL may be both good for productivity, protability and scalability.

    John
    Vembu SwisSQL
    www.vembu.com
  42. Interesting what was the reason (in general) to introduce own language for data access in CMP and JDO ?
    I think even the restricted SQL (but still the SQL) would be better here

    Dmitry Namiot
    Coldbeans
  43. Interesting what was the reason (in general) to introduce own language for data access in CMP and JDO ?

    > I think even the restricted SQL (but still the SQL) would be better here
    >
    > Dmitry Namiot
    > Coldbeans
    The answer is simple. These so called Object/Relational mappers are not truly Object/Relational, but Object/Tabular.
    The relational model establishes several types of relations: base relations (tables in SQL), views, snapshots, result set, intermediate result sets, temporary relations.
    The only mechanism that RDBMS provide for accessing these relations (any type) is SQL.
    The problem with "O/R" mappers is that they do not see result sets as relations, and thus require another nonstandard (SQL-like) language to provide for the lack of support for result sets.
    A true O/R mapper should allow mapping to any type of relation, and the closest is have seen IMO is iBatis. SQL/Mapping is the only way to provide true O/R mapping for a SQL RDBMS.

    BTW, I personally think CMP is an atrocity and JDO is ugly (though not as bad as CMP), but there are some (pseudo) O/R mappers that may be good, especially when a straightforward mapping is possible. One of them is Hibernate (at least version 2.0, I never tested 1.x), TopLink looks like a well designed product too although I have not used it (only have seen some docs).

    Regards,
    Vegeta
  44. <quote>
    BTW, I personally think CMP is an atrocity and JDO is ugly (though not as bad as CMP), but there are some (pseudo) O/R mappers that may be good, especially when a straightforward mapping is possible. One of them is Hibernate (at least version 2.0, I never tested 1.x), TopLink looks like a well designed product too although I have not used it (only have seen some docs).
    </quote>

    You may want to check out JDX, a lightweight yet powerful O/R Mapper from Software Tree. JDX provides an elegant, non-intrusive and flexible mapping solution for simple cases as well as complex situations involving relationships, collections and class-hierarchies.

    Interestingly, instead of inventing a new query language, JDX leverages the full power of SQL in predicate specification with useful extensions like path-expressions. So, the predicate is defined in SQL syntax that uses class attribute names, and path-expressions which are more natural to specify at the object model level. JDX internally translates the predicate as per the mapping specification to an appropriate WHERE clause. Thus you get the best of both worlds - object and relational.

    JDX 4.0 was released recently with many innovative features and an enhanced JDXStudio, a highly intuitive GUI that simplifies mapping specification, verification, schema generation, and reverse engineering Java classes from existing schema. Here is a link to the related theserverside thread:


    -- Damodar Periwal
    Software Tree, Inc.
    Simplify Data Integration
    http://www.softwaretree.com
  45. I'd like to pick up on two points that have been put forward by others.

    Firstly it was inferred that JDO will use the lowest-common-denomenator of SQL in order to achieve portability across databases.

    This is not true. Each JDO implementation is at liberty to use DB-specific sql if appropriate. My experience is that the established commercial implementations, specifically those which are targeting the enterprise market, are using DB-specific SQL to good effect in order to achieve scalability. Portability is in no-way compromised by this performance enhancement which occurs within the JDO implementation.

    Secondly someone asked what the high level benefits of JDOQL were.

    One which I find very useful is that the reslt of a JDOQL Query is a collection of persistent objects. That is a true java.util.Collection, and the contents are domain objects that were previously persisted and now meet the query filter. This is fully polymorphic. It is also dynamic, as is SQL, so there's no requirement to pre-define static query paths as there is in CMP.

    Kind regards, Robin.
  46. It is not news that EJB should not be used for persistance, but we can do better than JDO and better than JDBC.

    > It should say O/R or E/R is the big question.
    > Some people with simple small applictions can do O/R such as JDO.
    >
    > E/R (Entity Relationship)or SQL based DAO's include iBaits, Scaffolding, CommonsSQL, etc. Look, before E/R - SQL DAO's major applications used JDBC so that they can write complex/fast applications with SQL.
    >
    > I think anyone who thinks that you can write complex/fast applications without masterring SQL is not realistic, so my 2 cents is that any O/R based, such as JDO/Hibrenate will have to be replaced by a SQL E/R DAO.
    >
    > If you used to do EQL, or HQL, no reason to do OQL, when you can do ANSI SQL once and not have to swtich arround.
    >
    > .V
    I think you are using the wrong temrinology here (almost everyone does here, anyway).
    E/R modeling is not directly related to SQL or the relational model. It is a modeling technique that provides a procedure to express the model as a set of tables (base relations), but it does not introduce the abstract concept of "relation" the relational model does. E/R modeling does not provide a theoretic foundation to express queries. It only expresses entitiy sets and relationship sets that express relationships among entities. CMP is completly based on E/R modelling (you have Entities and Relationships). In the relational model the relevant concept is the relation (there is no distinction bewteen entities or relationships). Hibernate is also very E/R like, but is more object oriented and much, much better designed.
    To me, what you call O/R should be called E/R or pseudo O/R or O/T (Object/Tabular).
    And what you call E/R or SQL mapper is what should really be called O/R.
    I also agree completely with you on that there should be no need to use a proprietary non-standard query language instead of SQL to access a SQL database.
  47. SQL vs JDOQL[ Go to top ]

    Hi all

    "I also agree completely with you on that there should be no need to use a proprietary non-standard query language instead of SQL to access a SQL database."

    The problem is that SQL is very proprietary and definitely non-standard. Very simple statements are portable but as soon as you start with outer joins and blobs and so on things are different for every database.

    JDOQL is a higher level more productive query language. JDOQL queries are basically normal Java boolean expressions with some restrictions to ensure they can be expressed in SQL. The JDO implementation is responsible for generating the SQL to match the database in use. The implemention also has to work around all the nasty incompatibilities and bugs in JDBC drivers.

    It is true that most applications are written for a single database. However it is common for different projects to use different databases. You may be using Oracle today, Sybase next month and later MySQL. All of these databases are very different.

    Cheers
    David
    JDO Genie - High Performance JDO for JDBC
  48. SQL vs JDOQL[ Go to top ]

    Hi all

    >
    > "I also agree completely with you on that there should be no need to use a proprietary non-standard query language instead of SQL to access a SQL database."
    >
    > The problem is that SQL is very proprietary and definitely non-standard. Very simple statements are portable but as soon as you start with outer joins and blobs and so on things are different for every database.

    You are right in that different vendors provide proprietary SQL extensions, but I think porting from a JDO product to another is harder than porting outer join/lob statements from a database to another. And porting from a JDO product to another is more likely to happen than porting from a database to another.

    >
    > JDOQL is a higher level more productive query language. JDOQL queries are basically normal Java boolean expressions with some restrictions to ensure they can be expressed in SQL. The JDO implementation is responsible for generating the SQL to match the database in use. The implemention also has to work around all the nasty incompatibilities and bugs in JDBC drivers.

    Can you elaborate on why JDOQL is more productive than SQL? I have not used JDOQL. Could you give several concrete examples that show JDOQL being really more productive than SQL? Please, also consider performance and capability issues. There is no point in investing in Oracle to use it as if it supported only what MySQL supports.
    What if I have an application that requieres the use of a proprietary database SQL extension that is not supported by JDOQL, What does JDO do in that case? Some (pseudo)O/R mappers (like Hibernate) provide some hooks to use SQL more or less easily.


    >
    > It is true that most applications are written for a single database. However it is common for different projects to use different databases. You may be using Oracle today, Sybase next month and later MySQL. All of these databases are very different.

    It is more common that you consider using JDO X today, JDO Y next month and later Hibernate or TopLink or Cocobase. Of course, this is just my opinion. Changing databases frequently in a production environment does not look like a recommended practice to me (maybe it is in a testing environment).

    Regards,
    Vegeta
  49. SQL wrapping rascals[ Go to top ]

    As a reformed DBA, I'm biased towards a solution that keeps developers aware of SQL and query optimization. While developers may be able to move faster in an object paradigm, long term support suffers as it's harder to track down what the wrapper is doing. DBAs get saddled with the task of debugging O/R wrapper SQL. When performance heads south, what refactorings do you have available in any of these tools. You move into a world of straight JDBC, stored procs, wrapped objects, etc... Each solution solves about 80-90% of problems leaving the last bit as some other fall back solution like straight JDBC or vendor extension. People are arguing the last 10-20%.

    I'd like to see the J2EE spec come out with peer options to CMP to include JDO and Fast Lane Reader pattern (JDBC or stored proc). If JDO was in the spec how would this argument change. If Apache supported an EJB container where would we be?
  50. Why I like Hibernate[ Go to top ]

    As a reformed DBA, I'm biased towards a solution that keeps developers aware of SQL and query optimization. While developers may be able to move faster in an object paradigm, long term support suffers as it's harder to track down what the wrapper is doing. DBAs get saddled with the task of debugging O/R wrapper SQL. When performance heads south, what refactorings do you have available in any of these tools. You move into a world of straight JDBC, stored procs, wrapped objects, etc... Each solution solves about 80-90% of problems leaving the last bit as some other fall back solution like straight JDBC or vendor extension. People are arguing the last 10-20%.


    That's why I like Hibernate:
    http://hibernate.bluemars.net/hib_docs/reference/html/best-practices.html
    "Use hand-coded JDBC in bottlenecks.

        In performance-critical areas of the system, some kinds of operations (eg. mass update / delete) might benefit from direct JDBC. But please, wait until you know something is a bottleneck. And don't assume that direct JDBC is necessarily faster. If need to use direct JDBC, it might be worth opening a Hibernate Session and using that SQL connection. That way you can still use the same transaction strategy and underlying connection provider. "

    You can use Hibernate query language 90% and fall back to the raw SQL when it's REALLY necessary.
  51. SQL vs JDOQL[ Go to top ]

    1) JDOQL is datastore agnostic. This means that you can easily change the persistence mechanism to XML, flatfile, ODBMS, MIDP RecordSet without spending a week or more just learning the basics of a datastore's language

    2) JDOQL is Java centric. You do not need to pay for developers who are both SQL and Java aware. The Java that you and we all know is the same language used to query the datastore.

    3) Using SQL, while powerful, can cripple the performance of most large scale applications if done poorly. What percentage of developers understand properly batching statements, properly managing datasets as to not update entire rows when there is only a small change, caching datastore hits, efficient blob handling, etc? Forget about complexity at the object level such as inheritance, sophisticated object locking mechanisms, etc.

    JDO provides a high level of performance and scalability for most every application. In addition, the benefits gained in rapid development, low maintenance costs, as well as deployment flexibility far outweigh the perceived performance loss (which is more or less nonexistent as most JDO implementations support raw SQL when necessary). Changing databases is not an uncommon practice... how many times have any of us had clients requesting that your application run on X DB which they have partnerships with? Rebuilding your application from the dataobject level up, vs. simply changing a .properties file in JDO provides huge benefits.

    In a proper -Object- oriented application, using JDO can be far more efficient than writing pure SQL. Object, query, and relational caching mean that your application can run without touching the datastore the majority of the time.

    --
    Steve Kim
    SolarMetric, Inc.
    http://www.solarmetric.com
  52. SQL vs JDOQL[ Go to top ]

    Stephen,

    I really liked your comments about JDO.

    I'm trying JDO first time, and I'm very happy with it.

    While using JDO, I think OO. It's sure JDOQL is not the fatest, but using JDO I can increase the general performance of application by spending time in OO logic and not in queries.
  53. OJB is the wave of the future.[ Go to top ]

    I have been using OJB and it rocks. Getting it to work is a little tricky. Once you have it going, good bye CMP. the remote aspect of entity beans itself a convoluted concept because of session facade pattern. CMPs are extremely hard to write deployment descriptors for and no tool generates dependable deployment descriptors. They all suck. Container managed relationships is a pain in the ass to deal with.
    I am glad I don't have to use that crap anymore.
  54. SQL vs JDOQL[ Go to top ]

    \Kim\
    1) JDOQL is datastore agnostic. This means that you can easily change the persistence mechanism to XML, flatfile, ODBMS, MIDP RecordSet without spending a week or more just learning the basics of a datastore's language
    \Kim\

    For a different perspective, look at things from the point of view of large development team writing enterprise applications. And consider the performance, business, and integration issues such an enterprise app is likely to entail.

    You really think such people are going to consider swapping out their Oracle database for a flatfile, XML, or ODBMS? Think of integration - most enterprise apps must share a database with many other systems and tools (for a trivial example, think of report writers like Crystal Reports). Think of performance - many apps are tuned for fairly specific requirements. You don't just pull out DB2 or Oracle and swap in an ODBMS and expect similar performance results.

    The bottom line is that the fundamental datastore behind you is _important_ to know. Not just from a Java perspective, but a business one. It may seem technically neat to be able to read from an XML file and Oracle with the same code, but it's inherently uninteresting for many large apps.

    \Stephen Kim\
    2) JDOQL is Java centric. You do not need to pay for developers who are both SQL and Java aware. The Java that you and we all know is the same language used to query the datastore.
    \Stephen Kim\

    This concept is, in my experience, a huge mistake which will bite teams in the ass at some point in time. I shudder when I think of developers working with a database who don't know SQL. It's fine when everything works, but when something breaks or is a performance hog, such people are going to be completely clueless and utterly worthless in finding the problem. Real developers working on real applications need to fire up the old sqlplus command line or RDBMS GUI tool and actually see what its application's SQL is doing and how the database is reacting to it.

    What you're advocating is developer's using a complex and expensive resource blind.

    \Stephen Kim\
    Changing databases is not an uncommon practice... how many times have any of us had clients requesting that your application run on X DB which they have partnerships with? Rebuilding your application from the dataobject level up, vs. simply changing a .properties file in JDO provides huge benefits
    \Stephen Kim\

    Why on earth would you need to rebuild your application from the dataobject level up? SQL can easily be configurable in plain old JDBC or EJB use, which eliminates trivial syntax issues (e.g. don't hard code all your SQL in your Java source). Views can be very handy (although I can see how a none-database aware person wouldn't know about those :-). Yes, there may be some changes in code from database to database, but an app developer can generally handle them without much muss or fuss. And frankly I wouldn't trust a JDO vendor to always do the right thing with a given database e.g. a JDO implementation will have bugs, and those bugs _will_ differ from database to database and vendor to vendor!

    \Stephen Kim\
    In a proper -Object- oriented application, using JDO can be far more efficient than writing pure SQL. Object, query, and relational caching mean that your application can run without touching the datastore the majority of the time.
    \Stephen Kim\

    Either one can be "more efficient", depending. And I think you're discounting many real environments where JDO-style caching is effectively impossible - because in many real environments there are _multiple_ writers to the database. Or the environment is large enough with enough volume that syncing a cluster of app servers to ensure cache coherence is problematic compared to just trusting the RDBMS.

    JDO may have its uses, and there are places where it makes sense, but I think the reasons you've stated are the worst possible ones to put forward.

        -Mike
  55. SQL vs JDOQL[ Go to top ]

    Mike: "You really think such people are going to consider swapping out their Oracle database for a flatfile, XML, or ODBMS?"

    As someone in a financial services company said to me the other day, "There may be companies that can just put their data in an Oracle database, but unfortunately we don't have that luxury." The reason that they said that is because most of their data comes from mainframes. And integration is the biggest part of their work.

    The reason I give that quote is because JDO is about how you give reasonably good developers tools to handle 90% of the data integration work *on the Java side* (there's still work on the object design and the mapping side that needs to be done by someone who's fairly knowledgeable on both sides) and allow vendors to provide the plugs to/through J2CA, JDBC, etc. If you can avoid having to do custom integration work for each project, you can probably cut the cost per project by 75% or more. That *may* mean that the project in production isn't quite as efficient as if their top developer dude had hand-coded the integration, but there are only so many "top developer dudes" (and dudesses) and usually they are busy putting out a fire somewhere else ;-).

    JDO is not the mythical "silver bullet", but it's a bullet worth having in your belt.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Easily share live data across a cluster!
  56. SQL vs JDOQL[ Go to top ]

    \Cameron\
    As someone in a financial services company said to me the other day, "There may be companies that can just put their data in an Oracle database, but unfortunately we don't have that luxury." The reason that they said that is because most of their data comes from mainframes. And integration is the biggest part of their work.
    \Cameron\

    I work in such an environment right now. The majority of so-called "Front office" work is happening on various Unix platforms, but the bulk of post trade processing is happening on Tandem and IBM z-series mainframes.

    In that environment, which is pretty representative of the setup in a financial services company that's been around for any length of time, integration is a key issue, as you said. And anything that can help ease that is a good thing.

    But - keep in mind that performance and data integrity in such an environment is _equally_ important to development time and convenience, and custom monitoring and recovery techniques also play a very big role.

    \Cameron\
    The reason I give that quote is because JDO is about how you give reasonably good developers tools to handle 90% of the data integration work *on the Java side* (there's still work on the object design and the mapping side that needs to be done by someone who's fairly knowledgeable on both sides) and allow vendors to provide the plugs to/through J2CA, JDBC, etc. If you can avoid having to do custom integration work for each project, you can probably cut the cost per project by 75% or more. That *may* mean that the project in production isn't quite as efficient as if their top developer dude had hand-coded the integration, but there are only so many "top developer dudes" (and dudesses) and usually they are busy putting out a fire somewhere else ;-).
    \Cameron\

    In the environment you've described, which seems to closely match what I've described, overly generic layers which hide too much of a real resource always becomes a headache eventually. The generic solution, like JDO, works fine until it _doesn't_ work fine, and then working around the generic solution becomes a gigantic bottleneck. In fact, I've worked in many groups in advanced stages of production (e.g. in production for over a year) where tricking out some third party component took up 25% of the development time. And I've seen such overly generic solutions get ripped out after spectacular production failures which resulted in down trading desks or failing trades at settlement - and the senior managing director cut a wide, bloody strip off the people who were pushing the generic solution.

    Financial companies are always looking for a way to leverage others' work, cut their development time, and generally avoid reinventing the wheel. But that's _always_ a secondary consideration. The main consideration is always to make sure the system works, is highly robust, and fast as hell. The main reason to use third party solutions is not generally to save development time, but strictly because third party solutions are sometimes more proven in the field, and "local" developers are unlikely to put together a robust enough solution. But finance companies have been bit many times by technologies and products which didn't live up to claims, and they are quite willing to build their own when it clearly shows the lowest risk and highest level of control.

    Keep in mind - in really top financial firms that are doing it right (IMHO), developers routinely make 6 figures and can get yearly bonuses of 50% of their base salary if the market's doing well. As such there's usually a much larger percentage of "top developer dudes", and they're expected to deliver, not bet the company on a shaky generic solution which might crap out in production in the worst possible circumstances.

    As I've said - JDO seems to have its uses, and may get used (even in the financial community :-). But financial developers are always going to need (not want, need) a fine level of control of their resources to get the fault tolerance and performance the business demands. If they ignore that, the firm can literally lose millions of dollars in a single day. Sadly, financial firms have all the political issues of any big organization, and some groups in such firms sometimes start valuing developer convenience over the needs of the firm. And sometimes those groups even have their time in the sun, and their views get wide spread. But those groups oriented in that way _do_ get hit hard when their convenient solutions fail, the SMD goes on a rampage, and it generally hits the fan.

        -Mike
  57. SQL vs JDOQL[ Go to top ]

    Mike, what can you say about Spring framework ?

    In our project we used direct JDBC, no O/R tools...in fact we had to tune every single SQL statement that was performed on Oracle database. Though I've got a feeling that we could use some caching tool for SQL, that was fetching every time the same data.
  58. SQL vs JDOQL[ Go to top ]

    I think SQL and UDF is a very good way for JAVA applications too.
    I have used a few O/R mapping tools, but the best result was with JDBC ... .
    Mapping tools are good, if it is not your first project with this tool,
    it helps to reduce JAVA code, but developer needs to write XML, QL's, learn
    to use build tools, read new exception stack traces ... and use JDBC workarounds at the same time.
    We have started a project on SF, I think it must be interesting for developers: http://voruta.sourceforge.net,
    we want to find a way to integrate O/R tools and "legacy" SQL/UDF.
  59. SQL vs JDOQL[ Go to top ]

    Mapping tools are good, if it is not your first project with this tool,

    > it helps to reduce JAVA code, but developer needs to write XML, QL's, learn
    > to use build tools, read new exception stack traces ... and use JDBC
    > workarounds at the same time.

    I'll take these one at a time....

    > but developer needs to write XML

    Try this for size, which marks up my.package.MyClass as persistence-capable and accepts default values for persistence modifiers.

    <jdo>
    <package name="my.package">
    <class name="MyClass">
    </class>
    </package>
    </jdo>

    In the simplest case, JDO persistence descriptors are trivial. Where you are exploiting the O/R Mapping to an existing schema, then <extension> tags come in and I agree these are unweildy. But you can still hand-code a JDO persistence descriptor whilst GUI tools abound for those who prefer them (ask your vendor what GUI tool they provide).

    The equivalent XML descriptors for EJB are now much too complex for mere mortals to hand code (accolades to those who can still do so!).

    Of course XDoclet and other techniques can limit the extent to which hand coding is necessary....

    > QL's

    JDOQL becomes quite intuitive to Java developers. You must understand that in its current for, JDOQL filters a set (Collection or Extent) of objects and results in a subset of that "candidate" set. Given you use it for this purpose, JDOQL filter expressions are very expressive. The ability to navigate references within the query filter beats SQL from a language point of view, and the fact that queries cannot modify persistent instances or invoke arbitrary methods makes it easy for vendors to create high performance SQL specific to the target data store (or other native query language where SQL is not supported).

    And for those hard-to-express queries you can tunnel SQL through the JDO connection to share transaction isolation if you wish....

    > learn to use build tools

    The enhancement step can be done through ANT. If you're not using ANT I'm tempted to say you should be. However you can probably integrate enhancement into your IDE's build process with a little thought and in conjunction with your chosen JDO vendor.

    > read new exception stack traces

    Yes, if there's a problem you may see JDO implementation classes in the stack trace. However line numbering is maintained for debugging your persistence-capable classes, which is a big benefit over source-code enhancement.

    > and use JDBC workarounds at the same time

    You can use JDBC workarounds if you need to subject to support from your chosen JDO implementation. I see this as a benefit, because now you have all the gains of JDO with the "veto" clause that lets you do it your own way if you want to.

    Kind regards, Robin.
  60. SQL vs JDOQL[ Go to top ]

    I agree, It must be trivial to write XML, learn JDO stuff, buy GUI, use custom task for Ant and to find workarounds if something doe's not performs, I do not
    ignore JDO and O/R stuff, I help to develop this kind of opens ource solutions myself. JDO is a very good experiment, I have a few implementations on my disk,
    I believe we will have more open source JDO, O/R mapping implementations in future, but I do not believe this stuff can "replace" old good SQL.
    I am one of "financial developers" too and I 100% agree with Mike.
  61. SQL vs JDOQL[ Go to top ]

    \Robin Roos\

    [Responses on JDO issues]

    \Robin Roos\

    The issue for most application developers isn't so much the feature-set that any component has out of the box - it's how it can be used in their own environment, and how much work that will entail. In my own context - first, let's carve out entity beans from the discussion, and eliminate the concept of moving from an RDBMS to some non-RDBMS data store. Those two aren't applicable to me and the environments I've worked in.

    With those two pieces removed, and figuring in the complexity of the applications I work on (along with integration and performance and robustness issues), frameworks like JDO typically require more work and more code then straight JDBC, JDBC encapsulated by Stateless Session Beans, or some custom code-gen based solutions I've seen.

    This may sound absurd, and for some types of projects it would be. But in my experience, the "defaults" you're citing are rarely applicable in a complex application. As an application grows, you find the default behavior is less and less applicable to what you need, and you spend a significant amount of time "bending" the tool to your needs - effectively, you're fighting it. In comparison, some JDBC and SQL code may be hairy and complex, but at least it's so in a straightforward way.

        -Mike
  62. SQL vs JDOQL[ Go to top ]

    A well thought out and well written reply, Mike, my compliments.

    For clarification, the defaults which JDO applies are roughly the following:

    Fields which are transient, static or final are not persistent.
    Fields of types unrecognized by JDO are not persistent.
    All other fields are persistent (stored in the database).

    Fields whose types are "the usual suspects" (primitives, wrappers, String, Date, Locale, and a few others) and arrays are part of the default fetch group. These are fetched from the data store when the object is first fetched.

    Other fields (associations to persistence-capable types, and all Collections) are not part of the default fetch group and are loaded lazily.

    These settings can be overriden on a per-field basis as required, so a "transient" field can be made "persistent", a collection put into the default-fetch-group and an array loaded lazily, etc.

    However I acknowledge your critique that whatever the defaults, one often ends up needing to override them and that starts to increase the complexity of the chosen solution.

    I would be very interested to hear the sorts of changes that might make JDO more appropriate for your context. I ask because I'm a member of the JDO Expert Group, and I'm putting together my suggestions for JDO 2.0.

    Kind regards, Robin.
  63. SQL vs JDOQL[ Go to top ]

    Robin,
    A JCP member asking for community input? How I have seen everything.
    Since you asked here are suggestions for JDO 2.0 or 3.0:
    - Make ANSI SQL 92 or ANSI SQL 88 or 99 required. Deprecate OQL in any case.
    - Require support for Stored Procedures, most large compelx sites do it and you should design for the rule (not design for exception)
    - Require support for CLOB/Blob. (Maybe required full JDBC 3.0)
    - Handle tabluar data!!!!
       SQL returns a list of columns, it does not return a string or an int. Not even a list of Strings.
    I represent what my DAO (iBatis) returns as a Collection (rows) of List (columns). This way MQ or other strange things (non SQL DB) can work.

    It is (mathematicaly) un-natural not to be able to work with sets.

    Maybe allow for a option to return a Disconected RowSet (for low level work).

    So DAO (JDO in this case) should return tabluar data (collection of lists) and do stored procedures and ANSI SQL (executed at db not at JDO, its not JDO's problem. If some DB is not ANSI SQL 88, 92, or 99, that is their problem). Also allows us to tune querry using vendor proratory extencions (Transact SQL, plSQL, pgSQL, etc.)

    Lets say you need to page through data, SQL has Limit/Offset, a good way.

    Also, work the JDO through typcall examples of production enviromnet, for example ones in Joe Celko book on SQL Puzzels and Answers. For example, a nested tree with a self join. Correated querrys. Covered Indexes.
    I am almost sure that any open source would love to test out JCP ideas before they are set in stone, if JCP would only take consider the input.
    basicPortal.sf.net would.

    Invlude stress testing features (dbMonster and DieselTest) come to mind.

    As for Cross platform siliness, we know how to go from Oracle to MS SQL using stored procedures allready.

    Maybe just license iBaits.

    .V
  64. SQL vs JDOQL[ Go to top ]

    Hello everyone

    I'm going to comment on Vic's points in turn. The result must necessarily be quite lengthy. If you take the time to read this I look forward to your comments.

    > A JCP member asking for community input? How I have seen everything.

    I'm not special and I don't live in an ivory tower - I merely employ some of my time and expertise to help steer the Java platform in areas where I believe I can contribute constructively. If you wanted to contribute then you probably could - have you tried?

    > Since you asked here are suggestions for JDO 2.0 or 3.0:

    Ok, lets bear in mind that JDO is a standard for "Object Persistence". It is not a standard for "Object-Relational Mapping". Under JDO 2.0 I expect us to standardize the majority of tags necessary to describe O/R, but that is not JDO's primary purpose. It is a credit to JDO that it has been designed so that vendors can implement back-ends against both Object and Relational data stores. As far as I have seen, none of the O/R Vendors are complaining that JDO is too object-focussed, although that is often cited as a criticism in forums such as this.

    > - Make ANSI SQL 92 or ANSI SQL 88 or 99 required.

    SQL in its various forms is good at retrieving columns of data from a relational database. JDO provides an object-level abstraction of your domain, relieving the developer of this low-level burden. Some vendors will support direct SQL, either through the javax.jdo.Query interface, or through exposing the underlying (usually JDBC) connection.

    Did you know that the JDO spec facilitates queries in different query languages? To be compliant a JDO implementation must support JDOQL. However it is possible to use pm.newQuery() to create a Query object whose query language is SQL. This is supported by the JDO specification, although no JDO vendor is under obligation to implement a SQL query parser.

    I do not anticipate the JDO spec mandating SQL queries. However you will choose an implementation that does support O/R mapping features if you need them.

    Would it satisfy you if a sub-spec, "ORJDO", were to mandate support for SQL query parsing and O/R mapping semantics? All the O/R implementations would then become "ORJDO" compliant. Applications written using these extra features would be portable only across "ORJDO" implementation, whereas applications that did not exploit these features would be portable across all "JDO" (including "ORJDO") implementations.

    The net effect of this would merely be a marketing issue. Technologically the "ORJDO" specification would be very thin, specifying O/R mapping tags, requiring support for SQL as a query language, and identifying the runtime type of the object returned from the execution of such a query.

    In this forum I do not represent the JSR12 committee, and my comments should not be construed in any way as a consensus of that committee, but in the interests of open discussion what do you think?

    > Deprecate OQL in any case.

    Can you be more precise? OQL is part of OMG. OQL is neither supported nor mentioned by JDO. OQL facilitates the invocation of arbitrary methods of objects in a query, and affecting that with relational schema would probably require the instantiation of objects merely to exclude them from the query result on the basis of the method invocation.

    JDOQL does not facilitate the invocation of arbitrary object methods within filter expressions, and mandates that the filter execution be non-mutating. This enables JDOQL expressions to be cleanly and efficiently expressed in a number of native data store query languages, including SQL (for Relational databases) and the appropriate object query language (perhaps OQL) for Object databases.

    I perceive JDOQL as being an elegant solution to filtering subsets from a set of Objects. That is its purpose. JDOQL is not intended to return field-level data. However JDOQL is weak in two areas: Aggregation and Projection.

    Currently before the expert group are proposals for extending JDOQL and the appropriate methods in the javax.jdo.Query interface to support both Aggregation and Projection.

    Another possible extension would be the provision of type-safe means to build query expressions through method invocation semantics, as an alternative to the String query filter expressions whose validity can only be tested at query parse time. This is not high on the agenda, but has been suggested and does have some merit.

    > - Require support for Stored Procedures, most large complex sites do it and you should design for the rule (not design for exception)

    Some JDO implementations already support the mapping of JDO Queries (javax.jdo.Query, not JDOQL in this case) to stored procedure invocations. The developer writes a Stored Procedure whose resultset is a single column that is the primary key for the matching rows. (Multi-field keys could easily be accommodated). The Stored Procedure is invoked through the Query API. The Query runtime takes the resultset of keys and builds a collection of persistence-capable instances corresponding to those keys. The result returned to the client is thus true to the JDO Query semantics, whilst the developer has retained full control of the query logic.

    Although the developer has full control of the query, this approach may not be efficient in some circumstances, since the JDO implementation must still fetch the default-fetch-group fields for each object from the data store. However I'm sure the vendors supporting this feature would do their best to fetch this data in a paged fashion where possible, to avoid issuing more SQL statements than necessary. The point is that the cache used by the stored procedure is probably not shared by the JDO implementation, but that's already more detail than I want to get into here....

    So, if this notional "ORJDO" standard were to specify the means by which an application could invoke a Stored Procedure in a portable manner, as (a) something that returns a result set of keys as per javax.jdo.Query, or (b) the arbitrary invocation of a Stored Procedure that might change the data, to be invoked in the Persistence Manager's active transaction (which might be the J2EE transaction where appropriate), would that help?

    > - Require support for CLOB/Blob. (Maybe required full JDBC 3.0)

    What Java type do these represent? If you want to store images, then let the JDO community know which Java classes you use to represent these and ask for them to be supported. If you want to store Audio, do the same. For reference, most JDO implementations can usually store any Serializable object as its serialized form in a blob (off hand I'm not sure that this is actually addressed by the standard), but I'm sure you want a more comprehensive capability than this.

    > - Handle tabular data!!!!

    Can you be more specific with this one? If you mean that JDO should manage Rows and Columns then I beg to differ, but perhaps you had something else in mind.

    > SQL returns a list of columns, it does not return a string or an int. Not even a list of Strings.
    > I represent what my DAO (iBatis) returns as a Collection (rows) of List (columns). This way MQ or other strange things (non SQL DB) can work.

    JDO is already data store-agnostic, so the same JDO-based application could work on any supported data store, relational or not. But you know that already....

     
    > It is (mathematically) un-natural not to be able to work with sets.


    It is un-natural not to work with Objects. Objects supersede sets, having more advanced semantics and being capable of more-closely representing a given business domain.
     
    > Maybe allow for a option to return a Disconnected RowSet (for low level work).

    Yes, a paging structure, specifically for representing pages of large result sets, would be great. This would have applicability for HTML pages and Swing clients amongst others.

    > So DAO (JDO in this case) should return tabular data (collection of lists) and do stored procedures and ANSI SQL (executed at db not at JDO, its not JDO's problem. If some DB is not ANSI SQL 88, 92, or 99, that is their problem). Also allows us to tune query using vendor proratory extencions (Transact SQL, plSQL, pgSQL, etc.)

    Tabular data in this form does not provide an object-level abstraction or solve the object-relational impedance mismatch. If you want tabular data, and you do not want objects, you should probably not use JDO for that particular task.
     
    > Lets say you need to page through data, SQL has Limit/Offset, a good way.

    This is a performance benefit over other approaches. JDO should accommodate the requirement (limiting query results, or skipping initial results). Vendors should be left to exploit the performance optimizations in the underlying data store, or implement their own strategy if native support is missing.
     
    > Also, work the JDO through typcall examples of production enviromnet, for example ones in Joe Celko book on SQL Puzzels and Answers. For example, a nested tree with a self join. Correated querrys. Covered Indexes.

    Where the examples warrant an Object-level abstraction, I'm sure presenting a JDO-based solution would be illustrative. Where the examples merely serve to illustrate complex SQL usage, but do not warrant an Object-level abstraction, there would be no point in doing them in JDO. Furthermore, the "failure" of JDO to address such tests would not be to JDO's detriment. Remember, JDO is an "Object Persistence" standard and whilst I will work to accommodate and standardize issues specific to JDO's usage in a Relational context, I will not advocate turning JDO into another JDBC driver.

    > I am almost sure that any open source would love to test out JCP ideas before they are set in stone, if JCP would only take consider the input.
    > basicPortal.sf.net would.

    There is nothing to stop representatives of Open Source communities from becoming involved in the JCP. However the new revision of the JCP process makes all interim specification drafts public, where they previously were not available outside JCP itself. The increased transparency this will provide should make it easier for the public to try out new specs (and new spec versions) and comment on them without having to be part of the actual specification team.

    > Invlude stress testing features (dbMonster and DieselTest) come to mind.

    Is this the remit of JDO? I don't think so. It is not up to the spec to show how implementations should be load tested. The JCP process mandates a TCK, which guarantees compatibility of JDO implementations from a functional perspective. The JCP does not mandate performance testing.

    Thus an implementation is not required to be quick in order to be JDO-compliant. Coincidentally I believe this area is being addressed by the JDO community outside the specification itself.

    > As for Cross platform siliness, we know how to go from Oracle to MS SQL using stored procedures allready.

    JDO provides an Object-level abstraction of persistence. This has tremendous benefits from the perspectives of developer productivity, application design, architectural applicability, maintainability, etc. Depending on the implementation and chosen data store, these benefits might not incur a performance penalty. Any penalty that is incurred should be weighed against the benefits JDO provides, and it is up to each potential user of JDO technology to perform due diligence in this regard. However you should also be aware that in some cases, JDO might significantly increase performance.

    One of the benefits that JDO provides, as a virtue of its object-level abstraction, is portability of applications across different data store paradigms. This benefit is important to some users, specifically ISVs who market applications and allow the client to choose the data store. Also people wishing to get very high performance in some circumstances might choose an object database. Architecturally, the decision of the deployment environment can be deferred and taken at an appropriate time: the deployment environment is not immediately constrained to be J2EE/SQL as it is at the beginning of so many application development projects.

    But that doesn't mean data store portability is seen as important to everyone. Neither is data store portability the fundamental reason for JDO's existence - it is just one intended benefit of JDO's object-level abstraction.

    > Maybe just license iBaits.

    ;-)

    Kind regards, Robin.
  65. SQL vs JDOQL[ Go to top ]

    Thanks for a reply.
    With respect I disagree. (as you do with me :-) )

    I think OMG does not represent how I use DAO, and trying to work it in will just ... not fit. I have learned to avoid OMG.

    I work with sets (set theory, unions, intersections) of tabular data. A list of employees, a list of items on an invoice, a tax from ( a list of income AND a list of deductions).

    This is what my forms look like. For example NetLedger.com is a realistic application. (Free to sign up and view forms). Any of those typycall forms would have to be custom work arrounds in JDO, I might as well use JDBC Discounected RowSet.
    I think ADO.net is usefull, allowing a C# and WebForms developer to do work without OMG; and matches the developer need. A good API.

    At least you heard one voice. I hapen to specialize in failed project recovery and in general one of the things I do on a project recovery (after fixing real requirments) is rip out all the layers of OMG, VO, DTO.... it creates complexity on a big project. Then I do simple OO, "is a/ has a" of nested form beans... that each has a DAO; done!

    Also, I tell the client that it is simply false to say that you can do a larger project without someone (tech lead?) _mastering_ ANSI SQL. I show them Joe Celko way of solving an issue and the project suceeds. Set theory works. SQL Engines work.

    And Java is not even excelent at doing Web GUI. Yes, you could do JSF in Java... I think a good practice is to do ECMA script in browser (javascript, action script).
    Consider a nested tree (self join, maybe complex stored proc.) in ANSI SQL (in PostgreSQL Engine),
    a Java Bean (backed by a Collection- on Resin Engine, with DAO autocache/flush)
    and a JavaScript tree (executing in Netscape Browser), when users clicks, the tree expeands in browser, not submit to server.
    So not Java everywhere, Java where works, on server. Not Java for GUI or Java Query execution path.

    To me JDO is mini-Ejb. If OMG is some reaserch academic theory, that is a not usefull to a corporate application developer that wants somethings simple to work fast. Missleading coporate developers that this is what experienced app. developers do is contraversial at least.

    iBaits.com got it (almost?) perfect, under open source licese, KISS.

    Peace, I am out.
    .V
    (shame less plug: I am teaching an _advnaced_ best practices web development with Struts online in a few weeks, MONEY back gurantee, sign up at baseBeans.com)

    Robin, my private e-mail is vic_cekvneichNOSPAM at baseBeans dot com, if you wish to coninute private)
  66. SQL vs JDOQL[ Go to top ]

    Sorry for the quicky response, but I'm short on time and one thing struck me here....

    \Roos\
    Ok, lets bear in mind that JDO is a standard for "Object Persistence". It is not a standard for "Object-Relational Mapping". Under JDO 2.0 I expect us to standardize the majority of tags necessary to describe O/R, but that is not JDO's primary purpose. It is a credit to JDO that it has been designed so that vendors can implement back-ends against both Object and Relational data stores. As far as I have seen, none of the O/R Vendors are complaining that JDO is too object-focussed, although that is often cited as a criticism in forums such as this.
    \Roos\

    The above is a fundamental disconnect between my views (and also how many large financial apps are built) and the drive behind things like JDO.

    I'm not interested in "Object Persistence". Objects and OO programming in general has turned out to be great for modelling and coding the runtime structure of a program and its data. And IMHO OO _sucks_ when modeling the persistent structure of data - where as relational shines.

    Something I've seen again and again is that the model you use at runtime has a number of assumptions and requirements built into it which are (hopefully) optimal for runtime behavior and maintainability of the code. But the requirements for storing and accessing data are often in direct conflict with runtime needs. A structure and API that's "fast" for querying 10 million "rows" of persistent data will be rather unwieldy for runtime use of in-memory constructs - and vice versa.

    Throw in the fact that persistent data is almost never accessed just by one "program" or even just one "language", but in fact typically many systems will need to store data and retrieve it from a common place, and the idea of applying OO to the data store seems even less attractive.

    Put another way - it would be damn weird to program in a relational model, and languages which work that way haven't ever really caught on for either speed or convenience reasons.

    Likewise - storing data using an OO model is damn weird, and in my experience doesn't work for both speed and convenience reasons - and also because of the need of multiple accessors/storers in multiple languages, and size/robustness reasons.

    Decades of research have resulted in the relational model being the best for storing and retrieving data. It may seem bizarre or sub-optimal to those used to objects - but the industry has tried other persistence formats for a long time (hierarchical, object, etc), and they always come back to the RDBMS.

        -Mike
  67. Single store, multiple programs[ Go to top ]

    Throw in the fact that persistent data is almost never accessed just by one "program" or even just one "language", but in fact typically many systems will need to store data and retrieve it from a common place,


    This is what a middle-tier is supposed to be all about. To be frank, I don't believe it's good that multiple programs can all access the same data store directly. If I want people accessing my data store, I want them doing it through my business logic. That doesn't mean that they have to write in my language - use another language that runs on my virtual machine, or use a combatible protocol (IIOP, SOAP, etc...)

    It's really just an extension of the principle of opaque interfaces and data hiding. Why people fight so hard against this, I'll never know.

    God bless,
    -Toby Reyelts
  68. Single store, multiple programs[ Go to top ]

    \Reyelts\
    This is what a middle-tier is supposed to be all about. To be frank, I don't believe it's good that multiple programs can all access the same data store directly. If I want people accessing my data store, I want them doing it through my business logic. That doesn't mean that they have to write in my language - use another language that runs on my virtual machine, or use a combatible protocol (IIOP, SOAP, etc...)

    It's really just an extension of the principle of opaque interfaces and data hiding. Why people fight so hard against this, I'll never know.
    \Reyelts\

    I'd be in hog heaven if I worked in an environment that was amenable to such an idea. But the reality is the following "stuff" hits the same data store for varying reasons (reasons business, legacy, and technical):

       - Report writers like Crystal Reports
       - J2EE app
       - Batch jobs in various languages
       - CISCs stuff
       - C programs
       - C++ programs
       - perl scripts & misc utilities
       - various tools used by DBAs

    It'd be real, real nice to force all access through a JVM one way or another, but it isn't going to happen in my environment in my lifetime. If you can do it in your environment, you are truly blessed. For the rest of us, it's a reality we live with.

    As an aside...remember that 10 years from now your JVM/J2EE based system is going to be a crusty old codger that the new kids in the new framework/language will go to great lengths to avoid - but the data store will likely still be there and accessible directly from this new whizzy majig. :-)

        -Mike
  69. Single store, multiple programs[ Go to top ]

    Mike Spille wrote:
    [i]
       - Report writers like Crystal Reports
       - J2EE app
       - Batch jobs in various languages
       - CISCs stuff
       - C programs
       - C++ programs
       - perl scripts & misc utilities
       - various tools used by DBAs
    [/i]

    What prevents somebody from writing a version of Crystal Reports that works with JDO?

    When you mention "batch jobs", I'm envisioning ETL tools like DataStage. My experience with ETL tools is that it would have been quicker to write the equivalent task from scratch without involving the ETL tool.

    I don't understand how CICS (what I think you meant) causes a problem here.

    I've already addressed C, C++, Perl, etc...

    The tools that DBAs use are mostly provided by the vendors themselves.

    God bless,
    -Toby Reyelts
  70. Single store, multiple programs[ Go to top ]

    The environments I've worked in over the past <mumble> years have been highly heterogenous - in terms of hardware, OS's, and languages. In many cases, companies have made big investments in existing third party tools and existing in-house code. And they understand how those tools and that code works, and are comfortable with them.

    Now - when someone writes a J2EE application, they do not just throw all those tools and all that existing code away. They cannot, for any number of reasons, mandate that all access to data will be through Java or a JVM - or even through web services or something similar. Beyond political and budgetary considerations, there are many technical reasons why "everything Java" just doesn't work. Certainly web services will never fly for batch processes (performance!), and even J2EE solutions tend to fall over dead if you try to throw a hundred-thousand-row (or larger) dataset at them all in one gulp. And no one will throw out all the RDBMS data manipulation tools they're using (whether home grown or third party) and write new ones from scratch to use Java and webservices. Or existing "feed" systems which want to pick up data from this new J2EE system may not be amenable to Java or web services (it may be written in C, or COBOL, or any number of other languages) with alot of code behind it. Such feeders can often easily be adapted to pull new data out of an RDBMS, but pulling from a web service or Java could entail a complete re-write, or a bulky (and initially buggy) side-process written in Java that emulates what the original does.

    I could go on and on. The point is that most big companies aren't going to say "java and webservices only, direct RDBMS access is verboten". If they did, they'd frankly go out of business.

         -Mike
  71. Single store, multiple programs[ Go to top ]

    In many cases, companies have made big investments in existing third party tools and existing in-house code. And they understand how those tools and that code works, and are comfortable with them.

    I think you've missed my point. Whatever system is in place, should remain in place, until it's just not cost-effective to maintain it anymore. New systems will be written with new tools, and the new systems will have to communicate with old systems - that's a fact of life. I simply said that a common data store doesn't seem like a very robust means of communication between systems.

    God bless,
    -Toby Reyelts
  72. Single store, multiple programs[ Go to top ]

    \Reyelts\
    I think you've missed my point. Whatever system is in place, should remain in place, until it's just not cost-effective to maintain it anymore. New systems will be written with new tools, and the new systems will have to communicate with old systems - that's a fact of life. I simply said that a common data store doesn't seem like a very robust means of communication between systems.
    \Reyelts\

    Often it's the only effective way they can communicate :-/

    Also, even for new systems, it's rare for a company to go completely with all new tools as well. Someone may think it's great to use Java for everything, or develop an uber Web service layer that can cover all uses of the data - but in reality, most companies will have a suite of tools (in house and third party) that can do an awful lot with raw RDBMS data, and the company will have neither the time, budget, nor general inclination to throw all that stuff out to go Java-only or web-service only. And, in fact, that goal may be impossible altogether - it's next to impossible for any home-grown API to match the data loading, querying, and data mining capabilities that a modern RDBMS offers. Something that a given tool can do with ease against, say, Oracle might take an awful lot of work to build into your custom business API, or to expose as a Webservice, and in many cases no custom API is going to perform well enough to even come close what the RDBMS can do and the timing the business needs.

        -Mike
  73. ORJDO[ Go to top ]

    Ok, so Toby agrees that an ORJDO spec which extends (adds to but does not detract from) the JDO spec would be beneficial and aid market acceptance of JDO in O/R environments.

    What do the rest of you think?
  74. Single store, multiple programs[ Go to top ]

    Mike wrote:

    [i]As an aside...remember that 10 years from now your JVM/J2EE based system is going to be a crusty old codger that the new kids in the new framework/language will go to great lengths to avoid - but the data store will likely still be there and accessible directly from this new whizzy majig.[/i]

    I don't think so. I don't go writing programs that read directly from IMS or DBASE databases.

    God bless,
    -Toby Reyelts
  75. Single store, multiple programs[ Go to top ]

    \Reyelts\
    I don't think so. I don't go writing programs that read directly from IMS or DBASE databases.
    \Reyelts\

    So tell me - how well does your new Java code pick up the "legacy" business logic & RDBMS access written in a C/C++ program? Do you make sure your Java code never hits the RDBMS, but instead _always_ goes through the C code for access?

    Likewise - 10 years from now, will all new code written in the snazzy new language go through your (now antiquated) Java code, or will it hit the RDBMS directly? My money is on the latter.

         -Mike
  76. Single store, multiple programs[ Go to top ]

    Do you make sure your Java code never hits the RDBMS, but instead _always_ goes through the C code for access?

    Yes, rather then re-write a bunch of working existing business logic, I try to go through the existing system. (If it ain't broke, don't fix it?) When that system becomes too crufty to maintain (40 years from now?) I'll have to migrate the data into a new persistent store and write a new business layer.

    God bless,
    -Toby Reyelts
  77. ORJDO[ Go to top ]

    Robin wrote:

    > Would it satisfy you if a sub-spec, "ORJDO", were to mandate support for SQL query parsing and O/R mapping semantics?

    Wow, that is almost exactly what I posited in the JavaLobby forums a day or so ago. I think you've got this right on the money Robin. I believe it's necessary to bring at least some standard semantics directed towards O/R into JDO. Creating a separate "sub-specification" will demonstrate JDO's commitment to supporting O/R, allowing that support to become strong, while still keeping JDO a persistent-store agnostic technology overall.

    God bless,
    -Toby Reyelts
  78. SQL vs JDOQL[ Go to top ]

    I would be very interested to hear the sorts of changes that might make JDO >more appropriate for your context. I ask because I'm a member of the JDO >Expert Group, and I'm putting together my suggestions for JDO 2.0.


    1) Make this API public.
    https://jsecom16.sun.com/ECom/EComActionServlet/LegalPage says:
    "Transaction stopped. The selected product(s) cannot be provided to your location."
    2) Make this API more realistic.
    I do not think migration from RDMS to flat files is realistic use case.
    3) Remove some useless stuff like XML and Enhancer.
    It doe's not help for innovations, I do not think XML is the best way for metadata and persistence can be implemented without code transformations.
    4) try to implemet realistic application with JDO and without workarounds before to declare JDO as standard.
  79. SQL vs JDOQL[ Go to top ]

    1) Make this API public.


    There are 14 commercial implementations of JDO, as well as 4 open source projects underway. Download one of those if you are not permitted to access the Sun Download Centre. Indeed, download one anyway since the reference edition (which I presume you were trying to obtain) is not intended for general use. It only supports file storage anyway so I doubt it's of interest to anyone other than a JDO Vendor checking for functional correctness of their implementation.

    > 2) Make this API more realistic.
    > I do not think migration from RDMS to flat files is realistic use case.

    The RI uses file-system storage. It is not intended for use outside the vendor community. Obtain an implementation that targets the data store of your choice.

    > 3) Remove some useless stuff like XML and Enhancer.
    > It doe's not help for innovations, I do not think XML is the best way for metadata and persistence can be implemented without code transformations.

    Feel free to implement PersistenceCapable yourself in source code or through other means. You'll find the process tedious, error-prone, and confusing to debug.

    Alternatively use an enhancer, which does the implementation of PersistenceCapable predictably every time, can be invoked as part of your build script (e.g. Ant), and maintains the original source code line numbering to ease the process of debugging your code.

    > 4) try to implemet realistic application with JDO and without workarounds before to declare JDO as standard.

    JDO is in the "Late Early-Adopter" phase. The technology is stable and is being used on commercial development projects, both underway and deployed. However JDO usage is not yet "widespread". Why not ask the vendors for some of their client references?

    Kind regards, Robin.
  80. SQL vs JDOQL[ Go to top ]

    Thanx for replay,
    "Make this API public"
     It doe's not help me to trust JDO as standard (I can download M$ "standards" without any problems)

    "Make this API more realistic"
    Why RI uses file-system storage ? I do not kow, but I do not believe JDOQL is implemented in RI too. I konw some open source implementations and all of them innovate. See this http://tjdo.sourceforge.net/docs/direct_sql_queries.html .
    I know a few ways to implement the same examples using JDBC
     or with framework like http://voruta.sourceforge.net, implementation is more simple.
    I do not think transparence is usefull for developers http://hibernate.bluemars.net/38.html.

    See http://www.xorm.org for "Remove some useless stuff like XML and Enhancer" It uses XML to store metada, but this kind of JDO can be implemented without it. I think it is better to define the object model for metadata in JDO, not a file format.
    Persistence can be implemented without code transformations.
    I do not think byte or source code transformations is the best way to implement this suff, let implementation to find this way.
  81. SQL vs JDOQL[ Go to top ]

    Hi Juozas

    I'm sorry that you don't trust the JCP, which governs Java and under which JDO is a standard.

    The JDO RI uses a file system data store because this is all that is required in order to exhibit JDO functionality that is true to the specification. The RI certainly does support JDOQL, and the TCK verifies both JDO and JDOQL compliance of implementations.

    The biggest problem that would face an ORJDO sub specification is the inherent complexity of the reference implementation it would require.

    I read your link (http://hibernate.bluemars.net/38.html) and agree with its contents. I too would be wary of a standard whose advocators had not used it. JDO is a standard, yet the vendors innovate very effectively. This innovation includes performance and scalability aspects as well as value-added functionality.

    JDO is now in the "Late Early Adoptor" phase. Whilst it was in the "Early Adopter" phase I regularly had sub-24 hour turn arounds from various vendors as they provided reliable fixes to problems I had identified that day or the day before.

    > I do not think transparence is usefull for developers

    I'm surprised to hear that. Perhaps you aren't aware of all the benefits that come under the banner of "Transparent Persistence". However you are free to choose whether or not to use JDO, and free publicly to articulate the reasons for your decision.

    > I do not think byte or source code transformations is the best way
    > to implement this suff, let implementation to find this way.

    JDO does not mandate enhancement. It is offered as a productive and reliable way for developers to implement PersistenceCapable. That is all. Nothing in the spec requires an enhancement activity, and vendors are free to innovate.

    However enhancement is widely accepted as the best available solution, even though some products enhance on-the-fly at runtime (typical of ODBMS implementations).

    One day the JVM will do this for us. Until then tool support is preferable to hand-coding the methods of PersistenceCapable.

    > Thanx for replay

    My pleasure, Robin.
  82. SQL vs JDOQL[ Go to top ]

    \Stepan Samarin\
    Mike, what can you say about Spring framework ?
    \Stepan Samarin\

    I have no direct experience using it, but I like what I've read here and there (unfortunately, I couldn't find any docs on the supplied link).

    In general, I really like pragmatic frameworks that codify common usage of J2EE components. Something which imposes a slightly higher level of semantics, centralizes common behaviors, and gets rid of error-prone boilerplate. Spring seems to live up to this, but obviously I can't comment on specifics.

    I'm less enthusiastic about something like JDO because it hides too much by default IMHO.

        -Mike
  83. SQL vs JDOQL[ Go to top ]

    Mike and Stepan,

    I am a developer on the Spring Framework project and I just wanted to let you know that we are preparing for our 0.8 release which includes some documentation and a tutorial. Please check back in a few weeks. http://sourceforge.net/projects/springframework

    > \Stepan Samarin\
    > Mike, what can you say about Spring framework ?
    > \Stepan Samarin\
    >
    > I have no direct experience using it, but I like what I've read here and there (unfortunately, I couldn't find any docs on the supplied link).
    >

    --Thomas
  84. SQL vs JDOQL[ Go to top ]

    Stephhen,
    iBaits.com DB layer lets you use custom tuned SQL and does caching and flushing automaticaly.

    It's also open source (unlike JDO).

    .V
  85. SQL vs JDOQL[ Go to top ]

    1) JDOQL is datastore agnostic. This means that you can easily change the persistence mechanism to XML, flatfile, ODBMS, MIDP RecordSet without spending a week or more just learning the basics of a datastore's language

    And why would I want to change my persistence mechanism from an advanced RDBMS to flatfiles or XML. It is absurd to design enterprise applications thinking that would be a possibility.
    As I said before, it is much more likely that I want to switch to another (pseudo)O/R mapper and that is not trivial. I have a (pseudo)standard query language, but a nonstandard mapping tool.

    >
    > 2) JDOQL is Java centric. You do not need to pay for developers who are both SQL and Java aware. The Java that you and we all know is the same language used to query the datastore.
    If I am doing a professional database application I need a professional who knows SQL. This argument is only valid for small, non-professional applications where data integrity and performance is not a big issue.
    If the application is in Java, I need a professional who knows Java.

    >
    > 3) Using SQL, while powerful, can cripple the performance of most large scale applications if done poorly. What percentage of developers understand properly batching statements, properly managing datasets as to not update entire rows when there is only a small change, caching datastore hits, efficient blob handling, etc? Forget about complexity at the object level such as inheritance, sophisticated object locking mechanisms, etc.

    Like I said, that is true if you use non-professional people to develop applications. If you are doing an enterprise aplication using a database like Oracle is because you need the performance and capabilities Oracle provides. There is no point in using a lest common denominator functionality with Oracle that allows the application to be ported to flat file persintence.


    >
    > JDO provides a high level of performance and scalability for most every application. In addition, the benefits gained in rapid development, low maintenance costs, as well as deployment flexibility far outweigh the perceived performance loss (which is more or less nonexistent as most JDO implementations support raw SQL when necessary). Changing databases is not an uncommon practice... how many times have any of us had clients requesting that your application run on X DB which they have partnerships with? Rebuilding your application from the dataobject level up, vs. simply changing a .properties file in JDO provides huge benefits.
    >
    You should isolate your SQL statements from your business logic, so that it is easy to change it, but that does not mean that not isolating JDOQL statements would be a good practice. If your application is complex enough so that porting the SQL would be too hard then you will probably have to port your database schema definition and you should check if the porting is possible at all.
    For example, if your application uses the AS OF clause of Oracle SELECT, then you are going to have a hard time porting it to MySQL or flat files.
    The porting will be much more than changing a "simple" configuration file.
    If you have to make an application that from the start is intended to work with several RDBMS, then you should use SQL that is as standard as possible (which is not too hard) or use a SQL porting tool like SwissSQL.
    If your application is intended to support flatfiles and RDBMS then I think that would be a valid use for JDO. Of course, such an application would not have any performance and data integrity requirements.

    > In a proper -Object- oriented application, using JDO can be far more efficient than writing pure SQL. Object, query, and relational caching mean that your application can run without touching the datastore the majority of the time.
    >
    I doubt it.
    Like Mark said, JDO may have its uses, and there are places where it makes sense, but I think the reasons you've stated are the worst possible ones to put forward.
  86. SQL vs JDOQL[ Go to top ]

    You've stated all the reasons why people everywhere have already turned in droves to products like JDO, proprietary O/R mappers, and CMP. There should be no limits on how you design your application, afraid of rewriting 100s of lines of SQL and Java code because some CTO somewhere shook hands with Larry Ellison. And having trained developers and worked for several major consulting firms, most enterprise projects are lucky to get a development DBA, let alone a theoretical army of Java and SQL experts. And even if you did, isn't their time better spent making a better UI, smoother workflows, smarter rules engines, and altogether more usable product? If raw speed were number one, wouldn't we all be programming in native C still? We program in J2EE because we can exploit the standard building blocks like JDO, EJB, servlets, etc. to build products that may not be baddest hot-rod on the block, but products that instead can be built quickly, perform at a high level, and are easily maintained through the lifespan of the product. Let JDO deal with niggly details and instead focus on the -business- of your enterprise.
  87. SQL vs JDOQL[ Go to top ]

    \Stephen Kim\

    [...]
     And even if you did, isn't their time better spent making a better UI, smoother workflows, smarter rules engines, and altogether more usable product?

    [...]

    Let JDO deal with niggly details and instead focus on the -business- of your enterprise.
    \Stephen Kim\

    This is all well and good until JDO _doesn't_ do what you need it to do, and you need to spend a large amount of time working around it. To turn your argument around a bit - what good is all that work on UI/workflows/engines/usability if your persistence mechanism craps out or is too slow? The fact is, for most business apps, everything you've mentioned is _worthless_ if the persistence mechanism doesn't work well. Persistence is the bedrock that every thing else relies on and is built on top of (perhaps with messaging forming the other fundamental building block). If it's not doing its job, literally nothing else matters. As such - you shouldn't ignore persistence in pursuit of loftier goals. Instead, you'd better make sure you know persistence inside and out and that it really _is_ the bedrock of your app, so that those loftier goals will acutally work.

         -Mike
  88. SQL vs JDOQL[ Go to top ]

    Hi Mike

    "This is all well and good until JDO _doesn't_ do what you need it to do, and you need to spend a large amount of time working around it."

    You make some good points. However remember that most (if not all) JDO O/R mapping vendors make it easy to use direct JDBC calls in conjunction with their products. You can use JDO, get the productivity and performance benefits for 90%+ of the project and do the 10% that really needs direct SQL/JDBC yourself.

    Cheers
    David
    JDO Genie
  89. SQL vs JDOQL[ Go to top ]

    Hi Vegeta

    "You are right in that different vendor provide proprietary SQL extensions, but I think porting from a JDO product to another is harder than porting outer join/lob statements from a database to another"

    No way - getting blobs to work properly on all the databases JDO Genie supports was a big hassle. Not to mention all the JDBC driver bugs we have run into :)

    To port from JDO to JDO, all you have to do is redo your mapping from your model to the database tables. We provide a GUI tool for this. You should not have to change your application code.

    "Can you elaborate on why JDOQL is more productive than SQL? I have not used JDOQL. Could you give several concrete examples that show JDOQL being really more productive than SQL?"

    Well developers write (restricted) Java boolean expressions. Thats a much higher level of abstraction that SQL.

    An example:

    public class Order {
        private Date orderDate;
        private List lines = new ArrayList(); // inverse OrderLine.order
        ...
    }

    public class OrderLine {
        private Order order;
        private Iten item;
        private int qty;
        ...
    }

    public class Item {
        private String code;
        ...
    }

    To get all orders after a given date containing a line for item with a specific code with at least a certain qty:

    Query q = pm.newQuery(Order.class);
    q.declareParameters("String c, int n, java.util.Date d");
    q.declareVariables("OrderLine v");
    q.setFilter("orderDate > d && lines.contains(v) && v.item.code == c && v.qty > n");
    q.setOrdering("orderDate descending");

    The SQL for an ANSI type database:

    select a.orderNo, a.customer_id, a.order_date, a.orderNo, a.jdo_version
    from ordr AS a
    where a.order_date > ?
      and exists (
          select 1
          from order_line AS b
               JOIN item AS c ON (b.item_id = c.item_id)
          where a.orderNo = b.orderNo
            and c.code = ?
            and b.qty > ?)
    order by a.order_date DESC

    On MySQL (which does not have subqueries) we will use a "select distinct with outer join" to run this JDOQL query.

    "Changing databases frequently in a production environment does not look like a recommended practice to me"

    I was not talking about changing the database for a given application. What I meant was that different projects frequently target different databases. You may be working on Oracle now but the next project might be MySQL. JDO makes it much easier for developers as it hides the different SQL dialects and bugs in JDBC drivers and so on.

    Cheers
    David
    JDO Genie
  90. JDO vs. CMP: their comparison table is good...I don't think there's more that can or should be said about it...but here's a UML question:

    In their sample chapter 1, the first UML diagram...that's more like a DB relational diagram. Shouldn't be modelled, in a proper UML diagram, as an "association class" connected to the association between "actor" and "movie", which is many to many?

    In essence, it adds attributes to that association, doesn't it?
  91. There a lot of people here saying JDO is not scalable and its good only for small applications and not of enterprise app. In special, they talk about performance of database querys, because using JDO they cant optimize querys. JDO Vendors what can you tell me about it ?
  92. Optimizing JDO Queries[ Go to top ]

    Hi Erik

    "There a lot of people here saying JDO is not scalable and its good only for small applications and not of enterprise app. In special, they talk about performance of database querys, because using JDO they cant optimize querys. JDO Vendors what can you tell me about it ?"

    There is certainly nothing "not scalable" about JDO. JDO O/R vendors provide extensions to control how SQL is generated. Many vendors also integrate powerful caching features (caching of instances and whole queries).

    JDO Genie supports eager fetching with outer (and inner) joins, multiple fetch groups and other optimizations. All of this is controlled through the .jdo meta data. This give you a lot of control over the SQL without having to write it yourself.

    With JDO Genie you can specify the fetch group to use for a JDOQL query at runtime. Fetch groups can be chained together making it possible to fetch a whole slice of your object graph with a single query. All of these things can be easily setup using our GUI Workbench.

    In practice the default queries and fetch behaviour is usually good enough. You will probably only need to optimize a small percentage of your application. Our Workbench can download and analyze events from a running JDO Genie server to help with this process.

    And like other JDO O/R vendors, we provide support for direct use of JDBC for the (very few) cases where SQL is the best tool for the job. This includes APIs to evict from the level 2 cache after bulk SQL updates and so on. So you do not have to worry about being "stuck". If you are really up against the wall you can still drop down to JDBC.

    Cheers
    David
    JDO Genie - High Performance JDO for JDBC
  93. This topic keeps coming up but I'm glad it does because each time there seems to be more and more JDO supporters and less and less CMP supporters. I know in my case I tried a project with CMP and the increased complexity just wasn't worth it (I was mainly using CMP as an O/R mapper). I discovered JDO and haven't even thought about using CMP since. I really like the Bitter EJB book, it really makes the point that EJB, especially Entity Beans, aren't for everyone. I know BEA, IBM, and even SUN would have us belief so but it's just not true.

    Also there isn't an open source JDO implementation (well not one that is mature, at least not yet I know one is on the way) but I'm using KODO and the price is very affordable. It's per developer, not per CPU.

    Michael
  94. It seems that the community's interest in JDO is growing very fast, so I thought you may be interested in a real world JDO case study of a large JDO based application/product that has been deployed in several countries.

    The case study explains the reasons that lead us to the decision of using JDO as persistence solution. It provides also an discussion of the advantages of using genuine object models for data modelling, instead of mixing the object oriented programming paradigm with relational data models and a lot more lessons learned from the experience of working with JDO and genuine object-oriented data models.

    An especially important point that gets discussed here is that application servers are only a subset of the environments where JDO fits in (many people forget this point). Our experience showed us that JDO may have the potential to bring Java also as a strong player the desktop if you combine it with metainformation and adequate GUI components.

    You can download the case study at JDOCentral: CastorTPV 1.2 Case Study

    P.D.

    Although this application (CastorTPV) and Exolab's "Castor JDO" persistence framework (which is not compliant with SUN's specification) sound somewhat alike, there is no relationship between them.

    Alberto López
  95. In a recent meeting with BEA sales & pre-sale architect, I posed the JDO question, to which I was assured that JDO was very much NOT on BEAs agenda.

    An interesting stance?

    As an aside I know of one leading CMP persistence manager (not fully blown app server) that is moving its internal persistence to JDO. It is still fully CMP 2.1+ compatible and works with any app server.
  96. Aaron,

    That may well be the stance of BEA now. If enough people want it then BEA can simply buy one of the JDO companies to fill the gap.

    Dan
  97. Aaron,

    If you had asked that question to the exact same people at the exact same vendor 16 months ago, you would have been told that Toplink was the right way to go for persistence. Magically, one incident like the Toplink acquisition by Oracle and now Toplink is no longer the right answer for BEA.

    What do you think BEA's answer will be 16 months from now as JDO vendors continue to succeed? For the record, I work for SolarMetric, and we now have 150 companies that are paid customers of Kodo JDO, our implementation of the JDO spec. These companies range from the Fortune 500 to small 2 man consulting shops.

    Bottom line is that market-driven companies like BEA, IBM, Sun, Oracle, Borland, etc. will react to whatever will help them raise their stock prices.

    Neelan Choksi
    SolarMetric, Inc.
    www.solarmetric.com
  98. I always get puzzled when reading threads discussing whether EJB is a good persistence API or not. EJB have never been designed to be a persistence API. It is a component model. This why is doesn't support object-level features such as inheritance, fine-grained persistence, etc.

    The reason why EJB provide a persistence solution is the same as why it provides a solution to remoting, security and transactional issues : it has to allow a component to be reused, i.e. fit different application contexts (need/don't need transactions in some application, need/don't need transactions in another application, need/don't need persistence in another one) *without code modification* (i.e. through component and component assembly descriptors). CMP is just here as one of the required services to allow a component to save its state without depending on some specific persistence solution/API that would not be encountered in another application context.

    So, to me, comparing EJB with JDO doesn't make sense. The goals are not the same, the API levels are not the same. A more simple statement is that we used JDBC/EJB for persistence while waiting for an object-level persistence API (not a state-saving API like serialization). Now that JDO is out, we can use JDBC and EJB for what they have been designed for, that is field-level persistence and reusable components, respectively.

    People interested in discussing EJB as components will be able to find a thread I started about this topic at http://www.theserverside.com/discussion/thread.jsp?thread_id=18546
  99. I always get puzzled when reading threads discussing whether EJB is a good persistence API or not. EJB have never been designed to be a persistence API. It is a component model. This why is doesn't support object-level features such as inheritance, fine-grained persistence, etc.

    >

    That's because a lot of people jumpt o EJB because of the persistence API. In my opinion EJB is very overused, and it's CMP is one of the reasons. A developer needs an O/R mapper and starts with EJB CMP. Then they need session beans to talk to the entity beans. Then they need all kinds of design patterns to overcome EJB complexity, and in the end they end up with an architecture totally drive by the decision to use CMP for persistence!

    Michael
  100. JDO is a good choice[ Go to top ]

    JDO is a good choice