In this first part of a series of blog entries, Ilya discusses persisting enterprise data in XML. This is an overview of what else is to come in the following posts. XML has been embraced throughout the industry for various reasons, but it has yet to make it big as a true native persistence technology.

I think most enterprise developers today look at XML as an intermediary transport format, configuration file format, and other applications where readability of data and interoperability is a must. It's been embraced more as an integration technology and for good reasons of course. There has always been a feeling that persisting application data can be done in XML, being that your domain model fits into the benefits criteria of XML Schema storage.

I think the biggest benefit of persisting data in XML is if your schema is dynamic and requires constant, and at times, unpredictable change. Yes, there are ways around it with relational schemas, like having generic tables that store field name/value pairs, etc... But that's an ugly way of covering up a true limitation of the relational schemas. XML schema is a dynamic hierarchical model that perfectly fits that particular use case.

What do you think about persisting data in native XML databases/file stores. What were some of your experiences?

What do you think about persisting data in native XML databases/file stores. What were some of your experiences?

We have used XML serialization for our object model using XStream for some time now, and it is definitely very very nice. Arbitrily complex models, which have no predefined schema (e.g. objects with hashmaps with objects with lists of random objects), can be stored. Schema migration is a matter of doing XSL's on the content.

I too have been using this in one of my implementations and it is very easy to use and performs very well.

I've used Java's built-in XMLEncoder/XMLDecoder for persisting to XML files with great success. Many Java objects get persisted without any effort and where effort is required it generally amounts to adding a few public methods to expose properties to persist -- or denoting some as transient in BeanInfo to avoid doubly persisting a property where multiple methods expose the same property.

I've used Java's built-in XMLEncoder/XMLDecoder for persisting to XML files with great success.

Many Java objects get persisted without any effort and where effort is required it generally amounts to adding a few public methods to expose properties to persist -- or denoting some as transient in BeanInfo to avoid doubly persisting a property where multiple methods expose the same property.

Jess, there are many very capable XML/Object mapping technologies out there. For some reason they keep being called persistence technologies. It's like if Hibernate generated SQL code on save, without actually persisting the object graph. There is a big difference between mapping objects to sql (and in this case objects to XML) and actually persisting object graphs. Also, I think when people start talking about persisting XML, it's mostly within a CLOB-type field in a relational database. Again, I was talking more about true native XML databases, that support XQuery and a myriad of other XML specific technologies and persist the data in a native XML format, and not break things down into an relational model behind the scenes, as a lot of RDBMS vendors do to support XML storage. Ilya

From my experience, saving XML as CLOB/BLOB in relational db does not really have any business value except that it can serve some purpose like xml logging. One of my big concerns about persisting XML data in DB (relational/xml) is versioning of xml data. One example is that it is hard to query data across different versions of same type of xml data in DB. My 2 cents. Rick

I've used Java's built-in XMLEncoder/XMLDecoder for persisting to XML files with great success.

Many Java objects get persisted without any effort and where effort is required it generally amounts to adding a few public methods to expose properties to persist -- or denoting some as transient in BeanInfo to avoid doubly persisting a property where multiple methods expose the same property.

Jess, there are many very capable XML/Object mapping technologies out there. For some reason they keep being called persistence technologies. It's like if Hibernate generated SQL code on save, without actually persisting the object graph. There is a big difference between mapping objects to sql (and in this case objects to XML) and actually persisting object graphs.

Also, I think when people start talking about persisting XML, it's mostly within a CLOB-type field in a relational database. Again, I was talking more about true native XML databases, that support XQuery and a myriad of other XML specific technologies and persist the data in a native XML format, and not break things down into an relational model behind the scenes, as a lot of RDBMS vendors do to support XML storage.

Ilya

XMLEncoder/XMLDecoder are naturally used for persisting XML object graphs, e.g. into flat files or CLOBs. It's not a nature means of persisting to an XML database as far as I can see -- but then again I've not used an XML database as the world is still quite relational. [XMLEncoder/XMLDecoder certainly don't lend themselves to ORM as I see it either, though the underlying Encoder/Decoder might...]

I've used Java's built-in XMLEncoder/XMLDecoder for persisting to XML files with great success.

Many Java objects get persisted without any effort and where effort is required it generally amounts to adding a few public methods to expose properties to persist -- or denoting some as transient in BeanInfo to avoid doubly persisting a property where multiple methods expose the same property.

Jess, there are many very capable XML/Object mapping technologies out there. For some reason they keep being called persistence technologies. It's like if Hibernate generated SQL code on save, without actually persisting the object graph. There is a big difference between mapping objects to sql (and in this case objects to XML) and actually persisting object graphs.

Also, I think when people start talking about persisting XML, it's mostly within a CLOB-type field in a relational database. Again, I was talking more about true native XML databases, that support XQuery and a myriad of other XML specific technologies and persist the data in a native XML format, and not break things down into an relational model behind the scenes, as a lot of RDBMS vendors do to support XML storage.

Ilya

XMLEncoder/XMLDecoder are naturally used for persisting XML object graphs, e.g. into flat files or CLOBs. It's not a nature means of persisting to an XML database as far as I can see -- but then again I've not used an XML database as the world is still quite relational. [XMLEncoder/XMLDecoder certainly don't lend themselves to ORM as I see it either, though the underlying Encoder/Decoder might...]

Jess, take a look at JiBX (http://jibx.sourceforge.net/) Yes, persisting to flat files and/or clobs is a different thing, vs. persisting to a native XML facility that support s XQuery and it's data model, though you're never really accessing XML in it's serialized form. My next post will describe in more details, as I see the need to point out the differences. Thanks.

In the past I have used eXist NXD. I found it to be pretty good. The main benefit we saw was that we were able to store xml files that were produced from an application into eXist NXD. We were then able to XQuery and generate reports from the XML natively. Other applications and processes were able to XQuery eXist (using same queries) and transform the result into whatever format was needed. Something that would have been really cool was if we could have had the application that produced the xml files store the objects directly into eXist, but alas the application was a vendor-purchased application and we had no option to make that big of a change in their design. I'm sure my examples don't scratch the surface of the possibilities of applying NXD to applications, but that's what we used it for. Looking forward to futher blog entries.

In the past I have used eXist NXD. I found it to be pretty good. The main benefit we saw was that we were able to store xml files that were produced from an application into eXist NXD. We were then able to XQuery and generate reports from the XML natively. Other applications and processes were able to XQuery eXist (using same queries) and transform the result into whatever format was needed. Something that would have been really cool was if we could have had the application that produced the xml files store the objects directly into eXist, but alas the application was a vendor-purchased application and we had no option to make that big of a change in their design. I'm sure my examples don't scratch the surface of the possibilities of applying NXD to applications, but that's what we used it for. Looking forward to futher blog entries.

Matt, did your application require transaction support? Being that eXist currently doesn't support transactions, was that not of your requirements or did you accomplish that in some other way? There are two sides of storing XML data, one is just a matter of persisting it as it's used as some intermediary format for communication (which I believe is your case), and the other is actually persisting your domain objects in an XML schema, which is where the Object/XML mapping comes into play. Thanks. Ilya

We did not have any type of transactional requirements. We were just persisting it to use it for intermediate communication as you said. We also used it (actually not in production) to store large xml docs during application processing.

More at: http://www-306.ibm.com/software/data/db2/9/

More at: http://www-306.ibm.com/software/data/db2/9/

Any idea on how it supports locking/concurrency? Locking in the native XML storage world is a different beast compared to the relational world, though I'd like to better understand how they support it. I couldn't find any documentation on IBM's site. Ilya

I've no experience with it at all, just saw the page . .but maybe the best bet would be the manuals, PDF's at: http://www-306.ibm.com/software/data/db2/udb/support/manualsv9.html There is also a brand-new draft Redbook for v9 that covers the free downloadable "Express" version - it is at: http://www.redbooks.ibm.com/redpieces/pdfs/sg247301.pdf Truely, it would be nice to hear some experiences with it.. regards - garyh

What do you think about persisting data in native XML databases/file stores.

I think it is the essential ingredient to help winning the 'inefficiency' contest: who will make things the most inefficient, non-performant, and cumbersome way.

What do you think about persisting data in native XML databases/file stores.

I think it is the essential ingredient to help winning the 'inefficiency' contest: who will make things the most inefficient, non-performant, and cumbersome way.

Konstantin, can you please explain? Is there some bad experience behind that claim? What's inefficient about storing files in an XML native store? Traditionally native XML databases had many inefficiencies, I agree. That is changing. XML indexing and storage on internally, has been optimized in many scenarios to scale and perform beyond a relational store. Of course that all depends on your data and whether it's best represented in a hierarchical manner or rectangular. Ilya

Konstantin, can you please explain? Is there some bad experience behind that claim?

Yes, bad experiences. In way too many cases use of XML makes things harder than it would be with POJOs. Sometimes XML makes things easies for short term, but acts like time bomb in the long run pretty much like simplicity of CGI and Perl based web sites and applications. Of course if someone has to use XML (dig with a spoon) then XML database and other XML oriented tools are necessary (wow, that is the Shovel!). PS: Never mind that there are excavators are available and there might be even no need to dig that pit in the first place :) PS2: there are few legitimate places where XML can be beneficial, but I estimate it like 5% or less.

Yes, bad experiences. In way too many cases use of XML makes things harder than it would be with POJOs.

But these are different. POJOs is about building your domain model, XML is about persisting it and/or using it as a communication medium. I'm not sure I'm understanding your point of view. When using a POJO based approach for your domain model, you still have to persist it somewhere. In your case, you're most likely persisting it in a relational model. But my write up deals with the use cases where your domain model closely resembles a hierarchical format vs. a ractangular one. There is a impedance mismatch with XML/OO as there is with relations/oo models, but my argument is the fact that many domain models today are best represented in XML, rather than in a rectangular relational model. I mean, it's the same as the case for OO databases. Also, in a SOA based architecture, where XML is already used to marshal your object for RPC, why have another level of going to and from a relational model vs. persisting in native XML.

Sometimes XML makes things easies for short term, but acts like time bomb in the long run pretty much like simplicity of CGI and Perl based web sites and applications.

Can you be more specific? I'm not trying to be hard here, but comments like that get us no where and I nor anyone reading this is getting any more knowledge. I'd love to hear your experiences.

there are few legitimate places where XML can be beneficial, but I estimate it like 5% or less.

Where did you get this number? Ilya

XML is about persisting it and/or using it as a communication medium. I'm not sure I'm understanding your point of view.

IMO in many cases RDBMS are better choice for persisting data. And sometimes LDAP, and sometimes filesystem, and sometimes plain text. POJOs of other lean data structures (CORBA, ICE, and alike) make very good communication medium.

SOA based architecture, where XML is already used to marshal your object for RPC

You mistakenly equate SOA to XML based SOA implementation. There are other ways to implement SOA: CORBA, RMI, Jini, Grid technologies. Please see note on P 135 of "Service-Oriented Architecture" by Thomas Erl. He is honest in his advises regarding XML based SOA. Why marshal to XML if DataStructure can be transferred by other means? DataStructure _can_ be stored in ObjectDatabase or in Relational. XML is not silver bullet. Can we agree on that?

XML schema is a dynamic hierarchical model that perfectly fits that particular use case.

"dynamic hierarchical model" sounds almost like oxymoron. Dawn of hierarchical databases was caused by their rigidness. I would say that rigidness is endemic to hierarchy of any kind.

IMO in many cases RDBMS are better choice for persisting data. And sometimes LDAP, and sometimes filesystem, and sometimes plain text.

I don't disagree that they might be more stable ways of persisting it right now, due to the investment of billions of dollars and time by RDBMS vendors. But I disagree that it's a better choice of persisting data. Just like OO databases where always a better choice of persisting objects, XML databases are better for persisting XML. OO databases didn't make it to mass adaption not due to the fact the RDBMS was a better choice, but rather because of some not so good decisions and directions that where taken by groups/companies embracing it. One of them was the fact that they didn't standardized query language like SQL to relational. Yes, there was OQL, but it was never fully finilized/implemented.

POJOs of other lean data structures (CORBA, ICE, and alike) make very good communication medium.

Yes, and so did EDI at one time. There were many good technologies that had their own benefits that never truly where adapter by the masses, though CORBA came pretty close). Binary storage formats where also pretty popular at one point, but even Microsoft is now storing their Word, Excel, etc... data in XML. I guess the interoperability benefits beat the incremental efficiency.

You mistakenly equate SOA to XML based SOA implementation. There are other ways to implement SOA: CORBA, RMI, Jini, Grid technologies.

I don't, I think you're misunderstanding the benefits of XML technologies. It's not about whether it's efficient, better, etc..., it's the fact that it has been embraced and standardized upon by most vendors and technologies, though finally giving us a common communicatin medium that you can bet is most likely supported by some other vendor in one shape or another. XML standars have come a long way to finally providing a common communication medium that no other technology really ever did across industry standards.

Why marshal to XML if DataStructure can be transferred by other means?

It's not about senselessly (un)marshaling, but rather abiding by your data model. If your data model is better represented in a hierarchical manner, than XML is your way to go, IMO. Also if your application requires to abide by some current industry standard that is represented in XML for interoperability, why use a relational store add another level of marshaling, since you would have to represent that data as XML at one point or another anyways.

XML is not silver bullet. Can we agree on that?

I never said it was. You might be misunderstanding me again. I'm not saying drop relational stores, rather I'm saying if your application currently represents it's domain model in XML, why add another impedance mismatch and now also have to support it as a relational structure.

"dynamic hierarchical model" sounds almost like oxymoron. Dawn of hierarchical databases was caused by their rigidness. I would say that rigidness is endemic to hierarchy of any kind.

Why does it sound like an oxymoron? When you can have dynamic hierarchies within a strict hierarchy, etc... If your application needs to support it, why fight it? Say you have a non-predetermined model that allows dynamic configureation of state, basically fully dynamic objects, not predetermined at compile time. A good example of that would be, dynamic forms that are configured at runtime, and have to be typed. Forms, might even have subforms, etc... (composition), and all is dynamic, and they all have to enforce foreign key relationship integrity. In a relational world, it's mostly represented as a huge table, that would keep the attribute name, value, it would not be strongly typed. Doesn't fit into the rectangular model very well, does it. Of course it can be done. You can also store XML in an OO database, and relational data in an XML database. You can write huge programs in a procedural language, just as you can in OO. All I'm saying here is that all data models do not fit the relational structure and if it doesn't, why fight to fit it in there. Ilya

I don't, I think you're misunderstanding the benefits of XML technologies.

Ilya, I base my conclusion about your confusion on your words:

Also, in a SOA based architecture, where XML is already used to marshal your object for RPC,

Please provide clarification on what you mean rather than question my ability to see benefits of of XML technologies asking why I think XML technologies are useful in 5% of cases. I also invite you to reread you words and laugh with me:

It's not about whether it's efficient, better, etc...,

Is it mean that the XML technologies are worse than others and inefficient....

it's the fact that it has been embraced and standardized upon by most vendors and technologies,

Hmm, feodalism and communism were widespread at some points in history, so what?

though finally giving us a common communicatin medium that you can bet is most likely supported by some other vendor in one shape or another. XML standars have come a long way to finally providing a common communication medium that no other technology really ever did across industry standards.

Really? Have you seen recent interoperability tests? Oh, and last time I checked my TV and Radio signals were not in XML form :)

Also if your application requires to abide by some current industry standard that is represented in XML for interoperability, why use a relational store add another level of marshaling

Keyword here is _current_. There is no guarantee that it will stay current. While RDBMS standards are very well time tested. Also as you pointed out earlier XML is not efficient, not better etc. :)

All I'm saying here is that all data models do not fit the relational structure and if it doesn't, why fight to fit it in there.

All I am saying is: - in many cases XML is not the best way to do business; - it has been pushed too far and now used where it should not. Pretty much like EntityBeans were few years back;

Ilya, I base my conclusion about your confusion on your words:

Constantin, I don't know if you've again misinterpreted the context of my reply, but I was never getting personal with your, but you seem to reply in a defensive manner. I won't let this post get down to that level, at least not on my end.

Also, in a SOA based architecture, where XML is already used to marshal your object for RPC,

Please provide clarification on what you mean rather than question my ability to see benefits of of XML technologies asking why I think XML technologies are useful in 5% of cases.

What I meant is whether you agree with it or not, majority of todays SOA architectures are being designed and developed using XML technologies. Again, please don't start telling me why it's a bad idea, because it's simply reality and if you want to participate in such architectures for business transactions, etc..., you'll have to bite your tongue and play nice.

I also invite you to reread you words and laugh with me:

It's not about whether it's efficient, better, etc...,

I'm rereading my words and find nothing laughable there, but everyone has a different sense of humor. Are you still the guy that screams about why we should use C and C++ without managed memory, because it's more efficient? Or do you find that depending on the requirements, it's sometimes (and sometimes more than others) OK to say that efficiency and speed is not as crucial as say, maintainability and cost? Maybe we should write an SOA architecture in assembly and binary protocols. It's beat the efficiency of Java and CORBA.

Is it mean that the XML technologies are worse than others and inefficient....

I know what you mean, and then I don't:-) I understand your opinion, but I disagree. And you haven't provided any sufficient explanation as to why, to make me consider otherwise.

Hmm, feodalism and communism were widespread at some points in history, so what?

Oh, come on, are you serious? I know I'm from Russia, as I believe you are, but you don't have to try to find common ground based on communism examples. Just like the rule of who ever mentions Hitler in the thread looses, same should apply to communism.

Really? Have you seen recent interoperability tests? Oh, and last time I checked my TV and Radio signals were not in XML form :)

That's mostly because they don't have as great of a demand in interoperability between industries, etc... Their technology is contained within the same industry and the other companies must comply. You can't achieve the same with cross-industry support. EDI was also efficiently used in transportation and logistics, but 90% of folks who worked with it, would never want to look at that format again, and it wasn't even as bad as the binary format your proposing above.

All I am saying is: - in many cases XML is not the best way to do business; - it has been pushed too far and now used where it should not. Pretty much like EntityBeans were few years back;

And all I'm saying is I'd like for you to explain why. I've listed several reasons why I think it's a great way to achieve communication between different technologies/industries. You just keep saying that it's bad and providing analogies and no concrete points. Constantin, seriously, I'm not being personal here. I initially replied because I wanted to hear your experiences, not merely your opinion. Maybe we just misunderstood each other.

but I was never getting personal with your, but you seem to reply in a defensive manner

IMO it is OK to get personal (without name calling of course) since we all express our personal opinions and it is OK to point that a person does or does not understand something.

majority of todays SOA architectures are being designed and developed using XML technologies.

Let me remind that SOA is ‘architectural concept’ and it has been in use for a long time before current marketers tied it to XML. I would say that majority of Service Oriented Systems today are _not_ XML based.

Again, please don't start telling me why it's a bad idea, because it's simply reality and if you want to participate in such architectures for business transactions, etc..., you'll have to bite your tongue and play nice.

…efficiency and speed is not as crucial as say, maintainability and cost

I say that correctness goes first and maintainability goes second on pair with convenience to develop. And this is exactly where WS* stack is not that great at all. The beaten to death argument of ‘loose’ coupling mislead many because it does not tell exactly what kind of coupling is loose. No matter what technology we use we still have tight ‘content’ coupling and XML based stack is very weak on enforcing content contracts. Yes, it is possible to validate XML documents and messages but it brings system on the knees almost instantly. XML parsers – still have bugs despite XML promise was to free most of developers from writing parsers.

You might be misunderstanding me again. I'm not saying drop relational stores, rather I'm saying if your application currently represents it's domain model in XML, why add another impedance mismatch and now also have to support it as a relational structure.

Let me repeat myself: I am not saying to drop XML, I rather saying “use XML where it makes sense and question carefully if my current or proposed use of XML makes sense”. What you fail to realize is that I am ally of XML technologies :) seriously, I try to prevent their abuse and misuse that will lead to the inevitable backlash.

Maybe we should write an SOA architecture in assembly and binary protocols. It's beat the efficiency of Java and CORBA.

performance!=efficiency Efficiency is much broader concept

but you don't have to try to find common ground based on communism examples.

If people want to conduct a productive dialog then they have to try to find some common ground and work out differences.

Again, please don't start telling me why it's a bad idea, because it's simply reality ….

Do I need to explain why this is very bad attitude? By the way RDBMS is the prevalent form of storage – ||it’s simply reality||

Just like the rule of who ever mentions Hitler in the thread looses, same should apply to communism.

“Those who cannot remember the past are condemned to repeat it.” If rules were absolute then you … win? :)

Let me remind that SOA is ‘architectural concept’ and it has been in use for a long time before current marketers tied it to XML. I would say that majority of Service Oriented Systems today are _not_ XML based.

Well, yes, I agree with that, though I think these SOA architectures were contained within an organization and/or industry constraints (of course there were exceptions). Most of the other technologies, though CORBA came close, were never as widely embraced and standardized upon. I think today is when we're starting to see the true adaption of SOA, though why we're starting to discover many issues surrounding it. It's still over-hyped, no disagreement here:-)

Let me repeat myself: I am not saying to drop XML, I rather saying “use XML where it makes sense and question carefully if my current or proposed use of XML makes sense”. What you fail to realize is that I am ally of XML technologies :) seriously, I try to prevent their abuse and misuse that will lead to the inevitable backlash.

Ok, I agree. And my write up was centered towards applications that have a domain model that is already either represented as an XML schema, due to an industry standard, and though they must abide by it, and/or domain models that best fit the XML schema hierarchical storage concept. I, in no way, advertised the demise of RDBMS systems, rather said that they are sometimes used when they might not be the best choice. As is also the case the other way around.

performance!=efficiency Efficiency is much broader concept

Yes, I know that, though they are in most cases parallel, when you achieve efficiency, in most cases, performance improvements are also inherent.

If people want to conduct a productive dialog then they have to try to find some common ground and work out differences.

:-) Ok, fine. Which part of former USSR did you come from? :-)

Do I need to explain why this is very bad attitude? By the way RDBMS is the prevalent form of storage – ||it’s simply reality||

Yeah, and I'm not saying don't use it. It's probably useful in majority of the cases, but there are some that are not. Just like your TV and Radio example. I wouldn't recommend using XML there.

“Those who cannot remember the past are condemned to repeat it.” If rules were absolute then you … win? :)

I'm not trying to win, I'm trying to learn from experiences, whether they are mine or yours. Ilya

What do you think about persisting data in native XML databases/file stores.

I think it is the essential ingredient to help winning the 'inefficiency' contest: who will make things the most inefficient, non-performant, and cumbersome way.

It is still far more inefficient, non-performant, and cumbersome to deal with impedance mismatch and mapping problems on all levels of your application than going the XML way. Of course, you may wish to stay away from XML entirely, but it can be hard sometimes, as is the opposite, but you can still make smarter choices how you choose to use XML and how you build applications that work with it.

Hi Ilya This is Mansoor from UAE, could you please tell me whether the XML Persistence can satisfy all transactional requirements of a complex usecase such as create,search,detlet,modify,etc. in a master-detail scenario? At present I am planning to use XML for persistence while database connection does not exist. In the later part I would read from the XML and persist in the Database when connection exists. Please suggest any XML API/Technology/Tool,etc. which would match the Database Persistence.

Hi Ilya

This is Mansoor from UAE, could you please tell me whether the XML Persistence can satisfy all transactional requirements of a complex usecase such as create,search,detlet,modify,etc. in a master-detail scenario? At present I am planning to use XML for persistence while database connection does not exist. In the later part I would read from the XML and persist in the Database when connection exists. Please suggest any XML API/Technology/Tool,etc. which would match the Database Persistence.

Mansoor, I need to better understand your requirements. CRUD scenerio you described above is pretty basic functionality. Not sure if you can share the details with this list, if not, just go to my page and contact me there, then we can discuss. I'd like to hear what you're doing. http://ilya.typepad.com/about.html Ilya