Our problem is mapping our legacy database schema to XML in a maintainable way – allowing for easy handling of future changes to the schema of either database or XML.

In our case we need to disseminate large sets of information in several formats - therefore XML and XSLT are a natural, now the problem is mapping our legacy database schema to XML in a maintainable way – allowing for easy handling of future changes to the schema of either database or XML.

Our solution – create a hierarchical value object model – where each value object apart from having the usual value properties has a toXML() method which serializes the object to XML according to the corresponding part of the XML schema.

By having a hierarchical model a top level value object may contain collections of lower level value objects and within it's toXML() method it would call upon those objects toXML() method therefore allowing for very clean localization of the database schema to XML schema mapping.

Therefore by adding XML "awareness" as methods to the value objects – the server work stays equal setting in the properties of the objects, and the client work becomes trivial – receive any value object and dump it to XML while still having the flexibility of combining and ordering it as wanted – which you don’t have if you pass only XML back to the client as well In our case there is the need to access properties of the value objects apart from generating XML.

This pattern creates self contained value objects where all of the mapping code is located where it logically "belongs" - within the object - where the properties are kept.

Hi Noam,
we are having the exact same solution. A lot of our classes are created/instanciated out of Configuration Files. We actually went a little further and use XML Based constructors, to initialize the classes directly out of an XML element. This works perfectly fine, and we actually developed a little framework around it, helping to read/write the XML-Elements. Yet there are two thíngs that I like to change the next time:
 - use a standard XML-Binding Network. We looked at some Systems (i.e. Zeus, Castor) and were uncontent to see, that they all require the usage of bean like method signatures to work properly. This would have forced us to develop unwanted Class Signatures and XML-DTDs to implement a non-standard XML-Marshalling Framework. I think things will change with the JDK 1.4. Having the Serialization built directly into the system, might be a motivation to actually change all the classes and documents.
 - use a builder pattern to seperate the creation of the class and the class itself. In some of our classes the XML-Adaptor code is actually longer and more complicated than the actual logic of the class itself. So I think I will have seperate builder classes, as defined by the GOF.

The wrap up, it seems unlogic for me to go through the trouble of adapting the XML- and Java-representation of Information just to comply with the needs of Marshalling Frameworks, for domains with a constant Information Structure. Building you're own Adaptors has proven to be faster and the design of the classes and documents is more straight forward, meaning the maintenance of the code is a lot easier.

Of course I lack the experience of devloping code for dynamic information structures, nor have I ever used a predesigned framework for larger objectgraphs, and I'd love to hear feedback from people who have actually used such software.

Stefan

Research on dynamic systems was done long time ago before J2EE or even Java were popular.

Adaptive Object Model
AOM heavily uses type object pattern.

Excellent resources on metadata

One more intresting reading about Metadata
by two fathers of metadata Foote and Yoder ;)

Instead of XML Aware Value Objects, why not use XML-Empowered Value Objects that work with the DOM directly? Yes, yes, heavy weight, but that is a question of whether your hardware and performance requiremets allow it. In many cases they do, and it the cases they don't, you need more hardware anyway. And if you are going to convert to XML your data anyway, I don't see the point in passing it through Java data structures, which are convenient on a low-level access to data attributes and properties but fall flat on a larger scale when you have to look up and access objects in your tree. The DOM coupled with XPath expressions is far more powerful. I am not sure if the Java DOM supports data types, but that is easily fixeable with some small helper classes.

Our application does rerquire quite a lot of resources as is:
a diferential data extraction creates a 25-50 Meg XML file.
try holding that in a DOM tree...
a complete extraction is about 2-3 Gig XML file which is NOT created by holding a tree in memory - not even of binary objects - instead each sub item is created, outputed and discarded.
JDOM has proven right for the job, yet I must say we only use it's very basic functionality - constructing a tree of elements and attributes, setting schema and namespaces, and then outputting either the whole document or just a branch.

I agree with the few folks who have favored the "clean" approach. I just read the JAXB specification and API docs for the first time, and I am not happy about the requirement that objects implement special interfaces before they can be marshalled/unmarshalled to and from XML. I am willing to accept a peformance hit when performing this operations if it means that maintenance of the marshalling code can be consolidated in one framework.

Cacheing of the reflective metadata required to marshal instances can help alot with performance, and doesnt break your separation of concerns. I have in the past lost the debate about whether objects should know how to render themselves, persist themselves, or what have you, and it resulted in bloated classes that were punishing to maintain, even for the small application we were building.

I think that time spent designing a marshalling framework that is fully capable of introspecting the managed objects is well worth the effort. The less code required, the better, as always. I hope that the JAXB team can be persuaded to abandon their current stance in upcoming revisions of the draft specification. The forces encouraging the "dumb managed object" approach seem to be fairly obvious.

This problem is really several distinct issues I recon.

1) The technical issue of updating code throughout all systems to reflect schema changes.

2) The business issue of ensuring that only changes are rolled out in a highly structures, controlled and tested way.

Often schema changes are dictated and accepted with no risk analysis, eg changing a column width can blow out vast chunks of a system if there are implicit dependancies.

Another view on the same problem is once of services/data feeds. I bet you find yourself using the XML classes as a data feed mechanism as well as a publication system, which means you are immediatly into protocol versioning (or object serialisation versioning). eg a feed written a year ago is V1 and you are now on v2 which has dropped 2 columns and added a third. What should you do?

As ever, the solution depends on the scope of the project, your time scales etc. My only recommendation at this stage is to ensure EVERY XML stream includes a version number specific to the data being serialised. This gives you the option of supporting old versions if you want to.

Jonathan

We have been through the toXml()/fromXml() route, and then swapped to to interrogating objects using reflection to output XML based on the accessor method names. This solves issue 1 for us, and as we are only using the XML internally issue 2 does not apply to us.

However, I have encountered elsewhere the issue of controlling changes to a published interface that maps on to an underlying internal interface that is more fluid. This can happen with an API, and it can happen if you allow direct access to your database for reporting purposes. Where I have seen this the approach taken has been to code all those mappings manually.

My ideas around this have been: to use a custom Javadoc doclet to generate mapping classes. Many mappings could be done by using type information, but doclets would allow the use of custom javadoc tags to control the mapping where a default is not what is wanted. I like the idea of storing the mapping details in the source code, while not adding to the bulk of the actual class. Hopefully programmers making changes will see the mapping info and realise their importance.

This approach should also help with the detection of unwanted changes to the API. Changes to the mappings, will be reflected in the serialVersionUID values of the classes. These can be computed using the jdk program 'serialver'. If these values change then the existing API has been broken.

JAXB could probably be used in the building of these mapping classes. The embedded javadoc tags could give the DTD and schema info required. I haven't had a chance to play with JAXB, so I haven't thought this through yet.

Please read this article by Mark Pollack about code generation using Javadoc on JDC. http://developer.java.sun.com/developer/technicalArticles/Programming/Javadoc/

Reg

>doclet to generate mapping classes. Many mappings could be
>done by using type information, but doclets would allow
>the use of custom javadoc tags to control the mapping
>where a default is not what is wanted. I like the idea of
>storing the mapping details in the source code, while not
>adding to the bulk of the actual class.

Have you looked at XDoclet? (http://sourceforge.net/projects/xdoclet) This is just the sort of thing it was developed for; there's already (undocumented) support for Castor in there, or you could do your own mapping classes with a custom template.

To clarify, as Jonathan pointed out, there are several issues targeted by the pattern.
Our architecture is rather standard:
App server - middle tier (web server) - client

We had the following requirements:
1. Mirroring the data from our legacy db to XML, so that:
   A. the middle tier or end client can perform XSLT transformations to several other formats.
   B. the middle tier can access some of the data fields directly - preferably without needing to parse the XML - for formatting of the views.

2. a highly configurable end user output - yet a rather stable XML schema - we will be publishing the data as XML as well - and don't want to be changing versions more than absolutely necessary. (see Jonathan's point 2)

3. keep maintenance simple - one code base of db to XML mapping for both server and middle tier. (see Jonathan's point 1).

The "standard" solutions were much heavier than we needed as they provide a configurable two way binding while we wanted a one way only mapping, db to XML, and rather fixed - we get all the configuration flexibility we need from using XSLT.

The strength of this pattern is that on the server side all looks like we're using the usual light weight value objects.
And on the client we get BOTH the value objects and the XML - without needing XML parsing libraries on the middle tier.
As a bonus we also get efficient network usage as the serialization is of binary value objects and not the full blown XML.

Before coding toXml() or similar methods in data objects you need to think whether it will not result in coupling data objects with third-party libraries. Personaly I do not think that it is data object's responsibility to know how to store them in the XML-based storage especially if you use the same data objects both at client and server side's - this way client also needs to have XML libraries although conversions to/from XML are usually done at server side only. Also what if you need to store your objects in different form, e.g. LDAP, some relational database or some other formats? Would you code methods toLDAP(), toMySpecificFormat() and so on in the same object??? I would prefer separating data objects from their representations in XML or some other formats to keep data objects as light as possible. I would write some input/output stream or similiar writer/reader classes for storing data objects to XML and reading them from XML. Thanks for attention.

 I agree. I believe XML requires a more orthogonal
 persistence mechanism, via a Marshaller or Codec, to
  stream your object to and from XML. Often you can't
  pre-generate the object you need to stream ( as in JAXB)
 or it is not easily touched ( perhaps your value objects
  were already generated for some other persistence scheme,
  like JDBC or entity beans ). This approach also makes
  versioning easier ( different codecs for different
  versions or variants of your XML ).

The VISITOR pattern addresses exactly this issue, allowing operations to be performed on the elements of an object structure to be defined in a class separate from the elements themselves.

Yes, but in visitor pattern you have some restrictions
1) you should have hierarchy of subjects
2) this hierarchy shouldn't change very often, because then you need to change visitor interface for adding more "visit" methods for new subjects.

What about using a decorator pattern instead? the basic object classes would not contain any logic beyond data and business rules. Then decorator classes could be created to add the toXML() method to the various objects.

Like so
 Business b = new Business();

 XMLBusiness x = new XMLBusiness(b);

String xml= x.toXML();

of course a decorator would have to be created for each class type in the hierarchy
Jeff Anderson

I think in this case most appropriated pattern is Strategy. You can create strategies for various data sources.

I used the Composite pattern together with the Proxy to achieve this in a project. Alan Holub had some interesting insights into this in a series he had in Java World last year.
Every object that needed to represent itself should implement a "Renderable" interface. A builder is then populating the object's attributes with whater "Renderer" is needed for the task. For example XMLRenderer, or HTMLRenderer. Each attribute has a method "render()" that actually delegates it to the pre-configured renderer. A composite object performs that task just by iterating through all its "children" and ask them to render themselves.

In the end I think it's just a variation of the model described previously. It is complex to build it but once you have it on mapped out nicely it is nice, elegant and flexible.

Thanks for all of the input - here's example code of what we ended up with:
We've started using JDOM and found a very elegant way of doing what we need.

I have an interface:
public interface XAValue {
public String toXML();
public Element getElement();
}

Now when you want to create a full blown XML document you can simply construct a JDOM Document and add the root element using the getElement() method on the root value object.
Then you can do whatever you want with the Document, output it, perform transforms on it, etc. using standard JDOM methods.

Using the JDOM Elements as the mapping tool allows us to keep the objects light weight as the Element is only created on demand and therefore does not cross the wire while gaining all the benefits of a standard API (plus the conversions to DOM and SAX that JDOM provides)

Regards,
Noam

Code example - In the value objects the interface is implemented as follows:

public class ID implements XAValue{

public String number;

public ID(String number) {
this.number = number;
}

public String toXML() {
return Outputter.getInstance().outputString(getElement());
}

public Element getElement(){
Element elem = new Element("id", "");

elem.addContent(new Element("number").addContent(number));

return elem;
}
}

Where Outputter is simply a singleton wrapper for the JDOM XMLOutputter:

public class Outputter extends XMLOutputter{
private static Outputter ctxt;
public Outputter() {
super(" ", true, "UTF-8");
}
public static Outputter getInstance(){
if (ctxt ==null)
ctxt = new Outputter();
return ctxt;
}
}


and a more complex value object would look like:

public class Representative implements XAValue{
public XAValue id;
public XAValue agentType;
public XAValue contactInfo;

public Representative(XAValue id,
XAValue contactInfo,
XAValue agentType) {
this.id = id;
this.contactInfo = contactInfo;
this.agentType = agentType;
}

public String toXML() {
return Outputter.getInstance().outputString(getElement());
}

public Element getElement(){
Element elem = new Element("representative", "");

elem.addContent(id.getElement());
elem.addContent(agentType.getElement());
elem.addContent(contactInfo.getElement());

return elem;
}
}

//Notice how the toXML() method is the same.

>We've started using JDOM and found a very elegant way of >doing what we need.


Hi!
I'm just curious to hear if you have had any problems with the approach you described? Has JDOM been working alright? (It's just a beta isn't it?)

Regards,
Jöns

JDOM is very expensive affair even when compared to DOM parsing. Would it be right decision to use, specially as the latency of operation would directly add to response time for the end user.

What about generating SAX events instead of Strings: i.e.
define
void toSAX(ContentHandler ch)
instead of
String toXML()

You can then pass any ContentHandler: to generate String (text) output, a DOM tree, or pipe to a XSLT engine that can handle SAX events...

We did the same thing in a project.
In our implementation the XML mapping was stored in an XML configuration file according to our own specs (I know but there was no standart yet) And the XML was then created by a reflection type of factory. This kept maintainability high and kept direct mapping code non existent.

Cees Habraken

Have you look at JAXB and Sun's RI for it?.

Look at Castor...http://castor.exolab.org/ seems to have more features the current JAXB....

I think you definetly should take a look at "Adaptive Object Model". It's aproach how to create dynamic, "self aware" data object model.
Main idea behind "Adaptive Object Model" is type object pattern.

In this model you have:

1) attribute type - type of attribute aka class
2) attributes - which represents instances of attribute type
3) entity - set of attributes
4) EntityType - defines what kind of attribute types entity
   contains

At attribute type level you can create strategies for attribute formatting, editing (converting from text), maping to JDBC statements, marshaling and so on.
You can introduce your own types like SSN, AccountNumber, Money, PhoneNumber, EmailAddress ...

At entity level type you can create more complex strategies for mashaling, maping ...

In this model you can write code like this:

moneyAttributeInstance.format() - produces "user friendly" representation of money as text (Note: that inside format method delegates format call to type FormattingStrategy)
moneyAttribute.setFromTtext("10.100")-
(Note: that inside format method delegates format call to type JDBCMapingStrategy)

Giedrius

Ups typo :) I though about JDBC but wrote about EditingStrategy
Correct version:

moneyAttribute.setFromTtext("10.100")-
(Note: that inside setFromText method delegates call to attribute type EditingStrategy)

Each attribute has reference to it's type.

Giedrius

Check out XchainJ.com -- these guys have a GUI to map between XML / Java / database schema then their runtime processor takes care of the coding (don't need SAX/DOM/JDOM/JDBC). Cool!

My solution to the problem was to use JAXB. It required development of a DTD, and optionally a transform specfication for each table.

I looked at similar solutions like Castor, as well as, homw grown stuff. But what drove me towards this solution was that given Sun's seal of approval, I could be assured that is would be a standardized solution.

Another major advantage that I found was that I could subclass from the class generated by the transform compiler. This allows me to instantiate a different classes for client operations and another for my EJB land value object, using XML to go from one to another.