The Smart Value Object (SVO) allows server components to track client-side modifications of business objects in a rich client/J2EE server environment, by using the latest features offered by bytecode processing tools.

When developing client/server applications, the usual way is to implement the Value Object (VO) pattern, through a set of Java objects representing a lightweight graph of persistent server objects, that the server sends to the client for user modification.

The Smart Value Object project intents to go a little further by:

* transparently managing concurrency in a multi-user environment,
* analyzing the actual modifications done on the value object, to optimize data exchange and persistence issues,
* allowing a simple way of defining the graph of retrieved VOs.

The design makes the SVO independent from the persistence layer (entity EJB, JDO, Hibernate, ...).

This project is part of the Bright Side Framework (http://www.bs-factory.org), whose goal is to provide ready-to-use high level components to quickly build business J2EE applications accessed by rich java/Swing clients on HTTP.

Started in November 2003, the goal is to have a first version running in January 2004.

Currently, we are at the prototype stage, testing implementations. We encountered difficulties around the ClassLoading of our modified VO in a J2EE server and had to work around it by performing a post compilation modification of VO instead of a dynamic runtime modification. We will work again on this deployment part in the future.

The modification of the bytecode adds versioning info to the VO to allow a server service to verify the concurrency. It also modifies the access to the attributes by adding a field interceptor to flag the VO as dirty when needed. In case of a graph of VOs using collections or maps, these are replaced by our specific implementations in order to flag the added or removed dependent objects.

We have not figured out yet the way of defining the size of the VO graph.

As you see this project is in its early stage. We're interested in your opinion and ideas, so don't hesitate to give your feedback and share your experience

Read the full story on http://forge.objectweb.org/forum/forum.php?forum_id=365

CarrierWave (http://carrierwave.sourceforge.net/) has the same goal, but a more general approach.

In addition to "value object" (better known as data transfer objects in the J2EE world) generation, CarrierWave also queries for DTOs (and has great support for defining the object graph closure), executes actions/commands and integrates with persistence mechanisms.

It's basically an automated DTO service, and should be used to implement the business facade or service layer on top of a POJO domain model. It can also help to separate the presentation layer developers from the developers on the "server side". One party provides a "view" on the business objects (using JavaDoc tags on domain POJOs) and finder methods, the client developers use that interface to work independently. The transfer objects are automatically generated, so this is source generation instead of bytecode hacks.

It's still obvious that it is an in-house framework of some company that was open sourced, so it needs more help and attentation. I recommend reading the patterns on the website first, to get an idea what its all about. Don't let yourself be distracted by the names and terms (Image, Imageable, etc.), once you ignore that, its easier :)

I looked quickly at the CarrierWave project. This project seems to be far more complicated than the Smart Value Object one (more than 200 classes).

Our goal is to provide some facilities in a particular deployment target : a java client using serializable VO retrieved from a server.

We don't adress the finder problems. We think that SQL is the best way to present datas in a tabular way. The user then can choose the domain object he wants to edit. The SmartValueObject is used for this domain object graph, transparently analysing the user modifications.

We chose bytecode modification to be the less intrusive in the development process. Our first implementation performs the modifications during an ant task but we hope to do it at runtime. The client code can modify the VO as we would do with any "normal" javabean. The server has then the possibility of analysing the VO to check if some modifications has been done and where.

We will soon provide a first webstart demo application to show it in action.

Gaetan Zoritchak,
http://www.bs-factory.org

While not exactly the same thing, the SDO of JSR 235 does a similar thing. It should be able to track changes made on the client side and send a diff document back to the server where it could be applied to a backend, for example, using optimisitic locking on a SQL backend.

Billy

Some of the features are the same : tracking the client modifications, using optimistic concurrency but that's all (because there is not much more in the SVO project ;-) ).

Our API is again far less complicated. The client code has nothing special to do and the server code can perform this kind of call :

    SmartAnalyser.isDirty(myVO); //Checks client modifications on this object
    SmartAnalyser.removedIterator(myVO.getChildren()); //Iterates on the removed children

The modifications are tracked by field interceptions. Collection, Maps and Set interfaces are replaced by custom implementations to track the modifications done on dependant objects.

Gaetan Zoritchak,
http://www.bs-factory.org

> SmartAnalyser.isDirty(myVO); //Checks client modifications on this object
> SmartAnalyser.removedIterator(myVO.getChildren()); //Iterates on the removed >children

Looks nice. But can you check which "properties" have been changed?

For example, you have complex VO with 100 fields (coming from different tables)
Is there a way to find which field was modified?
So, instead of writing back the whole list of 100 fields, you write only changed ones?
Also, how do you represent DB NULL values? Do you always use Object or you can use primitive types (e.g. "Integer" vs "int")?

Thanks for the reply.

Currently, we flag the whole SVO as dirty when it has been modified on a single field. The main reason is that we rarely use VOs with 100 fields. We prefer composing a graph of small specialized VOs. Then when one is marked as dirty, we must persist the modification so it doesn't really matter to go deeper in the analysis.

The SVO can use primitive types.

The correspondance for the DB NULL values is a null object. For instance, a NUMERIC type will be represented by Long instead of a primitive long allowing to distinguish the 0 and the null values.

Gaetan Zoritchak,

Gaetan,

What is the benefit of using SVO's over factory driven DTO's?

For example, let's say I have a Web form, which requires a set of objects to update. First, I'll send the request to the server along with a list of required fields. Secondly, a DTO factory would iterate the field list and create the dynamic objects. Thirdly, the dynamic objects would "get" the data from the entity domains. Finally, the dynamic objects, which are a graph image of the entity domain is return to the Web form.

If the graph image is updated then the specific object within the graph is flag for update. Also, the application server ( Weblogic ) could check for deltas and only apply the changed fields.

Note:
The above scenario works with a Cocoon Web interface layer, so reading dynamic objects graphs are simple.

Also, an added overhead of sending a "known" list of fields to the server is required. But, no big deal because you got to know this anyway...

How does your SVO compare with BEA'a and IBM's SDO specification?

I really appreciate your efforts regarding the "distributed" data object opportunities; this surely is disconcerting "real world" J2EE projects out there.

Hi George,

The use case you describe is near from the SVO specification. Our approach is different mainly because we often develop rich client applications and rarely web client applications (see our website).

When developping rich client application you really prefer to use static interfaces to access your domain objects. This allows you to reuse UI components based on interfaces, apply some business validators ( the same that you would use on the server side), ... and benefit from the classic advantages of static interfaces (code completion, compile-time type cheking, ...).

The SDO specification wants to propose the both types of access to data, static and dynamic, but the main one is the dynamic one. The static interfaces are, for instance, generated from XML schemas. This part of the specification is not very clear to me but I imagine it a little bit complicated ( a proxy accessing the data objects and data graphs).

To compare the SVO to the SDO specification I would say that the SVO implements transparently the Data Object and the Data Graph. The changes tracking is embedded is the SVO. Currently we haven't implemented a Mediator Service but provide a analyser that helps to analyse the changes before persisting the objects.

With the SVO we don't expect to resolve all kind of data transfert (like the SDO does) but optimize the way of doing it for a java client accessing a java server.

HTH,
Gaetan Zoritchak,

Yes, the news about SDO (https://www.theserverside.com/home/thread.jsp?thread_id=22607) hit us just after the "design phase" of the project. It's good news that there are standardization efforts, though JSR 235's final release is scheduled for fall 2004.

Regarding the differences to JSR 235, it's basically a question about having statically typed data (DTOs) or generic, dynamically typed containers (SDO). Difficult to say which one's better, but sticking to basic POJOs sounds easier and more intuitive to me, although the SDO API is not a very complicated one.

 Jan

We had to deal with this problem when we wrote our CMS, since we use an applet as the client. What we did was to send graphs of mixins (=partial objects) to the client, where the graph size was determined by a per-thread "loading policy" (e.g. "if a load occurs then eager load X,Y,Z interfaces with reference depth 3"). Then, on the client we intercept all calls to the model using AOP, and when the client clicks on "Save" the list of calls are sent to the server and is then reapplied to the object model there.

In other words, we are not sending *state* back to the server. We are sending the calls. Why? For several reasons:
1) since we use AOP some method invocations have side-effects, and it is important that they trigger properly. Updating the state only would not trigger these.
2) methods are associated with security restrictions. By sending state, or state chunks a la value objects, these are not checked properly on the server. By sending the calls we can have fine-grained security associations on a per-call basis.
3) We don't have to have any advanced algorithms for calculating diffs and sending these.
4) We have to worry less about calls that make changes to collections. If one client calls "addFoo" on the model and then another calls "addFoo" on the same model then the underlying collection will automatically include both new Foo objects. With state synchronization techniques one would have to use some kind of merge logic, or invalidate one call due to optimistic locking. Our model can take more "abuse" in this sense.
5) The same method can easily be applied on the server to synchronize with other servers for replication purposes. I.e. method calls are capture on client and sent to the server, where they are captured again and sent to all servers in a cluster. All side-effects trigger appropriately and the algorithms are truly stupendously simple.

All in all, this approach has worked quite well for us. Of course, if you are very picky with concurrency issues there are some things to deal with in the above method, but for many applications it is "good enough". We have not had any problems with it so far anyway.

One more thing: since we are capturing the model calls on the client we can also do transparent "undo" management on the client. I.e. any call which modifies the object model can be transparently undone. It's pretty neat.

Rickard, this is a very interesting approach. My concern about this is that you have a choice between either:

(a) stateful middle tier
(b) re-retrieving the modified objects from the database when the log of operations is passed back to the server (before applying the operations)

I've always hated the notion that most object persistence mechanisms make it impossible to perform an update upon a row without a previous select upon the same row in the same database transaction. With Hibernate's support for detached object graphs, (esp. if we are using optimistic locking), we can update the state of an instance by simply passing the modified object back to the server and asking Hibernate to propagate changes to the database (including a version check). We don't need the useless extra select.

This seems to me to be a significantly more efficient approach in terms of database traffic. Though I gotta admit, theres some things I really like about your aproach....

Obviously, using bytecode tricks to track changes more precisely would help us be even more efficient. I've been thiking about this for a while; its nice to see someone has actually implemented the idea.

>to check if some modifications has been done and where

Why do you need to this analysing ? Generate log on client and send it to server (modifications only).

>
> (a) stateful middle tier
> (b) re-retrieving the modified objects from the database when the log of operations is passed back to the server (before applying the operations)

Yup, and we have both. For a) we are using pools with weak references in order to maximize our usage of the memory (i.e. so objects are thrown out fairly rarely), and for b) we are using a serialized persistent hashtable (now Jisp, and tomorrow JDBM), which means that the overhead to (re)load an object is more or less none.

> I've always hated the notion that most object persistence mechanisms make it impossible to perform an update upon a row without a previous select upon the same row in the same database transaction. With Hibernate's support for detached object graphs, (esp. if we are using optimistic locking), we can update the state of an instance by simply passing the modified object back to the server and asking Hibernate to propagate changes to the database (including a version check). We don't need the useless extra select.

Right, but see above. With our approach the cost of the above is most of the time none (i.e. it is in memory) and when it is not then the cost is low since the hashtable is really fast to load from. The hashtable manager is in-process so there are no TCP calls either.

/Rickard

Hi Rickard,

We had to deal with this problem when we wrote our CMS, since we use an applet as the client. What we did was to send graphs of mixins (=partial objects) to the client, where the graph size was determined by a per-thread "loading policy" (e.g. "if a load occurs then eager load X,Y,Z interfaces with reference depth 3"). Then, on the client we intercept all calls to the model using AOP, and when the client clicks on "Save" the list of calls are sent to the server and is then reapplied to the object model there.

Interesting approach but it has drawbacks that won't make it applicable in certain cases. Gavin mentioned the potential SELECT overhead but I would be more worried about the extreme use of optimistic concurrency.

What I mean by "extreme" is that unlike typical "relational optimistic concurrency", you are using "graph optimistic concurrency". In other words, a series of calls can end up modifying quite a few tables, thus increasing the risk of a concurrency exception.

Another problem is with undo.

If you get a rollback, it might be hard to inform the client exactly what part of the request was denied and they might have to submit everything again, which might be costly.

Anyway, your approach is definitely something to have in mind when trying to design such a system.

--
Cedric
http://beust.com/weblog

> Another problem is with undo.
>
> If you get a rollback, it might be hard to inform the client exactly what part of the request was denied and they might have to submit everything again, which might be costly.
>
It is not a problem, client can send savepoints himself too.

>
> What I mean by "extreme" is that unlike typical "relational optimistic concurrency", you are using "graph optimistic concurrency". In other words, a series of calls can end up modifying quite a few tables, thus increasing the risk of a concurrency exception.

Yes, correct. But, for the vast majority of those cases we don't care if concurrent transactions are modifying the same objects. If one transaction does "setFoo" and another does "setBar" on the same object the state as a whole is still consistent. So, we don't worry about it. But our application is perhaps "special" in this case.

> Another problem is with undo.
>
> If you get a rollback, it might be hard to inform the client exactly what part of the request was denied and they might have to submit everything again, which might be costly.

Right, we would undo the entire queue, but the exception can contain information about what happened and why. But, rollbacks happen EXTREMELY seldom, and is mostly if there's a system error (e.g. out of diskspace or similar).

> Anyway, your approach is definitely something to have in mind when trying to design such a system.

Well, it was very simple to implement and has the nice features I described above. As you are well aware of there is no such thing as "perfect" in software development, only "tradeoffs". I would be most interested to hear how the approaches discussed in this thread deals with, for example, AOP-style method invocation side-effects. With our approach it comes "for free", but how would they be accomplished if state only is transferred? (I'm assuming (rich) client/server here)

Hi Rickard,

I'm not sure if I can quite follow you, but which side effects do you have in mind ? For now we're not even using a 'real' AOP framework but a custom bytecode modification which does precisely one thing: updating the state of a "version" object which is attached to the POJO when doing a modificating write access on the object. When doing the state transfer there are 2 possibilities: the state has changed or it hasn't (for a given object). As a recent post pointed out, this means a rather coarse granularity, but we estimate that it should be sufficient for now.

 Jan

There's a bunch of different things that can happen as side-effects of invoking methods which leads to a changed state. For example, it can trigger asynch. events and it can trigger scripts (e.g. "when attribute foo is updated, run this serverside Javascript"). If all you're doing is thinking about state transfer then all such AOP-ish side-effects will not trigger properly. When I call a method on an object on the client the "state change" is important, but it is not at all the only thing that is important about the call.

I like the idea of capturing the client method calls and applying them directly in the server on the same domain objects. Unfortunately I imagine that it is not as simple and that you have to adapt the client method calls to the server implementation.

For instance with a CMP persistence a :

    bar.addFoo( new Foo());

should be replaced by something like

    bar.addFoo( FooLocalHome.create());

Despite that point, I see another advantage in your design with the possibility of serializing the client log in an XML doc allowing its deployment in a web services architecture.

Anyway, we’ll release a first version of the Smart Value Object with our first design and eventually improve it later with those new ideas :-).

Gaetan Zoritchak,

I wanted to say possibly instead of eventually. I definitely have to improve my english ;).

Hello,

Rickard writes:
# In other words, we are not sending *state* back to the
# server. We are sending the calls.
Isn't that a terribly high price to pay? In effect you are saying
that what once was the complete state is not the complete state
anymore, it has to be augmented with context information (method
calls that have side effects). What a loss! For example, state
sync schemes like cluster-wide caches (like Coherence) would be
of far less use.

IMHO, state should be in variables, not in "call invocation traces".
If the latter is a consequence of AOP, then maybe the usefulness
of AOP is less than it was thought?

Cheers from Munich,
    Henrik Klagges

> # In other words, we are not sending *state* back to the
> # server. We are sending the calls.
> Isn't that a terribly high price to pay? In effect you are saying
> that what once was the complete state is not the complete state
> anymore, it has to be augmented with context information (method
> calls that have side effects). What a loss! For example, state
> sync schemes like cluster-wide caches (like Coherence) would be
> of far less use.
>
> IMHO, state should be in variables, not in "call invocation traces".
> If the latter is a consequence of AOP, then maybe the usefulness
> of AOP is less than it was thought?

It's a tradeoff, like anything else. We prefer to use AOP and the benefits it brings, and have not seen any negative side effects of sending calls in order to do synchronization. YMMV, of course.

/Rickard

SVO, SDO or some other objects but it would be great if xdoclet generated such kind of object instead of simple DTO (and no AOP).

We could generate the complete SVO with XDoclet from tags on a CMP for instance. We just have to put a dirty flag when modifying a field from a setter, using some special implementations of Maps and Collections and adding a concurrency field.

But we hope to use the SVO the more transparently possible and that's why we decided to implement it by using bytecode modification on existing VOs. We currently working on the SVO implementation and have left the J2EE deployment problem for the next step. Aspectwerkz had worked on this problem and proposes some solutions that we may use in the future.

Gaetan Zoritchak