Only a few high-level UML  diagrams survive the life cycle of the application

We need models to represent the software design. But even in large projects of about 500 man-months, I observed that only 3-4 sequence diagrams really matter and have a chance of surviving the entire life cycle of the application. Those 3-4 sequence diagrams (and class diagrams that represent their static time relationships), usually represent the high level design of the application.

Or, look at it this way:

Any decent enterprise application will not have 20 generic call flows. There will be one or two generic (or abstract) call flows, which all the concrete use cases implement. Let us take a simple Struts / EJB application. The generic flow will be something like - an action class calling a validator and then calling a stateless session bean, which in turn calls a domain class, which will call a DAO. All the use cases of the application just implement this flow with concrete classes that are specific to that use case.

Do you agree?

If you do not, I would like to hear about applications that have 20 different generic call flows and survived for 5 years after the first release.

If you agree with me, we are boiling down to 3-4 class and sequence diagrams even for large enterprise applications comprising several thousand classes.

It is hard to find good UML documentation that is kept in sync with the evolving code

You might say that in addition to the few high-level diagrams, you want to document all the use cases of the system for training or documentation purposes. During my 15 years of experience in the enterprise software world, I don't remember seeing well 'maintained' UML documentation ever. First of all, good diagrams are difficult to produce and are not found that often. Secondly, they are out of sync with the code most of the time. Most of my experience is with large banks, insurance companies, Auto companies, etc. Those environments are just too hectic and their resources are limited (difficult to believe, but true) for 'maintaining' good documentation.

So am I suggesting that we get rid of UML?

No. We need visual models to represent complex systems. Human brains are at their best when processing visuals. 

So what is a reasonable solution to easily produce and maintain UML models?

1. Start with drawing high level diagrams on paper or whiteboard for the generic call flows. These diagrams should not exceed 3 or 4 even for large applications.
2. After these diagrams are stable, you might use the UML drawing tools. Or just scan the hand drawn tools and make them easily accessible to everyone in the team. 
3. For the diagrams at the next level of abstraction (any useful models should have different levels of abstraction), generate the UML from source code or at runtime. You can generate both class and sequence diagrams using both methods.
4. If you hate drawing the diagrams, simply write the code for the shell classes and generate the high-level UML class and sequence diagrams as well. This way there won’t be any UML to maintain at all.

The source code is the truth - generate the UML from the code

Can you argue against the statement that the code is the truth? If you do not, why not generate the models from the source code itself? I am not suggesting the round-trip engineering, by the way. I am just suggesting a one way trip - from code to models.

Problems with the generated UML

Problems with generating the class diagrams: When we hand draw a class diagram, we show the relations between the classes involved in a use case. Most existing class diagram generating tools allow the user to drop the Java classes (the source code) into the tool and the tool automatically shows the relations between the classes. The problem with this approach is, how does one know about the classes involved in a use case to begin with?

The tools that generate the UML diagrams at runtime can solve this problem by first capturing all the classes participating in a use case and then generating the class diagram showing the relations between those classes only.

Problems with generating the sequence diagrams: Sequence diagrams by definition show the runtime relationships between the classes in a use case. Static sequence diagrams generated from the source code can never show the interactions between the concrete classes at runtime.

The tools that generate the rutime sequence diagrams solve this problem by capturing the concrete classes called at runtime. However, the generated sequence diagrams can become verbose and defeat the very purpose of models, which is to highlight the important design aspects and filter out the unimportant details.

Choose the tools that offer the necessary filtering mechanisms to view the generated diagrams at different levels of abstractions. When debugging a defect, you may want to see the low level interactions in a particular tier of the application. On the other hand, some times you may just want to find out all the important business classes called in a use case. Good tools should allow you to cature the runtime interactions for a use case and apply different filters on the same sequence diagram to view it at both high and low levels of detail as necessary.

Conclusion

In this age of agile development, it will be costly to spend time on drawing UML diagrams. Generating UML diagrams on the fly as they are needed would save the time to produce the diagrams and also keep the diagrams always in sync with the source code.

This argumentation is quite common and frankly it to my experience it boils down to the "I don't to do anything but code" attitude. It mostly originates from senior developers who can afford a fire and forget mentality because they do not cover the full product life cycle but, after the initial project  has passed, move on to the next project and let maintenance be some other people's problem.

Architectural modeling (and software engeneering in general) is not about creating diagrams. It is about creating an architectural, sustainable model, of which UML diagrams are just a simple view on the various domain aspects. I often come across the point where I recognise that this important difference is not recognised. Such a model answers long-term questions about reusability, modularity, functional completeness. It is the basis of agreement between developers, testers, product owners, customers.

We don't do this because we can not maintain it? Come on, we don't want to maintain it, is the real reason. And while we are at it. The code is the truth? For developers, yes. Not for testers, product managers, customers, 3rd party partners, etc. Besides developers, nobody understands the code. UML based architecture is understood by far more. So why not turning the page and making the model the truth from which we generate the code, test cases, interface descriptions, etc.? Some years ago MDA was big on the tables and it was exactly the right approach. Modern tools provide support for the whole cycle.

Because it is not code and it is not desired. Because there is no broad experience with it and because we start to feel comfortable with the "code is the truth" approach.

The higher the complexity of a product and its lifecycle (and I have seen quite a lot of complexity) the higher is the need for software engeneering. In the whole debate I realize that exactly this engeneering aspect is dropping more and more from the tables.

This might sound la bit rantish. So I leave it up for each of  its own to make the best out of it.

Oliver

+1

In the end UML is about communication, both now and later.

Good documentation at the begining gets the entier team on the same page. Updated diagrams keep them there as the application evolves. Maintained documentation makes it much easier to explain to new people how things work and how they were built.

Good documentation at the begining gets the entier team on the same page. Updated diagrams keep them there as the application evolves. Maintained documentation makes it much easier to explain to new people how things work and how they were built.

They are all valid statements. My question is, how much of that do you see happening in reality? Please comment on the real world enterprise application maintenance under the conditions like one of the below:

a) A few senior developers maintain multiple applications of  several tens of thousands of lines of code

b) A code base of such magnitude being maintained by a combination of on shore and off shore teams

c) Large applications where developers often change

d) Large enterprise applications where developers are always under pressure to meet the demands of business users

e) Agile teams maintaining and enhancing large code bases

I am really interested in real world stories of UML usage in teams maintaining large code bases under such conditions.

--Choudary.

The higher the complexity of a product and its lifecycle (and I have seen quite a lot of complexity) the higher is the need for software engeneering. In the whole debate I realize that exactly this engeneering aspect is dropping more and more from the tables?

I conclude from this that you are promoting the idea that "engineering" is always and inextricably connected with grapically "designing" a system. In fact, I get the feeling that, if its not graphical, its not engineering. And as you may expect, I strongly object to this: to me, the process of software development is about "design" (in the sense of creatively inventing solutions), and in that sense, "engineering" from the very start up to the point where the compiler sets in. ONe indication is the fact that usually the people performing the different tasks along the production line basically have the same education, namely "software engineering" degrees. And I would continue the argument by saying that the text (source code) form is as much a software design tool as any other one. To carry this even further, it is the most appropriate one, because it is the only one that is completely unambiguous and always in sync with reality. And, if you have a good engineering tool like, for example, Eclipse IDE, you can get higher-level views like class hierarchies etc. extracted from the code - even graphically, if you like.

In summary, I would say that, from the standpoint of software production and maintenance, UML is mostly irrelevant. A good requirements paper is a very valuable asset, and it may contain a few diagrams here and there, but their value is mostly illustrative. I wouldn't even call them true engineering artefacts, because of potential incompleteness and ambiguity (but I am willing to concede this point)

In summary, I would say that, from the standpoint of software production and maintenance, UML is mostly irrelevant. A good requirements paper is a very valuable asset, and it may contain a few diagrams here and there, but their value is mostly illustrative. I wouldn't even call them true engineering artefacts, because of potential incompleteness and ambiguity (but I am willing to concede this point)

I strongly agree with Oliver. Your approach may only well suited for small-scale sw development (e.g. when stakeholders do have a common understanding of the problem space and there is no communication overhead between them), otherwise it will be the source of serious problems. You cannot map problem space to solution space using a programming language which is itself a solution space artifact. Architecture is a different (higher) abstraction level than source code. While using UML is not a requirement for creating and communicating architecture, good diagrams are very valuable for it.

I have always found that the fact that we feel the need to "hijack" a job description from a completely different trade is a sign for the immaturity of ours. Just think of a "mechanical engineering architect". Anyway, what you refer to as architecture is part of the solution space as well. I am not saying that UML is useless here - however, its just an illustrative tool which is worlds apart from "graphical construction" like a real (building) architect does

as well. I am not saying that UML is useless here - however, its just an illustrative tool which is worlds apart from "graphical construction" like a real (building) architect does

I'm not an architect, but I think using UML is similar to "graphical construction". An architect does not model the entire building, just some parts needed for the construction of the building itself. There are civil, mechanical etc. engineers, who creates plans based on his design. Without their work, the building cannot be built.

I'm not an architect, but I think using UML is similar to "graphical construction". An architect does not model the entire building, just some parts needed for the construction of the building itself. There are civil, mechanical etc. engineers, who creates plans based on his design. Without their work, the building cannot be built.

One important difference between UML and the artifacts delivered by a "real" architect is that you will normally always be able to trace the finished building back directly to the initial plans. The plans are complete and unambigous engineering artifacts on which the subsequent activities are based only to augment the details. I have never seen a software project where you could trace the finished product back directly to any significant number of UML diagrams. Drawing diagrams, in the end, is a far too unwieldy means for specifying algorithms. The idea of specifying software with pictures was born in the mind of some non-technical manager that thought "there must be an easier way", and a technical consultant who said "give me your money and I'll tell you". The easier way has not been found to this day, and there have been quite a numer of attempts.

Again: I am not agains UML. All I am saying is that they have their place in early phases of development/specification, but they are not true engineering artifacts in the sense of being unambigous and complete. Software is too complex to lend itself to such means.

1) Exist.

2) Has master and Ph.D. programs.

3) Is called Architectal Engineering.

4) Work mostly in civil engineering field.

Hardware Mechanical Architects were replaced by the new fields of Electrical Engineering, System Engineering, and Software Engineering as methods of dealing with increasing levels of complexity.

To carry this even further, it is the most appropriate one, because it is the only one that is completely unambiguous and always in sync with reality.

I completely agree. If the code is wrong, the application fails. If the model is wrong, there is a chance that it would be caught while coding. I don't see any other artifact being as nearer to the truth as the source code.

It's pretty common to see well-functioning applications without any design. It is not uncommon to see applications where no requirement specs exist (and even the business analysts are not sure of the business rules). But, can we maintain an application without the source code :)

--Choudary.

It totally agree with that.

Think about the full life cycle of the software, think about all stakeholders, think about how you produce the software (from client to IT to subcontractors).

But yes, you've to think about how to maintaint good models (and not diagrams !) in sync with the "reality"/the code.

And my 2 cents, sequence diagrams are not my preferred diagram to design an algorithm, use activity diagram.

Oliver,

I hope you are not missing my larger point - I am not saying UML is useless; I am just questioning the heavy investment (both in terms of money and effort) in drawing the UML as opposed to generating them from the code.

>>It mostly originates from senior developers who can afford a fire and forget mentality because they do not cover the full product life cycle

I am not one of them. I spent more of my time maintaining large applications than building them. I  often feel that the most current thought leaders in the industry fall in the category you described and it does not serve the community well. People who get paid big often build new applications as opposed to maintaining them and might not understand the issues involved with the day-to-day maintenance.

>>Because it is not code and it is not desired. Because there is no broad experience with it

Probably. I can't oppose this statement strongly. I always used and see the UML diagrams as a means to think, understand, communicate and review the design of the application. I didn't see modeling as the centerpiece of the application.

UML models being used by the testers and customers? I agree that some high-level models are certainly helpful when talking to the customers and testers. But my question is, in a large, 10 strong developer, one year project, how many UML diagrams would you share with customers and testers?

My guess is less than 5. The deployment and component diagrams certainly help. The class and sequence diagrams? I doubt it.

What I want to stress is, almost 80% of the money on an application is spent on maintenance. The models and tools should help the people maintaining the code too, not only those who developed the initial system. UML models at somewhat lower levels of abstraction (with more details or for each specific use case) will help 'maintainers' better. But at the same time, it is pretty hard to draw and maintain the UML models for all use cases of the system. This kind of diagrams can be generated quickly and be viewed at different levels of abstraction more easily.

In other words, I am not ditching models or UML. But I am not really sure about models being the centerpiece. I argue that generated models (if they are good) provide better value over the entire life cycle of an application than the hand drawn UML models.

By the way, I am the developer of MaintainJ, a tool that generates sequence and class diagrams at runtime. I could be biased, but my opinions are from what I directly see in the industry. I am not pretending I have seen all. I repect your different opinion.

Choudary.

The only aspect of UML that functional are the sequence diagrams and given how terra-bad the rest of it is I suspect it's me that doesn't see the obviously superior methods of conveying that information.

UML is a visual modeling language that doesn't use visuals. This fails a FMEA at the concept stage.
Consider circuit diagrams and regard how each different element has a distinctly different iconic representation.

The same shapes, lines, and connectors between everything visually shows that everything is the same.
Information is conveyed when you encounter something different than expected.
Since UML fails to visually differentiate anything it necessarily fails to (visually) communicate any information.

It's garbage. Stop using it and stop promoting it so a void forms that a better solution can fill.

Good luck generating a useful activity diagram from the source code.

My experience is that if UML can be autogenerated from source code, you're doing it wrong. The point of UML is not to document all the code in pictures (i.e., 100 step sequence diagrams stink), but to get all the various stakeholders in sync with the architecture of your system.

I've seen lots of UML<->Code tools come and go over the years. Some tried to generate code from the model. Some tried to generate model from the code. They both were not complete solutions.

I find that limited UML is pretty useful is letting product owners, developers, testers, even customers (sometimes they care about how an API works if they want to use it in their site) know what you're trying to do. So having a couple of use case, activity and sequence diagrams is useful for this and saves hundreds of lines of descriptive text.

Now the caveat to my opinion is that I also think if you're spending over 30 minutes on a picture you get extremely diminishing returns. I'd rather get a quick sketch out quickly and get feedback than take weeks building a perfect UML architecture (I don't think these really exist). I only use UML for my pictures, because why not? If you're going to sketch out a deployment diagram to let the security folks know what you want to install you may as well use UML.

Comical aside- I was interviewing an architect once who claimed UML sucked and he creates his own intuitive notation that doesn't require any explanation to other developers, testers, etc. He then spent 10 minutes trying to explain his class and diagram notation to me.

Brian,

I agree with your points.

One clarifications about my post - I am advocating generating sequence and class diagrams at runtime. Not from the source code at static time.

Allow me to pose another question:

Let us take an actively maintained large enterprise application where about 100 developer-months effort is spent every year. What percentage of that effort would be:

a) simple bug fixes (localized changes like HTML/Javascript, a logic fix in a single class or adding a new form field, etc.)

b) simple enhancements like adding a new screen that is pretty similar to an existing screen or adding a new feature that is pretty similar to another existing feature

c) major changes that impact many parts of the application

My contention is that the first two maintenance activities accounts to about 50%. Do you see any utility for UML models as you describe in the first two maintenance activities?

In smaller size project I use UML (and other similar tools) as communication tools to help developers and stackeholders in development or understanding the design. These documents may or may not be a part of an overall documentation but they still help everyone to do a better job.

That would be OK for self-contained applications. Otherwise it would simply ignore system functional architecture that is meant to support the new application.

http://caminao.wordpress.com/engineering/models-perspectives/

Well, from my point of view there is a fundamental problem in your statement: you consider UML and code to be on the same level of abstraction. And that is the actual a problem. As a means of communication between stakeholders and developers a model has to be on a higher level of abstraction than code. Otherwise it is of no use. And this is the case for any kind of model being it UML or a textual DSL (Domain Specific Language). It does not matter.

Typically developers not realizing that argue in the way you do. I guess this comes from the legacy of the early UML tools trying to generate code from the models and at that time that understanding was not widely spread. So the code generated usually sucked but it lead to the impression that a "rectangle in a UML model" is the same as a class in a OO-Programming Language.

You can use models in a smarter way. As other commenders have already said and as discussed in Eric Evans book "Domain Driven Design" use a model as a means of communication with the stakeholders to enforce a common understanding. Model on a high level of abstraction and later on enhence the model with technical hints supporting your target platform. Then a code generator can be used to produce at least the most boilerplate code and configuration files that needs to be created and maintenanced anyway. There are open source generators available. And with the Eclipse Modeling Framework writing your own one is no big deal anymore. Bottom line, this is not so difficult as it was 10 years ago. The tooling has improved significantly.

Working this way the model also becomes a kind of source artifact and I guarantee you it will be kept up to date because it is less work to make a little change in the model than to try to change all the code and configuration files generated by hand.

I have successfully used this approach in agile environments and it turned out that the agile team even became "more agile". Less bugs and a higher quality of the code base was just another side effect. The size of a project does not really matter. The question is more whether one is using modeling in a smart way or not.

Ingo

very interesting topic. no correct answer. Sometimes looking at code makes sense and if u r starting from scratch then building a model im UML AND maintaining makes much more sense. if you have new developers join the team - are you ready to waste their time reading the code vs. first resding a visual design doc and thus making it fast for them to understand the code. They will need to read the code no matter what but if they see UML design doc there is time that can be saved for sure.

Now can the community share a few names of tools they have used to forward engineer - UML to code AND reverse engineer - Code to UML ( especially run time sequence diagrams) ? i just want to know if there are tools out there that i m not aware of.

Always use UML and MDA for forward enginnering, because your models should have a higher abstractions level than your codes or configurations.

I successfully use following tools for this purpose:

1. MagicDraw for creating the UML models. 

2. AndroMDA and ArchitectureWare for generating the codes and configurations.

Lofi.

http://lofidewanto.blogspot.com

Agree with you Ingo 100%!

Especially in huge projects where you have hundreds of Business Objects, you won't survive and can't be agile without UML and MDA.

BTW.: I use the term of MDA for generating code based on UML models.

Another point: I'm a visual type of person, so I prefer UML with diagrams instead of textual modeling stuffs.

Lofi.

http://lofidewanto.blogspot.com

..And you have managed to use 3 buzzwords in the title of your post. Congratulations. Now add a litte SCRUM, XP and SOA and you are all set.

For me they (UML, MDA, agile) are not buzzwords anymore, they are the real way of life as a developer... I cannot imagine developing a good software (small and big) without those things anymore...

Cheers,
Lofi. 

Well, from my point of view there is a fundamental problem in your statement: you consider UML and code to be on the same level of abstraction. And that is the actual a problem.

Typically developers not realizing that argue in the way you do.

 

Ingo,

You either don't read well or I failed to make my point.

I think I very well understand that any complex thing can be represented at different levels of abstraction. I started the post with 'high-level UML diagrams' as the title of first paragraph. In paragraph titled 'So what is a reasonable solution to easily produce and maintain UML models?', I clearly suggested ways to produce the UML at different levels of abstraction.

I want to make my point clearer regarding generating the models - I am suggesting generating the models at runtime and not from the source code at static time. I don't think EMF helps you to generate the runtime models. I also mentioned that with good tools, one can view the same runtime generated model at different levels of abstraction.

Working this way the model also becomes a kind of source artifact and I guarantee you it will be kept up to date because it is less work to make a little change in the model than to try to change all the code and configuration files generated by hand.

I have successfully used this approach in agile environments and it turned out that the agile team even became "more agile".

I always want to hear the real world stories. Could you give me an idea of the size of the project you mentioned above? Is it a new project or a major enhancement to an existing application? How long did you maintain the same application after it went to production?

It is easy to maintain the static model in sync with the code bases, but I am somewhat skeptical how diligently this is done in real world.

I partially agree.

Like the author, I have never seen complete UML models that were in sync with the code after some time. But IMO using a model and good code generators one can produce a lot of high quality code very fast. Especially when it comes to typical crud like operations there are many excellent tools available. Unfortunately I have experienced, that many companies are reluctant to use such tools exactly for the reason that they think it is not possible to keep the model and (generated) code in sync. And there are right! That is very difficult, if not impossible. But who cares! Try to generate as much as possible lines of source code in the beginning of the project, and then say goodbye to your initials model, continue with the (partially generated) source code, and use reverse engineering tools to produce graphical models for documentation purposes, from the source code.
Last note, I have used this approach often, but not using UML as the model, but Java itself, see the easyMda project hosted at Google, if you want to find out how this is done.