The Hard Part of SOA


News: The Hard Part of SOA

  1. The Hard Part of SOA (20 messages)

    ZDNet posted a reference to a Brenda Michelson blog entry that talks about the "hard part of SOA," focusing on "the 'Hard Part of SOA' as identified by enterprise practitioners and lead thinkers/technologists (not marketers) at application infrastructure vendors." As the ZDNet reference indicates, vendors tend to gloss over crucial problems like this.

    The three problems, with some potential solutions for suggestions edited out for space:
    1. Service Definition

    Many enterprises find it difficult to determine the correct bounds (job and tasks) and granularity (collaborations) of business services. Correctness is viewed in terms of both re-use and agility. Business services should be reusable (shared) within a line of business, across lines, and in some cases, at the enterprise level. Business services should be easily changeable (replace, augment, compose) to meet business demands.

    Enterprises have erred equally on creating services at too fine a grain, and too large a grain. Erring on the fine-grained side still allows for re-use, through service collaboration, but creates performance inefficiencies to perform actual work (bounds). Erring on the large-grained side creates application- or use-specific services, rather than common business services.

    2. Semantic Understanding

    For humans, machines, and the combination, semantics has always been a challenge. While it is relatively easy to receive information (conversation, message, data exchange) it is not always easy to discern the sender’s true meaning. For example, when the word “customer? is used, does it mean end consumer, prospect, or purchaser?

    As information exchange between applications and businesses became prevalent, industry and company dialects were defined to assign common terms, syntax and meaning to information elements. Given the initial purpose (technical integration) of these terms, the definition and administration has largely been the purview of technologists (geeks). As such, many of the terms are cryptic, derived from the application and information sources, rather than a true business dialect.

    With SOA, the need for semantic understanding becomes even more pronounced, because of the multitude of service interactions, and the self-describing nature of service-oriented languages for service descriptions, contracts, policies, and orchestrations.

    3. SOA Program

    The mission of an SOA program is to articulate and execute an SOA strategy that makes sense for your business. Common activities include evangelizing SOA, setting and enforcing standards (technical, methods), selecting application infrastructure and tooling, building out common (system-oriented) services, finding and conducting pilot projects, and stewarding the service catalog.

    Architects, technical leaders, and business solution project leaders say the biggest challenge in developing a SOA program is balancing the natural tension of executing business projects (today, on-time, on-budget) with the need for architectural integrity.

    Obviously, the simplest thing to do is give your architecture team a head start. Fund, staff, and start the SOA program three to six months ahead of the first business project that needs it. The shorter the head start, the less critical that first business project should be.
    What do you think? SOA architectures tend to be presented as a sort of silver bullet, while glossing over problems as enumerated by Ms. Michelson. Is her analysis fair? What can vendors do to address these problems better?

    Threaded Messages (20)

  2. Vendors can't do much...[ Go to top ]

    ...except provide flexible tooling that can be exploited in a variety of contexts. This they may sometimes do fortuitously, but not by design.
  3. Silver Bullet[ Go to top ]

    There is something to SOA but like most new TLAs and new hot tools, it's often being implemented by the over-optimistic and in general with really thinking the whole thing through.

    I work with services that are really thin wrappers to the mainframe down to getting mainframe error codes returned on 'successful' transactions.

    It seems to me that this is a common disease in IT. A new language or tool comes out with a lot of language. It's use is then rushed into production. A few years later, once a lot of people really understand the tool, it becomes apparent how badly the initial implementation was botched. But then it's too late: the people with the purse strings aren't willing to pay to reimplement something that already exists because the usually don't understand the cost of maintaining such things.
  4. Personally I'm fairly comfortable with levels of granularity, defining cononical forms, and communicating domain models and symantics.

    The huge unknow for me is how exactly the data architecture _really_ works.

    If anyone knows of a book or some online literature that would help me learn, please, please point me in the right direction :)

    The unknowns for me:
    When your business functions are represented as services (instead of monolithic applications), what happens to your data integration? Do you really have to throw out centralised databases and use EII/ data gateways to access all of your data?

    IMHO the realities of volumes/ performance and transactional complexities make this a pie in the sky approach.

    How about reporting?
    Do we:
    -throw away our traditional reporting tools (crystal/ actuate) and build reports manually, calling web services?

    -use the traditioal reporting tools, but source them to web services/ EII platforms instead of the databases directly? (performance and productivity again precludes this as a serious option)

    -use the traditional tools but have multiple data sources (losing the benefits of loose coupling - each time a service changes, your reporting investment breaks)

    -build OLAP data stores (data marts/ warehouses) with feeds from the decentralised databases (where-as the EII vendors claim their tools make OLAP a thing of the past)

    -have a centralised database providing the data for your services (in which case we again lose the loose coupling benefits, as the services are now tightly coupled through the shared database schema)

    There is a lot of hand wavy marketing material floating around talking about how EII / data gateways are going to solve all these problems, but the detail is lacking and I find myself seriously doubting that this problem space is well understood (let alone resolved).

    I've done a lot of SOA based on pub sub in the past (which I believe is vastly superior to point-to-point models). I've always had to compromise with the data architecture. In some cases the data is sourced from a service, which is elegant. In other cases (high volume or reporting), I've had to tightly couple services to a shared database schema.

    In large environments, OLAP is required, but comes with a loss of agility, and high operational/ maintenance costs.

    Anyway - there's my rant... I'm not interested in any "holier than thou" flames, but if anyone can point me at resources to improve my understanding of the data architecture in relation to SOA - thanks :)
  5. The huge unknow for me is how exactly the data architecture _really_ works.

    I doubt you're going to be flamed for this, on the contrary, most people who've had this question haunting them when they think of SOA or are told to architect a project based on SOA will probably pop up on this board and may possibly give us some direction.

    As for me personally, reading your post was like : OMG! I'm not the only one!.

    I would also like to know about UI architecture, especially when talking about 'composite applications'.

    Any thoughts anyone?

  6. I would also like to know about UI architecture, especially when talking about 'composite applications'.Any thoughts anyone?Cheers,Ryan
    Hiya Ryan,
    I agree - UI architecture is also an interesting question. Certainly coding stand-alone applications that bundle a collection of web services KOs your agility and re-usability.

    Portet servers however are quite a good match. Sure - currently building a decent portlet application is liking having your teeth pulled out.

    That said - they do allow you to bundle small focussed applications into a portel desktop, personalised to each user. I think the vendors quickly noticed the alignment of portels with BPM.

    Even the very earliest BEA BPM demos featured the generation of portlets based on web services. Oracle's demos also feature this these days. IBM?? they should but they have sooo much to demo I always leave their presentations bewildered by the huge range of products that have flashed past my eye balls.
  7. You're so right! This has been something playing in the back of my mind for a long while - most organisations have stovepipes of systems, all maintaining their own data! Along come the SOA guys and tell us we can integrate all this stuff really easily, but hang on what about maintaining data integrity across those stovepipes?

    I attended a certain large IT organisations SOA launch here in the UK last September, there was a presentation on their EII offering - as you say lots of hand wavy marketing and little substance. Very disappointed.

    Again, if there's any material out there on this aspect I'd be very interested.
  8. I have used web services for some time, and defined service layers long before the entire SOA was defined at the enterprise level.

    In my experience, this author has nailed a couple of key points. To the first point, I add that not only should it be deemed acceptable - but rather added into project plans - to go back and rename elements, services and even app/db code in order to maintain continuity from the bottom to the top. I cannot explain how many times - miscommunications between those whom develop the business use-cases and those whom develop them cause loss of time/money because data mappings are far from clear. This adds to the idea that the better the implementers know the business, the better the project will go. Unfortunately of course, this is not always possible. To answer the TSS question, I guess the best thing that can be done would be for the vendors to contract a business-project-champion into the development of the project in order to achieve success. This person should be sympathetic towards the needs of the developers, but still demand appropriate business requirements, such as on-time and on budget.

    My 2 cents.
  9. Well, I have not read the post in depth but, if I have properly catched the sense, there is nothing SOA specific.
    This is the hard part of domain analysis, wonderfully described in many books, such as "Object Oriented Modeling and Design" from Rumbaugh & friends (the one I like).
    The real problem is that there is always little thinking and too much point-and-click-to-generate-wsdl-and-the-job-is-done approach.
    There real problem is that architectures are (not)designed out of real-world context. I mean that it is stated that "here we put a web service to do this, there we put a web service to do that" and so on, without deep analysis. Then the ball passes to the programmer.....
    As quite (too) often happens, nothing really new under the sun.

  10. +1
  11. Unless you have a measureable and cost justifiable vision and map of what will be different when you are done. The self perpetuating gap between IT and management of IT: What do you want? Manager: What can I have? IT: You can have anything. Manager: What is available?...the answer is not SOA - The answer is what makes more money or drastically changes service delivery, product design and operations that yields X result. SOA used as a wrapper or band aid on legacy systems is gtreat if it substantially grows value by reducing time to deployment and avoiding fork lift software/vendor changes - But IT needs to make the value and alternatives clear. Business managers should be asking for more capabilities and functionality but most do not know what possibilities are enabled through SOA - IT must highlight creatig smarter services, embedded rules that can trigger price increases or decreases based on published business objectives connected to in-line systems, products or services delivered with in-fathomable combinations and variations based on a client's specific needs, profile, value and history - and yes the list goes on - but management has not seen the list - IT's fault and Management has not asked for this level of sophistication and change in business practices - Managements fault. Sorry for the rant - The hard part of SOA is beginning with the end in mind based on a highly valuable change. Michael R Hoffman, CLIENTxCLIENT
  12. I agree. Despite arguments for or against SOA, IT teams in general need to be more strategic about their decisions and remember technology exists in most of our lives to support a business. Our technology decisions should be based on business justification and the old concept of "value added" not some article someone read in the waiting room of the dentist office. :)

    With respect to SOA, I think the two overarching problems it faces are 1) there seems to be a universal impedance mismatch between what different architects and developers understand SOA to be and 2) there are already emerging "standards" like BPEL and ESB that are interrelated with a root technology that still has little consensus understanding which compounds the situation.

    I've heard people try to sell SOA like it's a silver bullet that will bring the world of software together in unity irrespective of platform, language or system implementation concerns. We simply aren't there yet, nor do we always need to be doing something that grandeur unless the problem warrants that type of a solution.
    In addition to the article, I would add that consideration should be made for the difficulties a team will face implementing a SOA from both a client interoperability perspective and an OO perspective. Hardly anyone ever mentions this. When architecting a SOA, it can be very tempting to move away from OO strategies and implement things in a procedural manner, kind of like we did back in the days of EJB2. We start throwing our religion out the window because of ancillary considerations a technology imposes on us and makes it hard to leverage what we know works very well.

    I don’t think SOA is bad, I just think it should be a tool in our tool belt that we use when it’s balanced against the requirements. Sure you could use your new hammer to drive a screw into a board… but why not use your old screwdriver when it’s much more appropriate.
  13. The Hard Part of SOA[ Go to top ]

    What a lot of people who've only discovered SOA in the last 5 years or so don't seem to understand is that SOA extends from Component-Based Architecture. SOA is not just about services.

    CBA came about sometime in the 1980's (for me circa 1987) as a way to implement OO designs in non-OO languages for performance reasons and it was also found to be more conducive to reuse due to their more rigorous contracts, specs, etc. (relative to objects) and the fact that they tend to provide more value relative to the effort it takes to learn them.

    Not all components make good services because they can be too fine-grained; however, you can easily compose services from them. Services are really facades that realize use cases.

    So, if you extend some of the better CBA methodologies to SOA and remember how services are defined (use cases), you can cobble together an effective approach to SOA.

    BTW - Catalysis is a good CBA methodology to start with.
    Also, if you doubt what I say about SOA and CBA - check out the OMG - SOA view, the IBM reference architecture for SOA, a paper from 2002 by Rational describing implementing SOA with components, etc. There is plenty of supporting evidence.
  14. ...actually understanding, believeing and taking on board that it isn't about Web Services.....

    If SOA is an olympic sized swimming pool, Web Services (and all it's related stuff like BPEL, ESB et al) would be a litre of water in the pool.

    SOA starts, and indeed ends, at the contract.

    In going round full circle, it runs the gamut of SLA's, type based lookup, interface v implementation (WS, incidentally, choose to mix impl [SOAP] with the interface, in such a hilarious fashion, that it could make a headline on the 'Daily Show with Jon Stewart', so that they became inseparable or to many WS heads - the One!), re-use (how many WS' have WSDL specifically for each customer, instead of a single service codebase designed to transform vendor specific data, from multiple different vendors with differing formats, into an internal canonical model), componentisation, maintainability, The Network Fallacies, etc.

    Most importantly, once concern I have is that SOA is not just for those communication between businesses - it should be fully, and intrinsically, part of the infrastructure of your own _internal_ business, right down to the metal. There is too much emphahsis that SOA is about everything oustide your business - that's also why there is far too much emphasis on getting buy-in from management, and not enough hype-free, developer-friendly material for SOA.

    Quite simply, WS hijacked the SOA train.....

  15. The Hard Part of SOA[ Go to top ]


    Thanks for bringing my post to the attention of theserverside readers. I agree with many of the comments, especially the importance of having a business reason to pursue SOA (or any technology, architectural strategy). I have a "business-driven" soapbox myself.

    Re: methodology, I'm not advocating reinvention, or getting bogged down by model mania. There is much to be leveraged from OO and CBD. In my own work, I leverage CRC in service definition. What does frighten me is "right-click service definition" and implementation partner specific methodologies. That just moves us backwards.

    Re: hype. It's out of hand. But expected, I suppose. Folks that have implemented solutions in the enterprise know better. You just can't buy SOA in a box.

  16. RE: The Hard Part of SOA[ Go to top ]

    I think it's yet another (and good) article trying to expose some of the real aspects of any SOA implementation. The lack of cohesive service definition is certainly one of the biggest pains and reasons of surprises found in later stages of any Integration Implementation (be it via SOA or via anything); and I think, with so much business stake invested in existing definitions, having a cohesive and uniform service definition throughout all the industry verticals would still take a number of years to come...

    Another very good point that I personally liked is that in real world implementation almost all the SOA projects are so intimidating that a lot of time goes in getting a "buy-in" from all the stake holders and the actual implementation time left to the architects is much less than it really should be.

    The "tips from the field" are really great! I would like to add to some of them.
    Re-Use: Along with reusability one should take into acount the complexities "re-usability" brings. This mainly pertains to understanding how verious modules interact with each other as they are reused across the enterprise.
    Simplify: A very key point here is "Don't buy a box of SOA". With SOA becoming a famous buzz word with all the integration suites trying to map their existing technologies and tweaking them a bit here and there and trying to polish it as a "true" implementation of SOA - architects shouldn't be bogged down with the intimidating effort of chosing the right vendor. I really like the idea of "finding what products you need and mapping your options to them". In other words "re-use what you have" and "use best of breed technology modules" based on "what you think is right for your SOA implementation" because "there is no SOA implementation that's fully out of the box and works for all businesses".

    A very important point is that most of the recent industries already have enough technologies in place to tweak a bit and fit them into SOA based architecture, thereby cleaning the small amount of mess they made over the years... but not that enough to totally dump all their efforts! SOA has a lot to do with the mindset and the thought and envisioning of your enterprise integration and it's not something like a new technology or compliance framework.

  17. The Hard Part of SOA[ Go to top ]

    Here's a recent ComputerWorld article on SOA at Wells Fargo.


    Cameron Purdy
    Tangosol Coherence: Clustered Shared Memory for Java
  18. The Hard Part of SOA[ Go to top ]

    Who's going to flame Cameron for blatant marketeering!
  19. The Hard Part of SOA[ Go to top ]

    it is like to steal candies to a child.....
    (Hope it has the same italian meaning)

  20. The Hard Part of SOA[ Go to top ]

    Who's going to flame Cameron for blatant marketeering!

    Oh, come on! They don't even mention Coherence until the second page ;-)

    The Wells Fargo SOA was a project that I was personally involved with, and so I have both a personal interest in it and gained more knowledge of enterprise SOA in financial services from the process. Their SOA initiative was one of the driving factors for our JSR 236/237 implementation (our grid-based CommonJ Work Manager), which coincidentally got us involved with the JSR itself, which is now availble for public comment at Doug Lea's site:

    Enjoy :-) .. if you find it interesting, there's an event at JavaOne on Wednesday at 4pm on the topic. I don't know where yet, but if you are interested, drop by our booth (#638) at the show and ask me.


    Cameron Purdy
    Tangosol Coherence: Clustered Shared Memory for Java
  21. Correction[ Go to top ]

    The BoF is on Thursday at 9:30PM local time:

    Session Id: BOF-0980
    Session Title: JSRs 236 and 237: Concurrency Utilities for the Java™ EE Platform in Practice
    Track: Core Enterprise
    Room: Hall E 134
    Date: 18-MAY-06
    Start Time: 21:30