Java Development News:
Enterprise Service Bus: Yet Another Paradigm Shift or Better Orchestration of Old Technologies
By Frank Teti
01 Aug 2006 | TheServerSide.com
In my opinion, some of the more interesting marketing battles within the IT industry are fought over the enterprise software market. In that space, business fundamentals rule the day and some things never change - that is, if you control the API, you control the market. Essentially, the vendor(s) with the best technical strategy, commitment to the market and ability to execute usually dominate. The others either fall by the wayside, are acquired or settle for their technology to be resold by larger vendors.
While SOA and Web services are standards-based to a large degree, the Enterprise Service Bus (ESB) "specification" is, in many ways, wide open. Although that can be a good thing, too many standards can hamper software developer productivity. An SOA, including an ESB, either using a commercial product, open source or custom made is key to having a truly scalable, enterprise-wide solution. A modern ESB is capable of dynamically transforming Web service payloads in a deterministic way through a metadata-enabled, policy-driven configuration environment and intelligently routing the messages to the corporate pipeline.
Nevertheless, while meta-data and model-driven programming are the wave of the future, whether the implementation is a Spring Web Flow, a standard Web service or an ESB, most organizations have limited tolerance for protracted analysis and system design lifecycle phases, which are required to configure these systems properly. It is my opinion that this work ethic needs to be seriously reconsidered.
Participants in the SOA Software Market
Not since the client-server paradigm shift and early days of J2EE has a market materialized with so much potential. In this market, vendor strategies range from automating the lofty, C-level, SOA governance process (i.e. CA's Clarity) to providing search capabilities that utilize the XQuery standard (i.e. Mark Logic).
XQuery is a technology under development by the World Wide Web Consortium that is designed to query collections of XML data -- not just XML files, but anything that can appear as XML, including relational databases. XQuery has broad support from IBM, Microsoft, and Oracle as well as application server vendors such as BEA and Software AG. These vendors have clearly indicated that SOA is a primary market focus.
Albeit, to brand yourself as an SOA vendor, it appears that all you really need to have is a track record for building service-based software. Generically speaking, a service can be an MQSeries Manager, an EJB (Enterprise Java Bean), a basic HTTP service or robust asynchronous messaging environment. For example, TIBCO, a traditional middleware vendor strong in the financial services industry with a solid reputation for delivering financial data, is competing with the dominant players and moving into the business process automation segment of this market. Oddly enough, I know of one large pharmaceutical company that is using Tibco's Business Works as an "interim" SOA solution, before deciding on IBM or BEA. This organization is clearly underestimating Tibco's ability to become a major SOA platform.
A Generic Service Model
In general, an SOA that utilizes an ESB exposes a service as a proxy. Communication from the client to the ESB or ESB to the business service can be over HTTP, JMS, SMTP or FTP.
Vendor Selection Process
One way to determine which vendor(s) are appropriate for your organization is to know your Web services "system" Use Cases, such as:
- The need for disparate data integration (Oracle Fusion Middleware, IBM Websphere ESB)
- A high-performance, guaranteed delivery message integration backbone (Tibco BusinessWorks, WebMethods Fabric, Sun/SeeBeyond, IBM Websphere, BEA AquaLogic)
- A secure SOA infrastructure, including service monitoring, hub integration and business process reuse (IBM Websphere and Tivoli, Oracle Fusion Middleware, Tibco Business Works, BEA AquaLogic)
- A low-end, no-coding, configuration environment, an alternative to the custom, home-grown solution (Sonic, Fiorano, Cape Clear)
- A network services gateway for B2B, process orchestration and protocol brokering (IBM, Tibco, WebMethods, BEA)
Today, an SOA architecture that utilizes a "packaged" ESB is clearly at the top of the list of the enterprise software vendors, such as IBM, Microsoft, BEA and Oracle. ESBs are essentially all about integration and in some ways there is nothing new here at all -- routing, transformation and system integration.
What is different are the considerable number of customers facing Web Service standards that work in conjunction with a modern ESB in a seamless fashion. Not since the ill-fated DCE (Distributed Computing Environment) has there emerged a model that can truly support the disparate technologies of the enterprise. However, just like DCE, which had a reputation for having long lead times for software releases and being hard to administer, a true SOA represents a complex software architecture.
So, beware the mythmaker. The mythmaker usually recommends that you have to tear down the old technology to make room for the new. However, Web service and application point-to-point integrations are time consuming and expensive projects. A technical strategy is required to move the organization to an SOA over the longer run that includes integration with legacy technologies. For organizations of any scale, a methodology and release strategy is required; any other approach would be a fool's errand. It is better to face the fact that provisioning, integrating and managing services is hard work. This isn't like build stunning, flashy, experience design Web sites; integration implementations are, for the most part, painstakingly tedious, operator-less processing style programming efforts.
Strategy and Methodology
SOA and Web services are, for the most part, an improvement in B2B technologies and standards for the Internet, including XML. An SOA doesn't necessarily mean you must expose all services as Web services, but there's a strong case for implementing B2B functionality as a Web service. So, during the discovery phase of an SOA project, a good candidate for a Web service is B2B functionality. Conversely, the technology doesn't have to be limited to B2B. It does represent a bridge technology for communicating between disparate environments; however, the SOA model, which uses Web service as its enabling technology, addresses a more complex problem set.
Although, even season architects disagree on:
- What constitutes a solid Web service?
- What types of applications or processes should be converted into a Web Service?
- How fine-grained should the Web service be?
- When and how should an ESB be utilized in the architecture?
- What protocols should be relied on?
Indeed, a colleague of mine stated that to be a service it should be extendable. In my opinion that is exactly what you don't want; reuse of Web services is not gained through extendibility, but by making the service available to many consumers. This is an age old object-orientation controversy: When to extend an object and when to embed it.
Nevertheless, many organizations that have implemented an enterprise-wide portal technology have realized through that process that a certain amount of guidance is required to insure success. At the minimum, identify a set of SOA artifacts before pursuing an enterprise-wide SOA and develop a roadmap.
The Technology Landscape: Vendor Strategy and Tactics
While the SOA software industry clearly has room for many participants, at the enterprise-wide software grade, the usual suspects will dominate. In the March 16, 2006 issue of Network Computing, BEA's AquaLogic Service Bus (ALSB) was the Editor's Choice during a vendor lab test of eight ESB suites, with Oracle being the runner-up. However, this survey focused on core ESB features, and not enterprise-wide features such as ubiquitous integration, process management, workflow and activity monitoring where IBM excels.
ALSB is a comprehensive enterprise grade ESB, which could serve as the heart of your SOA infrastructure. BEA has combined messaging and management in a single environment. BEA enjoys a reputation in the J2EE software market for being out in front of the other vendors with respect to J2EE spec compliance. Now BEA has turned its attention to providing a comprehensive SOA infrastructure, while again supporting the important standards, such as WS-Policy, WSDL, XQuery and WS-Security. BEA also has added a certain amount of their own technology to the SOA mix. In ALSB terms, pipelines have stages in which logical choices and transformations can occur. A transformation is where you need to change the format of the data from one type to another type or perhaps augment or even split the message/document in the bus. These transformations are part of a pipeline which is processing the message/document as it moves through various stages.
BEA, like IBM and Oracle, is starting to be able to leverage other components within its software portfolio. ALSB uses a lightweight version of WebLogic Portal for administration and operation. This is probably a plus for BEA, which in the past tended to put too much functionality in its application server. Other enterprise-grade vendors tend to have a more distributed approach to software engineering.
Even though BEA would consider itself to be a complete SOA infrastructure, architects will still see the need to augment the ESB with tool adapters, possibly more powerful transformation and extraction tools, specialized B2B components, and possibly another BPM engine, even though BEA recently purchased Fuego for BPM. BEA, through their portal acquisition of Plumtree, clearly sees the need to sell to the business user and not just the technologist, as I believe they have successfully done in the past. The SOA business model is an enterprise-wide model and has to be implemented with that type of scope, if it is to be successful. Only senior business analysts truly have that kind of perspective; BEA is now trying to woo that constituency.
IBM has, in reality, been in the SOA market for a long time. In some organizations, the existence and use of MQSeries represents the ESB and for those organizations will remain an interim ESB. However, IBM has had a series of modern process orchestration tools starting with the acquisition of CrossWorlds, which was re-branded to WBI (WebSphere Business Integrator) and then Server Foundation, which really never got any traction. IBM has had a difficult time differentiating WBI from ETL type tools, given its large software portfolio.
Their current incarnation of ESB is Websphere ESB. Because IBM has a number of hardware platforms to support, IBM tends to make a market for their technology well in advance of a viable product being shipped. One brilliant strategic decision that IBM did make was to standardize its WebSphere tooling around Eclipse, a popular IDE in the Java developer community.
The next generation Eclipse-based tool is specifically designed to build and deploy business processes based on SOA. IBM states the tools are easy to use and require minimal programming skills. However, while IBM taunts its "on demand" abilities, in reality, it suffers from a reputation that always puts it squarely behind BEA with respect to ease of implementation. Indeed, understanding IBM's marketing terminology is perhaps a metaphor for understanding its software applications. In fairness to IBM, IBM eventually delivers a robust system that can be used for enterprise-wide deployment.
If you are an Oracle DB shop and your basic SOA system Use Case revolves around integration with disparate data sources, Oracle Fusion Middleware appears to be a solid platform. From a data-centric perspective, while vendors such as IBM and BEA focus on (XML) data analysis of in transit and data at rest using XQuery and XPath, Oracle BI and analytic capabilities are still centered on the DBMS. Oracle's Fusion Middleware does include SOA process and management, so a viable SOA infrastructure is achievable.
Perhaps Oracle's best strategic decision of the year was to not purchase JBoss, as was indicated by the rumor mill. EJB 3.0, JBoss's area of specialization, is a bloated spec that is encumbered by the need to support existing functionality (i.e. deployment descriptors) as well as new functionality (i.e. annotations), which JBoss viewed as mandatory even though the constructs do virtually the same thing. Today's architects are more interested in building services with lightweight data-centric containers. Oracle is now in position to acquire several of the more nimble SOA pure-plays in order to augment and complete its SOA offering.
Today, SOA and ESB is an area where there is maximum hype and minimum understanding regarding how the holistic model will be designed and implemented. In larger organizations, the enterprise architects may be planning to move to IBM or BEA in the future, but they are still in their own product evaluation and learning phases. Therefore, enterprising technologies within the organization are building a case for implementing "interim" custom solutions by wrapping Spring services or using other open source technologies such as IONA's Celtix, an open source ESB.
IONA's model is to give away their software and sell their professional services in order to "help organizations take advantage of open source and to ensure the successful adoption of SOA." While technologies like Spring and IONA represent the nuts and bolts of an SOA, companies such as Savvion are providing free copies of their Process Modeler. The tool provides model simulation and functionality for collaboratively building executable business processes. SOA is clearly not a packaged environment and one way or another an organization will have to assemble a certain amount of technologies and technical expertise to pull it off.
The OASIS (http://www.oasis-open.org) SOA technical committees focus on creating standards to help interoperability for industry computing environments. They don't provide a "how to" for developing the whole enterprise system from beginning to end. Each organization has to decide and tailor the appropriate software methodology for their purposes.
The OASIS SOA TC will develop a Reference Model. This is primarily to address SOA being used as a term in an increasing number of contexts and specific technology implementations. Sometimes, the term is used with differing - or worse, conflicting - understandings of implicit terminology and components. This Reference Model is being developed to encourage the continued growth of different and specialized SOA implementations while preserving a common layer of understanding about what SOA is.
Pushing the Edge of Technology
I know of an insurance company that was analyzing 300 slightly different transformations and they wanted to know if there was a way in which a vendor-purchased ESB could either template or build transformations on the fly. Basically, they wanted to create XQueries, based upon a set of business rules and make the transformation process totally dynamic. There are a number of ESBs that would be appropriate here that support XQuery. For example, XQuery is used in several of BEA products: WebLogic Integration, ALSB and Data Services Platform. Essentially, the operative words that surface the need for a service bus are routing, transformation and mapping. The open question then becomes is it for enterprise use or is it needed for one specific department. I find that many Architects are now grappling with this decision and that if the need is departmental, then it does not require an ESB.
Savvy architects today tend to look for certain distinct patterns that indicate the need for a Web service or ESB. Currently, I am working with a major healthcare provider that is interested in significantly enhancing an existing application that routes and queues open claims and claims adjustments to a number of claims management systems. Claims data is routed based on certain business rules (that is, area of expertise, training, dollar amounts, etc.) to adjusters. There is limited transformation required. Additionally, new requirements state a need to access an external environment for provider reference data. This clearly indicates the need for a Web service.
Managers on the project envision the new application as a significant modification of the existing J2EE application that is five years old. From a software perspective, this is a lifetime. I see the need for a modern ESB, given the non-standard constructs used in the existing system, such as threading, lack of separation of business logic from the presentation layer and polling versus MDB-style listeners. We are still in the design phase, so a decision regarding the reference architecture is still under way. I'll keep you posted on the outcome.
A recent Network Computing poll indicated that about 54 percent of respondents said they have implemented ESB or will do so this year. Although, out of this group, it appears to be indeterminate how many respondents actual have already implemented an ESB. Granted, a concise definition of ESB could be an elusive. Basically, an ESB has routing, transformations, and protocol support, and orchestration and integration capabilities. Respondents indicated that the biggest barrier to ESB integration was the lack of technical expertise and security concerns. Fortunately, any lifecycle methodology worth a salt would address these issues during the scope or design phase.
Develop a Service-Oriented Architecture Methodology
Use Spring Web Flow with IBM WebSphere Application Server 5
Use Enterprise Generation Language in a Service-Oriented Architecture
About the Author
Frank Teti is an industry analyst and principal architect. He can be reached at firstname.lastname@example.org.