Enterprise Service Bus (ESB): Lasting concept or latest buzzword?

The enterprise service bus (ESB) has begun to gain popularity and could supplant proprietary enterprise application integration (EAI).

Guest Commentary
Enterprise Service Bus (ESB): Lasting concept or latest buzzword?
by Nicolas Farges, Head of R&D, TechMetrix

A new term has appeared in the field of application integration: the  "Enterprise Service Bus" (ESB). This term was used by the Gartner Group to define a new type of application integration middleware: "An Enterprise Service Bus (ESB) is a new architecture that exploits Web services, messaging middleware, intelligent routing, and transformation. ESBs act as a lightweight, ubiquitous integration backbone through which software services and application components flow." (Source: Roy Schulte, Gartner)

This definition is very similar to that of  "Enterprise Application Integration" (EAI) tools, apart from three significant terms, which we shall come back to later: Web services, ubiquitous and lightweight.

Enterprises are only just starting to measure the impact of application integration on their information systems and organization. The more advanced companies have recently started deploying EAI solutions or have launched pilot projects. Given the speed with which trends seem to succeed each other, it is not unreasonable for us to wonder whether the ESB concept is actually a legitimate one.

The emergence of the ESB Concept is closely linked with the lasting trends that have been slowly transforming the EAI market for the last few years: standardization of infrastructures with J2EE, Microsoft .NET and Web services, and the redistribution of roles of integration software component vendors.

In this article, we shall look at the motivations behind ESBs, and their likely impact on the middleware and software market.

What's wrong with traditional EAI solutions?

An EAI solution is designed to integrate applications according to the principle of loose coupling, whereby applications can, to an extent, evolve and operate independently from one another. The stack of middleware usually provided by EAI solutions includes:

  • Connectors, able to introspect the data model and services of a given application, and to open them up to other applications connected to the EAI bus. A connector generally handles some or all of the following tasks: authentication, transaction, conversation management, security context, rights management, and so on.
  • Asynchronous middleware (or bus) which handles guaranteed asynchronous delivery of messages for applications connected to the bus. Security aspects are also managed by this layer.
  • Service manager, able to apply added-value services handled by the integration architecture and not the functional side of applications (e.g. routing, message transformation, transcoding, etc.)
  • Process manager, able to orchestrate inter-application message exchanges in order to execute business processes. The presence of a human workflow tool is also desirable here.
  • Repository, to guarantee consistency and configuration of components deployed in the EAI solution.
  • Administration console, able to supervise the operation of technical components in the EAI solution, along with business processes.

Vendors of EAI solutions have, through their R&D investments and, more often, through external acquisitions, managed to feature the full abovementioned middleware stack in their solution. Functional richness is therefore not a concern for enterprises investing in this type of solution.

However, EAI solutions are criticized for certain shortcomings:

  • Proprietary development interfaces: The integration logic (process, transformation rules, workflow services, etc.) cannot today be ported between EAI solutions. This fact means that enterprises are closely linked to vendors with regard to a structural aspect of the IS, sends the price of specialized consultants through the roof, and weakens the durability of integration-related developments.
  • Proprietary communication protocols: Although there is no difficulty in transferring and monitoring messages within an EAI solution deployed on a wide scale, the same does not hold true between EAI solutions from different vendors. At present, it is not possible to supervise a business process from end to end if it's executed on two separate EAI platforms. While the SOAP/HTTP does provide a bridge between EAI solutions for interoperability, it only tackles the rudimentary issue of the sessionless RPC, overlooking the other fundamental architecture services, (security, guaranteed delivery, etc). This limits interoperability between two enterprises with different EAI solutions.
  • Lack of consistency between application servers and EAI solutions: Vendors often put forward the argument that connectors are able to integrate existing applications without altering their code. This is true. However, the fact that an application communicates with an "intelligent" network often impacts on its functional scope, which often needs to be improved. For example, an application may have to communicate with the EAI workflow manager, or simply deliver a message to a remote server connected to the EAI solution's bus after a transaction has been executed. Interfaces for access to integration services via application servers are often proprietary (e.g. in the form of C or Java APIs). Some standard XML interfaces may be available, such as SOAP/HTTP, but are still too rudimentary to fully meet requirements.
  • Excessive license cost: The main EAI vendors base their strategy on functional richness. The resulting high R&D costs lead to high prices, which make EAI solutions inaccessible to all but the large corporations. An EAI middleware stack is also voluminous (e.g. number of connectors), and costly to maintain.
  • Lack of scalability: Most EAI solutions have based their architecture on the principle of a centralized, hub-and-spoke architecture. Expanding an EAI architecture on a large scale becomes difficult with this model. The move to a bus-type architecture, which deports all or part of the central integration logic to "smart" connectors based on a distributed asynchronous middleware, has nonetheless been achieved by the leaders on the EAI market. We should point out, however, that a bus architecture is more complex to deploy and administer, but can also be configured as a hub & spoke architecture.

In addition to the problems related purely to EAI solutions and the other standards in force, difficulties may arise in relation to the design and implementation of this type of architecture. For example:

  • Difficulty reconciling the data models and security models of each application
  • Difficulty making the application modules of an architecture functionally independent, in the form of components
  • Communication difficulties between the different teams involved in an EAI project
  • Political difficulties relating to redistribution of power caused by the urbanization of the IS
  • Difficulties for developers and administrators to integrate a new architecture paradigm, in an economic context marked by reduced budgets and workforce

For these reasons, "35% of EAI projects are not completed in accordance with deadlines and budgets", estimates Forrester Research, and the term "EAI" is has found itself the blacklist of buzzwords that IT managers no longer want to hear. Many EAI solution vendors have also banned this term, to replace it with others such as "application integration", "integration architecture", etc.

Whatever the term or the technology used, the problem of application integration remains a difficult, long-haul affair. Strictly EAI-related difficulties can nonetheless be resolved by vendors and standardization bodies. The market does seem to be moving in this direction, giving rise to a new time of software: ESBs.

ESBs, or the quest for the universal integration middleware

Before we look at the market solutions that claim to offer ESBs, let us sketch out the ESB in its fullest form (able to meet both the internal and external integration problems of enterprises).

The diagram below shows an ESB. It does not invent any new architecture concepts, but instead uses standard Web services interfaces between its components, thus differentiating it from a proprietary EAI solution:

  • Web service-compatible integration server: An application wishing to invoke a component on the network or expose its services on the network will do so via an integration server designed for the purpose. Quality of service aspects relating to the service's exposure on the network will be declared by the application, and managed transparently by the integration server.
  • Standardized interfaces between applications and the integration server: An application created using a given language can invoke the services of the integration server (e.g. delivery of a message to the server http://ACME.com:5555/endpoint1 delivered once only, Triple DES encryption and request for notification of receipt, in the context of a distributed transaction). This service is requested using a standard API linked to the language used (e.g. Java, C# or Perl), which isolates the application from any proprietary integration server API.
  • Standard Web service communication protocol which natively integrates the notions of quality of service which are essential for wide-scale application integration (confidentiality, integrity, non-repudiation, guaranteed delivery, transaction, sessions, administration, etc). This 100% XML protocol makes it possible to progressively integrate other integration servers in the architecture, from other vendors, and to integrate a network of compatible servers from a partner, without harming quality of service. This is where the notion of "ubiquity" comes in, as put forward by Gartner.
  • A container for Web services and orchestration incorporated in the integration server mentioned above, in order to support various components relating to the integration logic (message transformation, enrichment, management of long processes, etc). These components are additional standards for Web services, which isolates them from the proprietary aspects of the container.
  • A standard administration protocol, enabling a third-party administration console to supervise a Web-Service compatible server or a network of Web service servers, thanks to a standardized XML administration protocol.

This holy grail of the "universal middleware" has been sought many times - just think back to DCE or CORBA, which encountered many problems. The concept of ESB is another attempt to achieve this ideal, and there are three reasons why ESB has a good chance of meeting its goal, at least to an extent:

  • Application integration is today at the heart of many strategic interests: e-CRM, extended enterprise, automation of business processes, etc.
  • The route followed by Web service standards takes a bottom-up approach: the layers in the Web services stack may be used as required, according to their maturity. The basic technical line-up of SOAP-WSDL/HTTP limited to the RPC, will be gradually extended by optional functions. This pragmatic approach is encouraging, and will enable many companies to avoid needless complexity ¯ hence Gartner's use of the term "lightweight".
  • The XML language is used at the heart of Web services: Web services are based on the W3C's XML schema recommendation, which is unanimously supported by the middleware industry.

So, the ESB architecture is inventing nothing new, offering something as standard that has been offered for years by proprietary middleware. The success of the ESB will therefore depend on the market's ability to deliver reliable Web service standards, and truly interoperable implementations.

The stack of Web service standards is still a house of cards...

It is usual to associate Web services with the SOAP/WSDL/UDDI standards trio, together with HTTP. This trio only resolves basic application integration problems, i.e. the notions of service contract, stateless RPC and directory. Below is a list of the architecture services required by ESBs, as well as the solid standards proposed by the main standardization bodies and vendors:

Service

Related standard(s)

Transport

HTTP

Messages

SOAP

Service contract

WSDL

Directory

UDDI

Delivery guarantee

HTTPR, WS-Reliability, ebXML Messaging Service

Graphical interface

WSUI, WSIA, WSRP

Encryption

XML Encryption, WS Security

Signature

XML digital Signature, WS Security

Single Sign On

SAML

Access control list

XACML

Transaction

BTP, XAML, WS-Transaction, WS-Coordination

Management of public/private keys

XKMS (XKISS, XKRSS)

Process representation

BPEL, BPML, XLANG, WSFL, WSCI, WS Choreography

Administration

OMI

XML dialects of business messages

fpML, CML, etc

XML dialects of business processes

RosettaNet, etc

This overview is incomplete, it being difficult to give a complete list of standards as they are issued by a number of different standardization bodies (the most active being the W3C and OASIS) or vendor consortiums (IBM, Microsoft, BEA, Sun, etc). Many standards meet head-on in competition, and a number of them are likely never to be implemented. All these standards are playing for huge stakes, as they are the key to the adoption of Web services, and therefore to the rise in sales of compatible middleware.

The standardization bodies therefore see very ambiguous relationships playing out between software vendors. A consensus is required in order to establish standards that will send software sales skyrocketing, and every vendor is trying to use their powers of persuasion in order to get a head start when it comes to implementation. The battle will be even tougher given that standardization will no doubt be followed by the appearance of open-source solutions, to the great advantage of customers and service companies.

Still missing from the Web services implementations on the market are the vital standards covering the guarantee of asynchronous message delivery and security. The HTTPs standard does not provide high enough security for Web services and the RPC connotation of SOAP/HTTP contradicts the notion of loose coupling on which Web services are based. The Reliable Messaging and Web service Security standards (in particular, based on XML Digital Signature and XML Encryption from the W3C) from the OASIS are excellent candidates to address these issues, and are likely to enable Web services to take off. While it is true that the other standards are important, they do remain secondary in terms of requirements.

The propagation of standards will inevitably be followed by a streamlining of middleware vendors, and a redistribution of suppliers' roles...

The impact of ESBs on the integration middleware market and on software vendors

The current vision of an integration solution whose components come from the same middleware vendor has now been overturned. Standards, which are more or less mature, are converging towards a "compound" view of the ESB, which will be an assembly of software components from specialized vendors:

  • Connector standards will make it possible to obtain an application connector from the vendor best positioned to develop it (e.g. Oracle DBMS JDBC connector from Oracle Corp, JCA R/3 connector from SAP, JCA CICS connector from IBM, etc). Part of the integration middleware market will then legitimately fall into the camp of specialized system and software vendors.
  • Administration standards will enable integration platforms to be administered from an external environment such as Patrol or TNG,
  • Business process standards will make it possible to use third-party process managers, orchestrating Web services and workflow, based on a standardized Web service bus
  • Messaging standards will make it possible to free oneself from the asynchronous middleware used, which will become interchangeable
  • ... and so on..

We are already seeing vendors of EAI solutions leaving the connector market and moving towards application servers. The asynchronous messaging system is also becoming a commodity component. Examples of such positions are multiple: WebMethods is opening its system to J2EE JBoss and is clearly putting its money on Web services, SeeBeyond is offering a new version of its architecture, called ICAN, completely based on JCA and compatible with J2EE application servers and the latest Web service standards, Vitria is integrating the Tomcat server, Sonic Software has based its integration solution on Web services, XSL, JMS and JCA. Some newcomers such as SonicXQ or Kenamea are already embryonic forms of ESB. We should point out that, on the whole, application and asynchronous messaging system vendors are very well placed to address the ESB market. BEA's solution is a good example of a unified middleware stack handling different integration problems. The limits between application servers, message-oriented middleware and EAI solutions will therefore fade.

As well as the creation of connectors, the software market is also undergoing a gradual, profound transformation. The strategic turnaround by Siebel illustrates this trend particularly well. Instead of deploying its business components in a confined technical environment, Siebel will extract the business processes from its software package in order to deploy them in standard form in EAI and Web services environments, so as to resolve the integration problems traditionally encountered in CRM projects. To do this, Siebel is using partner integration solutions, particularly those of WebMethods and SeeBeyond. SAP is also following a similar approach with its xApps, although it is less clear on the technical architecture that will be used. JD Edward and Peoplesoft have not escaped the tidal wave, and are respectively offering EPI (Extended Process Integration) and AppConnect. The gradual disappearance of barriers between applications will bring some vendors to reconsider their functional scope. A new hand looks likely to be dealt...

The vision of an ESB that we describe in this article will take some time to become reality, as experience has shown that this type of architecture takes a long time to standardize. The future of EAI solutions, in their current guise, is far from certain. ESB is therefore a lasting movement which falls into the architecture standardization logic that began with J2EE and .NET, and is continuing with Web services.

Nonetheless, neither ESBs nor EAI can reduce the crucial analysis phases that must come prior to any integration project, in order to study the urbanization of the information system. Initially, and if feasible, it would therefore be reasonable to embark on these analysis phases, without investing too much in over-extensive integration solutions, and to start building the foundations that will one day hold an ESB.


Copyright 2003. Originally published by TechMetrix Research, reprinted with permission. TechMetrix is a technology-oriented analyst firm focused on e-business application development needs. TechMetrix is also backed by its parent company, a European global system integrator - SQLI - with more than 800 developers in the field.


More on this topic

  • Looking for free research? Browse our comprehensive White Papers section by topic, author or keyword.
  • Are you tired of technospeak? The Web services Advisor column uses plain talk and avoids the hype.
  • For insightful opinion and commentary from today's industry leaders, read our Guest Commentary columns.
  • Hey Codeheads! Start benefiting from these time-saving XML Developer Tips and .NET Developer Tips.
  • Visit our huge Best Web Links for Web services collection for the freshest editor-selected resources.
  • Visit Ask the Experts for answers to your Web services, SOAP, WSDL, XML, .NET, Java and EAI questions.
  • Couldn't attend one of our Webcasts? Don't miss out. Visit our archive to watch at your own convenience.
  • Choking on the alphabet soup of industry acronyms? Visit our helpful Glossary for the latest lingo.
  • Discuss this article, voice your opinion or talk with your peers in the SearchWebServices Discussion Forums
This was first published in July 2003

Dig deeper on Enterprise Services Bus (ESB)

Pro+

Features

Enjoy the benefits of Pro+ membership, learn more and join.

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

-ADS BY GOOGLE

SearchSoftwareQuality

SearchCloudApplications

SearchAWS

TheServerSide

SearchWinDevelopment

Close