What is the right way to implement an ESB within a SOA?
Enterprise Service Buses, or ESBs, have been a controversial technology since the term was first used in 2002. In fact, there are some who would even argue that it's not a technology, it's an architectural approach, and that's part of the problem.
SOA was the hot topic in the early to mid 2000's, and vendors wanted to capitalize. ESB products wound up being the technology most closely associated with SOA, leading some to believe that either an ESB was mandatory to adopt SOA (it isn't) or that by buying an ESB, you'd magically have SOA (you won't).
Even worse, no two ESBs were alike. Some ESBs were built from scratch, some were simple rebranding of existing integration (EAI) suites, some were delivered as a network appliance, and others were positioned as the next generation application server. It's no wonder we're still asking questions about how to utilize one properly.
While others may have a different opinion, my advice is that an ESB is best utilized for mediation and operational policy enforcement. It can be thought of as an intelligent proxy or router that sits between service consumers and services providers in the same way that a load balancer sits between a user's browser and the Web servers providing the content. A mediator is not something that contains business logic, it is something that enforces the operational policies associated with a service interaction.
This can include routing, security, traffic shaping (throttling), monitoring, metric collection, and some basic transformations associated with versioning. This does not include orchestration or hosting of business logic associated with services, even though there are ESBs on the market that do this. (For a more detailed list, please see this post on ESBs from my blog.)
My reasoning behind this is to maintain a separation of concerns. Business logic and process orchestration are tasks that involve a development team. They are meant to be coded using tools for a developer, whether that is an IDE or a BPMN modeling tool, and then deployed onto an application server, Web server, or process execution engine. Policy enforcement is an operational task.
Do you have developers configuring firewalls or setting load balancing pools? Probably not. They get the policy information they need from a development team, but the configuration is done by operations. Consider this when selecting your ESB vendor, because there are some out there that require Eclipse to configure them, which is certainly atypical for an operations team.
If you view your ESB as a developer's tool, what belongs on the ESB versus the application server? Whose choice is it? It is very likely to become complicated.
If you agree with my opinion on policy enforcement and mediation, and that this is an operational configuration activity and not a development activity, your next question may be, "Do I even need it?" It's true that firewalls and load balancers can handle much of this, but that's typically at a TCP/IP or HTTP header level, and not anything that may involve message inspection.
If you need to get deeper into the message structure, whether SOAP headers or XML or JSON payloads, some of the ESB technology may be a better fit, especially if some basic transformation of the message is needed to support different versions of the messages and interfaces involved (be careful with this as transformation policies can quickly turn as complicated as source code).
The other question is whether you'll need some of the less common areas of policy enforcement, such as traffic throttling. For internal use, this might be overkill. For external clients, where you don't have any visibility into how the service consumers are written, it may be more important. This introduces the concept of domains. You need to think of the environments where your service consumers run and where your service providers run in terms of domains subject to certain policies. Where the interaction crosses a domain, you may need an ESB.
The easy case is inside the firewall versus outside the firewall. It makes sense to have a mediation point for enforcing policies when that boundary is crossed. Inside the company, you may choose to do the same thing when calling between geographically dispersed data centers, or between an ERP environment (where standards of the vendor software must be followed) and a custom coded environment (where more flexibility exists).
What those domains are, and which boundaries are important are a decision for your organization, and it will vary, but if you are able to identify those domains, it will help you properly position your ESBs and avoid potential bottlenecks of policy enforcement. Move those enforcement points so that they only see traffic that is subject to those policies.
To sum it up, ESBs can play a valuable role in your environment if used properly. Make sure you understand whether you choose to use it as a tool for operations or a tool for developers, and make sure that you actually have a need for the powerful capabilities it can provide.
This was first published in September 2011