Explaining integration

By Philip Howard 03 Feb 2004 | IT-Director.com, special to SearchWebServices.com

Digg This!     StumbleUpon     Del.icio.us

Market Analysis

Explaining integration
When Bloor Research first set up its practice model, integration was divided between me and my colleague Peter Abrahams. Amongst other things, he was to look after application integration, while I would focus on data integration. However, at no time was it explicitly stated where application integration ended and where data integration began. The reason for this was that it was by no means clear to us where any such dividing line should be placed. As my next project is a report on data integration I have had to devote some effort to attempting to clarify this position in my own mind and the following represents some of the conclusions I have reached.

The first and most obvious way in which you might differentiate between application and data integration is that the former uses messaging and the latter does not. This might once have been valid but it is not so any more.

If you think about types of data integration then there is the traditional ETL (extract, transform and load) functionality, which effectively acts in batch mode.

Then there is near real-time data transfer. This is typically provided not by the ETL technology itself but by the data connection technology that underpins it. For obvious reasons, such functionality requires change data capture to give it decent performance and this is what vendors like Striva (now part of Informatica) and Attunity provide.

Finally, there is true real-time capability, which is clearly a growing requirement. Now, both Ascential and Informatics, for example, have real-time engines but the truth is that more and more companies are using messaging to update data warehouses (for example) where they require real-time capability.

However, there is a fundamental difference between messaging and conventional data integration: the latter is a pull technology whereas the former uses a pull-based approach. If you think about it, this must be the natural approach to real-time integration: changes and updates must be propagated as they occur rather than by using some technology that goes and asks if some change has been made, since the latter must involve some sort of time delay.

Nevertheless, just because you introduce push technology does not mean you have to use messaging. It should be equally feasible to use database replication. However, this would mean replicating updates to the ETL tool for transformation purposes rather than to another database, which is the usual purpose of replication. Thus this is not generally considered as an option. Nevertheless, there is one ETL vendor, GoldenGate, which does precisely this: combining ETL with its own replication technology.

Does this get us a lot further? Well, it suggests that EII (enterprise information integration) is getting squeezed between EAI (enterprise application integration) and ETL, so we might lose an unnecessary initialism. It also suggests a negative: that messaging does not help to define the distinction between data and application integration. Perhaps we can say that application integration is initiated by an operational application and data integration isn't, or that the data passed during EAI is used directly by the recipient application whereas it is only indirectly used in data integration environments. If anybody has a more specific (and concise) definition, I would be pleased to hear it.

For more information:

• Looking for free research? Browse our comprehensive White Papers section by topic, author or keyword.
• Are you tired of technospeak? The Web Services Advisor column uses plain talk and avoids the hype.
• For insightful opinion and commentary from today's industry leaders, read our Guest Commentary columns.
• Hey Codeheads! Start benefiting from these time-saving XML Developer Tips and .NET Developer Tips.

• Visit our huge Best Web Links for Web Services collection for the freshest editor-selected resources.
• Visit Ask the Experts for answers to your Web services, SOAP, WSDL, XML, .NET, Java and EAI questions.
• Couldn't attend one of our Webcasts? Don't miss out. Visit our archive to watch at your own convenience.
• Choking on the alphabet soup of industry acronyms? Visit our helpful Glossary for the latest lingo.
• Discuss this article, voice your opinion or talk with your peers in the SearchWebServices Discussion Forums.

Digg This!     StumbleUpon     Del.icio.us