BPM and SOA projects: Avoiding common failuresDate: Nov 18, 2010
Business process management (BPM) and SOA projects have become increasingly interconnected in today's enterprises. SearchSOA recently held a virtual trade show on BPM and SOA to explore strategies for integrating SOA and BPM in the modern enterprise. The following screencast was taken from a session at that show.
In his session, "Avoiding Common Failures with BPM and SOA," independent consultant Michael Rosen covers four areas where BPM failure occurs and how to bolster struggling BPM projects with solid SOA techniques. Avoid falling into common failure modes with SOA and BPM projects. Discussion topics include:
- having an effective architecture in place
- using common semantics across business process definitions and services
- understanding enterprise context for services and business processes
- implementing BPM projects in an Agile fashion.
Read the full transcript from this video below:
BPM and SOA projects: Avoiding common failures
Jack Vaughan: Welcome to the SearchSOA.com, virtual trade show on pragmatic [DPM] or SOA. My name is Jack Vaughan; I'm the editor for SearchSOA.com. I'll be your moderator for this session of our event. We're looking at strategies that work today for those that need to make business process management happen in the enterprise.
If you experience any technical difficulties you can email our team at virtual seminar support at techtarget.com. I'll say that again; that's virtual seminar support at techtarget.com.
Now, in the last year, service-oriented architecture has increasingly been apprised of the problems of business process management or BPM. Our next presenter is well qualified to talk you through what is sometimes a bit of a mine field. Mike Rosen is an independent consultant providing advice and assistance on the design and implementation of SOA business application and enterprise architecture. He's the Director of Cutter Consortium's Business & Enterprise Architecture practice, and has years of experience in architecture and design of applications from global corporations involving all kinds of integration technology over a good period of time.
Mike will discuss how to avoid failure modes of BPM and SOA. In the trade of [software] architect these are sometimes called anti-patterns, things not to do. We think it's a good way to help you to figure out how to do the right thing. Mike, thanks for being with us. Why don't you get started.
Mike Rosen: Thanks a lot, Jack, and welcome everybody. What I want to talk about today is some failure modes that I see happening quite often in projects that are related to SOA and BPM. And so, the four failure modes, or the recommendations for avoiding these failure modes that I'm going to go through are first, having architecture, and we'll talk a lot about architecture. Then, use common semantics across your services and in the definitions for your process. Understand enterprise context that these things fit within. And then, implement the BPM and SOA projects in an agile fashion.
Now, if any of you were lucky enough to tune in to the earlier presentation by [Sandy], it was a very nice presentation. My style will be quite different. I have almost entirely pictures instead of words. And I tend to try to keep the pictures moving. I want to start the presentation with sort of a fictitious example. So, imagine that two insurance companies have merged together, and so now we want to make some sense of having those companies merged, and what we want to do is give people an incentive, if they, say, a homeowners policy, to buy an automobile policy, if they have an automobile policy to buy a homeowner's policy. You want to get some cross sell; you want to have the synergy of having these two companies together.
So, how do we go about doing that? So we define let's say a new business process to come up with the quote and apply this enterprise pricing policy to it. But the reality is that pricing and quoting and things are still implemented on our existing legacy systems. So, how do we go from this new business process to those existing systems? The typical scenario is that we define this process, a model, and then our BPM tool has a mechanism that allows us to connect each activity in our business process to some existing system. So there's some mechanism that allows us to essentially reverse engineer the EPI on our application system, read in a COBOL copy book and expose it. And so we do this with drag and drop connection to the existing systems. And we do then some integration between systems, maybe data integration between two applications, or integration at an API level or maybe their more sophisticated applications that support integration through web service interfaces.
And this is all well and good, and after about three or six months we've got this process up and running, and everybody's happy. But the question is, what happens next? Right? And basically what happens next is that we discover that if we just had the information from here over there, as well, we could enhance our business process. Or, if we could connect these three things together instead of just being two we could have a smarter calculation of risk. Or, the next time we need to implement a business process we don't take into account that we've already done it and we essentially do that same thing, but in a slightly different way.
So, what we've seen from experience is that this approach, where we tightly couple up business process activities directly to the existing systems, doesn't scale. We end up with this mess here, what I sometimes call enterprise hairball. But it has a real name; it's been called the enterprise application spaghetti, for quite a long time. This is a slide that I borrowed from [Gartner] in 2000. So this problem has existed for a really long time. And what we believe is that there's a better way to do this integration; there's a better way using a service-oriented architecture in BPM approach to get away from adding to our integration mess, and instead, implement these processes in a way that's more agile.
So, we have a typical architecture and we're going to walk through this picture, starting at the bottom. We have all of those existing systems, the pricing, the quoting, and policy systems, from our two insurance companies. And what we're going to do is we're going to expose those systems as services, right? We're not going to simply wrap them as services; we're going to think about what aspect of those existing systems is important in our new business processes and we're going to probably do some transformation in terms of granularity and intonation models to provide a service interface that connects to all of those systems.
So we have now this integration layer of existing information capabilities now exposed as services within our enterprise. We start now to build up a layer of business services. So, we implement, say, a new business service for accessing policies. Here I have pricing underwriting policy identified as business service layer. And what we want is we want all of our business processes to use the policy service when they need to access policies. And we want to encapsulate the complexity of understanding the different types of policies we have, different types of policy systems, and where they exist we want to encapsulate all that complexity in the policy service. So, there's one place that we go to in the enterprise to get policy information, and that's where the complexity of the different policy systems are.
Once we've implemented our business processes along this line, then we have the ability to, let's say, move all the policies from one policy server to another. Or maybe, one system is good for homeowners and another is good for automobiles, and the third is good for inland marine. And so we want to migrate all of our policies to different systems, or maybe none of the policy service implementation that we got with those two insurance companies are any good, and we want to buy a third modern policy system. We can do any of those changes by replacing the existing resource system, that lower layer, with a different implementation, as long as we don't change the pager interface at that business service layer. What we're trying to do is insulate the business processes from the underlying legacy systems.
Finally, once we've got this business service layer built up we can start to add business processes, that blue layer on the top, which spans across the enterprise. We can have a pricing system which knows the policies that I have across all lines of business and applies my discount. We can have marketing system that lets you understand the policies that I have and some attributes of the customer and to propose specific campaigns or products. We can manage the customer across all of those accounts.
And then, we want all of this, SOA architecture, to fit within an overall N-tiered multi-channel enterprise architecture. Typically, we see this type of picture drawn as a layered SOA architecture, so business processes at the top, business services in the middle, integration services connecting to our operational resources. Now a layered architecture has a couple of characteristics, right? We separate different concepts in the layers and responsibilities, and then the individual layers interact with each other through interfaces. So the business process layer is responsible for defining business processes and breaking those business processes down into individual activities.
The business service layer is responsible for implementing those business activities as operations on different service interfaces. And the interaction between the process layer and the service layer is through that service interface, where the interface is defined by the semantics and granularity of function required by the business processes. So at this business service layer we have very high level services, which are defined to support business processes.
Now of course, the business process layer and the business service layer are more complicated than this. We may have multiple levels of business process. In the service layer we typically have three or four levels of service from the high level business service, a medium level of what I call the main service, and lower level, utility services. It might include software as a service, SAS services that we've purchased from other organizations. And those services can perform a variety of different roles, such as implementing task, managing information, and making decisions.
So, the question is, if this is the better way to go about solving this problem with integrating BPM into our enterprise, how do we do that? How do we go about implementing SOA? Some vendors, in the past at least, tend to believe all you need to do is go out and buy some sort of web service platform. So, we can go out and we can buy an enterprise service bus, and that's all we need.
A web service platform certainly helps, right? It gives us this service paradigm so that we can implement services in a way that's natural to the technologies we're using, and it solves a lot of the complexity problems of previous technologies. But it hardly addresses the complexity of the architecture that we need at the enterprise level.
Instead, we need a much more rich architecture. So, let's look at the characteristics of enterprise-wide SOA architecture. First, we have some sort of service bus, and whether we buy that as open source, however we get it, we need to clearly define the protocols and technologies that that service bus supports, so that whoever is using it uses it in the same way. Typically I recommend that people adopt the WSI Basic Profile Version 1.2. It doesn't really matter exactly which profile you use or what set of technologies you use. What matters is that you have them consistent across all of the services.
Next we have this question about when we face service what do we mean by a "service"? What are the different types of services? Do all of our services support some common interface that allows us to manage them, or to collect metrics about their performance? Do we have a common naming convention for services? Then, we need to define when we go and do this service encapsulation or service exposing of our legacy systems, how will we go about doing that? That's a very complicated problem. We want a standard way to do that.
So, we've sort of taken the typical web service picture, and we've given it bit more specificity in terms of the protocols that the service bus supports. And what a service is, including how we integrate existing applications, now we're at the point where we say, well, what do we want to do with those services? The answer is that we want to build up enterprise business processes. We want to support BPM with web services. So the question has to be answered by the architecture; what's the business process? What's the relationship of the business process to a service, and as we saw in the previous slide, it's that interface between activities of the business process level and the operations at a business service interface.
Once we define that, now we start to get into the question of how do we now take services and combine them together into processes that support the overall enterprise. So here, someone, typically an architect, has to ask some business questions. Questions that I like to ask are, what business are you in? What market does position, what strategy are you trying to achieve over time? Given that position and strategy in your competitive and regulatory environment, what are the products and services that you need to have to support that market position in that environment?
Next, what are the capabilities and information that you need to support the business processes that provide those business processes, products, and services?
And then, finally, the question is, what are the SOA services that you need to provide those business capabilities and information to support the business process that provide the products and services? So, we need to ask a high-level question, and create some sort of roadmap of services that we need over time. Now, those are sort of big questions, but typically, I find that in a few days or a week you can answer those questions and lay out a roadmap, or at least the context. And we'll look at this a little further in the presentation, what I call a service catalog or inventory. You can typically identify 100 or 150 basic building blocks within a week. So, I'm not talking about spending three years or even three weeks doing some huge high-level analysis in design. I'm talking about thinking ahead in a practical way.
The last thing that we need is, now that we've got this set of services that we can combine together to support business processes, I want to combine a bunch of services together that have something to do with the customer. To support a business process, those services need to have the same view of a customer. So I need some common semantics, some definition of the information that needs to be shared between services that then work together in a business process. I'm not talking again about a three-year effort defining every attribute of every business entity. I'm talking about defining the information that has to be shared between services to work together in business processes, which is a much more manageable and smaller subset of [inaudible 18:02]. But if we don't have the common semantics, it doesn't matter how good the services we have will be, because we won't be able to use them together.
So, that's somewhat of a technology view, if you will, of business process in SOA. From a business process or BPM level, we want a different view. We're not interested in all of those details. What we're interested in is this relationship between business processes that we're designing using our BPM tools, and the business services that support those activities through our interfaces. Now is you look at this picture, again, this is a fairly common picture.
What's missing? Well, what about the information? Often, when we look at business process and SOA, if we ignore the information aspect and we focus on the functional aspect. The information is complex. Every service has some set of information. Existing applications have some set of information, and what's important is that at the enterprise business process they have some common view, some integrated data, or what I previously called common semantics. So, we've introduced a new level of information that we use to pass between services inside a business processes, what I call semantic data. So, we're probably used to thinking about the difference between physical data and the data base view?, or what I call [domain] data in this picture.
So, we know that as we function, as we used objects in our applications, or we use database views, that that storage, that view is different than how it's stored in tables and rows in the database. Now, what we're talking about is introducing another level of information, which is some aggregated normalized rationalized subset of information that's duplicated across multiple stores, and we come up with one common [canonical] view of it, and that's the information that's passed, meted out those service interfaces.
So, the first thing that we need to do in addressing the failure modes, right, is to have an architecture that completely describes SOA at the enterprise level, that positions SOA in the multi-tiered modern Internet architectures, and that defines the position and responsibilities of services and business processes, and the use of common semantics between them. Having that architecture is good, but it's not enough to be successful in designing business processes.
So, the next thing we need to understand is how to place the enterprise context into our design process. So, here is a fairly complicated picture, sort of divided into three activities. At the architecture level we design those business capabilities by asking that set of questions, and we've described those business capabilities in terms of a service catalog. At the design level the BPM design tool has to connect to that service repository in order to provide the usability into existing services. On the right side, there, it should say, "SOA Design Tool." I must have done a cut and paste error. So, also, the SOA design too, the tool that we're using to build SOA services, must have the ability to publish those services into this repository in accordance with this business model, or this service catalog.
Now I don't want to spend a lot of time talking about run time, but I do want to discuss an important difference between design time and run time. At design time we use a repository. A repository lets us discover services and then evaluate them for appropriateness in our business process. Perhaps we can search for services, give a subset and then do a refined search on them, find a service, and then evaluate its interfaces and perhaps we can evaluate its test plan, its reliability, its [bug] rate, so that we can have confidence that it's the right service and that it's a good service.
That is very different than a registry, which is used at run time for looking up a service. Once we know the service we're going to use and we've designed it in our program we don't want to hard code its address into our program, into our business process. So, at invocation time the business process goes to this registry and says, I want to use this specific service; and the registry binds it to that business service.
One of the failure modes here is that people try to use this run-time registry as a design time repository and it just doesn't have the capabilities necessary to be successful. So, I like to organize my service inventory, or service catalog, in a variety of different ways. I position the services in it, say, at the different granularities, utilities, domain, business services. I sometimes divide it up into organizations, or business domains, or business processes. The exact organization is useful, but how you decide on the exact organization is how it best fits your organization, and your needs, your design processes. But what's important is that you have some version, some vision, of a sense of services that you need now and over time to meet your set of business requirements.
So, here's an example of a subset of service inventory for travel insurance. And the idea is that as I go to build a business process I can look at this service inventory, however it's organized or visualized, and I can see the services that already exist. And I can make sure that I build my business process around existing services. If I'm responsible for building a service, or enhancing a service, I can look at the overall set of services and I can see what responsibilities that service has and what it doesn't have. So I can see if I'm going to add a new service here that has something to do with the customer that that service is not going to take the responsibility for handling billing or the plaids to that customer. I can see not only what my service is supposed to do, but equally as important, what my service should not do.
Now this is part of a larger overall process where I start from the business architecture level, describing the strategy, goals that set me up with an overall business context, service inventory and information model, from which I define individual business process, which then leads to design of business services in this overall inventory and business documents that follow that enterprise semantic model.
Now, as architects, we can get carried away with stuff, which you're probably thinking right this very second. Now the challenge is, as architects, we have to strike a delicate balance, right? If we were trying to achieve certain enterprise goals, and here perhaps the enterprise goals are to enable individual organizations to build services that will be able to be combined together to support business processes. Or, to allow business process analysts and designers to build the business processes based on an existing flexible, agile set of enterprise capabilities. Those are smart enterprise goals, and so the architecture has to describe enough so that we meet those goals. If we prescribe too little, the goals are not met. If we prescribe too much, then architecture is then ignored. So, a challenge is to have just the right level of architecture to, and this is a balance that we're always going to be struggling for; what we want to do then is to phase our architecture in over time, right?
So, in terms of SOA, what we're going to do is we're going to start with some individual project. We're going to look for something that's a good candidate, that's important enough for people to care about, with an achievable scope, but not so important that if it fails, we're on a critical path. And then, we're going to implement that SOA service, and hopefully in the context of a business process project, we're going to figure out what works. We're going to roll those learnings back into our processes and organizational structure and governance processes. And then we're going to expand that out.
So, here is an actual roadmap from a project that I did a few years ago, illustrated in two phases. The first phase implements the very first service. This organization is just learning how to do SOA. So, how do we build a service? So, we put that architectural context in place and build the first service. They had four people working on this project for six months. At the end of six months, have a service which is complete. And so, now the idea is we're going to deliberately take three months to evaluate what we've learned in implementing that service, to start adopting some processes and governance as architecture that makes sense; and to look for opportunities to build more services.
Now we enter phase two, where in phase two we are going to have four teams of people building services. So, at the beginning of Phase One, we had one team of four people who knew a little bit, but not much. And in Phase Two, we take each one of those four, experts now, and assign them to a team to help that team implement new services. We're going to try to make sure that we use our first service in the second phase so we can learn how to do multiple versions of a service, learn how to enhance a service. We're going to expose some legacy systems as a service in this space, because we want to learn how to do that.
And so, again, it takes about six months for these four teams to come up to speed, and we again have a three-month re-factor cycle in here; so we're going to go back and figure out what worked. We're going to incorporate those lessons learned into our architecture, into our governance, into our processes. And at the end of these two phases, we're probably ready to do a larger scale SOA rollout. Again, at the beginning at zero, we have really very sketchy architecture, no process, and no people know anything. At the end of Phase One we have four people who are relative experts, and we have an initial architecture as process and governance.
At the end of Phase Two we have four teams of people who know how to build services, four experts who can be consulting and head out and expand that out into the organization. And perhaps, most importantly, we have enough time to validate, prove, and document our architecture and processes to a level at which they can be followed by other people. If we try to roll out SOA faster than this, before we've had those two cycles and matured our architecture and documentation. If we try to roll it out before that, we will probably fail.
So, this agile approach to SOA first in learning what to do and then rolling it out is important. But, there's another aspect to this approach, which is also very important. You have to understand that services allow us to provide incremental implementations. We have version one of a service, and that version one supports some small subset, potentially, of a much larger need. So here's an example of some customer information services at a financial organization. And back in 2001 there were four retail products that they wanted to combine together, so they created this customer information service, Version 1, that combined the customer information across these four retail products. That's Version 1 of the service.
Then, they started to add a few more retail products, and they created Version 2 of the service. And now they're supporting both Version 1 of the four original projects, and the Version 2, of the new ones. Over time, they rolled in a dozen different lines of business,  different products in a dozen different lines to producing this into the customer information services to the point where now they've got again, the customer information service up to Version 47 of this service that supports the entire enterprise. Now they could never have gotten an information model of the customer correct at the beginning if they tried to do every one, and they could have never built it all at once. And they could never have gotten the organization to adopt it.
So, the agile incremental approach is find some customers for this service, implement it, support their business process and their business needs, and then over time enhance that. If you provide value with that service more business processes will want to use it. So now, the next business process comes along, and wants to use that service, and you add the additional capabilities to support that. Over time you'll built up perhaps two or three simultaneous versions of the same service and eventually people will [inaudible 34:39].
So, to summarize again, the business process, BPM, is a great way for analyzing business problems and designing solutions for that. But alone, BPM doesn't give us the flexibility and agility we need across the enterprise. We need some platforms that provide business capabilities where we have one way to do one thing and one place to get the information. And SOA gives us the architecture and technology platform to take those business capabilities and information and make them available to business processes. So together, those two things really give us the capability to provide agile flexible enterprises.
So, why don't we take some questions?
Jack Vaughan: Well, can you hear me now?
Mike Rosen: Yes.
Jack Vaughan: If I could, I'd like to get a word from someone in the audience. The slide they're asking us to be sure and go over I think is 27, The Failure Mode.
Mike Rosen: The Customer Information Service slide? Is that the right one?
Jack Vaughan: Can you highlight again what issues are involved in this portion here?
Mike Rosen: Sure. The idea is that through our services it gives us a way to incrementally add capabilities and maybe more importantly, to incrementally bring in more business consumers, more business processes of that service. So in this slide, the financial institutions started with four retail products where they wanted to have a common customer information across those four products. So they built a set of customer information services that aggregated the customer information across those four retail products. So if you had a checking account and a savings account it looked like the bank had one customer with multiple accounts. Now, you sort of expect this today, but ten years ago that was not the norm.
Now that the version of that business service is built, and these four retail products are using it and they're starting to build new business processes, and starting to get cross sell, other retail products look at this and say, "Wow. We'd better be in there. Our customers are complaining that they have to go somewhere else to get information about them." So now, they bring in the second, fifth, six, and eighth product in the retail line of business, and the way that they do that is by coming up with Version 2 of the service, and then Version 3 of the service. And over time in this particular institution they actually came up with 47 incremental enhancements to the Customer Information Service, which allowed them to bring in 12 lines of business in over 80 different products into the common information services.
Jack Vaughan: I'm sorry, I'm going to move to the question. I guess in one part, it seems like you bring the business stakeholders in somewhat late in the day. If you have in mind this ample timeline you discuss, does anyone look at your architecture view and your timeline and say anything like that? I'd like to business people in here.
Mike Rosen: Well, the idea is that I'm going to build some kind of a service as part of a business process management initiative. So, identify the BPM project, right, and there's some real business need for some flexible business capability of the information, and we think SOA is the best way to support that business need for this specific project.
Now, I'm going to start to impose that SOA timeline for building that project to support that business service. Then, at the end of that project, we're going to look for more business projects, more BPM projects, that need more capabilities and service, and look for services as a way to support them. So, we're going to be supporting business processes at the same time as we're enhancing our internal capabilities to build SOA services.
We will have prior to that engage the business in the discussions about what is our business architecture, or what is our business model of capabilities and information that we're going to need to support.
Jack Vaughan: Okay, at one point you talked about a type of buzz a business service buzz. I've never seen that kind of thing.
Mike Rosen: Well, it's the same name for what we call typically the product class, is enterprise service buzz. And so, I'm using that enterprise service buzz as a way to make business services available across the enterprise. And the point that's important from an architectural level is that it exposes business services in a way that allows them to be combined together to support an enterprise processes. From an architectural level I'm not so concerned whether someone goes out and buys an ESB to do that or they build it on message cuing. So, from an architectural level the important concept is the many to many relationship business services and business process supported by the bus concept and not so much the particular implementation of that bus.
Jack Vaughan: Let's go to a question from a viewer, Andy. Should production support be a separate team from the service implementation team? What do you see in your experience?
Mike Rosen: Typically I see the production support is a separate organization. I think that there needs to be a close working relationship between the implementation team and the production support team. I personally think that implementers should be responsible for the bug fixes in the services, but they probably should be responsible for the day-to-day answering of particular production incidents. So, I typically see the implementation team as a second-tier support; production support is the first tier. Typically, they're organized under different management, but they need a very close working relationship.
Jack Vaughan: Here's a question from Murray, and it relates to Slide 9. How do you layer additional complexity into the architecture, things like security, user roles, and capacity?
Mike Rosen: Here is Slide 9. The security, users roles, capacity, in a brief presentation it's hard to drive all those complexities. Certainly, security is important at every level of this architecture. And so, you need to have a security architecture, and typically your SOA security architecture has to be some derivative or some enhancement to your enterprise security architecture. And then, the characteristics of the individual services have to comply with that security architecture as well as the processes that you use to build the services have to support the implementation of things that conform to that security architecture.
The users, if we think about how this picture fits into a typical interior architecture to where we have a presentation, application, enterprise, and resourced tier. This picture really only shows the enterprise and resource tier. So the way that we integrate end users and the way we support multi-channel and multi [different] devices is in a different layer of an overall application architecture. So the brief answer is this is just a small piece of a larger architecture, which is part of the challenge of this recommendation to have an architecture. Here I could only explain a little bit of it, but to be successful and integrate all those other complexities that architecture needs to be more complete.
Jack Vaughan: You talked about having useful data, having a worthwhile repository, separating the design time repository from the run-time repository. And the questions comes up that remind me of questions from the object era, and they've continued into the services era, is how do you define a customer? That sometimes can be a stumbling block for a project. How do you view ways of finding a fast, agile path to deal with those things? You've pointed out, to have the complexity you need, but not more. How does that work out in real world implementation, Mike?
Mike Rosen: My approach is to start small and to figure out what works for your organization, and then to expand it. And so, certainly there is a challenge in having the right kind of information. And understanding what information is useful for your organization and how that information supports the processes that are used in your organization. So, I think a lot of the failures in the past were that we spent a lot of time building these libraries or repositories of information without a thought to what part of the process they supported, and how they worked within the culture of your organization. So again, I like to start with something pretty simple. Sometimes you can start with something just as simple as an Excel spreadsheet for the first couple of services. And once we understand the process and start to add some changes to those processes as well, and understand the governance, then we can implement more sophisticated SOA and BPM repositories. Often the BPM tools come with some simple mechanism for doing that. I think the challenge is understanding the role of that process in a repository in the overall analysis of design and the implementation process, making sure that those two things are in sync.
Jack Vaughan: Well, at this point I'm going to ask the audience and the presenter Mike to appreciate that we did have a break in our business process there, but we are going to make much of this available after the fact. I'm telling people to remember the Universal Trade Show environment. You are able to view presentations in our conference hall, our exhibit hall. There is a networking center where you can ask questions of individuals, and a resource center. I'm glad I'm going to be able to pick up on this conversation a little later today and in the panel talk about some of the context under which these decisions and places have to be made. Thanks so much for sharing your thoughts on BPM and SOA.
Mike Rosen: You're welcome, and I look forward to talking to you later. Hopefully the technology will work better that time.
Jack Vaughan: Hear, hear.