As students go through our Licensed ZapThink Architect (LZA) course, they experience a series of "aha" moments, as we systematically tear down their preconceptions about what service-oriented architecture (SOA) is -- and what it is not. But perhaps the biggest aha moment of all, however, is when they realize that implementing SOA isn't traditional systems engineering (TSE) at all, but rather a fundamentally different approach to dealing with complexity in the IT environment. Needless to say, this realization is an especially big wakeup call for people with TSE backgrounds!
The fundamental shift in thinking is this: TSE focuses on building big systems out of small components, where the behavior of the resulting system depends directly on the properties of the components. Essentially, TSE boils down to a "connecting things" way of thinking about distributed computing, where integration is the central activity, and what you end up with when you're done with all the integrating is at best what you expected to build.
SOA, on the other hand, calls for an entirely different approach. In the SOA context, we focus on building and maintaining the Business Service abstraction, which supports inherently unpredictable behavior as the business composes services to support fundamentally dynamic business processes. Essentially, with SOA we're building for change, while with TSE, we're building for stability. The problem with stability, of course, is it only takes the business so far -- if the organization requires business agility, then they're much better off implementing SOA.
Business agility as an emergent property
Business agility, which ZapThink defines as being able to quickly and efficiently respond to changes in the business environment and to leverage such change for competitive advantage, is the primary strategic motivation for most SOA initiatives. What is it about SOA, however, that enables such agility? Unlike with TSE, where the properties of the resulting system depend upon the properties of the components that make up the system, the individual elements of a SOA implementation, including services, service compositions, policies, contracts, and the like, somehow contribute to the agility benefit, even though each of them don't exhibit business agility themselves.
We call properties that certain systems exhibit that their individual components do not emergent properties. In addition, we call systems that exhibit emergent properties complex systems. Complex systems theory is an exploding area of study today, because so many different systems in the world exhibit emergent properties. Emergent properties include everything from friction to traffic jams to the human mind, and the field of complex systems theory is gradually explaining many of the more subtle aspects of the world around us.
Complex systems engineering: The key to implementing SOA
Explaining natural phenomena is one thing; building a complex system is quite another. We call such a practice Complex Systems Engineering (CSE). If you're looking to engineer a system, where your desired outcome is an emergent property like business agility, then TSE won't do -- you need to take a CSE approach. As a result, it is imperative that architects looking to implement SOA take such an approach, because business agility is a critically important emergent property, and in many ways defines the success criteria for SOA initiatives. Leveraging complex systems best practices, therefore, may be able to give us some important insight into how to deliver on the business agility promise, and perhaps more importantly, how to avoid impediments that might prevent a SOA project from providing the required agility.
A fascinating paper from 2006 by David A. Fisher from Carnegie Mellon University's Software Engineering Institute provides a marvelous starting point for a discussion of SOA in the context of a particular type of complex system known as a system of systems. According to Fisher's paper, systems of systems are qualitatively different from traditional large-scale systems. Such systems of systems are far more complex, and involve independently managed and operated components. Furthermore, systems of systems depend upon other systems that are outside the control of their owners and users. As a result, TSE approaches and methods are often inadequate or inappropriate for systems of systems.
Systems of systems have unique characteristics that distinguish them from traditional monolithic systems, including higher levels of complexity, flexibility, and emergent behavior. Such characteristics result from the operational and managerial independence of their constituent parts, from independent evolution, and from the character of emergent effects, while traditional monolithic systems depend on central control, global visibility, hierarchical structures, and coordinated activities. As a result, such traditional systems are unable to exploit the advantages of emergent behavior like business agility.
Is a SOA implementation a complex system?
Many of the characteristics of such systems of systems line up quite neatly with how ZapThink sees SOA. The core SOA principle of loose coupling in particular leads to the system of systems requirement of autonomous systems. In fact, complex systems theory recognizes that a result of tightly coupling systems is the fact that accidental failures of individual subsystems lead to cascading failures across the entire system. Furthermore, systems of systems depend upon interoperation among systems, rather than integration, where integration of subsystems leads to a unified system, while interoperation among systems is the combination of autonomous systems into a system of systems.
Interoperation in systems of systems demands what Fisher calls a node-centric perspective, where each constituent (a service provider or consumer, say) views the system from its own individual perspective. For each node, a node-centric perspective describes the interactions with its immediate neighbors based upon available metadata. In this view, the behavior of the overall system of systems depends upon the interactions between service providers and consumers. An individual constituent node need have no knowledge of the details of other nodes that it doesn't interoperate with.
To achieve this interoperation, systems of systems should be interdependent with other systems beyond their boundaries, where it's impossible to predict the resulting outcomes of the architecture. Furthermore, requirements for systems of systems that interoperate are constantly changing and imprecisely known, as they typically are for SOA implementations. Interoperation also encourages distributed control, which aligns nicely with the business empowerment benefit of SOA.
It seems, therefore, that an architectural approach like SOA that leads to loosely coupled, autonomous, interoperable services in response to dynamic business requirements is a perfect example of a system of systems. Fisher, however, makes an interesting claim to the contrary: that SOA implementations, at least at the time he wrote his article, don't provide sufficient interoperation in systems of systems because they assume an absence of emergent behavior or at least fail to recognize and provide support for managing emergent behavior. The question of whether SOA leads to emergent behavior like business agility, therefore, depends upon whether complex systems theory applies to SOA at all.
Taking a CSE approach to SOA
If business agility is an example of emergent behavior, and we expect SOA implementations to provide such agility, then how can we say that SOA implementations assume an absence of emergent behavior? This conceptual disconnect is at the heart of the aha moment our students experience in our course. Far too often, people assume that TSE is sufficient for implementing SOA, and TSE thinking excludes emergent behavior as a possible outcome. We see this misconception all the time, whenever an organization believes that they must purchase integration software like an Enterprise Service Bus (ESB) in order to implement SOA. TSE means connecting things, so the obvious place to start is with something that helps you connect things -- but that approach misses the boat entirely.
Fisher's second point is also well-taken. To put this point in another way, we would need to manage a SOA implementation's emergent behavior in order to achieve the benefits that systems of systems exhibit. ZapThink hammers home this point whenever we talk about business empowerment. If we mistakenly conclude that the sole point of SOA is to build all these flexible services and then simply turn them over to the business to compose willy-nilly, then it's true we'd never be able to achieve the business agility benefit, because such a haphazard lack of control would lead to immediate chaos. On the contrary, the key to the business empowerment benefit of SOA is governance -- a set of organizing principles that tempers the emergent properties of the architecture, thus providing the essential management of emergent properties that SOA requires for us to consider SOA implementations to be complex systems.
The ZapThink take
There is an important insight here which is the essential point of this ZapFlash: for SOA to be successful, organizations must take a CSE approach to their implementation, including the implementation of SOA governance. Our natural tendency would be to take a TSE approach to governance, where we hammer out organization-wide policies and put in place a systematic governance infrastructure to enforce those policies. While such an approach might be integral to a traditional architecture that we've implemented with TSE approaches, taking such a heavy-handed approach to SOA governance will stifle the implementation's emergent properties, most notably business agility. ZapThink calls this situation the "big brother effect," where excess governance can lead to the failure of the SOA initiative.
Instead, it is essential to take a CSE approach to SOA governance. Represent policies as metadata, and leverage SOA governance infrastructure to manage and enforce those policies in the context of Services. As the SOA implementation matures, leverage SOA governance best practices to support overall IT governance, and in turn, corporate governance. This approach to SOA governance "in the broad" not only improves the organization's governance overall, it is also essential to promoting the emergent properties of the SOA implementation. Your SOA initiative depends on it.
