Companies today hope to gain a competitive edge based on their ability to respond quickly to threats and opportunities. That edge can come from highly responsive, near-real-time systems based on complex event processing (CEP) techniques used for financial applications on Wall Street. Now these CEP techniques are being used more widely in enterprise applications on Main Street. CEP examples abound. Fraud analysts are alerted when the same credit-card number is used in several cities almost simultaneously. Power-line fault histories and locations can be quickly compared against real-time weather feeds to alert utility maintenance teams about likely trouble. A health care provider can rapidly check a type of test regimen against a patient’s insurance coverage. CEP is a way to greater corporate innovation.
For many companies, the move to event processing means a shift in perspective on software architecture. Complex events have unique characteristics, and people involved in the field still argue about the definition of various terms that describe CEP. “Event-driven architecture” is sometimes used to express the way in which CEP problems are handled.
Although the inner workings of various CEP engines differ in composition, their structure may not particularly concern application development managers who will instead tend to focus more on how a CEP engine is programmed. “If you talk to users of CEP, they may well be interested primarily in what they can do with the user interface to the engine,” said David Luckham, emeritus professor of electrical engineering at Stanford University and author of Event Processing for Business: Organizing the Real-Time Enterprise (John Wiley & Sons, 2011).
To ease programming, CEP vendors support SQL-like, visual and related programming techniques. These fit well with existing programming skill sets. Sometimes CEP products are distinguished based on whether they support query based CEP or rule-based CEP. But tooling is where the development team meets CEP head on.
Islands in the event streams
The varied composition of the “databases” underlying different CEP engines is in part because of their varied lineages. Some hail from the world of rules engines, some from Business Activity Monitoring (BAM) and some from other backgrounds.
Event-driven systems may show kinship to relational databases or middleware messaging systems or to hybrids that merge both schools. Sometimes underlying databases are described as “streaming databases” or “event streams.” Vendors with CEP-related products include Espertech, IBM, Informatica, Microsoft, Oracle, Progress Software, RedHat, StreamBase, Sybase, Tibco and Vitria, among others.
In most CEP engines, data records are processed before they are stored. That makes it different than, say, data warehousing. CEP differs from the conventional RDBM in that a small amount of data may be matched against a large number of queries rather than a small number of queries being run against a large amount of data.
Events—simple or complex—can be analyzed as they “stream” through operating systems. There’s a special emphasis on handling time-series data and establishing time windows that show event data. Within these windows, new data are compared with known event patterns.
Complex events are collections of several simple observable events that are of interest. To cite a common example, you might want to see whether the same credit card was used simultaneously in two locations, a situation that could indicate fraud.
SQL semantics sometimes in CEP
Observers say some CEP engines tend to exhibit more affinity with conventional relational databases, while other CEP engines exhibit less such affinity. But from a programmer’s point of view, the differences may be narrowing.
“Some CEP products look more like DBMSes than others. But the differences are a lot less than they used to be,” said W. Roy Schulte, a vice president and distinguished analyst for Gartner Inc. “Five years ago, there was a big difference.” One driver for convergence is tooling that supports SQL-like means for dealing with the unique characteristics of CEP data streams. Staying with the familiar has benefits. But the nature of event-driven architecture is such that some CTOs and development managers will consider new programming models. This can be a difficult decision point. You may not want to change your programming style, but a new class of applications may actually best be handled by a new programming style, said Curt Monash, president of the Monash Research consultancy. “There is a programming model that revolves around events and taking action on them, which is different [than conventional data base management systems],” he said.
Speed and complexity
Where speed or complexity is the issue, interest in “how the engine works” may become more pointed. Here, most CEP engines vastly outpace general-purpose servers.
“The products are all optimized for performance in very demanding situations. In some cases, one is better than another. They also have rather different programming models,” Monash said. “There are ones that are stream based that emphasize visual modeling. These and others let you write code if you insist. There are rule-based and language based [programming models]. Each [product] has its clear roots or emphasis. And that is often a very natural basis for choosing between them.”
One may create a SQL-like query language with special capabilities to handle streaming data and continuous processing. Another may provide a graphical user interface for modeling, allowing team members to drag and drop event components to compose applications. Yet another may provide a graphical event-flow language.
Meanwhile, as with other upstart technologies, CEP may ultimately have difficulty finding a steady job in the mainstream—that is, if incumbent database and business intelligence technologies keep up. “CEP is a contender,” said Monash. “But it may not win.” CEP is an alternative to database technology. Outside a couple of niches, it’s not clear that such an alternative is needed, given how capable and varied database technology is.
Handling thousands of events per second
Where do the workings of the underlying CEP data-handler truly make a difference? One place to look is especially complex event architectures requiring high-speed and high-volume processing. “Some of the problems that arise in applying CEP involve the complexity of the event patterns being used,” Luckham said. “If you have to check several events and a state reference representing—say, the history of events in the past hour—then you have to both detect the pattern and see if the state is true,” he said. That task becomes a bigger issue when fragments of patterns are involved, he said. “You often get partial instances of patterns. You may have several or many partially matched patterns. What do you do with that in CEP? When you get into those types of problems, at that point the structure of the [underlying event database] becomes of the interest to the application specialist,” Luckham said.
The issue could well depend on the kind of problem you’re facing, he said. For example, if you’re trying to do arbitrage trading at several thousand events per second on several stock market feeds simultaneously, the way the CEP engine handles events will certainly make a difference.
About the Author
Jack Vaughan is editor in chief of TechTarget’s SearchSOA website. Contact him at firstname.lastname@example.org.