Popular social media platforms such as Facebook and LinkedIn have begun to influence Web application and integration architectures. One example is Storm, which has been open sourced by Twitter.
Storm is the data stream or complex event processing (CEP) engine Twitter acquired with its purchase of BackType Labs in 2011. It was designed from the ground up using a MapReduce-like architecture, which allows new servers to be brought on board as demand warrants.
In a way, Storm is positioned as a broad-based engine for allowing companies to apply stream processing to Web data, much as CEP engines have done for crucial financial applications over many years.
“This is just ‘Main Street’ catching up with Wall Street since they have been dealing with huge quantities of data in real-time for many years,” said Mike Gualtieri, principal analyst at Cambridge, Mass.-based Forrester Research Inc. “The interest in platforms like Storm, S4 and Flume is driven by big data and real-time analytics.” Outside of specialized Web applications, other CEP platforms are likely to dominate, he and others indicated. The commercial CEP engines such as Progress Apama, StreamBase and SAP Aleri are deeply embedded in many financial systems, Gualtieri maintained.
What is Storm? The Storm CEP engine was built by BackType on the back end of a social media platform for enabling comments. The service allows users to better manage information across multiple social media feeds. Twitter purchased BackType last year for both the social business and access to the Storm CEP engine for better event analysis of the Twitter stream. In a blog post, Nathan Marz, the lead developer of Storm at Twitter, said there are three broad use cases for Storm, including stream processing, continuous computation and distributed remote procedure call (RPC).
Web-based applications of the cutting-edge kind
Roy Schulte, vice president and distinguished analyst at Stamford, Conn.-based Gartner Inc., said the Web-oriented MapReduce approach used by Storm, S4 and Flume makes the most sense for cutting-edge Web-based applications that require new features or adaptors that are not available in commercial products. However, he emphasized, the bulk of the CEP market will continue to be dominated by vendors specializing in financial markets and by integrated BPM suites with CEP support. Players he mentioned included Oracle, IBM, Microsoft, Tibco and Software AG.
“I think a lot of these will be used for cloud-based apps running on Rackspace or EC-2 and meant for building very high scale, but not necessarily low latency kinds of applications,” Schulte said. “Most traditional CEP tools can be use used in the cloud, but have not been.”
Established enterprise CEP tools may not face a challenge from the upstarts, according to Schulte. “I don’t think mainstream companies are going to jump on it quickly unless they are a leading edge, risk tolerant company,” he said.
Extensions of Storm help manage complex processes
Innovative developers are excited about some of the progress in extending the development toolset for Storm. Maarten Ectors, head of Cloud and Disruptive Innovation at Finland –based Nokia Siemens Networks, said Storm is being extended with Trident tools that help manage Storm processes.
Trident helps to reduce the complexity of creating applications that intermix high throughput stream processing with distributed queries. It also includes tools for doing processing on top of databases.
In some areas, there may be concerns about Storm application programming interfaces (APIs). Its backer, Twitter, has been criticized for what some see as backtracking on the openness of some of its interfaces. “Companies that are looking for standards will want to wait and see if Storm becomes the default standard,” Ectors said.
Financial industry users will probably never use Storm, Schulte said. “In that marketplace, the developers are extremely concerned about the latency, and there is at least the impression that Storm does not support the kind of latency you need for algorithmic trading.”
There are also concerns about the maturity of Storm, and the need to hook into data feeds and support established industry protocols like Customer Information Control System (CICS). “It would be hard to use this in a front office stock application,” Schulte said. Storm shows better promise in emerging Web-based applications where organizations can afford to invest in good developers.
Other products like StreamBase, Apama, Tibco and IBM have development toolkits, dashboard building tools and testing tools. In contrast, Storm, S4 and H-streaming provide a basic framework for building CEP; it is up to developers to do the rest.
“The commercial toolsets with accessories and development tools are meant for companies that want to do less development themselves,” Schulte said. “But if you have a new type of application for which off-the-shelf extensions are only partially applicable, then maybe you should be using one of the newer open source products, because commercial products don’t have adaptors that are useful for a lot of new types of applications.”