Tip

Q&A: Dave Linthicum talks about commercial Web data services and distributed computing

 

Dave Linthicum, the CTO of Bick Group, is a well known cloud computing and SOA expert. He played a role in developing concepts involved with distributed computing including EAI, B2B Application Integration, and SOA. David Linthicum has authored 10 books, including "Enterprise Application Integration" and "B2B Application Integration," and is an active blogger.

SearchSOA.com: Is there a specific area of

    Requires Free Membership to View

distributed computing that you are focusing on these days?

David Linthicum: In the cloud, we are moving toward what I call Big Data. The idea is that because we can get access to an almost unlimited number of servers at the same time -- thousands of instances, which is far more than you could have on premises -- we can use things like MapReduce, the software framework introduced by Google, Yahoo, and Facebook, to support distributed computing.

So, in short, we in IT have been messing with this idea of distributed computing and data for the last 20 years but now we have access to resources that we didn't have in the past.
Dave Linthicum, CTO, Bick Group

 MapReduce takes a query from a mass of data and will distribute it to any number of servers and run the query in little pieces. It is like giving 20 people a book to read and then reassembling and reducing to get the results back. That addresses a couple of problems. A lot of my clients, for instance, are waiting three days to get a traditional query back. Sequential queries and relational databases are the problems. By contrast, the cloud is a highly scalable and elastic way of doing that so queries that took days can now be done in minutes.

Apache Hadoop is another thing that helps. It arose out of work by Yahoo, Facebook, and Google. They had a massive information processing problem and were using MapReduce. The inventor of Hadoop made a software framework distributed application that can work with thousands of nodes.

So, in short, we in IT have been messing with this idea of distributed computing and data for the last 20 years but now we have access to resources that we didn't have in the past -- and it has been driven by the real business needs of organizations like Yahoo, Google, and Facebook.

SearchSOA.com: How about the somewhat more conventional world of commercial Web data services, what can application developers and others do to more effectively harness these external resources for their internal needs?

David Linthicum: Well, there certainly are services out there that provide this -- say Xignite, which provides financial data on demand. Most cloud providers such as Amazon and the US government provide feeds and the value of [those feeds] is that you and your organization can leverage the information cost effectively. These services are typically RESTful Web services that are coupled into applications to serve a particular purpose. In this case, the purpose is providing information. A stock ticker example is very typical, but we can also leverage news services from another API and customer validation from another, all leveraged by the same composite application.

I would say a typical app of this type is risk analytics, where you would actually model a stock trade. Suppose you are trading millions in assets to try to get a better financial position. The information you need to do the risk analytics come from the web. You populate it through these APIs, which work through backend services, and it ends up providing you with, say, a 1-5 rating. Instead of having to maintain information out of Wall Street for a stock trading operation, you can now leverage third-party provider information and the API they are going to provide. This capability has been around for a long time, but people didn't really get into it until the cloud took off.

Another related trend is people moving from just having information as a service to cloud providing additional capabilities around that information. For example, Amazon provides some aspects of information-as-a-service, by providing some information using a Web services interface. So, why subscribe to another separate service when Amazon provides some and Google provides others? They are the aggregators now. In fact, they may end up buying up some of the older service providers.

SearchSOA.com: A few months ago you blogged enthusiastically about a new "consortium" that you were participating in. What is it and has it lived up to your expectations?

David Linthicum: That was something called the SOA Data Integration Architecture Community. It exists on LinkedIn and it provides a place for people to discuss the ins and outs of SOA. It is a place where people suggest, for example, different standards and ways to work with current data integration standards. It is a very loosely coupled group -- open and free -- but I think it is proving to be very valuable to people who want to create agile architectures such as SOA. Members come from many disciplines, including enterprise and IT architects, as well as IT managers.

This was first published in July 2010

There are Comments. Add yours.

 
TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.