Testing and certification of Web services

By CBDi Forum 11 Jul 2002 | CBDi Forum Limited, special to SearchWebServices

Digg This!     StumbleUpon     Del.icio.us

Testing and Certification

Is Trust the Achilles Heel of Web services?
Web services are not yet widely used because of security concerns. But there's an even bigger roadblock waiting just down the road -- it's called trust. The big issue is "Will the service work correctly every time when I need it?" As yet few are thinking about the issues of testing and certification. We suggest that testing and certification of Web services is not business as usual and that new solutions are needed to provide assurance that services can really be trusted. We identify some of the issues and challenges.

Given the less than perfect reliability of most systems, we might conclude that testing is the Cinderella of most industry and enterprise organizations. So what of Web services? Do we need to reinvent testing approaches and technologies once again? Or is it "business as usual" for developers and the test tool industry? After all, a Web service is just a URL, and we have already adapted testing to encompass Web sites, so are Web services any different? Similarly, we have experience of testing components, and a Web service is just another interface isn't it?

In theory we should be able to learn from our experience with black box components. Unfortunately, we didn't seem to learn much from the use of components. A service is in essence the same concept, just wrapped in more protocols that provide better understanding of how to use it. But time and time again, feedback from our members was that when they changed a single component they still retested the entire system. Even though the change wasn't supposed to alter the interface and the service it offered, or affect other components, they never quite trusted themselves enough. So best practice, motivated as much by the fact that they could (because everything was accessible and under their control), was simply to retest everything.

But with Web services you won't have visibility across the entire system. Web services enforce separation and the service truly becomes a black box and the consumer is now dependent only on the behaviour of the service to test their part of the system. The consumer will not even know the provider has changed a component of the implementation if the delivered service remains unchanged. The provider has a similar challenge, as they can only observe the behaviour of the consuming application via the messages it sends. So we are going to need huge levels of trust here, or more likely have to compensate by upgrading testing strategies to cope.

Right now many organizations are using Web services internally, and just like with black box components, they might be taking advantage of the privileged accesses to test the entire system implementations together. However this would be a major mistake because it would nullify the (flexibility) advantages that separation gives you.

In our report last month on Implementing Context Driven Services, we make the point that "With context-aware services, the designer no longer tries to predict and control everything that the user does, but creates a system that can respond dynamically to what the user is trying to do." To achieve this we have to move beyond the traditional software development mindset and enforce separation.

So there is a profound change in the Web services environment that has a major impact on testing and certification, because the tester does not have visibility and access to the entire system. Many organizations will have started to recognize some of these issues with EAI projects, but these are simplistic in comparison to more sophisticated Web service architectures.

Here is a quick look at some of the challenges that Web services introduce into the testing environment:

System Test
System testing is difficult if it can only be done from one perspective, consumer or provider. From the consumer's perspective, it might involve multiple providers. Even a single provider might be a front for multiple other service providers. This requires convergence of "system" and "integration" testing, with the potential for additional process complexity, where each party may be in a lower state of readiness than conventional integration testing.

Micro vs. Macro services
Do I certify the macro service or the micro service? As a consumer do I have the visibility of what micro services are making up the macro service at any one point in time. Is re-certification required if the component parts change while the delivered service remains unchanged.

Context based services
Web services will offer different behaviours based on context such as location, consumer, time, etc. New behaviours can be introduced with new business rules on a dynamic basis. How do provider and consumer ensure that the new behaviours are performing according to agreement.

Interface vs. implementation
As discussed, the implementation is not available to the consumer. Service level agreement testing is going to become a major area of focus.

Granularity
Coarse-grained services will be more common with Web services. It might be difficult in testing to identify just where exactly in the service the problem is.

Service Level Agreement
Testing non-functional requirements is much more important with external Web services. Is the response time, or throughput promised achievable? How does a consumer test the scalability of the providers live service without harming it?

Precise specification
What are you testing against? Currently existing protocol standards do not cover the complete behaviour of a Web service. We haven't yet fully resolved this with components either. Without visibility of the implementation, the only knowledge you have is the specification of the service.

Live servicess
As a consumer, how do you test a live service? You cannot submit an order to see if it works, as the order will be processed. Providers must therefore make some provision in the service that permits it to be tested without actually activating the business process it supports. And it can't just be a dummy service, as that proves nothing. It is the real service you need to test. Providers will need to think very carefully about what sort of testing harness they provide to consumers.

Continuous testing
You cannot test a service once and presume it will thereafter always work. The implementation can change without your knowledge. Perhaps consumers should fire off test packets at regular intervals to check that behaviour is consistent.

Which service?
Interfaces, and Service, are meant to be immutable. But services will change constantly. I can certify a service, but which service is being used? Further services may be substituted at runtime, again dependent on context, such as other collaborating services involved in the higher-level business transaction. Is this the service that I certified/used yesterday?

Service hell
In working with components we have learnt that there are many indirect dependencies that are not defined by the interface. Many developers will be familiar with DLL hell. As yet few have thought through the possibilities of service hell.

Security and authentication
External Web services will place additional responsibility to test that security and authentication mechanisms are working.

Certification
The pressure on consumers to test could be reduced if there were some certification process on providers. Certification was much discussed with components. Yet, I am still not aware of any large-scale independent certification programme for components, other than those run by publisher/distributors. Will anyone be willing to certify Web services? And what exactly are they certifying?

Though we are still in the early adopter phase for Web services the issues above should be relatively obvious. What is deeply worrying is that there is very little attention being given to these issues. Further, very similar issues existed with components that remain largely unresolved. We are however optimistic that these issues will now be addressed because Web services will force the issue. But the entire industry needs to recognize that this is NOT business as usual, because when you don't have access to the implementation you have to focus testing on interfaces and external dependencies.

Let us know what you think on this important topic. Will these issues get resolved? Or are they the Achilles Heel of Web services? Please send your feedback to lawrence.wilkes@cbdiforum.com and or david.sprott@cbdiforum.com.

Copyright CBDi Forum Limited 2002. The CBDi Forum is an analysis firm and think tank, providing insight on component and web service technologies, processes and practices for the software industry and its customers. To register for the weekly newswire click here.

For More Information:

• Looking for free research? Browse our comprehensive White Papers section by topic, author or keyword.
• Are you tired of technospeak? The Web Services Advisor column uses plain talk and avoids the hype.
• For insightful opinion and commentary from today's industry leaders, read our Guest Commentary columns.
• Hey Codeheads! Start benefiting from these time-saving XML Developer Tips and .NET Developer Tips.

• Visit our huge Best Web Links for Web Services collection for the freshest editor-selected resources.
• Visit Ask the Experts for answers to your Web services, SOAP, WSDL, XML, .NET, Java and EAI questions.
• Discuss this article, voice your opinion or talk with your peers in the SearchWebServices Discussion Forums.

Digg This!     StumbleUpon     Del.icio.us

RELATED RESOURCES 2020software.com, trial software downloads for accounting software, ERP software, CRM software and business software systems Search Bitpipe.com for the latest white papers and business webcasts Whatis.com, the online computer dictionary