Menu

Web Disservices: Microsoft's Misstep

September 24, 2003

Mark Pilgrim

Microsoft recently announced Microsoft.com Web Services, through which developers can get programmatic access to Microsoft.com services. Microsoft has big plans to expand the program to include querying MSDN, MS Technet, and MS Support sites. For now, however, the only service available is getting information on top downloads from Microsoft.com. But this is enough to get a taste of the architecture and for us to dig in and see how it might be improved.

Microsoft.com Web Services are based on SOAP. To call a method you need to post a SOAP message, with particular HTTP headers, to a specific URL on the server; the server responds with a SOAP envelope containing a SOAP body containing your answer. For example, the service defines a GetVersion method which takes no parameters and returns a string describing the version of Microsoft.com Web Services that the server is running. That transaction looks like this:

Host: ws.microsoft.com

POST /mscomservice/mscom.asmx HTTP/1.0

Content-type: text/xml

SOAPAction: "http://www.microsoft.com/GetTopDownloads"



<soap:Envelope

  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"

  xmlns:wsu="http://schemas.xmlsoap.org/ws/2002/07/utility"

  xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/07/secext">

  <soap:Header>

    <wsse:Security>

      <wsse:UsernameToken>

        <wsse:Username>DEVELOPERTOKEN</wsse:Username>

        <wsse:Password Type="wsse:PasswordDigest">PASSWORDDIGEST</wsse:Password>

        <wsse:Nonce>NONCE</wsse:Nonce>

        <wsu:Created>2003-09-08T05:52:36Z</wsu:Created>

        <wsu:Expires>2003-09-08T05:55:36Z</wsu:Expires>

      </wsse:UsernameToken>

    </wsse:Security>

  </soap:Header>

  <soap:Body>

    <GetTopDownloads xmlns="http://www.microsoft.com">

      <topType>Recent</topType>

      <topN>25</topN>

      <cultureID>en-US</cultureID>

    </GetTopDownloads>

  </soap:Body>

</soap:Envelope>

There are 3 important pieces of information here.

  1. The method name. This is stored in two places: in an HTTP header called SOAPAction, and as the name of the element within the <soap:Body>.

  2. The method arguments. These are stored as elements within the <soap:Body>: topType, topN, and cultureID. The acceptable values and ranges for these arguments are documented.

  3. The authentication information. Not just anyone can make calls to Microsoft.com Web Services. Well, actually, that's not true. Anyone can, but first you need to sign up at Microsoft.com for a developer token and a PIN. These, along with some other calculated information (documented here), go in the <soap:Header>.

The rest is always the same. The server and the URL to post to are always the same (the server dispatches on the SOAPAction header instead). The namespaces used in the SOAP envelope are always the same. It all looks complicated, but it just boils down to method name, arguments, and credentials.

Here's another example, this time to get a version string that describes the version of Microsoft.com Web Services that is running on the server:

Host: ws.microsoft.com

POST /mscomservice/mscom.asmx HTTP/1.0

Content-type: text/xml

SOAPAction: "http://www.microsoft.com/GetVersion"



<soap:Envelope

  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"

  xmlns:wsu="http://schemas.xmlsoap.org/ws/2002/07/utility"

  xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/07/secext">

  <soap:Header>

    <wsse:Security>

      <wsse:UsernameToken>

        <wsse:Username>DEVELOPERTOKEN</wsse:Username>

        <wsse:Password Type="wsse:PasswordDigest">PASSWORDDIGEST</wsse:Password>

        <wsse:Nonce>NONCE</wsse:Nonce>

        <wsu:Created>2003-09-08T05:52:36Z</wsu:Created>

        <wsu:Expires>2003-09-08T05:55:36Z</wsu:Expires>

      </wsse:UsernameToken>

    </wsse:Security>

  </soap:Header>

  <soap:Body>

    <GetVersion xmlns="http://www.microsoft.com">

    </GetVersion>

  </soap:Body>

</soap:Envelope>

Changes are highlighted: the SOAPAction and first child within <soap:Body> are now GetVersion; this method takes no parameters, so they are simply omitted.

Here's the response from the GetVersion request:

HTTP/1.1 200 OK

Content-Type: text/xml; charset=utf-8



<soap:Envelope

  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"

  xmlns:xsd="http://www.w3.org/2001/XMLSchema"

  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

  <soap:Body>

    <GetVersionResponse xmlns="http://www.microsoft.com">

      <GetVersionResult>Microsoft.Com Platform Services 1.0 Beta</GetVersionResult>

    </GetVersionResponse>

  </soap:Body>

</soap:Envelope>

Again, the namespaces and <soap:Envelope> are boilerplate; the real result is in the <GetVersionResult> element, a child of the <GetVersionResponse> element, which is in turn a child of <soap:Body>. In this case we've called a GetVersion method with no parameters, and the method has returned a single string: "Microsoft.Com Platform Services 1.0 Beta".

Whew.

The tools will save us

More Dive Into XML Columns

Identifying Atom

XML on the Web Has Failed

The Atom Link Model

Normalizing Syndicated Feed Content

Atom Authentication

I know what you're thinking. Either you're thinking, "that's a lot of boilerplate just to call a method and get a string back". Or you're thinking, "well, I don't care about what goes over the wire because I'll never see it. I'll just point my Web-Services-Enabled IDE to a WSDL file and call a method in a wrapper class that takes no arguments and returns a string." Both of these thoughts are true: SOAP-based web services are verbose, and there are tools that hide all of this messiness from you. For instance, Microsoft provides a WSDL file which is understood by savvy IDEs like Microsoft's own Visual Studio .NET. They also provide a Web Services Enhancements Service Pack for Visual Studio .NET which handles generating the required authentication elements from your developer token and PIN. And after spending thousands of dollars for development tools and updating all your service packs, you can indeed call a remote method with no parameters and get back a string, in about 5 lines of code, and you will never see any of this XML.

Of course, no other development environment that I know of (Python, Perl, or Java) has libraries that can generate the required authentication credentials, so it gets a little more complicated. And even once you manually generate the information, some of them (I tested Python specifically) make it difficult to add those credentials in the appropriate place in the <soap:Header> where Microsoft's server will look for it.

So I think this web services architecture is harder than it has to be for clients, unless you are blessed with perfect tools, in which case nothing is ever hard for you and you probably don't need to read web services articles like this. But let's look at this from Microsoft's point of view, from the server side. There are also a number of problems:

  • Every call uses HTTP POST. This seriously reduces the possibilities for any sort of caching; POSTs are never supposed to be cached by the server, the client, or by any intermediary proxies.

  • Every call POSTs to the same URL. This means that the web services producer cannot rely on standard web server logging to provide much useful information about usage. It would provide IP address and timestamp, but no information about the web services call itself. Another logging utility would need to be written that was application-specific (or at least SOAP-aware) in order to capture the SOAPAction header and extract the method arguments out of the<soap:Body>.

  • The server doesn't support gzip compression. This is actually independent of using POST or using a particular URL scheme. POST responses can be gzip-compressed, and SOAP messages actually compress very well. Microsoft just isn't doing it. It's a separate question whether your SOAP library knows enough about HTTP to send the appropriate Accept-encoding: header and decompress the response properly.

  • Microsoft is using an insecure authentication scheme. It does not send passwords in the clear, but it does require both the client and server to store plain text passwords somewhere. There are numerous authentication mechanisms that do not require this; even HTTP digest authentication allows both sides to store hashes of the password, instead of the password itself.

  • Also, the authentication is entirely one-way; the server can tell that the client knows the password, but the client can't tell that the server knows it. A malicious proxy could capture web services calls from the client and answer them itself, and the client would have no way of knowing that the results were not from Microsoft.com.

To sum up: Microsoft has implemented a set of standards-based web services in such a way that ignores most of the benefits of the Web and has made it as easy as possible with its own tools and as difficult as possible without them.

Is it possible to make web services easier? Is it possible to make them more like the Web?

An alternate approach

First, let me say that I have no problem with the actual data that Microsoft is returning from these web services calls. XML is a fine way of serializing complex data structures, and several of Microsoft's public web services methods return data structures much more complex than a single string. The WSDL file contains an XML schema that defines these data structures and datatypes and allows WSDL-aware tools to produce wrapper classes to translate these serializations into native data structures.

So +1 on XML, +1 on WSDL, +1 on XML Schemas. These are all good things, have sizable benefits, and are worth their cost in complexity. I am also in favor of HTTP, which is a fine way of requesting data and receiving responses, even XML responses. It's well understood, widely supported, and mature. +1 on HTTP.

But let's think about how we could provide the same set of services as Microsoft has offered but in such a way that we take advantage of the Web more directly. First, Microsoft is overloading a single URL, choosing to dispatch their methods on a custom HTTP header (SOAPAction). This method name could be put into the URL instead:

http://ws.microsoft.com/mscomservice/GetVersion

http://ws.microsoft.com/mscomservice/GetCultures

http://ws.microsoft.com/mscomservice/GetTopDownloads

And so forth. This would accomplish the same thing, but it would make logging more useful, since a standard IIS or Apache access log would contain the method name.

Second, those method arguments... None of these methods seem to take complex data structures; the most complex datatype they take is a GUID, which is simply a long string in a particular format. And as you can see from the sample requests, the server does not require declaring the datatype of each argument on each request; both the client and server are deemed to know them from the schema within the WSDL file. Each parameter is named and order doesn't matter. So, basically, we have a bunch of unordered, named key-value pairs that we're passing to a URL. So instead of

  <soap:Body>

    <GetTopDownloads xmlns="http://www.microsoft.com">

      <topType>Recent</topType>

      <topN>25</topN>

      <cultureID>en-US</cultureID>

    </GetTopDownloads>

  </soap:Body>

why not

http://ws.microsoft.com/mscomservice/GetTopDownloads?topType=Recent&topN=25&cultureID=en-US

Third, let's consider that insecure approach to authentication. It is susceptible to a number of attack types, and it requires both client and server to store sensitive information in plain text. It is not widely implemented in toolkits. There is also no programmatic way for a client to determine what authentication mechanism the server requires; it is documented in human-readable form, of course, but there is no way to write a generic client that speaks to this and to other web services without specific knowledge of each.

It appears to have been chosen solely as a way to show off a new standard which Microsoft had a hand in creating. I would replace it altogether HTTP digest authentication, which is more mature. Not only does digest authentication solve several security problems, it's more widely supported, and there is a standard, documented way for the server to announce that digest authentication is required (if a client tries to call a web services method without the proper credentials or without any credentials at all).

Given all this, what would our web service look like? Well, taking the GetVersion example above and making the necessary modifications, it would look like this:

Host: ws.microsoft.com
GET /mscomservice/GetVersion HTTP/1.0
Content-type: text/xml
Authorization: Digest Username="DEVELOPERTOKEN", realm="Microsoft.com web services", nonce="NONCE", uri="/mscomservice/GetVersion", qop="auth", nc="00000001", cnonce="CNONCE", response="RESPONSE", opaque="OPAQUE"

(The developer token is the same as the original example, and the other values are generated from the Microsoft-supplied PIN and other constants defined in RFC 2617.)

And the response would look like this:

HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8

<GetVersionResult xmlns="http://www.microsoft.com">Microsoft.Com Platform Services 1.0 Beta</GetVersionResult>

Note that this approach accomplishes exactly the same thing: we are calling a method with no parameters, getting a string back, and identifying ourselves appropriately to the server, just like Microsoft's SOAP service. The differences are where things go:

  1. The method name is now part of the URL instead of being in a custom HTTP header and in the <soap:Body>. This makes logging more useful.

  2. The method arguments are now query parameters in the URL instead of being stored as elements within the <soap:Body>.

  3. The authentication information is now transmitted in the standard Authorization HTTP header used by digest authentication.

There are further subtle benefits. If you look closely you'll see we're using GET instead of POST. This opens up all kinds of possibilities for controlled caching, both server-side and client-side. For example, if I call GetVersion the server could return an Etag or Last-Modified HTTP header along with the response. The next time I call GetVersion, I could pass along this etag, and if the data hasn't changed, the web services server could simply return an HTTP status code 304 "Not Modified" rather than returning the same response over and over. This is an ordinary and clever part of the Web.

Ah, but I know what you're thinking. You're thinking "well, I don't care about what goes over the wire because I'll never see it. I'll just point my Web-Services-Enabled IDE to a WSDL file and call a method in a wrapper class that takes no arguments and returns a string." Okay, you can still do that. WSDL supports HTTP GET bindings, so you could create a WSDL file that described this URL-and-query-parameters web service, just as easily as you can describe an everything-in-the-SOAP-envelope web service. Here is such a WSDL file. Try it in Microsoft's Visual Studio .NET; you'll find that it generates wrappers for the methods and classes with native data structures, which are virtually identical to the wrapper classes generated by Microsoft's original WSDL file. (In fact they're slightly simpler because there's slightly less cruft in the serialized XML responses. Other than that, they're identical.)

Microsoft wouldn't do this, but you should

Realistically, would Microsoft consider doing this? No, probably not. As you can see, the hypothetical version is much simpler over the wire, so there is some benefit from reduced bandwidth. But for developers using Microsoft tools (i.e. Microsoft's primary customers here), there is no other benefit since WSDL hides the complexities of either approach equally well. Everything is simple when you have perfect tools, and Microsoft makes a lot of money selling those tools. Microsoft has implemented its web services in as complex a manner as possible, and then it proceeds to sell developer tools that make calling those web services trivially easy.

However, you should consider the second approach, assuming you're trying to make money from your actual web services instead of from selling developer tools. The primary advantage of the second approach is that it is significantly easier to use when you don't have perfect tools -- in this case, that means non-Microsoft tools that haven't caught up to Microsoft's latest standards yet. Digest authentication is more widely supported than the UsernameToken scheme Microsoft is using; it's more secure and more extensible too. Query strings are easier to produce than SOAP bodies; even if you have a good SOAP library, they're still easier to debug. Developers win with reduced debugging costs, and everybody wins with reduced bandwidth costs.

Further reading: I've put together a live Microsoft.com Web Services gateway that acts as a proxy between Microsoft.com's SOAP-based web services and a hypothetical second approach similar to what I've described here. Live, clickable web services calls are provided; the gateway script takes the URL-and-query-parameters and translates them into the SOAP body that Microsoft's server requires, and sends it along, then takes the response and strips out all unnecessary SOAP envelope and returns just the XML the client needs. Sample client code is available to call the gateway from both C# and Python, along with a WSDL file, and the gateway script itself.