Catching Up with the Atom Publishing Protocol

December 7, 2005

Joe Gregorio

The Atom Syndication Format is now also known as RFC 4287. Atom is an internet standard in the same way that little things like SMTP and HTTP -- also known as "email" and "the Web," respectively -- are internet standards. (If you're just now coming to terms with the Atom Syndication Format, check out Uche Ogbuji's Agile Web XML.com column this week for some practical advice and running code.)

Progress on the Atom Publishing Protocol (APP) has lagged a bit, in part because it relies so heavily on the format specification. Now that the format is done, work has progressed rapidly on the protocol.

The Basics

Here is the general outline of how the protocol works; in particular, I'll point out design alternatives that the Working Group (WG) has considered and rejected and some of the reasoning behind those decisions. I'll also point out the lingering questions and open problems. Please be advised that this is just one person's viewpoint and that everything and anything can change before the WG puts its stamp of approval on a final version of the APP.

Here is an Atom Entry:

<entry xmlns="http://www.w3.org/2005/Atom">

  <title>Atom-Powered Robots Run Amok</title>

  <link href="http://example.org/2003/12/13/atom03"/>

  <author>

    <name>John Doe</name>

  </author>

  <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>

  <updated>2003-12-13T18:30:02Z</updated>

  <summary>Some text.</summary>

  <content type="xhtml">

    <div xmlns="http://www.w3.org/1999/xhtml">

      <p><i>[Update: The Atom draft is finished.]</i></p>

    </div>

  </content>

</entry>

There are two things to notice about this Atom Entry. The first is that it is a completely valid Atom document. Yes, that's right, an Atom Entry is a valid Atom document in its own right. The format actually defines two kinds of documents: one with a document element of atom:feed, the other with a document element of atom:entry.

The second thing to notice about this document is that it contains almost all of the information you would need to describe a weblog entry. You've got a title, author, summary, and the content. An Atom Entry is a good representation of a weblog entry, which shouldn't be terribly surprising, but it helps out a great deal when we go to design a RESTful web service for weblog editing. If you go back and read How to Create a REST Protocol, you see that we want to answer the four questions:

1. What are the resources?
2. What are the representations?
3. What are the methods supported?
4. What status codes can be returned?

The two major resources in this problem space are members and collections. There is a single member resource for each entry on your weblog. That member resource has an Atom Entry as its representation, and changing that Atom Entry representation is how the corresponding weblog entry is changed. The collection resource is a list of URIs of member resources. Put in our standard tabular form we get:

Back to the Future

This isn't very different from the protocol described in draft-gregorio-09, which is what the WG started with almost two years ago. The language of gregorio-09 is a bit rough, and certainly not proper specification text, but the basic ideas are there. What was then called the EntryURI is now called a member resource. In the intervening years the FeedURI and PostURI have merged into a single collection resource. The basic editing model hasn't changed: we use the four basic methods of HTTP (GET, PUT, POST, and DELETE) to shuffle Atom Entries around.

One of the big differences from gregorio-09 is the absence of SOAP. A SOAP binding may appear some day, but it was considered too much to have that in the core protocol. There were also some proposals to use WebDAV. While those proposals were not accepted, they did have some influence and the WG adopted the term collection from the WebDAV spec. While the APP is not based on WebDAV, there seems to be general agreement that we should peacefully coexist and that a WebDAV collection can also be an APP collection.

You've Got Issues

Now it may look pretty simple but there are quite a few issues that have to be dealt with.

Who Controls the Elements in the Entry?

When creating or editing an entry using an Atom Entry, there is the basic question of which side controls which element in the Atom Entry. For example, the client should probably be in control of the atom:title. On the other hand, two elements cause some concern when considered in the context of the protocol. The first is atom:id. As the spec says,

The atom:id element conveys a permanent, universally unique identifier for an entry or feed.

Right. So what does that mean when creating an entry? Does that mean the client has to create a globally unique atom:id just to POST a new entry? Should the server overwrite that atom:id just to guarantee that the atom:id is unique? Remember the use case for this restriction: entries that were duplicated in multiple feeds wouldn't be displayed more than once in your aggregator. How do I transfer my entries from one blogging system to another and keep the atom:id the same?

Consider the following scenarios:

We have collection A and collection B. What should happen if I copy an entry "a" from collection A to collection B? Should that entry be given a new universally unique atom:id or should it keep the old one? It would be nice if we kept the same atom:id because that would avoid users seeing duplicate entries in their aggregator.

If copying "a" from A to B was done because we we're migrating content from one system to another, or adding content into a different category, then keeping the atom:id is the desired behavior.

On the other hand, what if I start editing entry "a" in collection B while keeping the original in collection A unchanged? What if I changed it significantly? Then I would have two entries with the same atom:id but diverging content. That's not very consonant with the idea of a "permanent, universally unique identifier for an entry or feed."

There are also issues with atom:updated.

The atom:updated element is a Date construct indicating the most recent instant in time when an entry or feed was modified in a way the publisher considers significant.

Let's repeat that last bit for emphasis -- "was modified in a way the publisher considers significant."

We're designing a protocol here, what the heck are we supposed to do with that? Is the client or server considered "the publisher"? And how exactly do they determine "significant"?

Introspection

Remember I said at the beginning that collections contain members that have representations as Atom Entries? I lied. There are actually two types of collections, one that contains Atom Entries and the other called a media collection that contains any type of media: images, PDFs, etc. The mechanics of dealing with media collections are the same as entry collections: POST a representation to the collection to create a new member resource; GET, PUT, and DELETE on the member resources to edit. The only difference is that the representation of a member doesn't have to be an Atom Entry.

A weblog is more than just a single list of entries. There may be a media collection, or you could have a link blog, a book list, and a music list. How does your blogging client discover all the collections associated with your blog? And what if you have multiple blogs associated with a single account, like on some third-party weblog hosting services?

Here we have the battle of the formats. There have been proposals to use all of the following:

Custom Format

Actually many different custom XML formats have been proposed. Here is a sample of the format defined in the current draft.

<?xml version="1.0" encoding='utf-8'?>

<service xmlns="http://purl.org/atom/app#">

  <workspace title="Main Site" > 

    <collection title="My Blog Entries" 

      href="http://example.org/reilly/main" >

      <member-type>entry</member-type>

      ...

    </collection>

    <collection title="Pictures" 

      href="http://example.org/reilly/pic" >

      <member-type>media</member-type>

      ...

    </collection>

  </workspace>

  <workspace title="Side Bar Blog">

    ... 

  </workspace>

</service>

XOXO

One of the older microformats, XOXO has been proposed as a format for listing the collections and workspaces. Here is a fragment from the XOXO microformat draft specification:

<ol class='xoxo'>

  <li>item 1

    <dl>

      <dt>description</dt>

        <dd>This item represents the main point we're trying to make.</dd>

    </dl>

    <ol>

      <li>subpoint a</li>

      <li>subpoint b</li>

    </ol>

  </li>

You can see how it would be easy to describe groupings of collections and workspaces using such a format.

Atom Syndication Format

Atom feeds have been proposed in several forms, including one that uses one entry per collection, and others that use link elements to point to collections, workspaces, and other Atom feeds.

Enumerating

In the table above I said that doing a GET on a collection returns an Atom feed with what might be a subset of all the entries in the collection. So how do I go about listing the rest of the entries? Good question; apparently a painful question too. There have been proposals that have spanned almost every possible combination and permutation.

link/@rel="next". Since we are using an Atom feed to enumerate the entries in a collection we can use the link element with a rel-attribute value of "next" to point to the next n entries. Do that recursively so that you end up with a chain of Atom feeds all linked together going back in time. This has several advantages; the server can always give you a reasonable amount of data, the client can use ETags on the first feed in the chain to see if there have been any updates. Of all the solutions, this is the only one that can be served statically. Finally, this is how enumerating entries worked in gregorio-09. Currently it appears that this method of enumerating members will end up in the core.

The disadvantages are that it gives the client very little control and the client may have to do many GETs following the chain of feeds if it wants to enumerate every entry in a collection.

The remaining proposals used either indexes or dates or some combination. The index approaches, where the collection is treated like a giant array and you pass in values to select a subset, have some drawbacks. The first is that you really need the total size of the collection, but realize that that number may change between the time you read it and the time you actually use it. The second drawback is that since the number of entries returned is determined by the client, it is possible for the client to accidentally, or maliciously, request a huge number of entries. Not really a problem for a weblog which would have its APP service protected by authentication, but think about an APP-enabled wiki which may not be protected by authentication.

URI Templating. URI templating is part of the current draft but I believe it will be departing soon. This is a system where a string is given with brace-delimited keywords embedded in it. Those braces and keywords get replaced with values and that constructs a URI. For example, if the keyword was index and it accepted a range of index values separated by a dash, then this URI template:

http://example.org/find/{index}

could be filled in like so:

http://example.org/find/0-14

to request the first 15 entries in the collection. This is a mechanism I described and provided code for in a previous column, Constructing or Traversing URIs.

A variety of proposals used different sets of keywords that did index- or date-based queries. All of these have the disadvantages I stated previously for index- and data-based queries. In addition this approach has the disadvantage that the client has to get the URI template from somewhere before it starts working with a collection.

SEARCH. Yes, someone actually proposed extending HTTP with a new method for searching. Again, all the disadvantages of a date- or index-based approach, with the additional obstacles presented by trying to create a new HTTP.

GET with Range. HTTP already supports the idea of a partial GET where you use a Range header, but that is only specified in terms of bytes. There was a proposal for doing this using a new unit for Range that was in updated. This was actually in the specification for a while back in draft-ietf-atompub-protocol-03.

This has all the disadvantages of a date-based query with the added problem of its opacity. In order to find out if a collection supports the new Range unit you have to do a GET on the collection and look in the headers for an Accept-Ranges header with the right value.

"Just" Use Query Parameters. There was even a proposal to "just" use query parameters. While simple to implement and specify, this has a pretty major flaw in that it prevents the server from using query parameters for anything else. For example, if you had multiple collections that were all accessed through the same CGI application, and you passed in the collection you wanted via query parameter, that would have been impossible with this approach. The more complex URI templates were invented in part to solve this problem but their complexity was in part responsible for them being rejected.

Summary

While the WG has certainly worked with a great many proposals and alternatives I believe the lessons are pretty clear: when in doubt, go for the simplest thing that could possibly work and trust in the utility of hypertext. A simple format sprinkled with links contains a surprising amount of power.