XML Conformance Update

May 10, 2000

David Brownell

Overview

This article is an update to my earlier articles from 1999 that tested the XML 1.0 conformance of parsers. Since then, there has been notable development of the parsers, and updates to the XML 1.0 specification itself. With respect to the dozen Java parsers reviewed last year, there have been a number of interesting changes, both in their conformance and in their open-source status. Microsoft's MSXML parser has also received a significant update, with the release of "technology previews."

Conformance at OASIS and at W3C

Unfortunately the OASIS XML Conformance committee, responsible for the original XML 1.0 test suite, has been rather quiet since it released its first draft last summer. The tests have not been updated in any way, although several issues have arisen since that first draft was released. In fact, there was a recent (April 4) e-mail to the OASIS xmlconf list saying that the conformance committee needed to be "re-initialized," so it's not clear when the necessary updates to its categorizations may be provided.

On another front, the W3C has at last begun to address some of the specification problems that were found by parser writers and by conformance and interoperability testing. In January W3C updated the XML 1.0 specification by publishing over forty errata that had queued up over the last year, since the original seventeen errata published in early 1999. In subsequent months additional errata have been published. I'm surely not the only one who wants to see just one revised specification, with change bars, instead of such a huge list of errata! The W3C has promised to deliver a "corrected version of XML 1.0," according to their public activity page. It is important for that to be delivered soon.

There appears to be a pause in the OASIS work on XML infrastructure conformance. The ball is temporarily in the W3C's court to address bugs in the XML 1.0 specification that have been reported by its implementors and users. I'm not sure there's enough dialogue in that process: my personal preference would be for the W3C to adopt the IETF approach, and insist on interoperability and conformance testing processes with positive results (and multiple full implementations) before making recommendations.

Updating the Test Database

The next logical step is to ensure that the OASIS test suite captures all those changes to the XML specification resulting from the W3C XML 1.0 errata. Since OASIS hadn't published any such work, and since a year's worth of W3C changes was published in January, I prepared a private update and circulated it in early February (including to the OASIS chair). This updated version of the OASIS 1.0 test suite is the one used in this article. I'm not aware that any of the newer errata (mostly editorial) affect these tests.

My February update (download it here, 4,028,333 bytes) addressed most of the issues I knew about in the original OASIS suite (relative URIs for test documents broken, and a handful of miscategorized test cases). It also addressed many issues coming from the errata W3C published in January -- at least the ones that didn't involve new test cases. See the README in the update for the open issues known to me at that time; only perhaps one or two percent of the tests seem likely to be problematic.

I'll repeat my plea from the earlier articles: If you have issues with these test cases, address them to OASIS and/or W3C. Ideally, do it so that the issue and resolution are matters of public record, without closed-door restrictions on who knows about such issues. (Note that discussions on XML-DEV don't always seem to be enough to resolve such issues in the W3C context.) Open standards need fully transparent processes, and responsive hosting organizations.

XML Parser Testing

This section presents the results of testing with the revised test suits. Some of the parsers examined last year have been omitted from these tests: the field has been slightly narrowed by these removals, with the least conformant processors from last year not getting a second look. Omissions include:

• Lark, roughly as conformant as Oracle's non-validating parser, is no longer being maintained.
• Datachannel's parser has been desupported.
• Microsoft's XML Parser for Java (in their Java virtual machine) has not been bugfixed either.
• Although the Silfide SXP parser did get an update, it was a minor one (version 0.88 to 0.89) that didn't particularly improve its test results.

The software used to do the testing was the latest version of the harness used in 1999. For SAX/SAX2 Java parsers, only a few bug fixes were needed to the original driver, and a conversion to use the new SAX2beta API conversion was made. For Microsoft's MSXML3 parser, the ECMAScript test harness from the original article was used.

Before we proceed to the test results, I'll sneak a word in about some additional testing I've learned about. Richard Tobin has done some testing of the RXP processor, making some results available. I'd like to see more parsers test and document their conformance, and applaud Richard's work with RXP!

Summaries for each parser tested are presented below. Click on the test scores to find the complete output from the test program. I've also included some notes on the changes undergone by each parser since the previous testing.

Ælfred 2

This new version of Ælfred is not from its original vendors. Microstar, the originators of Ælfred, have been acquired, and neglected to maintain the parser. Ælfred 2 is now part of the SAX2 XML Utilities, created by myself. The package also includes a layer for performing validation on top of SAX2. DOM Level 2 support is unbundled.

Why have I resurrected Ælfred? Last year I needed an open source Java XML parser that I could redistribute and realistically bugfix. I ended up taking a parser that seemed orphaned: Ælfred. It became the first publicly available parser to fully support all the new functionality in SAX2alpha and SAX2beta. It's had conformance improvements (as you see here) and yet I think it has preserved the original "small, simple, fast" approach, including trading off some conformance in favor of simplicity. (That tradeoff didn't need to be large.)

The most technically interesting feature of this parser comes from the fact that, since SAX2 exposes additional elements of the "XML Infoset" through new callbacks, most validity constraints can be tested by a cleanly layered module. The "validating Ælfred" here uses a validation layer with the non-validating parser. That layer works just as well with other SAX2 parsers. It can also be used with other software that produces SAX2 events, even doing validation of content as it's dynamically generated!

Curiously, I noticed that many of the (few) validity constraints that can't be checked using such a SAX2 layer were also ones that XML parsers often got wrong, or consciously chose not to address. These relate to lexical constraints, both for parameter entities and for some of the standalone document rules. This is perhaps another argument to remove those constraints, or have an XML errata turn them into well-formedness constraints to bless the way XP and MSXMLL handle them.

Microsoft MSXML3 (March 2000 Technology Preview)

Microsoft has released a new MSXML3 SDK as a "Technology Preview." The focus is on support for XSLT and XPath, and a handful of parser bugfixes seem to have been included as well. DOM Level 2 support was not mentioned.

This is the only parser addressed in this article that isn't a standard SAX or SAX2 Java parser.

Microsoft has posted an interesting bug list, showing some open XML (and XSLT, XPath, ...) bugs. I applaud this move, which not all companies would do. Since consumers need ways to keep the companies honest, I hope that this sort of transparency becomes much more common. (If markets are conversations, as the Cluetrain folk tell us, the need for bugfixes is an inevitable topic.)

However, reading the bug list, you wouldn't think that any parser conformance bugs had ever been reported! There were no bugs listed (open or closed) for the legal documents this parser rejects (the "legal unicode chars not allowed" is marked as fixed, but it demonstrably wasn't ... even given the weak test coverage of such characters found in the OASIS suite) or illegal documents it accepts ("permits whitespace before XML declaration," "accepts illegal unicode chars," or various validation bugs). It's clear that this bug list has a few problems!

The direct end-user consequence of such bugs is that "Microsoft XML" is demonstrably different from official XML. I've had the unpleasant experience of looking at the XHTML parts of a site where the webmaster had validated using MSXML, and then found that every MSXML-validated page showed a fatal error when used with a conformant XML parser. The interoperability lessons are clear: bugs like those hurt only people using non-Microsoft, standards-conformant, software.

Oracle XML Parser for Java v2.0.2.7

Although Oracle's distribution includes an XSLT processor, it does not claim conformance with the final recommendation. Neither SAX2 nor DOM Level 2 support is provided, and this is not an open source package.

The test results show that this release hasn't addressed conformance very much at all.