Pull Parsing in C# and Java
May 22, 2002
In my first article in
this series, I wrote about porting a SAX application called RSSReader to the new Microsoft
.NET Framework XmlReader. After publication, I received a message from Chris
Lovett of Microsoft suggesting I revisit the subject. As he said, while the code I
presented
works, my approach was not optimal for the .NET framework; I was still thinking in
terms of
SAX event driven state machinery. A much easier way to approach this problem is to
take
advantage of the fact that XmlReader does not make you think this way; and,
thus, to write a recursive descent RSS transformer as outlined below.
Based on Chris' suggestions, I've also made some other changes, including changing
the
output mechanism to use the XmlTextWriter, which will take care of generating
well formed XHTML on the output side.
And following all that, in a reversal of our usual process, I'll port this code back to Java.
Here then, without further ado, is the new RSSReader, optimized for C#. I've given the entire listing here, follows by an explanation.
using System;
using System.IO;
using System.Xml;
using System.Net;
public class RSSReader {
public static void Main(string [] args) {
// create an instance of RSSReader
RSSReader rssreader = new RSSReader();
try {
string url = args[0];
XmlTextWriter writer = new XmlTextWriter(Console.Out);
writer.Formatting = Formatting.Indented;
HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(url);
WebResponse resp = wr.GetResponse();
Stream stream = resp.GetResponseStream();
XmlTextReader reader = new XmlTextReader(stream);
reader.XmlResolver = null; // ignore the DTD
reader.WhitespaceHandling = WhitespaceHandling.None;
rssreader.RSSToHtml(reader, writer);
} catch (XmlException e) {
Console.WriteLine(e.Message);
}
}
public void RSSToHtml(XmlReader reader, XmlWriter writer) {
reader.MoveToContent();
if (reader.Name == "rss") {
writer.WriteStartElement("html");
while (reader.Read() &&
reader.NodeType != XmlNodeType.EndElement) {
switch (reader.LocalName) {
case "channel":
ChannelToHtml(reader, writer);
break;
case "item":
ItemToHtml(reader, writer);
break;
default: // ignore image and textinput.
break;
}
}
writer.WriteEndElement();
} else {
// not an RSS document!
}
}
void ChannelToHtml(XmlReader reader, XmlWriter writer) {
writer.WriteStartElement("head");
// scan header elements and pick out the title.
reader.Read();
while (reader.Name != "item" &&
reader.NodeType != XmlNodeType.EndElement) {
if (reader.Name == "title") {
writer.WriteNode(reader, true); // copy node to output.
} else {
reader.Skip();
}
}
writer.WriteEndElement();
writer.WriteStartElement("body");
// transform the items.
while (reader.NodeType != XmlNodeType.EndElement) {
if (reader.Name == "item") {
ItemToHtml(reader, writer);
}
if (!reader.Read())
break;
}
writer.WriteEndElement();
}
void ItemToHtml(XmlReader reader, XmlWriter writer) {
writer.WriteStartElement("p");
string title = null, link = null, description = null;
while (reader.Read() &&
reader.NodeType != XmlNodeType.EndElement) {
switch (reader.Name) {
case "title":
title = reader.ReadString();
break;
case "link":
link = reader.ReadString();
break;
case "description":
description = reader.ReadString();
break;
}
}
writer.WriteStartElement("a");
writer.WriteAttributeString("href", link);
writer.WriteString(title);
writer.WriteEndElement();
writer.WriteStartElement("br");
writer.WriteEndElement();
writer.WriteString(description);
writer.WriteEndElement(); // end the "p" element
}
}
Explaining the Code
The Main entry point to the new RSSReader uses the System.Net classes directly
to setup a WebRequest. You also see the XmlTextWriter being
constructed, turning on indenting so we get a nice readable output. Then the
XmlReader and XmlWriter become arguments to a recursive descent
RSS parser; the top level method is called RSSToHtml().
The top level RSSToHtml() method first checks that we really have an RSS file,
by checking the root element name. MoveToContent() is a convenient way of
skipping the XML prolog and going right to the top level element in the document.
If the XML
document used namespaces, then we'd also want to match on the NamespaceUri
property; however, this particular XML document doesn't use namespaces. If we find
an
<rss> element, then we read the contents, calling
ChannelToHtml() when we find a <channel> element and
calling ItemToHtml() when we find an <item> element. Any
other element is skipped. This is all wrapped in the XmlWriter call to write
the root level <html> output element.
The ChannelToHtml() method does two things: it writes out the HTML head
element containing a <title> element, then it writes out the HTML body.
Notice here we can simply use the XmlWriter.WriteNode() method which copies the
<title> element from the input reader to the output, since an HTML
<title> is exactly the same as an RSS one. The HTML head element
terminates when we reach the first child <item> element or the
</channel>
EndElement token. In the HTML body we look for <item> elements
and call ItemToHtml().
The ItemToHtml() method writes out an HTML <p> tag, then reads
the <title>, <link> and <description>
elements out of the input. These input tags could arrive in any order, which is why
we have
to read them all before we can write the output. Once we have them we can write the
<a> tag, with <href attribute equal to the
<link> element, and content equal to the <title>, followed
by an empty <br> element and the description.
All in all, it seems like a much simpler way to deal with converting RSS to HTML. .NET's built-in XML parser is pretty neat.
Java Pull Parsers
But pull parsers are not unique to the .NET world. The Java Community Process is currently working on a standard called StAX, the Streaming API for XML. This nascent API is, in turn, based upon several vendors' pull parser implementations, notably Apache's Xerces XNI, BEA's XML Stream API, XML Pull Parser 2, PullDOM (for Python), and, yes, Microsoft's XmlReader.
So how would we implement this same program in yet another pull parser, the Common API for XML Pull Parsing, or XPP? Let's take a look.
package com.xml;
import java.io.*;
import java.net.*;
import java.util.*;
import com.alexandriasc.xml.XMLWriter;
import org.xmlpull.v1.*;
public class RSSReader {
public static void main(String [] args) {
// create an instance of RSSReader
RSSReader rssreader = new RSSReader();
XMLWriter writer = null;
try {
String url = args[0];
writer = new XMLWriter(new OutputStreamWriter(System.out),false);
XmlPullParserFactory factory = XmlPullParserFactory.newInstance();
XmlPullParser parser = factory.newPullParser();
InputStreamReader stream = new InputStreamReader(
new URL(url).openStream());
parser.setInput(stream);
parser.setFeature(XmlPullParser.FEATURE_PROCESS_DOCDECL,false);
rssreader.RSSToHtml(parser, writer);
} catch (Exception e) {
e.printStackTrace(System.err);
} finally {
try {
writer.flush();
} catch (IOException io) {
io.printStackTrace(System.err);
}
}
}
public void RSSToHtml(XmlPullParser parser, XMLWriter writer)
throws IOException, XmlPullParserException {
// equivalent to XmlReader.MoveToContent()
while (parser.next() != XmlPullParser.START_TAG
&& !parser.getName().equals("rss")) {
}
if (parser.getName().equals("rss")) {
writer.beginElement("html");
do {
parser.next();
if (parser.getEventType() == XmlPullParser.START_TAG
&& parser.getName().equals("channel")) {
ChannelToHtml(parser, writer);
} else if (parser.getEventType() == XmlPullParser.START_TAG
&& parser.getName().equals("item")) {
ItemToHtml(parser, writer);
}
} while (parser.getEventType() != XmlPullParser.END_DOCUMENT);
writer.endElement();
} else {
// not an RSS document!
}
}
void ChannelToHtml(XmlPullParser parser, XMLWriter writer)
throws IOException, XmlPullParserException {
writer.beginElement("head");
// scan header elements and pick out the title.
while (!(parser.next() == XmlPullParser.END_TAG
&& parser.getName().equals("channel"))) {
if (parser.getEventType() == XmlPullParser.START_TAG) {
do {
if (parser.getEventType() == XmlPullParser.START_TAG
&& parser.getName().equals("title")) {
while (parser.next() != XmlPullParser.END_TAG) {
if (parser.getEventType() == XmlPullParser.TEXT) {
writer.writeElement("title",null,parser.getText());
break;
}
}
break;
}
} while (parser.next() != XmlPullParser.END_TAG);
break;
}
}
writer.endElement();
writer.beginElement("body");
// transform the items.
do {
if (parser.getEventType() == XmlPullParser.START_TAG
&& parser.getName().equals("item")) {
ItemToHtml(parser, writer);
}
parser.next();
} while (parser.getEventType() != XmlPullParser.END_DOCUMENT);
writer.endElement();
}
void ItemToHtml(XmlPullParser parser, XMLWriter writer)
throws IOException, XmlPullParserException {
writer.beginElement("p");
String title = null, link = null, description = null;
while (parser.next() != XmlPullParser.END_DOCUMENT
&& parser.getEventType() != XmlPullParser.END_TAG) {
if (parser.getEventType() == XmlPullParser.START_TAG
&& parser.getName().equals("title")) {
if (parser.next() == XmlPullParser.TEXT)
title = parser.readText();
} else if (parser.getEventType() == XmlPullParser.START_TAG
&& parser.getName().equals("link")) {
if (parser.next() == XmlPullParser.TEXT)
link = parser.readText();
} else if (parser.getEventType() == XmlPullParser.START_TAG
&& parser.getName().equals("description")) {
if (parser.next() == XmlPullParser.TEXT)
description = parser.readText();
}
}
HashMap attributes = new HashMap(1);
attributes.put("href", link);
writer.beginElement("a",attributes);
writer.write(title);
writer.endElement();
writer.writeEmptyElement("br");
writer.write(description);
writer.endElement(); // end the "p" element
}
}
Most of our port was the reverse of our previous ports; for example,
changing Console.Out to System.out, making method names start with
lowercase letters, adding explicit throws clauses. The real meat of this port
is in two areas.
The Parser
First, we're using XmlPullParser as a rough equivalent of
XmlTextReader. One difference is that while we are able to instantiate an
XmlTextReader directly in C# (remember, Microsoft is a one-stop shop), we
have to use the Java XmlPullParserFactory to get a concrete implementation of
the XmlPullParser interface. This should be a familiar exercise for anyone
who's used JAXP or, for that matter, JDBC.
Once we have the parser, most of the method name equivalencies are obvious. Remember
that
in C# the == operator works just fine for strings, but in Java you
must use the .equals() method; otherwise you'll be comparing object
references rather than their values, not at all what we want to do. Also, you can't
use a String as the expression in a switch...case statement in
Java, so we've turned those into an if...else structure.
Another difference between the .NET XmlReader and the Java
XmlPullParser has to do with the way in which events are pulled out of the
XMLdocument. In the former, the ReadString() method will return all the text
for the current element; while in the latter, next() must explicitly be called
to position the parser at the text node before calling getText() or
readText() to read the text.
This may be a minor difference, but it tends to make our port a little more difficult.
To
better handle this requirement, I've changed several while loops into
do...while loops. This, unfortunately, makes it less than a simple port; the
logic has changed, but not considerably.
The Writer
Second, there is no XmlTextWriter in Java, so we're using Alexandria Software
Consulting's XmlHelper
package, which contains a class called XMLWriter. Besides the naming of
methods, XMLWriter operates almost identically to .NET's
XmlWriter, except for two details.
First, XMLWriter has the notion of a collection of attributes, whereas
XmlWriter requires you to write each attribute individually. In Java, we call
beginElement(), passing the name and the Map of attributes,
whereas in C#, we called WriteStartElement() followed by
WriteAttributeString().
Second, XMLWriter has a writeEmptyElement() method, where
XmlWriter requires you to call WriteStartElement() followed by
WriteEndElement(). However, .NET automatically collapses an empty element
into a short end element (in this case, <br />). .NET's way gives you the
flexibility of determining whether the element is empty at runtime. If, however, you
need to
force an end tag, you can call WriteFullEndElement() instead of
WriteEndElement().
Conclusion
A pull parser makes it much easier to process XML, especially when you are processing XML with a well-defined grammar like RSS. This code is much easier to understand and maintain since there's no complex state machine to build or maintain. In fact, this code is completely stateless; the pull parser keeps track of all the state for us. So in that sense a pull parser is a higher level way of processing XML than SAX.
Although my original code quite intentionally didn't do any error handling, error handling in a push model state machine adds even more complexity to an already complex model. The new RSSReader has clear placeholders for error handling code in the cases when the input doesn't comply with the expected RSS DTD.
Performance can be an important consideration in an XML parser. Notice the call to
Skip() (in the C# version) when we find elements we're not interested in. In
this case the XML parser can skip over entire subtrees of XML without having to call
us back
on every element, even ones we know we're not interested in. In this case we skip
over the
<image> elements and all their children. Second, in C# we could optimize
out all the element name string comparisons and make the atomized pointer comparisons
if we
used the XmlReader's NameTable to pre-atomize those strings.
Finally, using an XML writer makes our output generation more robust. For example, it will correctly convert special characters -- <, &, etc. -- into their respective entity references. Because it maintains its own state internally, it never forgets which element to close after a convoluted series of while loops. And it will always produce XML output in the consistent and readable format of your choice.
And now for the inevitable comparison between .NET's XmlReader/XmlWriter and
the equivalent functionality in Java. As usual, I'll say that in .NET, Microsoft has
provided it all for you and, thus, it is undeniably simpler to learn and use. The C#
version of our RSSReader is about 20% shorter than the Java version, which is great
unless
you work in one of those shops which still measures productivity in KLOCs. And the
readability of the code itself is much greater in C#, although that probably can be
chalked
up at least in part to my own lack of skill in that conversion from while to
do...while.
But the real bottom line remains that doing it the .NET way means that Microsoft provides all the standards-compliant tools that 90% of developers are likely to need, while the Java way still means putting together a solution from various pieces that you can scrounge from various sources. Some of those pieces come from the Java Community Process and thus represent peer-reviewed, formally approved APIs, but some come from a quick search of the Web, and in the end only you are qualified to judge their worthiness.