It is unfortunate to see XML.com running an article that endorses bringing the haphazard world of Tag Soup from HTML into the world of XML. The primary benefits of XML are its widespread, CONSISTENT usage which allows for the availability of several off-the-shelf tools and reduces vendor lockin.
Encouraging consumers of XML to support ill-formed XML reduces the power of XML and induces fragmentation. If we arbitrarily pick bits and pieces of a standard to support then we cheapen the technology and reduce it to worthlessness.
I'd hate to see XML on the 'web reduced to HTML during the browser wars with people simply checking if "it works well with Mark Pilgrim's program" or creating ill-formed markup simply to satisfy broken tools.
What we need to face up to is the fact that the MXL community has broken in half without quite realizing it, and the two halves are talking past each other in perfect incomprehension and increasing frustration.
One camp says that XML is about formal syntax, the other that says it is about informal semantics. Both are right, because XML can do both; they are talking about disjoint applications. It is not necessary for one side to "win", but for both sides to realize that XML is a sufficiently protean technology to do things that its originators did not foresee.
The tone of the article is based on the demonstrated realities of the RSS world, which I agree is depressing. Are you proposing a solution (other than the two I proposed)? Or are you just idly wishing that life was easier for developers?
In this case with RSS. I believe the providers of the feed should care enough for it to check that it is not broken. And if the document is broken, how much should you fix it? There might be bits and pieces missing or completely wrong. If the document is silently fixed, how are you to know what you are missing?
If the document has two root elements, which one would you choose? Both? Should open tags be closed? Maybe the content of the document still is broken?
On the other hand. It would be very convenient in an RSS client when an invalid document is encountered to have a pop-up asking "Fix broken document? Yes/No". But I think the key point is to inform the user that the document is broken!
And if the RSS feed is fed into something else a notation that the document has been modified must be included...
Should this bee seen in a bigger context. Should all XML documents be fixed by the parser? Only well-formedness or also if not valid?
From a programmers standpoint it is very nice to know that you can (and should?) throw away a broken document because parsing it otherwise probably would propagate errors.
It depends on what you consider to be the problem. From my perspective, the problem is websites that provide non-standards compliant XML in their RSS feeds while from yours it is consuming this XML even if it does not comply with the W3C XML 1.0 recommendation.
The solutions from my point of view would rely on pressuring sites and tools that produce invalid RSS feeds to correct them and creating tools like the RSS validator produced by yourself and Sam Ruby (which is an excellent contribution to the community).
The temporary benefit of being able to read ill-formed RSS feeds is outweighed by the harm caused to XML and the Web by fostering the idea that it is OK to produce and consume XML that does not conform to W3C standards. XML has been successful thus far because of the fairly strict adherence to standards by vendors, producers and consumers of XML documents. It is unfortunate that your article is attempting to undermine this even though your intentions are good.
re: "The temporary benefit of being able to read ill-formed RSS feeds is outweighed by the harm caused to XML and the Web"
The problem is that the benefit is accrued by the software vendor, and is direct and immediate, but the harm is caused to everyone equally, and is long-term and abstract. Direct and immediate wins every time.
Direct and immediate wins everytime reminds me of Hardin's arguments vis-a-vis the commons, since come under some controversy.
It is in the main a philosophical argument, but as such I can not see how it is a sensible one.
You say the direct and immediate wins everytime, implying that newsreaders will have to parse everything that proclaims itself RSS whether it is or not because of business pressures to do so. But if a public newsreader did not parse the RSS instead returning a broken message to the clients of said feed then would this not create direct and immediate pressures on feed authors and sites to produce valid xml, and would this not spur product sales for RSS producers that produced valid RSS?
Part of the reason for xml (which after all is a simpler set of rules than most other languages) that is not well-formed with RSS is of course that RSS (2.0 and pre 1.0) allows escaped html inside of the description element, a practice I believe much more likely to cause broken feeds. As I've harped on before this hampers the transportability of feeds across media, to for example a non-html email newsletter format, various phone media, or even specific browsers.
It seems to me that a vendor that produced both a RSS producer and consumer that could be relied on to produce only well-formed feeds could derive direct and immediate benefits against other vendors, because of reuse of xml in other media.
You write: "But if a public newsreader did not parse the RSS instead returning a broken message to the clients of said feed then would this not create direct and immediate pressures on feed authors and sites to produce valid xml"
No, it would not. The person who puts out the feed rarely reads it.
> "implying that newsreaders will have to parse everything that proclaims itself RSS whether it is or not because of business pressures to do so."
Exactly.
> "But if a public newsreader did not parse the RSS instead returning a broken message to the clients of said feed then would this not create direct and immediate pressures on feed authors and sites to produce valid xml"
No. You are punishing the wrong people. You are still operating under the mistaken impression that XML, in and of itself, is important. It is not. It is a means to an end. End users don't care. And they shouldn't have to care.
Look, I was in this position: I tried several news first-generation aggregators that only used real XML parsers. Feeds would go unreadable for days at a time, and by the time they came back I had missed dozens of articles. I tried to switch to another aggregator that could allow me to follow the sites I wanted to follow, but none satisfied me, so I ended up writing the parse-at-all-costs RSS parser and building a homegrown aggregator around it for my own use.
And I'm *technically inclined*. I *care* about XML. Imagine the reaction of an end user who isn't, and doesn't. They bought (downloaded/whatever) a program that purports to help them read all the news and follow all the sites that they care about. They like this idea. Then they find out that sometimes it doesn't work, sometimes sites that worked yesterday don't work today, and some sites don't work at all, because of something called "XML". They don't know from XML, they've never seen XML, they don't care about XML, but this stupid POS program is complaining and saying there's nothing it can do about this "XML" problem and suggesting, in its infinite wisdom, that the end user should take it upon themselves to work around this problem by sending an email to the site owner and waiting an indeterminate length of time before they can read the news they care about, if ever.
You're kidding, right?
Then the user hears about another aggregator, a direct competitor, which claims to be able to let them follow *all* the sites they care about. It doesn't complain; it doesn't whine; it doesn't suggest that they work around the developer's laziness by firing off emails to random people they've never met. It just works.
No, but that is because I'm not really viewing an aggregator as a tool in itself, I don't think aggregators have much of a business future. I think they're destined to become part of other products.
>Then the user hears about another aggregator, a >direct competitor, which claims to be able to >let them follow *all* the sites they care about. >It doesn't complain; it doesn't whine; it >doesn't suggest that they work around the >developer's laziness by firing off emails to >random people they've never met. It just works.
Again, I don't believe in aggregators as stand-alone tools, I believe that they will become part of more wide-ranging products.
If such a product has to do with handling XML of widely different formats then it cannot devote development resources to handling stuff that thinks it's XML but really isn't.
A product can provide add-ins to convert legacy formats, but I don't think badly formed RSS will qualify for such attentions.
If such a product is the object then the well-formedness of the XML becomes integral to the product, development will have to provide ways to error report problems with individual XML instances, such as those originating from a feed.
This is not developer laziness, but developer ambition.
Error reporting to a user has always seemed to me to be an exercise in the art of communication. If a non-technical user receives the error message
"XML error at
hello world
" then they might well be expected to say "This program sucks" if on the other hand they receive information like "Newsfeed at http://www.myinfo.com/newsfeed7 is not conforming to the technical standards for newsfeeds, if you would like to learn more click More Info" then I would expect the user to think something like "Frigging amateurs at www.myinfo.com" despite not automatically fixing www.myinfo.com for the user the program may still command market share if it does enough other things with various other XML technologies. This may cause you to think again that I'm kidding but I'm not, I think a lot of these problems stem from the technical communities believe that the end user is an idiot. The end user may not understand XML or any other standard, but I have faith enough in the intelligence of people to understand a claim that such and such a thing does not conform to a standard.
I've heard your arguments before from other people and don't agree with them. Thankfully, those of us who work on core XML technologies at Microsoft don't have this attitude towards XML and related standards simply because we want to gain "competitive advantage". If we did many of the gains that XML brings to the our users due to its reusability and ability to foster interoperability would be lost.
Your article highlights a mini tragedy of the commons. If XML applications that process RSS documents begin to lean towards processing ill-formed XML then when RSS files are reused such as many XML formats are wont to be (e.g. some mention using RSS for weblog archives, others have suggested using it as a general push technology) then this sloppiness and lack of standards adherence will creep into this avenues as well.
All in all it's interesting to read a column called Dive Into XML on a website called XML.com which encourages poisoning the XML in the name of "competitive advantage".
This has nothing to do with the tragedy of the commons (boy, there's an overused phrase). It has everything to do with the Robustness Principle that Postel nailed years ago in RFC 793: "TCP implementations will follow a general principle of robustness: be conservative in what you do, be liberal in what you accept from others." The same applies here: validators and programs that produce RSS should be as conservative as possible; end user tools that consume RSS should be as liberal as possible. They serve different masters.
I'm tired of arguing with you, Dare. Despite your misrepresentation, we can all see for ourselves that my article clearly demonstrates an actual problem, describes a workaround for consuming tools, and pushes for not one but two long-term social solutions (the centralized advocacy effort at Syndic8, and the decentralized solution of making non-well-formedness visible to the end user).
Meanwhile, it's ironic that you hold up Microsoft as the epitome of XML standards compliance. What short memories we have! Have a quick look back in the XML mailing list archives to see all the confusion their ultra-liberal MSXML parser caused with people who mistook it for an actual validating XML parser. ("Whatdya mean my XML's not well-formed? It looks fine when I open it in IE!") That was not the place to parse at all costs; this is.
Actually a number of our customers regularly praise the standards compliance of MSXML.
Unfortunately, we also have customers who mistakenly assume that viewing XML in Internet Explorer causes it to be processed by the validating XML parser instead of the well-formed XML parser which is not the case. This design decision was before my time but was most likely motivated by good intentions similar to yours about reducing user pain and ensuring that even invalid but well-formed XML was viewable in the browser. No one thought to think about what would happen downstream when people assumed that
viewable in IE == well-formed & validated XML
instead of just
viewable in IE == well-formed XML
Your attempted slur actually helps bolster my point as to why your article should not be encouraging supposedly "user-friendly" but standards unconformant behavior.
This reminds me of a post on xml-dev where some guy named Tim Bray talked about using MSXML to prove to people that their xml was not well-formed, it was an off-hand remark, but he said something along the lines of that people usually grasped that xml was not well-formed when he had them open it in IE and it told them there was a problem.
Of course I don't know if this Tim Bray character might be someone to listen to. probably not, but still, just saying.
Build in functionality to report back to feeds providing garbage data. Make it easy to report to sites that their feeds are causing a problem.
The path of silently dealing with garbage data leads to excessive amounts of development time being spent on a problem which should take virtually no time. Further, it reflects poorly on the aggregator when it does run into a feed it can't deal with. Instead of blaming the feed, users now blame the tool for not handling it.
If I can't reasonably rely on RSS being well formed and complying to an industry standard specification, I'm more inclined to simply remove the functionality than to enter an endless back and forth battle of regular expressions and garbage data.
Put a fence at the top of the cliff, not an ambulance force at the bottom. Tools which generate problems will eventually fall from favor. All things considered, 10% failure for such a technology seems promising. There was a time when it was hardly possible to find ANY well formed web pages.
As an end user, I want a news aggregator that works for whatever feeds I refer to, thus I am very grateful for Mark's efforts.
I understand the "well formed" arguments however, and the difficulties inherent in providing feedback. I suggest that we shame the poorly-behaving sites by publishing their URLs for all the world to see, then issuing a press release.
Many feeds have no contact information, so this can not be easily automated. Regardless, I believe efforts are underway to do exactly this (when possible) in the next release of Aggie. Users who care about such things can take the time to contact the content provider.
However, this does not negate the fact that, as an end-user product, the #1 responsibility of the software is to the end user. The end user wishes to read news, and has downloaded, installed, and possibly paid for a program to help them read news. If the program refuses to display news for reasons that the end user considers arcane and trivial, the user will find another program that does not throw such technical hissy fits.
Why not do both? If the XML validator fails, display an unobtrusive quality indicator like iCab (the smiling face in the throbber changes to a frown for malformed HTML), automatically send some sort of request to a tracking site and fall back to the error-prone all-costs parser.
The tracking site would be extremely valuable if it could track the buggy software instead of just individual sites. Feeding crawler with, say, the weblogs.com feed would probably give a pretty accurate indicator of the relatively quality of the RSS implementations. While the users may not care, the authors might be more motivated about getting unlisted from the hall of shame.
Previous Message
