HTML is something that has evolved. During the great browser-wars, the evolution was chaotic. We've also had the great XHTML-versus-HTML debate, essentially a reaction, sometimes an over-reaction, against the forces of chaos. Much has been written about the need for standardization. Unfortunately most writers have focused on trivia to such an extent that the important issues have been largely ignored. An important principal that many writers including the standards-writers themselves have ignored is that you don't make incompatible changes. Those who design programming-languages have known for as long as there have been programming-languages that adding new features is fine, changing the rules so that programs written under the old rules stop working isn't. How could the people at the W3C have failed to learn that lesson? HTML is a markup-language, not a programming-language, but the same principle applies since there were millions of webpages written in HTML, and invalidating all those webpages with a stroke of the standards-pen ought to have been unthinkable.
You may wish to argue that the W3C did not make incompatible changes to HTML, rather they kept HTML as it was, and created a new markup-language called XHTML. And that's true, however it is also a great deal less than the whole truth. The W3C has for years had an agenda of getting those using HTML to switch to XHTML, employing tactics that sometimes resemble forcing XHTML down our throats whether we like it or not, and in that light XHTML being anything other than an incompatible revision to HTML is in name-only. Just take a look at how the W3C validator treats those wanting to use the SGML parse-mode, as is needed for HTML, as opposed to using the XML parse-mode, and at the W3C response to my suggestion about a simple change to their validator that would be of benefit to users of SGML and HTML (as opposed to XHTML): markmail.org/message/afpc6odvokbrwprn. Their validator's warning-message and their response to that suggestion amount to saying if you're not using the XML parse-mode, ie: not using XHTML, then you deserve to be punished. In fairness the warning annoys and the suggestion would benefit only a small minority of HTML-authors, only those using a somewhat esoteric feature of their validator.
Incidentally the "custom-DTD" mentioned in that suggestion is primarily an attempt to bridge the gap between the HTML-language accepted by real-world browsers and the one defined by the W3C. To validate with that DTD (and DCL) is to parse in a way somewhat closer to the way real-world browsers parse HTML.
Hearing about HTML-5 was a pleasant surprise. Could this mean the W3C is no longer working toward having XHTML replace HTML? That writing in HTML is still respectable after all?
[This is unfinished.]
Send your questions, suggestions, bug-reports to firstname.lastname@example.org.