What Happened To My Beautiful En-dashes?!
First, a small introduction to the problem: We’re running stuff in UTF-8 all the way. A few sites we’re reading feeds from are using ISO-8859-1 as their charset, but they either supply the feed with the correct encoding specific or the feeds arrive as UTF-8. Everything works nicely, except for the mentioned-in-the-headline en-dashes. Firefox only shows 00 96 (0×00 0×96), but everything looks correct when you view the headlines and similiar stuff on the original site.
Strange.
The digging, oh all the digging.
After the already mentioned digging (yes, the digging) in data at the large search engines (ok, maybe I did a search or two), I discovered that the windows cp1252 encoding uses 0×96 to store endashes. This seems similiar! We’re seeing 0×96 as one of the byte values above, so apparently cp1252 is sneaking into the mix somewhere along the lines. Most of the clients using the CMS-es are windows, so they might apparently be to blame.
ISO-8859-1 enters the scene
As the sites (and feeds) provide ISO-8859-1 as their encoding, I thought it would be interesting to see what ISO-8859-1 defines as the representation for the byte value 0×96. Lo’ and behold: 0×96 is not defined in ISO-8859-1. Which actually provides us with the solution.
I welcome thee, Mr. Solution
When the ISO-8859-1 encoded string is converted into UTF-8, the bytes with the value 0×96 (which is the endash in cp1252) is simply inserted as a valid code sequence in UTF-8 which represents a character that’s not defined.
We’re saying that the string is ISO-8859-1, although in reality it is either cp1252 or a mangled version of iso-8859-1 and cp1252 (for the endashes, at least).
If you’re on the parsing end of this mumbo jumbo, one solution is to replace the generated UTF-8 sequence (0xc2 0×96) (converted from 0×96 i ISO-8859-1) with the proper one (0xe2 0×80 0×93):
-
$data = str_replace("\xc2\x96", "\xE2\x80\x93", $data);
And voilá, everything works.
Tags: converting, cp1252, encoding, iso-8859-1, parsing, PHP, Programming, rss, utf-8, xml