Michael Rueger wrote:
Philippe Marschall wrote:
Tudor Girba wrote:
Or more general, how can I find out what kind of converter I need for a given file?
...
You can pretend the problem doesn't exist and just use a random encoding aka the platform default encoding. That would work if all your files were are created on your computer and all your programs used the
At least for those files it will work in almost all cases.
For the others you might not notice the error.
platform default encoding. E.g. you don't download stuff from the internet or copy stuff off storage media (USB sticks, CDs, DVDs, ...).
Well, there are some heuristics, but nothing failsafe... If the file has cr-lf, it is probably a windows file http://en.wikipedia.org/wiki/ISO/IEC_8859-1#ISO-8859-1_and_Windows-1252_conf...
Maybe, maybe somebody actually got Latin-1 right and then you might corrupt it when treating it as code page 2152. Moreover there are more Windows code pages than 1252. And finally there are other places in the world besides central Europe.
You can check for a Unicode BOM.
Which nobody does (in Pharo), may not be there and only works for UTF-8/16/32.
.html should be UTF-8 these days.
Yeah totally: http://squeakland.org/ Keep in mind there are still commercial Smalltalk dialects today struggling with UTF-8.
Try reading UTF-8, if it fails try re-reading with platform encoding.
The funny thing is that XML partially solves the problem because the encoding can be put into the preamble. In fact it should if the file is not ASCII. You can make a guess whether Yaxo supports binary input and detecting the encoding either from the BOM or the preamble.
Hmm, to be honest I'm not sure myself, but I think I (with help from others) did some work a while ago to make it read files according to the encoding entry. But maybe my memory has the wrong encoding ;-)
Failed for the very simplistic attached file with the following code (XMLDOMParser parseDocumentFrom: (FileStream fileNamed: 'latin-1.xml')) elements first characterData I had to run the FileStream in text mode, the parser would blow up in binary mode. If my platform encoding was utf-8 it would probably have blown up. Works fine out of the box in Eclipse though. Cheers Philippe