I'm not sure, but maybe useful... * http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/ * https://github.com/chardet/chardet/tree/master/tests cheers -ben On Wed, May 3, 2017 at 6:18 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi Cyril,
I want to try to write such a detector. I'll get back to you.
Any chance you could give me (part of) a file that causes you trouble (one that is legal latin1, yet does not fail utf-8 while doing it wrong in utf-8) ?
Sven
On 3 May 2017, at 11:40, Cyril Ferlicot D. <cyril.ferlicot@gmail.com> wrote:
Hello,
We have a problem using Moose because we have files which we don't know the encoding. Currently we have this implementation to get the content of a file:
completeText self fileReference exists ifFalse: [ ^ '' ]. ^ self fileReference readStreamDo: [ :s | [ s contents ] on: Error do: [ [ s converter: Latin1TextConverter new; contents ] on: Error do: [ '' ] ] ]
But, we have a problem because we have currently some files at Synectique in ISO-8859-1. The problem is that #contents is able to read some of the files without throwing an error, but the content is not right because it is not the good encoding.
Thus I wonder if it is possible to get the Encoding of a FileReference in Pharo to be able to read the file with the right encoding? Something like the bash command `file -I myFile.txt`.
-- Cyril Ferlicot https://ferlicot.fr
http://www.synectique.eu 2 rue Jacques Prévert 01, 59650 Villeneuve d'ascq France