I'm not sure, but maybe useful...
* http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/
* https://github.com/chardet/chardet/tree/master/tests

cheers -ben

On Wed, May 3, 2017 at 6:18 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi Cyril,

I want to try to write such a detector. I'll get back to you.

Any chance you could give me (part of) a file that causes you trouble (one that is legal latin1, yet does not fail utf-8 while doing it wrong in utf-8) ?

Sven

> On 3 May 2017, at 11:40, Cyril Ferlicot D. <cyril.ferlicot@gmail.com> wrote:
>
> Hello,
>
> We have a problem using Moose because we have files which we don't know
> the encoding. Currently we have this implementation to get the content
> of a file:
>
> completeText
>�� self fileReference exists ifFalse: [ ^ '' ].
>�� ^ self fileReference readStreamDo: [ :s |
>�� �� [ s contents ]
>�� �� �� on: Error
>�� �� �� do: [ [ s converter: Latin1TextConverter new; contents ]
>�� �� �� �� on: Error
>�� �� �� �� do: [ '' ] ] ]
>
> But, we have a problem because we have currently some files at
> Synectique in ISO-8859-1. The problem is that #contents is able to read
> some of the files without throwing an error, but the content is not
> right because it is not the good encoding.
>
> Thus I wonder if it is possible to get the Encoding of a FileReference
> in Pharo to be able to read the file with the right encoding? Something
> like the bash command `file -I myFile.txt`.
>
> --
> Cyril Ferlicot
> https://ferlicot.fr
>
> http://www.synectique.eu
> 2 rue Jacques Pr��vert 01,
> 59650 Villeneuve d'ascq France
>