> Am 03.05.2017 um 18:10 schrieb Cyril Ferlicot D. <cyril.ferlicot@gmail.com>:
>
>> Le 03/05/2017 �� 16:41, Sven Van Caekenberghe a ��crit :
>>
>>> On 3 May 2017, at 12:18, Sven Van Caekenberghe <sven@stfx.eu> wrote:
>>>
>>> Hi Cyril,
>>>
>>> I want to try to write such a detector. I'll get back to you.
>>
>> I added the following (Zn #bleedingEdge):
>>
>> ===
>> Name: Zinc-Character-Encoding-Core-SvenVanCaekenberghe.49
>> Author: SvenVanCaekenberghe
>> Time: 3 May 2017, 4:30:44.081888 pm
>> UUID: fe8b083d-010b-0d00-9df5-fde304bccfdc
>> Ancestors: Zinc-Character-Encoding-Core-SvenVanCaekenberghe.48
>>
>> Add ZnCharacterEncoder class>>#detectEncoding: to try to heuristically and unreliably guess the encoding used by a collection of bytes
>>
>> Add ZnCharacterEncoderTests>>#testDetectEncoding
>>
>> Add #= and #hash to ZnSimplifiedByteEncoder and ZnEndianSensitiveUTFEncoder
>>
>> Always use canonical name in ZnSimplifiedByteEncoder class>>#newForEncoding:
>> ===
>> Name: Zinc-Character-Encoding-Tests-SvenVanCaekenberghe.31
>> Author: SvenVanCaekenberghe
>> Time: 3 May 2017, 4:31:09.469852 pm
>> UUID: 30ef8b3e-010b-0d00-9df6-4a9304bccfdc
>> Ancestors: Zinc-Character-Encoding-Tests-SvenVanCaekenberghe.30
>>
>> Add ZnCharacterEncoder class>>#detectEncoding: to try to heuristically and unreliably guess the encoding used by a collection of bytes
>>
>> Add ZnCharacterEncoderTests>>#testDetectEncoding
>>
>> Add #= and #hash to ZnSimplifiedByteEncoder and ZnEndianSensitiveUTFEncoder
>>
>> Always use canonical name in ZnSimplifiedByteEncoder class>>#newForEncoding:
>> ===
>>
>>
>> Now you can do the following:
>>
>> ZnCharacterEncoder detectEncoding: ((FileLocator desktop / 'some.data') binaryReadStreamDo: [ :in | in upToEnd ]).
>>
>> (FileLocator desktop / 'some.data') binaryReadStreamDo: [ :in |
>>�� �� | bytes encoder |
>>�� �� bytes := in upToEnd.
>>�� �� encoder := ZnCharacterEncoder detectEncoding: bytes.
>>�� �� encoder decodeBytes: bytes ].
>>
>> It works on the test file you gave me, but this process is just a guess, a heuristic that is unreliable and often wrong (especially for very similar byte encodings), see https://en.wikipedia.org/wiki/Charset_detection.
>>
>> You can give the whole contents to the detector, or just a header.
>>
>> I was a bit too optimistic though, this is basically an unsolvable problem. It is MUCH better to somehow know up front what the encoding used is, or to know something useable about the contents (like the header of HTML or XML).
>>
>> Sven
>>
>
> Thank you! I'll try this tomorrow. If it works well I wonder if we can
> still includes it in Pharo6. Since it's only a little feature unused in
> Pharo it should not break anything but it would be cool addition for Moose.
>
> But since it is feature freeze if people do not want I'll not push it
> for Pharo 6 :)
>

It shouldn't be included. There no such thing as side-effect-free change. Moose can load a newer version of zinc. That is how it is supposed to be.

Norbert

> --
> Cyril Ferlicot
> https://ferlicot.fr
>
> http://www.synectique.eu
> 2 rue Jacques Pr��vert 01,
> 59650 Villeneuve d'ascq France
>