2018-01-29 18:59 GMT+01:00 Sven Van Caekenberghe <sven@stfx.eu>:

Your file is not in UTF-8 but in UTF-16 !

This will do:

(FileLocator desktop / 'schinese.txt') readStreamDo: [ :in |
�� (ZnCharacterReadStream on: in binary encoding: #utf16) upToEnd ].

BTW, this is not valid JSON.

>From Windows, for sure ...

> On 29 Jan 2018, at 18:49, Cl��ment Bera <bera.clement@gmail.com> wrote:
>
> Hi,
>
> I am currently parsing lua and JSON-like file in Pharo. They contain both Simplified and Traditional Chinese characters for comments and for strings displayed in the UI. Lua files are parsed correctly. However the JSON-like files aren't.
>
> In attachment I put one of the problematic file with Simplified chinese characters (I've also copied the file at the end of the mail). The problem can be shown as follow in Pharo:
>
> 'schinese.txt' asFileReference readStream contents
>
> The contents function sends a UTF8InvalidText 'Invalid utf8 input detected' error.
> However my text editor correctly displays the file and it is correctly parsed by the Lua runtime (The program parsed has been deployed in production for years and works fine).
>
> What can I do to parse this file correctly from Pharo ?
>
> Thanks,
>
> Below is the file content, non Chinese people may not have the font to display the characters, note that I have no idea what is written in Chinese (Please don't hold me responsible if there are offending contents):
>
> "lang"
> {
>�� "Language"�� "Schinese"
>�� "Tokens"
>�� {
>�� "text_store_cd" "��! ��"
>�� "text_cannot_huidaoguoqu"�� "��~"
>�� "tips1" "��~"
>�� "tips2" "��50��"
>�� "tips3" "��"
>�� "tips4" "��25��"
>�� "tips5" "��10��BOSS��~"
>�� "tips6" "��~"
>�� "tips7" "��"
>�� "tips8" "��"
>�� "tips9" "��~"
>�� "tips10"�� "��"
>�� "tips11"�� "��~"
>�� "tips12"�� "��"
>�� "tips13"�� "��10��MVP��"
>�� "tips14"�� "��"
>�� "tips15"�� "��~"
>�� }
> }
>
>
>
> --
> Cl��ment B��ra
> Pharo consortium engineer
> https://clementbera.wordpress.com/
> B��timent B 40, avenue Halley 59650 Villeneuve d'Ascq

> <schinese.txt>