Are you sure that the file is encoded in utf8? Can you try stream := ZnCharacterReadStream on: (File named: '...') readStream encoding: 'utf8'. stream upToEnd. ? If that does not work, it could mean that the file is in another encoding... 2018-01-29 18:49 GMT+01:00 Clément Bera <bera.clement@gmail.com>:
Hi,
I am currently parsing lua and JSON-like file in Pharo. They contain both Simplified and Traditional Chinese characters for comments and for strings displayed in the UI. Lua files are parsed correctly. However the JSON-like files aren't.
In attachment I put one of the problematic file with Simplified chinese characters (I've also copied the file at the end of the mail). The problem can be shown as follow in Pharo:
'schinese.txt' asFileReference readStream contents
The contents function sends a UTF8InvalidText 'Invalid utf8 input detected' error. However my text editor correctly displays the file and it is correctly parsed by the Lua runtime (The program parsed has been deployed in production for years and works fine).
What can I do to parse this file correctly from Pharo ?
Thanks,
Below is the file content, non Chinese people may not have the font to display the characters, note that I have no idea what is written in Chinese (Please don't hold me responsible if there are offending contents):
"lang" { "Language" "Schinese" "Tokens" { "text_store_cd" "è´å£³ååºçååä¼æ¯å¤©éæºå·æ°! ä¸æ¬¡å·æ°å·å´æ¶é´" "text_cannot_huidaoguoqu" "ç°å¨ä¸è½ä½¿ç¨åå°è¿å»~" "tips1" "å©ç¨é¼ æ æ»è½®å¯ä»¥è°èè§è§è·ç¦»ï¼æ¹ä¾¿ä½ æ¥çåºå°å ¨è²~" "tips2" "éè¿å ¨é¨50å ³ä»¥åä½ è¿å¯ä»¥ç»§ç»æææ´å æé¾åº¦çæ å°½è¯ç¼æ¨¡å¼ï¼" "tips3" "æ¯å¤©æ©æ¨è´å£³ååºä¼éæºå·æ°åéæºææï¼çå¿ä½ æ³è¦çååï¼" "tips4" "æè¡æ¦çå25åå¯ä»¥è·å¾çå å¥ç« ï¼è±¡å¾çä½ å¨å¡é²æ¸¸æä¸çåè¶æ绩ï¼" "tips5" "æ¯10æ³¢æ人ä¼æä¸ä¸ªBOSSå ³å¡ï¼å®æ¯æ®éæ人æ´é¾å»æ~" "tips6" "é£è¡çæ人ä¸ä¼è¢«ç³å¤´é»æ¡ï¼æ以ä¸è½å©ç¨è¿·å®«æ¥å¢é¿æªç©ç线路~" "tips7" "å¼å±çæ¶åç¹å»å·¦ä¸æ¹çè±ééæ©å¾æ å¯ä»¥æ¥ç并éæ©ä½ æ¥æçè±éï¼" "tips8" "é身çæäººå¿ é¡»éè¿èç½ç³»åå¡çç §æå ç¯æè½è¢«åç°ï¼" "tips9" "ç¹å»å³ä¾§çåæå ¬å¼æé®å¯ä»¥æå¼åæé¢æ¿ï¼äºè§£é«çº§å¡çåæ以åå½åé 件ç¶æ~" "tips10" "é请好åä¸èµ·æ¸¸æï¼å¯ä»¥äºç¸åä½å ±å对æ强大çæ人ï¼" "tips11" "ç³æ¿ä¼å¯¹è¸©ä¸å»çæ人产çææï¼æ以æ好æ¾ç½®å¨æäººå¿ ç»ç路线ä¸~" "tips12" "为è±éè´ä¹°ç¾ä¸½çç¹æï¼å½ä½ å¯ä»¥ä¸åååæçæ¶åææé 件é½ä¼æç¹ææ示ï¼" "tips13" "æ¯åå伤害æé«çå¡å°ä¼è·å¾æå¤10å±çMVPå ç¯ï¼å¢å ç©çåéæ³è¾åºï¼" "tips14" "æ¯ä¸ªæçæåä¸å¤©æ¯èµå£ç»ç®æ¥ï¼å°ä¼æ ¹æ®ä½ è¿èµå£çæåé¢å丰åçè´å£³å¥å±ï¼" "tips15" "å¦æä¸ç¥éææ ·é 迷宫ï¼ä½ å¯ä»¥ç¹å»å³ä¾§ç迷宫æåæé®æ¥çæè å享æ¨èç迷宫~" } }
-- Clément Béra Pharo consortium engineer https://clementbera.wordpress.com/ Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq
-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13