On 09 Apr 2014, at 19:35, itlists@schrievkrom.de wrote:
Ok, forget the JSON stuff - it has nothing to do with the "problem".
Other way round:
My whole database and internal processing is done in UTF8. This is the most important point here to mention.
Why ? This means you forgo almost all String functionality, since UTF8 is a variable length encoding not really suitable to character by character processing.
Now the request comes into Zinc as mentioned below (the content of the request is a JSON string only):
HTML-Request (charset=UTF-8) =(sends)=> ZINC HTTP
Now Zinc sees the content of the body, knows that it is coded in UTF8 and creates a ZnStringEntity with UTF8Encoder.
Zinc HTTP =(builds)=> ZnStringEntity (with UTF8Encoder)
The instance of ZnRequest and its entity value is an instance of ZnStringEntity (with its encoder attribute is set to an instance to ZnUTF8Encoder).
Yes, of course, UTF-8 (a variable length binary encoding) is converted into native Pharo Strings (possibly WideStrings) containing Characters, each of which is encoded using a Unicode code point value.
I checked the content of the string attribute of the ZnStringEntity and this string is NOT encoded in UTF8 any more, but in either ISO8859-? or WIN1252.
Here you lose me (again) ;-)
I think, that this is ok for almost all people, because they work with some CodePages - but my internal processing assumes UTF8.
No, nobody works with code pages or any encoding, just native [Wide]Strings in pure Unicode.
I just fixed this for me by changing ZnStringEntity>>initializeEncoder to ALWAYS set the encoder attribute to ZnNullEncoder and now everthing is ok again. This means of course, that all apllication running with that source code work in UTF 8 only ...
OK, I think I understand, you want UTF-8 to remain UTF-8. What you did is one solution, but I think it is wrong to use a String to represent bytes. This case is actually already implemented server side for Seaside: ZnZincServerAdaptor>>#configureServerForBinaryReading "Seaside wants to do its own text conversions" server reader: [ :stream | ZnRequest readBinaryFrom: stream ] The #reader: option is used here to read everything binary, without decoding to Strings. You will get ZnZnByteArrayEntity objects back, containing the original binary representation. BTW, I think this is an interesting discussion. Regards, Sven
Marten
Am 09.04.2014 18:42, schrieb Sven Van Caekenberghe:
Marten,
On 09 Apr 2014, at 18:25, itlists@schrievkrom.de wrote:
Ok, if the browser sends POST/PUT request with a JSON structure it also sends charset = utf8 (in my case). That's ok, because for JSON this is more or less the default charset.
Zinc now seems to notice, that UTF8 charset is needed and creates a ZnStringEntity with an UTF8Encoder.
Now when my application tries to get the JSON string of that ZnStringEntity and builds the structure out of that string - and the strings are NOT UTF8, but converted to (?) ISO8859 ?
(NeoJSONReader fromString: (ZnEntity with: (NeoJSONWriter toString: { #message -> 'An der schönen blauen Donau' } asDictionary))) at: #message.
You must be doing something possibly wrong when you <<get the JSON string of that ZnStringEntity and builds the structure out of that string>> (how do you do that, BTW), so please write some code that demonstrates what is not right according to you.
Sven
-- Marten Feldtmann