[Pharo-project] Unicode/Latin1 handling in Pharo 1.3
Sven wrote
On my Mac I get: ... (stream contents at: 6) codePoint 71303420 ('Teilrückzahlung' at: 6) codePoint 252
Hi Sven, on Windows I get: (stream contents at: 6) codePoint 252 ('Teilrückzahlung' at: 6) codePoint 252 For $ü codePoint I also get 252 So on Windows (at least for my test) it does not have the "leadingChar". Bye T. -- Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
Can you paste the code you used to come to this result? Norbert Am 08.05.2012 um 14:46 schrieb Torsten Bergmann:
Sven wrote
On my Mac I get: ... (stream contents at: 6) codePoint 71303420 ('Teilrückzahlung' at: 6) codePoint 252
Hi Sven,
on Windows I get:
(stream contents at: 6) codePoint 252 ('Teilrückzahlung' at: 6) codePoint 252
For
$ü codePoint I also get 252
So on Windows (at least for my test) it does not have the "leadingChar".
Bye T.
-- Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
Here is an example with the file out of the equation: | stream | stream := (ByteArray readHexFrom: '5465696c72fc636b7a61686c756e67') asString readStream. stream next: 5. (ISO885915TextConverter new nextFromStream: stream) codePoint. => 71303420 | stream | stream := (ByteArray readHexFrom: '5465696c72fc636b7a61686c756e67') readStream. stream next: 5. ((ZnCharacterEncoder newForEncoding: 'iso-8859-15') nextFromStream: stream) codePoint. => 252 <rant>Notice how f*cked up TextConverters are: when given a binary stream, their natural input one would think, they just don't do any conversion!</rant> On 08 May 2012, at 15:30, Norbert Hartl wrote:
Can you paste the code you used to come to this result?
Norbert
Am 08.05.2012 um 14:46 schrieb Torsten Bergmann:
Sven wrote
On my Mac I get: ... (stream contents at: 6) codePoint 71303420 ('Teilrückzahlung' at: 6) codePoint 252
Hi Sven,
on Windows I get:
(stream contents at: 6) codePoint 252 ('Teilrückzahlung' at: 6) codePoint 252
For
$ü codePoint I also get 252
So on Windows (at least for my test) it does not have the "leadingChar".
Bye T.
-- Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
Am 08.05.2012 um 16:20 schrieb Sven Van Caekenberghe:
<rant>Notice how f*cked up TextConverters are: when given a binary stream, their natural input one would think, they just don't do any conversion!</rant>
Isn't that obvious? In a time where there was text or binary and your biggest problem was line ending conversion it is normal to do not any processing on a binary stream. It is its definition. The world has changed since then and since we have encodings there is no such thing as a text/ascii stream anymore. I think we have to slowly convert those things to modern stuff. How hard that is? Look at Grease it is a piece of modern software but still encodes string -> string. There are always caveats not to ignore. Norbert
Norbert, On 08 May 2012, at 16:52, Norbert Hartl wrote:
Am 08.05.2012 um 16:20 schrieb Sven Van Caekenberghe:
<rant>Notice how f*cked up TextConverters are: when given a binary stream, their natural input one would think, they just don't do any conversion!</rant>
Isn't that obvious? In a time where there was text or binary and your biggest problem was line ending conversion it is normal to do not any processing on a binary stream. It is its definition. The world has changed since then and since we have encodings there is no such thing as a text/ascii stream anymore. I think we have to slowly convert those things to modern stuff. How hard that is? Look at Grease it is a piece of modern software but still encodes string -> string. There are always caveats not to ignore.
Right! In the very latest Zn I added ZnCharacterReadStream and ZnCharacterWriteStream but I did not yet integrate/use them. These add character decoding and encoding to binary streams, here are some tests that show how to use them: testSimpleUTF8WriteStream | string bytes stream | string := 'élève en Français'. bytes := ZnUTF8Encoder new encodeString: string. stream := (ZnCharacterWriteStream on: #[] writeStream). stream nextPutAll: string. self assert: stream wrappedStream contents equals: bytes testSimpleUTF8ReadStream | string bytes | string := 'élève en Français'. bytes := ZnUTF8Encoder new encodeString: string. self assert: (ZnCharacterReadStream on: bytes readStream) upToEnd equals: string I think these could be useful in other places (FS) as well and help in getting rid of some old stuff. Mind you, these are currently still only experiments. Sven -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
Thanks sven Indeed this would be good. Right now we will continue to fix the problems we are looking (event sensor, canvasâ¦.) at but this is good that people work on other parts of the system. Stef
<rant>Notice how f*cked up TextConverters are: when given a binary stream, their natural input one would think, they just don't do any conversion!</rant>
Isn't that obvious? In a time where there was text or binary and your biggest problem was line ending conversion it is normal to do not any processing on a binary stream. It is its definition. The world has changed since then and since we have encodings there is no such thing as a text/ascii stream anymore. I think we have to slowly convert those things to modern stuff. How hard that is? Look at Grease it is a piece of modern software but still encodes string -> string. There are always caveats not to ignore.
Right!
In the very latest Zn I added ZnCharacterReadStream and ZnCharacterWriteStream but I did not yet integrate/use them. These add character decoding and encoding to binary streams, here are some tests that show how to use them:
testSimpleUTF8WriteStream | string bytes stream | string := 'élève en Français'. bytes := ZnUTF8Encoder new encodeString: string. stream := (ZnCharacterWriteStream on: #[] writeStream). stream nextPutAll: string. self assert: stream wrappedStream contents equals: bytes
testSimpleUTF8ReadStream | string bytes | string := 'élève en Français'. bytes := ZnUTF8Encoder new encodeString: string. self assert: (ZnCharacterReadStream on: bytes readStream) upToEnd equals: string
I think these could be useful in other places (FS) as well and help in getting rid of some old stuff. Mind you, these are currently still only experiments.
Sven
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
participants (4)
-
Norbert Hartl -
Stéphane Ducasse -
Sven Van Caekenberghe -
Torsten Bergmann