Status: New Owner: ---- New issue 3360 by sven.van.caekenberghe: TextConverter handling of binary streams is wrong http://code.google.com/p/pharo/issues/detail?id=3360 It seems that the way binary (#isBinary true) streams are handled by TextConverter and its subclasses is wrong. When given a binary stream, the core text converter methods (#nextPut:toStream and #nextFromStream:) simply do no longer encode or decode at all. Moreover, the unit test UTF8TextConverter>>#testPutSingleCharacter seems plain wrong. The actual encoded bytes should be #[97 226 130 172]. However, this behavior seems to be added by design, so it is hard to estimate the impact of changing this. It is currently very ugly to get a binary UTF-8 encoding, one has to write to a character stream and then turn those characters into bytes. I wrote an alternative UTF-8 encoder as a support class to the Zinc HTTP Components (http://www.squeaksource.com/ZincHTTPComponents.html) together with the following unit test: testUTF8Encoder "The examples are taken from http://en.wikipedia.org/wiki/UTF-8#Description" | encoder inputBytes outputBytes inputString outputString | encoder := ZnUTF8Encoder new. inputString := String with: $$ with: (Unicode value: 16r00A2) with: (Unicode value: 16r20AC) with: (Unicode value: 16r024B62). inputBytes := #[16r24 16rC2 16rA2 16rE2 16r82 16rAC 16rF0 16rA4 16rAD 16rA2]. outputBytes := self encodeString: inputString with: encoder. self assert: outputBytes = inputBytes. outputString := self decodeBytes: inputBytes with: encoder. self assert: outputString = inputString based on the helper methods: encodeString: string with: encoder ^ ByteArray streamContents: [ :stream | string do: [ :each | encoder nextPut: each toStream: stream ] ] decodeBytes: bytes with: encoder | input | input := bytes readStream. ^ String streamContents: [ :stream | [ input atEnd ] whileFalse: [ stream nextPut: (encoder nextFromStream: input) ] ] The new encoder code is simpler, but might not handle everything that is needed (leading chars, language codes), but is all that still needed ? Sven