On 28 August 2012 14:45, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:

On 28.08.2012 14:35, Igor Stasenko wrote:

...

or it was like that from the birth??

arrrrgghhhh...

| stream | stream := WriteStream on: (ByteArray new: 100).

UTF8TextConverter new nextPut: (Character value: 129 ) toStream: stream.

stream contents #[129]

This is WRONG! RTFM, about utf8 encoding, please! :)

---------

nextPut: aCharacter toStream: aStream | leadingChar nBytes mask shift ucs2code | aStream isBinary ifTrue: [^aCharacter storeBinaryOn: aStream].

in my case, stream is binary, so it goes directly to #storeBinaryOn:

storeBinaryOn: aStream "Store the receiver on a binary (file) stream" value < 256 ifTrue:[aStream basicNextPut: value] ifFalse:[aStream nextInt32Put: value].

This is not even close to UTF8. If character code is less than 256, it will store a single byte (wtf?), and if more than that, it will store 32-bit integer value in big-endian order (wtf raisedToPower: 2)..

i wonder , for what purpose we actually having this code path? this stuff is completely useless. according to implementation of storeBinaryOn: there's no way how you can read the same character value back. because it can be 1 byte or 4 bytes.. but you simply cannot determine which one. this is one of the reasons we using utf8 encoding, btw ;)

It's been this way from the start.

so, it was broken from the start? the point is, that i am talking specifically about UTF8Converter... i don't care about any grade of polymorphism/megamorphism/whatevermorphism inherited from TextConverters.. but by simplest logic, my ignorant expectation was that UTF8Converter should convert its input into utf8 form or die trying.. but not something which not even remotely related to utf8.. wtf? Because if i don't need it to be converted to utf8, then why i would bother using it in a first place?

You want to sanely encode to in-memory bytearrays, use the Zn converters. You want to conveniently switch between putting binary and encoded contents to a filestream (and do so efficiently), use the TextConverters.

conveniently? are you joking? :) conveniently is when i say things once, either: newstream := converter on: stream. x timesRepeat: [ newstream nextPut: oneElement ]. or stream addConverter: myconverter. x timesRepeat: [ stream nextPut: oneElement ]. but instead i forced to use it like: x timesRepeat: [ converter nextPut: oneElement onStream: stream ] it is neither convenient, nor optimal, because you operating with three entities in a loop: - stream, converter and element, while you really need only two. But aside of that, is there a reason why we must have 2 implementations of very same thing in our 'clean' system? If we have that , i think, we should stop calling it clean :) Sven told, why he implemented his own - just because the existing one is inherently broken (well, it was more polite form if i remember). Maybe Sven is too polite to put it straight or don't wants to shake the very foundations of our beautiful text conversion ecosystem.. But i don't mind to say it: TextConverters is piece of crap and we need to throw it out and replaced with something better or review/refactor/(choose your path).. So if some of you feel offended, i am sorry. You're a good guy/girl, who produced this code. The problem is that code doesn't follows "The principle of a least wtf". :) -- Best regards, Igor Stasenko.

Henrik Sperre Johansen wrote

their extensive usage limits the space for actual improvements within the space of backwards compatability.

http://code.google.com/p/pharo/#Pharo_manifesto :

Pharo manifesto - Better for the better - Beauty to learn from - Not backward compatible *** - Clean, lean and fast

Every wtf we leave in the system limits the human expression of ourselves and our users. For one thing, aesthetics count. Emotions are hugely important to the quality of one's work, and unless we are all inspired by a beautiful system, our discontent will be reflected in code written in and on top of Pharo. Also, while crusty old Pharo wizards may know where all these little land mines are that must be avoided, new users may waste hours/days before finding out that things don't work the way one would naturally expect them too. If we can operate on RPackage in a live system, breaking code committing and loading, we can change *anything*. I've definitely felt the pain in 2.0, but this is what we're working for. Let's rip the bandaid off and have the system of our dreams. If anything, I believe the question should *never* be "do we leave this crap in the system or not?", but merely "how do we ease the pain of making this better?", whether via deprecations, RB transformations, or whatever; and we seem to be making steady progress on that front. -- View this message in context: http://forum.world.st/Who-broke-UTF8TextConverter-tp4645473p4646399.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.

Yes !

...

Pharo manifesto - Better for the better - Beauty to learn from - Not backward compatible *** - Clean, lean and fast

Every wtf we leave in the system limits the human expression of ourselves and our users. For one thing, aesthetics count. Emotions are hugely important to the quality of one's work, and unless we are all inspired by a beautiful system, our discontent will be reflected in code written in and on top of Pharo. Also, while crusty old Pharo wizards may know where all these little land mines are that must be avoided, new users may waste hours/days before finding out that things don't work the way one would naturally expect them too.

If we can operate on RPackage in a live system, breaking code committing and loading, we can change *anything*. I've definitely felt the pain in 2.0, but this is what we're working for. Let's rip the bandaid off and have the system of our dreams. If anything, I believe the question should *never* be "do we leave this crap in the system or not?", but merely "how do we ease the pain of making this better?", whether via deprecations, RB transformations, or whatever; and we seem to be making steady progress on that front.

-- View this message in context: http://forum.world.st/Who-broke-UTF8TextConverter-tp4645473p4646399.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.

On 6 September 2012 13:48, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:

On 28.08.2012 23:11, Igor Stasenko wrote:

...

On 28 August 2012 14:45, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:

...

On 28.08.2012 14:35, Igor Stasenko wrote:

...

or it was like that from the birth??

arrrrgghhhh...

| stream | stream := WriteStream on: (ByteArray new: 100).

UTF8TextConverter new nextPut: (Character value: 129 ) toStream: stream.

stream contents #[129]

This is WRONG! RTFM, about utf8 encoding, please! :)

---------

nextPut: aCharacter toStream: aStream | leadingChar nBytes mask shift ucs2code | aStream isBinary ifTrue: [^aCharacter storeBinaryOn: aStream].

in my case, stream is binary, so it goes directly to #storeBinaryOn:

storeBinaryOn: aStream "Store the receiver on a binary (file) stream" value < 256 ifTrue:[aStream basicNextPut: value] ifFalse:[aStream nextInt32Put: value].

This is not even close to UTF8. If character code is less than 256, it will store a single byte (wtf?), and if more than that, it will store 32-bit integer value in big-endian order (wtf raisedToPower: 2)..

i wonder , for what purpose we actually having this code path? this stuff is completely useless. according to implementation of storeBinaryOn: there's no way how you can read the same character value back. because it can be 1 byte or 4 bytes.. but you simply cannot determine which one. this is one of the reasons we using utf8 encoding, btw ;)

It's been this way from the start.

so, it was broken from the start? the point is, that i am talking specifically about UTF8Converter... i don't care about any grade of polymorphism/megamorphism/whatevermorphism inherited from TextConverters.. but by simplest logic, my ignorant expectation was that UTF8Converter should convert its input into utf8 form or die trying.. but not something which not even remotely related to utf8.. wtf? Because if i don't need it to be converted to utf8, then why i would bother using it in a first place?

...

You want to sanely encode to in-memory bytearrays, use the Zn converters. You want to conveniently switch between putting binary and encoded contents to a filestream (and do so efficiently), use the TextConverters.

conveniently? are you joking? :) conveniently is when i say things once, either:

newstream := converter on: stream. x timesRepeat: [ newstream nextPut: oneElement ]. or

stream addConverter: myconverter. x timesRepeat: [ stream nextPut: oneElement ].

but instead i forced to use it like:

x timesRepeat: [ converter nextPut: oneElement onStream: stream ]

it is neither convenient, nor optimal, because you operating with three entities in a loop: - stream, converter and element, while you really need only two.

But aside of that, is there a reason why we must have 2 implementations of very same thing in our 'clean' system? If we have that , i think, we should stop calling it clean :)

Sven told, why he implemented his own - just because the existing one is inherently broken (well, it was more polite form if i remember).

Maybe Sven is too polite to put it straight or don't wants to shake the very foundations of our beautiful text conversion ecosystem.. But i don't mind to say it: TextConverters is piece of crap and we need to throw it out and replaced with something better or review/refactor/(choose your path)..

So if some of you feel offended, i am sorry. You're a good guy/girl, who produced this code. The problem is that code doesn't follows "The principle of a least wtf". :)

I originally wrote a detailed response, but gave up after awhile.

Let me just say that, at it's core, I don't think it's so much a problem of believing the current situation is good, but that the inextricable link between converters and filestreams, and how their extensive usage limits the space for actual improvements within the space of backwards compatability. (for a related example, there are multiple codepaths that rely on #binary not actually doing any encoding)

For a definition of better that means "less wtf's per line of code read/spent trying to use these things", I for one don't see a way for big improvements that does not involve breaking huge amounts of old code. At that point, a clean break to a more sensible architecture like XTreams, with clear separation of streaming/encoding responsibilites becomes a more attractive alternative in the first place.

Yes, i agree, that the situation with streams is not simple, where you can swipe them out of a system and replace with better stuff. But we should not tolerate the status quo and do some little steps for having better system in future. Slowly but steady. And one of the first steps, is to prevent new code from using the stuff we want to remove. When users asking what they should use for conversions, we should inform them to not use TextConverters :)

Creating backwards-compatability protols to a new framework might help somewhat, but it's still a massive undertaking which would cause huge amounts of pain to users. Sort of like the Settings-switch really, but at a much larger scale.

And so we keep on encountering a few wtf's once in a while, learn to deal with them, and then move on with our lives while hoping someone else picks up the torch :)

Cheers, Henry

-- Best regards, Igor Stasenko.