Hi Pavel,

On 31 Jan 2014, at 20:33, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:

> Hi,
>
> I was looking why we cannot do condenseChanges and the reason looks to be clear: we do not have proper methods to move backwards (messages #back and #oldBack) in UTF-8 files. The current implementation changes position by one byte but in UTF-8 files it can be up to six bytes. And it's worse. We do not have mechanism how a text converter could handle this.
> Obvious solution is to rewrite condenseChanges to backup position somehow and do not move backwards at all but it doesn't seem to be so easy.
> Any ideas?
>
> And why do we need working condenseChanges? Because when we unload non-kernel packages and load them back, the resultant changes file has about 51MB :-)
>
> Cheers,
> -- Pavel

Very interesting!

Actually, I think this is fairly easy to do. Here is a prove of concept. It does not implement the exact semantics of either #back or #oldBack, but it does give the elementary building block to make it possible.

Adding the following method:

ZnUTF8Encoder>>backOnStream: stream
� [ (stream back bitAnd: 2r11000000) == 2r10000000 ] whileTrue

Makes this possible:

| encoder stream |
encoder := ZnUTF8Encoder new.
stream := (encoder encodeString: 'Les �l�ves Fran�aises') readStream.
4 timesRepeat: [ encoder nextFromStream: stream ].
encoder nextFromStream: stream. " => $�"
encoder backOnStream: stream.
encoder nextFromStream: stream. " => $�"
3 timesRepeat: [ encoder backOnStream: stream ].
encoder nextFromStream: stream. " => $s"

Implementing #back would then be something like:

| char |
encoder backOnStream: stream.
char := encoder nextFromStream: stream.
encoder backOnStream: stream.
^ char

to simulate the #peek, but that might not be needed for the caller.

Of course, to do this for real would require a couple of good unit tests, as well as implementations for all encoders in the ZnCharacterEncoder hierarchy.

What do you think ?

Sven

--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill