I wonder why we need back and oldBack (reallyOldBack ?) in the first place. Condensing sources or changes should just consist in writing source of each method/class comment sequentially (forward), and remember the position of the stream... So this backward thing must be some poorly written part. Ah perhaps we are reading the metadata (author etc...) back from source position, is that it ? If so, then it would be possible to point to the beginning of metadata, and skip forward... 2014-01-31 Sven Van Caekenberghe <sven@stfx.eu>:

Hi Pavel,

On 31 Jan 2014, at 20:33, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:

...

Hi,

I was looking why we cannot do condenseChanges and the reason looks to be clear: we do not have proper methods to move backwards (messages #back and #oldBack) in UTF-8 files. The current implementation changes position by one byte but in UTF-8 files it can be up to six bytes. And it's worse. We do not have mechanism how a text converter could handle this. Obvious solution is to rewrite condenseChanges to backup position somehow and do not move backwards at all but it doesn't seem to be so easy. Any ideas?

And why do we need working condenseChanges? Because when we unload non-kernel packages and load them back, the resultant changes file has about 51MB :-)

Cheers, -- Pavel

Very interesting!

Actually, I think this is fairly easy to do. Here is a prove of concept. It does not implement the exact semantics of either #back or #oldBack, but it does give the elementary building block to make it possible.

Adding the following method:

ZnUTF8Encoder>>backOnStream: stream [ (stream back bitAnd: 2r11000000) == 2r10000000 ] whileTrue

Makes this possible:

| encoder stream | encoder := ZnUTF8Encoder new. stream := (encoder encodeString: 'Les élèves Françaises') readStream. 4 timesRepeat: [ encoder nextFromStream: stream ]. encoder nextFromStream: stream. " => $é" encoder backOnStream: stream. encoder nextFromStream: stream. " => $é" 3 timesRepeat: [ encoder backOnStream: stream ]. encoder nextFromStream: stream. " => $s"

Implementing #back would then be something like:

| char | encoder backOnStream: stream. char := encoder nextFromStream: stream. encoder backOnStream: stream. ^ char

to simulate the #peek, but that might not be needed for the caller.

Of course, to do this for real would require a couple of good unit tests, as well as implementations for all encoders in the ZnCharacterEncoder hierarchy.

What do you think ?

Sven

-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill