MultiByteFileStream problem
There is an annoying error in MultiByteFileStream, reading back when you have a unicode character. It is also the cause of some FileOut errors. Your can reproduce it this way: testString := 'abcdé'. filename := 'test.txt'. filename asFileReference ensureDelete. filename asFileReference writeStreamDo: [ :stream | stream nextPutAll: testString ]. f := 'test.txt' asFileReference readStream. f setToEnd. f skip: -1. f peek. Any ideas how to solve it? Avoiding reading back is not an option ;). Maybe making skip to go back one more byte to check if there is a unicode character around?. Thanks a lot.
Hi José,
On 17 Nov 2014, at 02:06, José Comesaña <jose.comesana@gmail.com> wrote:
There is an annoying error in MultiByteFileStream, reading back when you have a unicode character. It is also the cause of some FileOut errors. Your can reproduce it this way:
testString := 'abcdé'. filename := 'test.txt'. filename asFileReference ensureDelete. filename asFileReference writeStreamDo: [ :stream | stream nextPutAll: testString ].
f := 'test.txt' asFileReference readStream.
f setToEnd.
f skip: -1.
f peek.
Any ideas how to solve it? Avoiding reading back is not an option ;). Maybe making skip to go back one more byte to check if there is a unicode character around?.
Thanks a lot.
This is a known problem/issue/limitation of MultiByteFileStream and the TextConverters. The more modern/logical ZnCharacterEncoders can do this reliably (provided the underlying stream can itself move backwards and the current position is between whole characters). 'test.txt' asFileReference readStreamDo: [ :in | in binary; setToEnd. ZnUTF8Encoder new backOnStream: in; nextFromStream: in ] IMHO, the ability to move backwards and to play with positions without restrictions on streams only makes sense with in-memory streams, not with file or network streams. It plays badly with buffering too. I know this doesn't solve your issue but I hope it gives you a better idea of the situation. Where exactly do you have this issue ? Sven
Thank you Sven. I an trying to read a veeeery long log file, and extract the last 5 messages for the user to see them. Reading the whole file from the beginning is too expensive. So maybe I could try reading backwards from the end using binary mode and look for a lf... I have to think about it a little bit, and study your proposal, of course. 2014-11-17 9:26 GMT+01:00 Sven Van Caekenberghe <sven@stfx.eu>:
Hi José,
On 17 Nov 2014, at 02:06, José Comesaña <jose.comesana@gmail.com> wrote:
There is an annoying error in MultiByteFileStream, reading back when you have a unicode character. It is also the cause of some FileOut errors. Your can reproduce it this way:
testString := 'abcdé'. filename := 'test.txt'. filename asFileReference ensureDelete. filename asFileReference writeStreamDo: [ :stream | stream nextPutAll: testString ].
f := 'test.txt' asFileReference readStream.
f setToEnd.
f skip: -1.
f peek.
Any ideas how to solve it? Avoiding reading back is not an option ;). Maybe making skip to go back one more byte to check if there is a unicode character around?.
Thanks a lot.
This is a known problem/issue/limitation of MultiByteFileStream and the TextConverters.
The more modern/logical ZnCharacterEncoders can do this reliably (provided the underlying stream can itself move backwards and the current position is between whole characters).
'test.txt' asFileReference readStreamDo: [ :in | in binary; setToEnd. ZnUTF8Encoder new backOnStream: in; nextFromStream: in ]
IMHO, the ability to move backwards and to play with positions without restrictions on streams only makes sense with in-memory streams, not with file or network streams. It plays badly with buffering too.
I know this doesn't solve your issue but I hope it gives you a better idea of the situation.
Where exactly do you have this issue ?
Sven
I do not know which OS you are on. But I have solved a similar problem by calling tail -5 through OSProcess on Linux for a quick fix. tac file can also help to read a file in reverse. Phil On Mon, Nov 17, 2014 at 10:20 AM, José Comesaña <jose.comesana@gmail.com> wrote:
Thank you Sven.
I an trying to read a veeeery long log file, and extract the last 5 messages for the user to see them. Reading the whole file from the beginning is too expensive. So maybe I could try reading backwards from the end using binary mode and look for a lf... I have to think about it a little bit, and study your proposal, of course.
2014-11-17 9:26 GMT+01:00 Sven Van Caekenberghe <sven@stfx.eu>:
Hi José,
On 17 Nov 2014, at 02:06, José Comesaña <jose.comesana@gmail.com> wrote:
There is an annoying error in MultiByteFileStream, reading back when you have a unicode character. It is also the cause of some FileOut errors. Your can reproduce it this way:
testString := 'abcdé'. filename := 'test.txt'. filename asFileReference ensureDelete. filename asFileReference writeStreamDo: [ :stream | stream nextPutAll: testString ].
f := 'test.txt' asFileReference readStream.
f setToEnd.
f skip: -1.
f peek.
Any ideas how to solve it? Avoiding reading back is not an option ;). Maybe making skip to go back one more byte to check if there is a unicode character around?.
Thanks a lot.
This is a known problem/issue/limitation of MultiByteFileStream and the TextConverters.
The more modern/logical ZnCharacterEncoders can do this reliably (provided the underlying stream can itself move backwards and the current position is between whole characters).
'test.txt' asFileReference readStreamDo: [ :in | in binary; setToEnd. ZnUTF8Encoder new backOnStream: in; nextFromStream: in ]
IMHO, the ability to move backwards and to play with positions without restrictions on streams only makes sense with in-memory streams, not with file or network streams. It plays badly with buffering too.
I know this doesn't solve your issue but I hope it gives you a better idea of the situation.
Where exactly do you have this issue ?
Sven
Thank you for the idea Phil, but I'd like the same code to work both on OSX and Windows. I don't like calling OS commands, either (my fault, I know...) 2014-11-17 11:30 GMT+01:00 phil@highoctane.be <phil@highoctane.be>:
I do not know which OS you are on.
But I have solved a similar problem by calling tail -5 through OSProcess on Linux for a quick fix.
tac file can also help to read a file in reverse.
Phil
On Mon, Nov 17, 2014 at 10:20 AM, José Comesaña <jose.comesana@gmail.com> wrote:
Thank you Sven.
I an trying to read a veeeery long log file, and extract the last 5 messages for the user to see them. Reading the whole file from the beginning is too expensive. So maybe I could try reading backwards from the end using binary mode and look for a lf... I have to think about it a little bit, and study your proposal, of course.
2014-11-17 9:26 GMT+01:00 Sven Van Caekenberghe <sven@stfx.eu>:
Hi José,
On 17 Nov 2014, at 02:06, José Comesaña <jose.comesana@gmail.com> wrote:
There is an annoying error in MultiByteFileStream, reading back when you have a unicode character. It is also the cause of some FileOut errors. Your can reproduce it this way:
testString := 'abcdé'. filename := 'test.txt'. filename asFileReference ensureDelete. filename asFileReference writeStreamDo: [ :stream | stream nextPutAll: testString ].
f := 'test.txt' asFileReference readStream.
f setToEnd.
f skip: -1.
f peek.
Any ideas how to solve it? Avoiding reading back is not an option ;). Maybe making skip to go back one more byte to check if there is a unicode character around?.
Thanks a lot.
This is a known problem/issue/limitation of MultiByteFileStream and the TextConverters.
The more modern/logical ZnCharacterEncoders can do this reliably (provided the underlying stream can itself move backwards and the current position is between whole characters).
'test.txt' asFileReference readStreamDo: [ :in | in binary; setToEnd. ZnUTF8Encoder new backOnStream: in; nextFromStream: in ]
IMHO, the ability to move backwards and to play with positions without restrictions on streams only makes sense with in-memory streams, not with file or network streams. It plays badly with buffering too.
I know this doesn't solve your issue but I hope it gives you a better idea of the situation.
Where exactly do you have this issue ?
Sven
participants (3)
-
José Comesaña -
phil＠highoctane.be -
Sven Van Caekenberghe