I think that there are several issues:
1) Character with codePoint < 256 are unique, but Wider are not.
The characters are not stored in the WideString (which instead stores
the code points), but any enumeration using at: rather than wordAt:
will create an overhead. I presume that it would be better to have a
#codePointAt: defined as #byteAt: in ByteString and #wordAt: in
WideString, so we could still define generic methods in String.
2) A lot of ByteString methods are optimized with ad hoc optional
primitives, but very few WideString methods are.
3) Some of these optimized operations work with 256 element masks and
cannot be generalized to WideString that easily
4) Some Stream operations that copy one Collection to the stream
buffer or vice versa are also not optimized for mixed byte/word
collections
5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters
also are (because they can use more primitives in this case)
6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe
questionable from quality POV, but were not replaced from efficiency
POV.
I would say that a new VM with immediate Character values might help,
but that won't be enough.
Nicolas
2011/11/16 Sven Van Caekenberghe <sven@beta9.be>:
> Dennis,
>
> On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:
>
>> Hi!
>>
>> Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb file (136 lines) and several minutes to parse a 3.4Mb file (about 8K lines).
>>
>> Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case� So, to be short: is there a simple way to improve the performance? TIA
>>
>> --
>> Dennis Schetinin
>
> I am interested in this problem.
>
> Would it be possible to provide a test file (not that I can read Russian) and your test code ?
>
> I would like to try this myself.
>
> Sven
>