On 16.11.2011 21:45, Nicolas Cellier wrote:
I think that there are several issues:
1) Character with codePoint< 256 are unique, but Wider are not. The characters are not stored in the WideString (which instead stores the code points), but any enumeration using at: rather than wordAt: will create an overhead. I presume that it would be better to have a #codePointAt: defined as #byteAt: in ByteString and #wordAt: in WideString, so we could still define generic methods in String. 2) A lot of ByteString methods are optimized with ad hoc optional primitives, but very few WideString methods are. 3) Some of these optimized operations work with 256 element masks and cannot be generalized to WideString that easily 4) Some Stream operations that copy one Collection to the stream buffer or vice versa are also not optimized for mixed byte/word collections 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters also are (because they can use more primitives in this case) 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe questionable from quality POV, but were not replaced from efficiency POV.
I would say that a new VM with immediate Character values might help, but that won't be enough.
Nicolas If I may object to point 6: (Squeak) [FileStream oldFileNamed: 'Data.csv' do: [:stream | text := stream contents]] timeToRun 6753 (Pharo) [FileStream oldFileNamed: 'Data.csv' do: [:stream | text := stream contents]] timeToRun 7027 Reading without conversion, then converting with utf8ToSqueak: [FileStream oldFileNamed: 'C:\Users\Rydier\Dropbox\Pharo 1.4 latest jenkins\Data.csv' do: [:stream | stream converter: Latin1TextConverter new. text := stream contents utf8ToSqueak ]] timeToRun 9752
The efficiency drain in this example is, as Dennis points out, CSVParser adding scanned characters 1 by 1 to the WideString representing current value. Cheers, Henry