> (so you noticed the commented "rest reading" whose purpose is to load
> the whole file in memory)
>
> The major advantage is that you can just replace #collect: with
> #collecting: and get a stream of rows. Lazy is cool.
>
> Nicolas
>
> 2011/11/18 Nicolas Cellier <
nicolas.cellier.aka.nice@gmail.com>:
>> 2011/11/17 Nicolas Cellier <
nicolas.cellier.aka.nice@gmail.com>:
>>> 2011/11/17 Dennis Schetinin <
chaetal@gmail.com>:
>>>> So, the answer for the question "Is there a simple way to improve the
>>>> performance?" would be "No", right?
>>>>
>>>
>>> No, the bottle neck is CSVParser as Henrik and Levente already noticed.
>>>
>>> My machine executes your code in 21.7s
>>>
>>> So I decided to play with SqueaXTream
http://www.squeaksource.com/XTream/
>>> and inlined row decoding in a single method:
>>>
>>> CSVParser>>nextRow
>>> � � � �| row char line nextValue |
>>> � � � �line := stream nextLine readXtream.
>>> � � � �row := OrderedCollection new.
>>> � � � �nextValue := (WideString new: 32) writeXtream.
>>> � � � �line endOfStreamAction: [^row add: (nextValue contents); yourself].
>>> � � � �[[(char := line next) isSeparator] whileTrue.
>>> � � � �char = $" ifTrue:
>>> � � � � � � � �[[nextValue nextPutAll: (stream upTo: $").
>>
>> Err, it's line upTo: $"
>>
>>> � � � � � � � �line peekFor: $"] whileTrue:
>>> � � � � � � � � � � � �[nextValue nextPut: $"].
>>> � � � � � � � �[(char := line next) = separator] whileFalse].
>>> � � � �[char = separator]
>>> � � � � � � � �whileFalse:
>>> � � � � � � � � � � � �[nextValue nextPut: char.
>>> � � � � � � � � � � � �char := line next].
>>> � � � �row add: nextValue contents.
>>> � � � �nextValue reset] repeat
>>>
>>> This reduces execution time to 2.4 seconds
>>>
>>> I also tried to use SqueaXTream for the file itself (if you don't mind
>>> the ugly invocation);
>>> But it does not gain much because you file is essentially made of Wide chars...
>>> It would help only if you'd have more ASCII fields...
>>>
>>> MessageTally spyOn: [| stream |
>>> � � � �stream := (((FileDirectory on: '/Users/nicolas/Downloads/') /
>>> 'Data.csv') readXtream binary buffered ~> UTF8Decoder) buffered.
>>> � � � �[(CSVParser on: stream)
>>> � � � � � � � �useDelimiter: $;;
>>> � � � � � � � �rows ] ensure: [stream close]].
>>>
>>> This reduces execution time to 1.9 seconds though
>>>
>>> So I wouldn't say there's not plenty of room for performance ;)
>>>
>>> Nicolas
>>>
>>>> 2011/11/16 Nicolas Cellier <
nicolas.cellier.aka.nice@gmail.com>
>>>>>
>>>>> I think that there are several issues:
>>>>>
>>>>> 1) Character with codePoint < 256 are unique, but Wider are not.
>>>>> The characters are not stored in the WideString (which instead stores
>>>>> the code points), but any enumeration using at: rather than wordAt:
>>>>> will create an overhead. I presume that it would be better to have a
>>>>> #codePointAt: defined as #byteAt: in ByteString and #wordAt: in
>>>>> WideString, so we could still define generic methods in String.
>>>>> 2) A lot of ByteString methods are optimized with ad hoc optional
>>>>> primitives, but very few WideString methods are.
>>>>> 3) Some of these optimized operations work with 256 element masks and
>>>>> cannot be generalized to WideString that easily
>>>>> 4) Some Stream operations that copy one Collection to the stream
>>>>> buffer or vice versa are also not optimized for mixed byte/word
>>>>> collections
>>>>> 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters
>>>>> also are (because they can use more primitives in this case)
>>>>> 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe
>>>>> questionable from quality POV, but were not replaced from efficiency
>>>>> POV.
>>>>>
>>>>> I would say that a new VM with immediate Character values might help,
>>>>> but that won't be enough.
>>>>>
>>>>> Nicolas
>>>>>
>>>>> 2011/11/16 Sven Van Caekenberghe <
sven@beta9.be>:
>>>>> > Dennis,
>>>>> >
>>>>> > On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:
>>>>> >
>>>>> >> Hi!
>>>>> >>
>>>>> >> Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I
>>>>> >> found it really slow. It takes about 5 seconds to read a 53Kb file (136
>>>>> >> lines) and several minutes to parse a 3.4Mb file (about 8K lines).
>>>>> >>
>>>>> >> Profiler shows that nearly all the time is eaten by WideString >>
>>>>> >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken,
>>>>> >> this means that <primitive: 66> there does not work in this case� So, to be
>>>>> >> short: is there a simple way to improve the performance? TIA
>>>>> >>
>>>>> >> --
>>>>> >> Dennis Schetinin
>>>>> >
>>>>> > I am interested in this problem.
>>>>> >
>>>>> > Would it be possible to provide a test file (not that I can read
>>>>> > Russian) and your test code ?
>>>>> >
>>>>> > I would like to try this myself.
>>>>> >
>>>>> > Sven
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Dennis Schetinin
>>>>
>>>
>>
>