2011/11/17 Denis Kudriashov <dionisiydk@gmail.com>

Hello,

Where is converting strings from UTF8 encoding?

2011/11/17 Dennis Schetinin <chaetal@gmail.com>

Sure.�

I've loaded CSVParser from http://www.squeaksource.com/CSVParser.html and patched one method there:

nextRow

| row |
row := OrderedCollection new.

stream skipSeparators.
[self atEndOfLine]

whileFalse: [row add: self nextValue].
"stream skip: -1."

stream next = $, ifTrue: [row add: ''].
^ row

This is how I test it:

stream := FileStream readOnlyFileNamed: 'Data.csv'.
[(CSVParser on: stream)�
useDelimiter: $;;�

rows ] ensure: [stream close].

I attach the bigger data file. The shorter one can be produced by deleting rows.
�
2011/11/16 Sven Van Caekenberghe <sven@beta9.be>

Dennis,

On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:

> Hi!
>
> Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb file (136 lines) and several minutes to parse a 3.4Mb file (about 8K lines).
>
> Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case� So, to be short: is there a simple way to improve the performance? TIA
>
> --
> Dennis Schetinin

I am interested in this problem.

Would it be possible to provide a test file (not that I can read Russian) and your test code ?

I would like to try this myself.

Sven

--
Dennis Schetinin