Hello, Where is converting strings from UTF8 encoding? 2011/11/17 Dennis Schetinin <chaetal@gmail.com>
Sure.
I've loaded CSVParser from http://www.squeaksource.com/CSVParser.html and patched one method there:
nextRow | row | row := OrderedCollection new. stream skipSeparators. [self atEndOfLine] whileFalse: [row add: self nextValue]. *"stream skip: -1."* stream next = $, ifTrue: [row add: '']. ^ row
This is how I test it:
stream := FileStream readOnlyFileNamed: 'Data.csv'. [(CSVParser on: stream) useDelimiter: $;; rows ] ensure: [stream close].
I attach the bigger data file. The shorter one can be produced by deleting rows.
2011/11/16 Sven Van Caekenberghe <sven@beta9.be>
Dennis,
On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:
Hi!
Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb file (136 lines) and several minutes to parse a 3.4Mb file (about 8K lines).
Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case⦠So, to be short: is there a simple way to improve the performance? TIA
-- Dennis Schetinin
I am interested in this problem.
Would it be possible to provide a test file (not that I can read Russian) and your test code ?
I would like to try this myself.
Sven
-- Dennis Schetinin