2011/11/18 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
Also note that the more mainXtream Xtreams at http://www.squeaksource.com/Xtreams/ does not perform that bad...
This ones gets a score of 3.0 seconds with a regular MultiByteFileStream
nextRow     | row char line nextValue noValue |     line := stream nextLine reading.     row := OrderedCollection new.     nextValue := (WideString new: 32) writing.     noValue := true.
    [[[(char := line get) isSeparator] whileTrue.     noValue := false.     char = $" ifTrue:         [[nextValue write: (line ending: $") rest.         (char := line get) = $"] whileTrue:             [nextValue put: $"].         [char = separator] whileFalse: [char := line get]].     [char = separator]         whileFalse:             [nextValue put: char.             char := line get].     row add: nextValue conclusion.     nextValue := (WideString new: 32) writing] repeat]         on: Incomplete do: [:exc | ].
    noValue ifFalse: [row add: nextValue conclusion].     ^row
with a XTFileReadStream, just rewrite this:
MessageTally spyOn: [| stream separator | Â Â Â Â separator := $;. Â Â Â Â stream := (((FileDirectory on: '/Users/nicolas/Downloads/') / 'Data.csv') reading encoding: #utf8) "rest reading". Â Â Â Â (stream ending: Character cr) slicing collect: [:line | Â Â Â Â Â Â Â Â | row char nextValue noValue | Â Â Â Â Â Â Â Â row := OrderedCollection new. Â Â Â Â Â Â Â Â nextValue := (WideString new: 32) writing. Â Â Â Â Â Â Â Â noValue := true.
        [[[(char := line get) isSeparator] whileTrue.         noValue := false.         char = $" ifTrue:             [[nextValue write: (line ending: $") rest.             (char := line get) = $"] whileTrue:                 [nextValue put: $"].             [char = separator] whileFalse: [char := line get]].         [char = separator]             whileFalse:                 [nextValue put: char.                 char := line get].         row add: nextValue conclusion.         nextValue := (WideString new: 32) writing] repeat]             on: Incomplete do: [:exc | ].
        noValue ifFalse: [row add: nextValue conclusion].         row]]
For some reason, this was super slow (>20s), though my previous attempt without slicing was around 3.0s...
Ah Ah, replacing contentsSpecies with ^WideString in XTEncodeRead/WriteStream took it back to 3.2s. Nicolas
(so you noticed the commented "rest reading" whose purpose is to load the whole file in memory)
The major advantage is that you can just replace #collect: with #collecting: and get a stream of rows. Lazy is cool.
Nicolas
2011/11/18 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
2011/11/17 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
2011/11/17 Dennis Schetinin <chaetal@gmail.com>:
So, the answer for the question "Is there a simple way to improve the performance?" would be "No", right?
No, the bottle neck is CSVParser as Henrik and Levente already noticed.
My machine executes your code in 21.7s
So I decided to play with SqueaXTream http://www.squeaksource.com/XTream/ and inlined row decoding in a single method:
CSVParser>>nextRow     | row char line nextValue |     line := stream nextLine readXtream.     row := OrderedCollection new.     nextValue := (WideString new: 32) writeXtream.     line endOfStreamAction: [^row add: (nextValue contents); yourself].     [[(char := line next) isSeparator] whileTrue.     char = $" ifTrue:         [[nextValue nextPutAll: (stream upTo: $").
Err, it's line upTo: $"
        line peekFor: $"] whileTrue:             [nextValue nextPut: $"].         [(char := line next) = separator] whileFalse].     [char = separator]         whileFalse:             [nextValue nextPut: char.             char := line next].     row add: nextValue contents.     nextValue reset] repeat
This reduces execution time to 2.4 seconds
I also tried to use SqueaXTream for the file itself (if you don't mind the ugly invocation); But it does not gain much because you file is essentially made of Wide chars... It would help only if you'd have more ASCII fields...
MessageTally spyOn: [| stream | Â Â Â Â stream := (((FileDirectory on: '/Users/nicolas/Downloads/') / 'Data.csv') readXtream binary buffered ~> UTF8Decoder) buffered. Â Â Â Â [(CSVParser on: stream) Â Â Â Â Â Â Â Â useDelimiter: $;; Â Â Â Â Â Â Â Â rows ] ensure: [stream close]].
This reduces execution time to 1.9 seconds though
So I wouldn't say there's not plenty of room for performance ;)
Nicolas
2011/11/16 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>
I think that there are several issues:
1) Character with codePoint < 256 are unique, but Wider are not. The characters are not stored in the WideString (which instead stores the code points), but any enumeration using at: rather than wordAt: will create an overhead. I presume that it would be better to have a #codePointAt: defined as #byteAt: in ByteString and #wordAt: in WideString, so we could still define generic methods in String. 2) A lot of ByteString methods are optimized with ad hoc optional primitives, but very few WideString methods are. 3) Some of these optimized operations work with 256 element masks and cannot be generalized to WideString that easily 4) Some Stream operations that copy one Collection to the stream buffer or vice versa are also not optimized for mixed byte/word collections 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters also are (because they can use more primitives in this case) 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe questionable from quality POV, but were not replaced from efficiency POV.
I would say that a new VM with immediate Character values might help, but that won't be enough.
Nicolas
2011/11/16 Sven Van Caekenberghe <sven@beta9.be>:
Dennis,
On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:
Hi!
Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb file (136 lines) and several minutes to parse a 3.4Mb file (about 8K lines).
Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case⦠So, to be short: is there a simple way to improve the performance? TIA
-- Dennis Schetinin
I am interested in this problem.
Would it be possible to provide a test file (not that I can read Russian) and your test code ?
I would like to try this myself.
Sven
-- Dennis Schetinin