[Pharo-project] WideString performance
Hi! Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb file (136 lines) and several minutes to parse a 3.4Mb file (about 8K lines). Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case⦠So, to be short: is there a simple way to improve the performance? TIA -- Dennis Schetinin
Dennis, On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:
Hi!
Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb file (136 lines) and several minutes to parse a 3.4Mb file (about 8K lines).
Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case⦠So, to be short: is there a simple way to improve the performance? TIA
-- Dennis Schetinin
I am interested in this problem. Would it be possible to provide a test file (not that I can read Russian) and your test code ? I would like to try this myself. Sven
I think that there are several issues: 1) Character with codePoint < 256 are unique, but Wider are not. The characters are not stored in the WideString (which instead stores the code points), but any enumeration using at: rather than wordAt: will create an overhead. I presume that it would be better to have a #codePointAt: defined as #byteAt: in ByteString and #wordAt: in WideString, so we could still define generic methods in String. 2) A lot of ByteString methods are optimized with ad hoc optional primitives, but very few WideString methods are. 3) Some of these optimized operations work with 256 element masks and cannot be generalized to WideString that easily 4) Some Stream operations that copy one Collection to the stream buffer or vice versa are also not optimized for mixed byte/word collections 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters also are (because they can use more primitives in this case) 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe questionable from quality POV, but were not replaced from efficiency POV. I would say that a new VM with immediate Character values might help, but that won't be enough. Nicolas 2011/11/16 Sven Van Caekenberghe <sven@beta9.be>:
Dennis,
On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:
Hi!
Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb file (136 lines) and several minutes to parse a 3.4Mb file (about 8K lines).
Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case⦠So, to be short: is there a simple way to improve the performance? TIA
-- Dennis Schetinin
I am interested in this problem.
Would it be possible to provide a test file (not that I can read Russian) and your test code ?
I would like to try this myself.
Sven
So, the answer for the question "Is there a simple way to improve the performance?" would be "No", right? 2011/11/16 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>
I think that there are several issues:
1) Character with codePoint < 256 are unique, but Wider are not. The characters are not stored in the WideString (which instead stores the code points), but any enumeration using at: rather than wordAt: will create an overhead. I presume that it would be better to have a #codePointAt: defined as #byteAt: in ByteString and #wordAt: in WideString, so we could still define generic methods in String. 2) A lot of ByteString methods are optimized with ad hoc optional primitives, but very few WideString methods are. 3) Some of these optimized operations work with 256 element masks and cannot be generalized to WideString that easily 4) Some Stream operations that copy one Collection to the stream buffer or vice versa are also not optimized for mixed byte/word collections 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters also are (because they can use more primitives in this case) 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe questionable from quality POV, but were not replaced from efficiency POV.
I would say that a new VM with immediate Character values might help, but that won't be enough.
Nicolas
2011/11/16 Sven Van Caekenberghe <sven@beta9.be>:
Dennis,
On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:
Hi!
Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb file (136 lines) and several minutes to parse a 3.4Mb file (about 8K lines).
Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case⦠So, to be short: is there a simple way to improve the performance? TIA
-- Dennis Schetinin
I am interested in this problem.
Would it be possible to provide a test file (not that I can read Russian) and your test code ?
I would like to try this myself.
Sven
-- Dennis Schetinin
2011/11/17 Dennis Schetinin <chaetal@gmail.com>:
So, the answer for the question "Is there a simple way to improve the performance?" would be "No", right?
No, the bottle neck is CSVParser as Henrik and Levente already noticed. My machine executes your code in 21.7s So I decided to play with SqueaXTream http://www.squeaksource.com/XTream/ and inlined row decoding in a single method: CSVParser>>nextRow | row char line nextValue | line := stream nextLine readXtream. row := OrderedCollection new. nextValue := (WideString new: 32) writeXtream. line endOfStreamAction: [^row add: (nextValue contents); yourself]. [[(char := line next) isSeparator] whileTrue. char = $" ifTrue: [[nextValue nextPutAll: (stream upTo: $"). line peekFor: $"] whileTrue: [nextValue nextPut: $"]. [(char := line next) = separator] whileFalse]. [char = separator] whileFalse: [nextValue nextPut: char. char := line next]. row add: nextValue contents. nextValue reset] repeat This reduces execution time to 2.4 seconds I also tried to use SqueaXTream for the file itself (if you don't mind the ugly invocation); But it does not gain much because you file is essentially made of Wide chars... It would help only if you'd have more ASCII fields... MessageTally spyOn: [| stream | stream := (((FileDirectory on: '/Users/nicolas/Downloads/') / 'Data.csv') readXtream binary buffered ~> UTF8Decoder) buffered. [(CSVParser on: stream) useDelimiter: $;; rows ] ensure: [stream close]]. This reduces execution time to 1.9 seconds though So I wouldn't say there's not plenty of room for performance ;) Nicolas
2011/11/16 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>
I think that there are several issues:
1) Character with codePoint < 256 are unique, but Wider are not. The characters are not stored in the WideString (which instead stores the code points), but any enumeration using at: rather than wordAt: will create an overhead. I presume that it would be better to have a #codePointAt: defined as #byteAt: in ByteString and #wordAt: in WideString, so we could still define generic methods in String. 2) A lot of ByteString methods are optimized with ad hoc optional primitives, but very few WideString methods are. 3) Some of these optimized operations work with 256 element masks and cannot be generalized to WideString that easily 4) Some Stream operations that copy one Collection to the stream buffer or vice versa are also not optimized for mixed byte/word collections 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters also are (because they can use more primitives in this case) 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe questionable from quality POV, but were not replaced from efficiency POV.
I would say that a new VM with immediate Character values might help, but that won't be enough.
Nicolas
2011/11/16 Sven Van Caekenberghe <sven@beta9.be>:
Dennis,
On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:
Hi!
Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb file (136 lines) and several minutes to parse a 3.4Mb file (about 8K lines).
Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case⦠So, to be short: is there a simple way to improve the performance? TIA
-- Dennis Schetinin
I am interested in this problem.
Would it be possible to provide a test file (not that I can read Russian) and your test code ?
I would like to try this myself.
Sven
-- Dennis Schetinin
2011/11/17 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
2011/11/17 Dennis Schetinin <chaetal@gmail.com>:
So, the answer for the question "Is there a simple way to improve the performance?" would be "No", right?
No, the bottle neck is CSVParser as Henrik and Levente already noticed.
My machine executes your code in 21.7s
So I decided to play with SqueaXTream http://www.squeaksource.com/XTream/ and inlined row decoding in a single method:
CSVParser>>nextRow     | row char line nextValue |     line := stream nextLine readXtream.     row := OrderedCollection new.     nextValue := (WideString new: 32) writeXtream.     line endOfStreamAction: [^row add: (nextValue contents); yourself].     [[(char := line next) isSeparator] whileTrue.     char = $" ifTrue:         [[nextValue nextPutAll: (stream upTo: $").
Err, it's line upTo: $"
        line peekFor: $"] whileTrue:             [nextValue nextPut: $"].         [(char := line next) = separator] whileFalse].     [char = separator]         whileFalse:             [nextValue nextPut: char.             char := line next].     row add: nextValue contents.     nextValue reset] repeat
This reduces execution time to 2.4 seconds
I also tried to use SqueaXTream for the file itself (if you don't mind the ugly invocation); But it does not gain much because you file is essentially made of Wide chars... It would help only if you'd have more ASCII fields...
MessageTally spyOn: [| stream | Â Â Â Â stream := (((FileDirectory on: '/Users/nicolas/Downloads/') / 'Data.csv') readXtream binary buffered ~> UTF8Decoder) buffered. Â Â Â Â [(CSVParser on: stream) Â Â Â Â Â Â Â Â useDelimiter: $;; Â Â Â Â Â Â Â Â rows ] ensure: [stream close]].
This reduces execution time to 1.9 seconds though
So I wouldn't say there's not plenty of room for performance ;)
Nicolas
2011/11/16 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>
I think that there are several issues:
1) Character with codePoint < 256 are unique, but Wider are not. The characters are not stored in the WideString (which instead stores the code points), but any enumeration using at: rather than wordAt: will create an overhead. I presume that it would be better to have a #codePointAt: defined as #byteAt: in ByteString and #wordAt: in WideString, so we could still define generic methods in String. 2) A lot of ByteString methods are optimized with ad hoc optional primitives, but very few WideString methods are. 3) Some of these optimized operations work with 256 element masks and cannot be generalized to WideString that easily 4) Some Stream operations that copy one Collection to the stream buffer or vice versa are also not optimized for mixed byte/word collections 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters also are (because they can use more primitives in this case) 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe questionable from quality POV, but were not replaced from efficiency POV.
I would say that a new VM with immediate Character values might help, but that won't be enough.
Nicolas
2011/11/16 Sven Van Caekenberghe <sven@beta9.be>:
Dennis,
On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:
Hi!
Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb file (136 lines) and several minutes to parse a 3.4Mb file (about 8K lines).
Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case⦠So, to be short: is there a simple way to improve the performance? TIA
-- Dennis Schetinin
I am interested in this problem.
Would it be possible to provide a test file (not that I can read Russian) and your test code ?
I would like to try this myself.
Sven
-- Dennis Schetinin
Also note that the more mainXtream Xtreams at http://www.squeaksource.com/Xtreams/ does not perform that bad... This ones gets a score of 3.0 seconds with a regular MultiByteFileStream nextRow | row char line nextValue noValue | line := stream nextLine reading. row := OrderedCollection new. nextValue := (WideString new: 32) writing. noValue := true. [[[(char := line get) isSeparator] whileTrue. noValue := false. char = $" ifTrue: [[nextValue write: (line ending: $") rest. (char := line get) = $"] whileTrue: [nextValue put: $"]. [char = separator] whileFalse: [char := line get]]. [char = separator] whileFalse: [nextValue put: char. char := line get]. row add: nextValue conclusion. nextValue := (WideString new: 32) writing] repeat] on: Incomplete do: [:exc | ]. noValue ifFalse: [row add: nextValue conclusion]. ^row with a XTFileReadStream, just rewrite this: MessageTally spyOn: [| stream separator | separator := $;. stream := (((FileDirectory on: '/Users/nicolas/Downloads/') / 'Data.csv') reading encoding: #utf8) "rest reading". (stream ending: Character cr) slicing collect: [:line | | row char nextValue noValue | row := OrderedCollection new. nextValue := (WideString new: 32) writing. noValue := true. [[[(char := line get) isSeparator] whileTrue. noValue := false. char = $" ifTrue: [[nextValue write: (line ending: $") rest. (char := line get) = $"] whileTrue: [nextValue put: $"]. [char = separator] whileFalse: [char := line get]]. [char = separator] whileFalse: [nextValue put: char. char := line get]. row add: nextValue conclusion. nextValue := (WideString new: 32) writing] repeat] on: Incomplete do: [:exc | ]. noValue ifFalse: [row add: nextValue conclusion]. row]] For some reason, this was super slow (>20s), though my previous attempt without slicing was around 3.0s... (so you noticed the commented "rest reading" whose purpose is to load the whole file in memory) The major advantage is that you can just replace #collect: with #collecting: and get a stream of rows. Lazy is cool. Nicolas 2011/11/18 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
2011/11/17 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
2011/11/17 Dennis Schetinin <chaetal@gmail.com>:
So, the answer for the question "Is there a simple way to improve the performance?" would be "No", right?
No, the bottle neck is CSVParser as Henrik and Levente already noticed.
My machine executes your code in 21.7s
So I decided to play with SqueaXTream http://www.squeaksource.com/XTream/ and inlined row decoding in a single method:
CSVParser>>nextRow     | row char line nextValue |     line := stream nextLine readXtream.     row := OrderedCollection new.     nextValue := (WideString new: 32) writeXtream.     line endOfStreamAction: [^row add: (nextValue contents); yourself].     [[(char := line next) isSeparator] whileTrue.     char = $" ifTrue:         [[nextValue nextPutAll: (stream upTo: $").
Err, it's line upTo: $"
        line peekFor: $"] whileTrue:             [nextValue nextPut: $"].         [(char := line next) = separator] whileFalse].     [char = separator]         whileFalse:             [nextValue nextPut: char.             char := line next].     row add: nextValue contents.     nextValue reset] repeat
This reduces execution time to 2.4 seconds
I also tried to use SqueaXTream for the file itself (if you don't mind the ugly invocation); But it does not gain much because you file is essentially made of Wide chars... It would help only if you'd have more ASCII fields...
MessageTally spyOn: [| stream | Â Â Â Â stream := (((FileDirectory on: '/Users/nicolas/Downloads/') / 'Data.csv') readXtream binary buffered ~> UTF8Decoder) buffered. Â Â Â Â [(CSVParser on: stream) Â Â Â Â Â Â Â Â useDelimiter: $;; Â Â Â Â Â Â Â Â rows ] ensure: [stream close]].
This reduces execution time to 1.9 seconds though
So I wouldn't say there's not plenty of room for performance ;)
Nicolas
2011/11/16 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>
I think that there are several issues:
1) Character with codePoint < 256 are unique, but Wider are not. The characters are not stored in the WideString (which instead stores the code points), but any enumeration using at: rather than wordAt: will create an overhead. I presume that it would be better to have a #codePointAt: defined as #byteAt: in ByteString and #wordAt: in WideString, so we could still define generic methods in String. 2) A lot of ByteString methods are optimized with ad hoc optional primitives, but very few WideString methods are. 3) Some of these optimized operations work with 256 element masks and cannot be generalized to WideString that easily 4) Some Stream operations that copy one Collection to the stream buffer or vice versa are also not optimized for mixed byte/word collections 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters also are (because they can use more primitives in this case) 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe questionable from quality POV, but were not replaced from efficiency POV.
I would say that a new VM with immediate Character values might help, but that won't be enough.
Nicolas
2011/11/16 Sven Van Caekenberghe <sven@beta9.be>:
Dennis,
On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:
Hi!
Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb file (136 lines) and several minutes to parse a 3.4Mb file (about 8K lines).
Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case⦠So, to be short: is there a simple way to improve the performance? TIA
-- Dennis Schetinin
I am interested in this problem.
Would it be possible to provide a test file (not that I can read Russian) and your test code ?
I would like to try this myself.
Sven
-- Dennis Schetinin
2011/11/18 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
Also note that the more mainXtream Xtreams at http://www.squeaksource.com/Xtreams/ does not perform that bad...
This ones gets a score of 3.0 seconds with a regular MultiByteFileStream
nextRow     | row char line nextValue noValue |     line := stream nextLine reading.     row := OrderedCollection new.     nextValue := (WideString new: 32) writing.     noValue := true.
    [[[(char := line get) isSeparator] whileTrue.     noValue := false.     char = $" ifTrue:         [[nextValue write: (line ending: $") rest.         (char := line get) = $"] whileTrue:             [nextValue put: $"].         [char = separator] whileFalse: [char := line get]].     [char = separator]         whileFalse:             [nextValue put: char.             char := line get].     row add: nextValue conclusion.     nextValue := (WideString new: 32) writing] repeat]         on: Incomplete do: [:exc | ].
    noValue ifFalse: [row add: nextValue conclusion].     ^row
with a XTFileReadStream, just rewrite this:
MessageTally spyOn: [| stream separator | Â Â Â Â separator := $;. Â Â Â Â stream := (((FileDirectory on: '/Users/nicolas/Downloads/') / 'Data.csv') reading encoding: #utf8) "rest reading". Â Â Â Â (stream ending: Character cr) slicing collect: [:line | Â Â Â Â Â Â Â Â | row char nextValue noValue | Â Â Â Â Â Â Â Â row := OrderedCollection new. Â Â Â Â Â Â Â Â nextValue := (WideString new: 32) writing. Â Â Â Â Â Â Â Â noValue := true.
        [[[(char := line get) isSeparator] whileTrue.         noValue := false.         char = $" ifTrue:             [[nextValue write: (line ending: $") rest.             (char := line get) = $"] whileTrue:                 [nextValue put: $"].             [char = separator] whileFalse: [char := line get]].         [char = separator]             whileFalse:                 [nextValue put: char.                 char := line get].         row add: nextValue conclusion.         nextValue := (WideString new: 32) writing] repeat]             on: Incomplete do: [:exc | ].
        noValue ifFalse: [row add: nextValue conclusion].         row]]
For some reason, this was super slow (>20s), though my previous attempt without slicing was around 3.0s...
Ah Ah, replacing contentsSpecies with ^WideString in XTEncodeRead/WriteStream took it back to 3.2s. Nicolas
(so you noticed the commented "rest reading" whose purpose is to load the whole file in memory)
The major advantage is that you can just replace #collect: with #collecting: and get a stream of rows. Lazy is cool.
Nicolas
2011/11/18 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
2011/11/17 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
2011/11/17 Dennis Schetinin <chaetal@gmail.com>:
So, the answer for the question "Is there a simple way to improve the performance?" would be "No", right?
No, the bottle neck is CSVParser as Henrik and Levente already noticed.
My machine executes your code in 21.7s
So I decided to play with SqueaXTream http://www.squeaksource.com/XTream/ and inlined row decoding in a single method:
CSVParser>>nextRow     | row char line nextValue |     line := stream nextLine readXtream.     row := OrderedCollection new.     nextValue := (WideString new: 32) writeXtream.     line endOfStreamAction: [^row add: (nextValue contents); yourself].     [[(char := line next) isSeparator] whileTrue.     char = $" ifTrue:         [[nextValue nextPutAll: (stream upTo: $").
Err, it's line upTo: $"
        line peekFor: $"] whileTrue:             [nextValue nextPut: $"].         [(char := line next) = separator] whileFalse].     [char = separator]         whileFalse:             [nextValue nextPut: char.             char := line next].     row add: nextValue contents.     nextValue reset] repeat
This reduces execution time to 2.4 seconds
I also tried to use SqueaXTream for the file itself (if you don't mind the ugly invocation); But it does not gain much because you file is essentially made of Wide chars... It would help only if you'd have more ASCII fields...
MessageTally spyOn: [| stream | Â Â Â Â stream := (((FileDirectory on: '/Users/nicolas/Downloads/') / 'Data.csv') readXtream binary buffered ~> UTF8Decoder) buffered. Â Â Â Â [(CSVParser on: stream) Â Â Â Â Â Â Â Â useDelimiter: $;; Â Â Â Â Â Â Â Â rows ] ensure: [stream close]].
This reduces execution time to 1.9 seconds though
So I wouldn't say there's not plenty of room for performance ;)
Nicolas
2011/11/16 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>
I think that there are several issues:
1) Character with codePoint < 256 are unique, but Wider are not. The characters are not stored in the WideString (which instead stores the code points), but any enumeration using at: rather than wordAt: will create an overhead. I presume that it would be better to have a #codePointAt: defined as #byteAt: in ByteString and #wordAt: in WideString, so we could still define generic methods in String. 2) A lot of ByteString methods are optimized with ad hoc optional primitives, but very few WideString methods are. 3) Some of these optimized operations work with 256 element masks and cannot be generalized to WideString that easily 4) Some Stream operations that copy one Collection to the stream buffer or vice versa are also not optimized for mixed byte/word collections 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters also are (because they can use more primitives in this case) 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe questionable from quality POV, but were not replaced from efficiency POV.
I would say that a new VM with immediate Character values might help, but that won't be enough.
Nicolas
2011/11/16 Sven Van Caekenberghe <sven@beta9.be>:
Dennis,
On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:
Hi!
Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb file (136 lines) and several minutes to parse a 3.4Mb file (about 8K lines).
Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case⦠So, to be short: is there a simple way to improve the performance? TIA
-- Dennis Schetinin
I am interested in this problem.
Would it be possible to provide a test file (not that I can read Russian) and your test code ?
I would like to try this myself.
Sven
-- Dennis Schetinin
Thank you very much, I'll play with and surely benefit from your suggestions. 2011/11/18 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>
2011/11/18 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
Also note that the more mainXtream Xtreams at http://www.squeaksource.com/Xtreams/ does not perform that bad...
This ones gets a score of 3.0 seconds with a regular MultiByteFileStream
nextRow | row char line nextValue noValue | line := stream nextLine reading. row := OrderedCollection new. nextValue := (WideString new: 32) writing. noValue := true.
[[[(char := line get) isSeparator] whileTrue. noValue := false. char = $" ifTrue: [[nextValue write: (line ending: $") rest. (char := line get) = $"] whileTrue: [nextValue put: $"]. [char = separator] whileFalse: [char := line get]]. [char = separator] whileFalse: [nextValue put: char. char := line get]. row add: nextValue conclusion. nextValue := (WideString new: 32) writing] repeat] on: Incomplete do: [:exc | ].
noValue ifFalse: [row add: nextValue conclusion]. ^row
with a XTFileReadStream, just rewrite this:
MessageTally spyOn: [| stream separator | separator := $;. stream := (((FileDirectory on: '/Users/nicolas/Downloads/') / 'Data.csv') reading encoding: #utf8) "rest reading". (stream ending: Character cr) slicing collect: [:line | | row char nextValue noValue | row := OrderedCollection new. nextValue := (WideString new: 32) writing. noValue := true.
[[[(char := line get) isSeparator] whileTrue. noValue := false. char = $" ifTrue: [[nextValue write: (line ending: $") rest. (char := line get) = $"] whileTrue: [nextValue put: $"]. [char = separator] whileFalse: [char := line get]]. [char = separator] whileFalse: [nextValue put: char. char := line get]. row add: nextValue conclusion. nextValue := (WideString new: 32) writing] repeat] on: Incomplete do: [:exc | ].
noValue ifFalse: [row add: nextValue conclusion]. row]]
For some reason, this was super slow (>20s), though my previous attempt without slicing was around 3.0s...
Ah Ah, replacing contentsSpecies with ^WideString in XTEncodeRead/WriteStream took it back to 3.2s.
Nicolas
(so you noticed the commented "rest reading" whose purpose is to load the whole file in memory)
The major advantage is that you can just replace #collect: with #collecting: and get a stream of rows. Lazy is cool.
Nicolas
2011/11/18 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
2011/11/17 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
2011/11/17 Dennis Schetinin <chaetal@gmail.com>:
So, the answer for the question "Is there a simple way to improve the performance?" would be "No", right?
No, the bottle neck is CSVParser as Henrik and Levente already noticed.
My machine executes your code in 21.7s
So I decided to play with SqueaXTream http://www.squeaksource.com/XTream/ and inlined row decoding in a single method:
CSVParser>>nextRow | row char line nextValue | line := stream nextLine readXtream. row := OrderedCollection new. nextValue := (WideString new: 32) writeXtream. line endOfStreamAction: [^row add: (nextValue contents); yourself]. [[(char := line next) isSeparator] whileTrue. char = $" ifTrue: [[nextValue nextPutAll: (stream upTo: $").
Err, it's line upTo: $"
line peekFor: $"] whileTrue: [nextValue nextPut: $"]. [(char := line next) = separator] whileFalse]. [char = separator] whileFalse: [nextValue nextPut: char. char := line next]. row add: nextValue contents. nextValue reset] repeat
This reduces execution time to 2.4 seconds
I also tried to use SqueaXTream for the file itself (if you don't mind the ugly invocation); But it does not gain much because you file is essentially made of Wide
chars...
It would help only if you'd have more ASCII fields...
MessageTally spyOn: [| stream | stream := (((FileDirectory on: '/Users/nicolas/Downloads/') / 'Data.csv') readXtream binary buffered ~> UTF8Decoder) buffered. [(CSVParser on: stream) useDelimiter: $;; rows ] ensure: [stream close]].
This reduces execution time to 1.9 seconds though
So I wouldn't say there's not plenty of room for performance ;)
Nicolas
2011/11/16 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>
I think that there are several issues:
1) Character with codePoint < 256 are unique, but Wider are not. The characters are not stored in the WideString (which instead stores the code points), but any enumeration using at: rather than wordAt: will create an overhead. I presume that it would be better to have a #codePointAt: defined as #byteAt: in ByteString and #wordAt: in WideString, so we could still define generic methods in String. 2) A lot of ByteString methods are optimized with ad hoc optional primitives, but very few WideString methods are. 3) Some of these optimized operations work with 256 element masks and cannot be generalized to WideString that easily 4) Some Stream operations that copy one Collection to the stream buffer or vice versa are also not optimized for mixed byte/word collections 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters also are (because they can use more primitives in this case) 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe questionable from quality POV, but were not replaced from efficiency POV.
I would say that a new VM with immediate Character values might help, but that won't be enough.
Nicolas
2011/11/16 Sven Van Caekenberghe <sven@beta9.be>:
Dennis,
On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:
Hi!
Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb
file (136
lines) and several minutes to parse a 3.4Mb file (about 8K lines).
Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case⦠So, to be short: is there a simple way to improve the performance? TIA
-- Dennis Schetinin
I am interested in this problem.
Would it be possible to provide a test file (not that I can read Russian) and your test code ?
I would like to try this myself.
Sven
-- Dennis Schetinin
-- Dennis Schetinin
Great! This one works (with just renaming separator to delimiter; had also to replace *stream* upTo: $" with *line* upTo: $" in previous code) and improves reading time drastically: from many-many-many-many minutes to about 30 seconds! Thank you very much once again! :) 2011/11/18 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>
| row char line nextValue noValue | line := stream nextLine reading. row := OrderedCollection new. nextValue := (WideString new: 32) writing. noValue := true.
[[[(char := line get) isSeparator] whileTrue. noValue := false. char = $" ifTrue: [[nextValue write: (line ending: $") rest. (char := line get) = $"] whileTrue: [nextValue put: $"]. [char = separator] whileFalse: [char := line get]]. [char = separator] whileFalse: [nextValue put: char. char := line get]. row add: nextValue conclusion. nextValue := (WideString new: 32) writing] repeat] on: Incomplete do: [:exc | ].
noValue ifFalse: [row add: nextValue conclusion]. ^row
-- Dennis Schetinin
On 16.11.2011 21:45, Nicolas Cellier wrote:
I think that there are several issues:
1) Character with codePoint< 256 are unique, but Wider are not. The characters are not stored in the WideString (which instead stores the code points), but any enumeration using at: rather than wordAt: will create an overhead. I presume that it would be better to have a #codePointAt: defined as #byteAt: in ByteString and #wordAt: in WideString, so we could still define generic methods in String. 2) A lot of ByteString methods are optimized with ad hoc optional primitives, but very few WideString methods are. 3) Some of these optimized operations work with 256 element masks and cannot be generalized to WideString that easily 4) Some Stream operations that copy one Collection to the stream buffer or vice versa are also not optimized for mixed byte/word collections 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters also are (because they can use more primitives in this case) 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe questionable from quality POV, but were not replaced from efficiency POV.
I would say that a new VM with immediate Character values might help, but that won't be enough.
Nicolas If I may object to point 6: (Squeak) [FileStream oldFileNamed: 'Data.csv' do: [:stream | text := stream contents]] timeToRun 6753 (Pharo) [FileStream oldFileNamed: 'Data.csv' do: [:stream | text := stream contents]] timeToRun 7027 Reading without conversion, then converting with utf8ToSqueak: [FileStream oldFileNamed: 'C:\Users\Rydier\Dropbox\Pharo 1.4 latest jenkins\Data.csv' do: [:stream | stream converter: Latin1TextConverter new. text := stream contents utf8ToSqueak ]] timeToRun 9752
The efficiency drain in this example is, as Dennis points out, CSVParser adding scanned characters 1 by 1 to the WideString representing current value. Cheers, Henry
On Thu, 17 Nov 2011, Henrik Sperre Johansen wrote:
On 16.11.2011 21:45, Nicolas Cellier wrote:
I think that there are several issues:
1) Character with codePoint< 256 are unique, but Wider are not. The characters are not stored in the WideString (which instead stores the code points), but any enumeration using at: rather than wordAt: will create an overhead. I presume that it would be better to have a #codePointAt: defined as #byteAt: in ByteString and #wordAt: in WideString, so we could still define generic methods in String. 2) A lot of ByteString methods are optimized with ad hoc optional primitives, but very few WideString methods are. 3) Some of these optimized operations work with 256 element masks and cannot be generalized to WideString that easily 4) Some Stream operations that copy one Collection to the stream buffer or vice versa are also not optimized for mixed byte/word collections 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters also are (because they can use more primitives in this case) 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe questionable from quality POV, but were not replaced from efficiency POV.
I would say that a new VM with immediate Character values might help, but that won't be enough.
Nicolas If I may object to point 6: (Squeak) [FileStream oldFileNamed: 'Data.csv' do: [:stream | text := stream contents]] timeToRun 6753 (Pharo) [FileStream oldFileNamed: 'Data.csv' do: [:stream | text := stream contents]] timeToRun 7027 Reading without conversion, then converting with utf8ToSqueak: [FileStream oldFileNamed: 'C:\Users\Rydier\Dropbox\Pharo 1.4 latest jenkins\Data.csv' do: [:stream | stream converter: Latin1TextConverter new. text := stream contents utf8ToSqueak ]] timeToRun 9752
That's not "reading without conversion", you explicitly setting Latin1TextConverter there, try StandardFileStream instead. ;) Levente
The efficiency drain in this example is, as Dennis points out, CSVParser adding scanned characters 1 by 1 to the WideString representing current value.
Cheers, Henry
On 17.11.2011 11:26, Levente Uzonyi wrote:
On Thu, 17 Nov 2011, Henrik Sperre Johansen wrote:
On 16.11.2011 21:45, Nicolas Cellier wrote:
I think that there are several issues:
1) Character with codePoint< 256 are unique, but Wider are not. The characters are not stored in the WideString (which instead stores the code points), but any enumeration using at: rather than wordAt: will create an overhead. I presume that it would be better to have a #codePointAt: defined as #byteAt: in ByteString and #wordAt: in WideString, so we could still define generic methods in String. 2) A lot of ByteString methods are optimized with ad hoc optional primitives, but very few WideString methods are. 3) Some of these optimized operations work with 256 element masks and cannot be generalized to WideString that easily 4) Some Stream operations that copy one Collection to the stream buffer or vice versa are also not optimized for mixed byte/word collections 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters also are (because they can use more primitives in this case) 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe questionable from quality POV, but were not replaced from efficiency POV.
I would say that a new VM with immediate Character values might help, but that won't be enough.
Nicolas If I may object to point 6: (Squeak) [FileStream oldFileNamed: 'Data.csv' do: [:stream | text := stream contents]] timeToRun 6753 (Pharo) [FileStream oldFileNamed: 'Data.csv' do: [:stream | text := stream contents]] timeToRun 7027 Reading without conversion, then converting with utf8ToSqueak: [FileStream oldFileNamed: 'C:\Users\Rydier\Dropbox\Pharo 1.4 latest jenkins\Data.csv' do: [:stream | stream converter: Latin1TextConverter new. text := stream contents utf8ToSqueak ]] timeToRun 9752
That's not "reading without conversion", you explicitly setting Latin1TextConverter there, try StandardFileStream instead. ;)
Levente Latin1TextConverter does not do any conversion, no? ;)
[StandardFileStream oldFileNamed: 'C:\Users\Rydier\Dropbox\Pharo 1.4 latest jenkins\Data.csv' do: [:stream | text := stream contents utf8ToSqueak ]] timeToRun 9221 Better? ;) Cheers, Henry
On Thu, 17 Nov 2011, Henrik Sperre Johansen wrote:
On 17.11.2011 11:26, Levente Uzonyi wrote:
On Thu, 17 Nov 2011, Henrik Sperre Johansen wrote:
On 16.11.2011 21:45, Nicolas Cellier wrote:
I think that there are several issues:
1) Character with codePoint< 256 are unique, but Wider are not. The characters are not stored in the WideString (which instead stores the code points), but any enumeration using at: rather than wordAt: will create an overhead. I presume that it would be better to have a #codePointAt: defined as #byteAt: in ByteString and #wordAt: in WideString, so we could still define generic methods in String. 2) A lot of ByteString methods are optimized with ad hoc optional primitives, but very few WideString methods are. 3) Some of these optimized operations work with 256 element masks and cannot be generalized to WideString that easily 4) Some Stream operations that copy one Collection to the stream buffer or vice versa are also not optimized for mixed byte/word collections 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters also are (because they can use more primitives in this case) 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe questionable from quality POV, but were not replaced from efficiency POV.
I would say that a new VM with immediate Character values might help, but that won't be enough.
Nicolas If I may object to point 6: (Squeak) [FileStream oldFileNamed: 'Data.csv' do: [:stream | text := stream contents]] timeToRun 6753 (Pharo) [FileStream oldFileNamed: 'Data.csv' do: [:stream | text := stream contents]] timeToRun 7027 Reading without conversion, then converting with utf8ToSqueak: [FileStream oldFileNamed: 'C:\Users\Rydier\Dropbox\Pharo 1.4 latest jenkins\Data.csv' do: [:stream | stream converter: Latin1TextConverter new. text := stream contents utf8ToSqueak ]] timeToRun 9752
That's not "reading without conversion", you explicitly setting Latin1TextConverter there, try StandardFileStream instead. ;)
Levente Latin1TextConverter does not do any conversion, no? ;)
[StandardFileStream oldFileNamed: 'C:\Users\Rydier\Dropbox\Pharo 1.4 latest jenkins\Data.csv' do: [:stream | text := stream contents utf8ToSqueak ]] timeToRun 9221
Better? ;)
Better, but not good enough :). I got the following in Squeak: [ FileStream readOnlyFileNamed: 'Data.csv' do: [ :file | file contents ] ] timeToRun. 2429 [ StandardFileStream readOnlyFileNamed: 'Data.csv' do: [ :file | file contents utf8ToSqueak ] ] timeToRun. 2288 Significant amount of time is spent finding the current leadingChar in both cases. Using 0 as leadingChar in Unicode >> #value: gives the following results: [ FileStream readOnlyFileNamed: 'Data.csv' do: [ :file | file contents ] ] timeToRun. 1152 [ StandardFileStream readOnlyFileNamed: 'Data.csv' do: [ :file | file contents utf8ToSqueak ] ] timeToRun. 1247 Levente P.S.: To avoid measuring disk latency, the file was already in memory during the tests.
Cheers, Henry
On 17.11.2011 11:53, Levente Uzonyi wrote:
On Thu, 17 Nov 2011, Henrik Sperre Johansen wrote:
On 17.11.2011 11:26, Levente Uzonyi wrote:
On Thu, 17 Nov 2011, Henrik Sperre Johansen wrote:
On 16.11.2011 21:45, Nicolas Cellier wrote:
I think that there are several issues:
1) Character with codePoint< 256 are unique, but Wider are not. The characters are not stored in the WideString (which instead stores the code points), but any enumeration using at: rather than wordAt: will create an overhead. I presume that it would be better to have a #codePointAt: defined as #byteAt: in ByteString and #wordAt: in WideString, so we could still define generic methods in String. 2) A lot of ByteString methods are optimized with ad hoc optional primitives, but very few WideString methods are. 3) Some of these optimized operations work with 256 element masks and cannot be generalized to WideString that easily 4) Some Stream operations that copy one Collection to the stream buffer or vice versa are also not optimized for mixed byte/word collections 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters also are (because they can use more primitives in this case) 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe questionable from quality POV, but were not replaced from efficiency POV.
I would say that a new VM with immediate Character values might help, but that won't be enough.
Nicolas If I may object to point 6: (Squeak) [FileStream oldFileNamed: 'Data.csv' do: [:stream | text := stream contents]] timeToRun 6753 (Pharo) [FileStream oldFileNamed: 'Data.csv' do: [:stream | text := stream contents]] timeToRun 7027 Reading without conversion, then converting with utf8ToSqueak: [FileStream oldFileNamed: 'C:\Users\Rydier\Dropbox\Pharo 1.4 latest jenkins\Data.csv' do: [:stream | stream converter: Latin1TextConverter new. text := stream contents utf8ToSqueak ]] timeToRun 9752
That's not "reading without conversion", you explicitly setting Latin1TextConverter there, try StandardFileStream instead. ;)
Levente Latin1TextConverter does not do any conversion, no? ;)
[StandardFileStream oldFileNamed: 'C:\Users\Rydier\Dropbox\Pharo 1.4 latest jenkins\Data.csv' do: [:stream | text := stream contents utf8ToSqueak ]] timeToRun 9221
Better? ;)
Better, but not good enough :). I got the following in Squeak:
[ FileStream readOnlyFileNamed: 'Data.csv' do: [ :file | file contents ] ] timeToRun. 2429
[ StandardFileStream readOnlyFileNamed: 'Data.csv' do: [ :file | file contents utf8ToSqueak ] ] timeToRun. 2288
Significant amount of time is spent finding the current leadingChar in both cases. Using 0 as leadingChar in Unicode >> #value: gives the following results:
[ FileStream readOnlyFileNamed: 'Data.csv' do: [ :file | file contents ] ] timeToRun. 1152
[ StandardFileStream readOnlyFileNamed: 'Data.csv' do: [ :file | file contents utf8ToSqueak ] ] timeToRun. 1247
Levente
P.S.: To avoid measuring disk latency, the file was already in memory during the tests. I fail to se how even those values are significant enough to warrant using utf8ToSqueak rather than let the converter take care of it, which was the point I contended in the first place.
Cheers, Henry
On Thu, 17 Nov 2011, Henrik Sperre Johansen wrote:
On 17.11.2011 11:53, Levente Uzonyi wrote:
On Thu, 17 Nov 2011, Henrik Sperre Johansen wrote:
On 17.11.2011 11:26, Levente Uzonyi wrote:
On Thu, 17 Nov 2011, Henrik Sperre Johansen wrote:
On 16.11.2011 21:45, Nicolas Cellier wrote:
I think that there are several issues:
1) Character with codePoint< 256 are unique, but Wider are not. The characters are not stored in the WideString (which instead stores the code points), but any enumeration using at: rather than wordAt: will create an overhead. I presume that it would be better to have a #codePointAt: defined as #byteAt: in ByteString and #wordAt: in WideString, so we could still define generic methods in String. 2) A lot of ByteString methods are optimized with ad hoc optional primitives, but very few WideString methods are. 3) Some of these optimized operations work with 256 element masks and cannot be generalized to WideString that easily 4) Some Stream operations that copy one Collection to the stream buffer or vice versa are also not optimized for mixed byte/word collections 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters also are (because they can use more primitives in this case) 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe questionable from quality POV, but were not replaced from efficiency POV.
I would say that a new VM with immediate Character values might help, but that won't be enough.
Nicolas If I may object to point 6: (Squeak) [FileStream oldFileNamed: 'Data.csv' do: [:stream | text := stream contents]] timeToRun 6753 (Pharo) [FileStream oldFileNamed: 'Data.csv' do: [:stream | text := stream contents]] timeToRun 7027 Reading without conversion, then converting with utf8ToSqueak: [FileStream oldFileNamed: 'C:\Users\Rydier\Dropbox\Pharo 1.4 latest jenkins\Data.csv' do: [:stream | stream converter: Latin1TextConverter new. text := stream contents utf8ToSqueak ]] timeToRun 9752
That's not "reading without conversion", you explicitly setting Latin1TextConverter there, try StandardFileStream instead. ;)
Levente Latin1TextConverter does not do any conversion, no? ;)
[StandardFileStream oldFileNamed: 'C:\Users\Rydier\Dropbox\Pharo 1.4 latest jenkins\Data.csv' do: [:stream | text := stream contents utf8ToSqueak ]] timeToRun 9221
Better? ;)
Better, but not good enough :). I got the following in Squeak:
[ FileStream readOnlyFileNamed: 'Data.csv' do: [ :file | file contents ] ] timeToRun. 2429
[ StandardFileStream readOnlyFileNamed: 'Data.csv' do: [ :file | file contents utf8ToSqueak ] ] timeToRun. 2288
Significant amount of time is spent finding the current leadingChar in both cases. Using 0 as leadingChar in Unicode >> #value: gives the following results:
[ FileStream readOnlyFileNamed: 'Data.csv' do: [ :file | file contents ] ] timeToRun. 1152
[ StandardFileStream readOnlyFileNamed: 'Data.csv' do: [ :file | file contents utf8ToSqueak ] ] timeToRun. 1247
Levente
P.S.: To avoid measuring disk latency, the file was already in memory during the tests. I fail to se how even those values are significant enough to warrant using utf8ToSqueak rather than let the converter take care of it, which was the point I contended in the first place.
It's not worth using #utf8ToSqueak in this case to get better performance. This is also true for all other cases, because of the optimizations done to text converters (in Squeak UTF8TextConverter uses the same method as #utf8ToSqueak when possible). The only thing you can save with it performance wise is the creation of the converter and the optional lookup of the converter class, which is insignificant in most cases. Levente
Cheers, Henry
Sure. I've loaded CSVParser from http://www.squeaksource.com/CSVParser.html and patched one method there: nextRow | row | row := OrderedCollection new. stream skipSeparators. [self atEndOfLine] whileFalse: [row add: self nextValue]. *"stream skip: -1."* stream next = $, ifTrue: [row add: '']. ^ row This is how I test it: stream := FileStream readOnlyFileNamed: 'Data.csv'. [(CSVParser on: stream) useDelimiter: $;; rows ] ensure: [stream close]. I attach the bigger data file. The shorter one can be produced by deleting rows. 2011/11/16 Sven Van Caekenberghe <sven@beta9.be>
Dennis,
On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:
Hi!
Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb file (136 lines) and several minutes to parse a 3.4Mb file (about 8K lines).
Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case⦠So, to be short: is there a simple way to improve the performance? TIA
-- Dennis Schetinin
I am interested in this problem.
Would it be possible to provide a test file (not that I can read Russian) and your test code ?
I would like to try this myself.
Sven
-- Dennis Schetinin
Hello, Where is converting strings from UTF8 encoding? 2011/11/17 Dennis Schetinin <chaetal@gmail.com>
Sure.
I've loaded CSVParser from http://www.squeaksource.com/CSVParser.html and patched one method there:
nextRow | row | row := OrderedCollection new. stream skipSeparators. [self atEndOfLine] whileFalse: [row add: self nextValue]. *"stream skip: -1."* stream next = $, ifTrue: [row add: '']. ^ row
This is how I test it:
stream := FileStream readOnlyFileNamed: 'Data.csv'. [(CSVParser on: stream) useDelimiter: $;; rows ] ensure: [stream close].
I attach the bigger data file. The shorter one can be produced by deleting rows.
2011/11/16 Sven Van Caekenberghe <sven@beta9.be>
Dennis,
On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:
Hi!
Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb file (136 lines) and several minutes to parse a 3.4Mb file (about 8K lines).
Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case⦠So, to be short: is there a simple way to improve the performance? TIA
-- Dennis Schetinin
I am interested in this problem.
Would it be possible to provide a test file (not that I can read Russian) and your test code ?
I would like to try this myself.
Sven
-- Dennis Schetinin
There's no need for :) 2011/11/17 Denis Kudriashov <dionisiydk@gmail.com>
Hello,
Where is converting strings from UTF8 encoding?
2011/11/17 Dennis Schetinin <chaetal@gmail.com>
Sure.
I've loaded CSVParser from http://www.squeaksource.com/CSVParser.htmland patched one method there:
nextRow | row | row := OrderedCollection new. stream skipSeparators. [self atEndOfLine] whileFalse: [row add: self nextValue]. *"stream skip: -1."* stream next = $, ifTrue: [row add: '']. ^ row
This is how I test it:
stream := FileStream readOnlyFileNamed: 'Data.csv'. [(CSVParser on: stream) useDelimiter: $;; rows ] ensure: [stream close].
I attach the bigger data file. The shorter one can be produced by deleting rows.
2011/11/16 Sven Van Caekenberghe <sven@beta9.be>
Dennis,
On 16 Nov 2011, at 19:30, Dennis Schetinin wrote:
Hi!
Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I found it really slow. It takes about 5 seconds to read a 53Kb file (136 lines) and several minutes to parse a 3.4Mb file (about 8K lines).
Profiler shows that nearly all the time is eaten by WideString >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, this means that <primitive: 66> there does not work in this case⦠So, to be short: is there a simple way to improve the performance? TIA
-- Dennis Schetinin
I am interested in this problem.
Would it be possible to provide a test file (not that I can read Russian) and your test code ?
I would like to try this myself.
Sven
-- Dennis Schetinin
-- Dennis Schetinin
I will have a look somewhere in the next couple of days. On 17 Nov 2011, at 05:15, Dennis Schetinin wrote:
Sure.
I've loaded CSVParser from http://www.squeaksource.com/CSVParser.html and patched one method there:
nextRow | row | row := OrderedCollection new. stream skipSeparators. [self atEndOfLine] whileFalse: [row add: self nextValue]. "stream skip: -1." stream next = $, ifTrue: [row add: '']. ^ row
This is how I test it:
stream := FileStream readOnlyFileNamed: 'Data.csv'. [(CSVParser on: stream) useDelimiter: $;; rows ] ensure: [stream close].
I attach the bigger data file. The shorter one can be produced by deleting rows.
participants (6)
-
Denis Kudriashov -
Dennis Schetinin -
Henrik Sperre Johansen -
Levente Uzonyi -
Nicolas Cellier -
Sven Van Caekenberghe