On 17.11.2011 11:11, Henrik Sperre Johansen wrote:
On 17.11.2011 10:38, Henrik Sperre Johansen wrote:
The efficiency drain in this example is, as Dennis points out, CSVParser adding scanned characters 1 by 1 to the WideString representing current value.
Cheers, Henry
There is something strange going on with CogVM's though, on an old VM the CSVParser use about 17 secs on Data.csv, while it's 131 secs on Cog 2508... It looks like primitive 61 has a *really* slow path for variable word classes?
Cheers, Henry ... or not. (Cog) ws := WideString new: 100000. char := Character value: 5021. [1 to: 100000 do: [:i | ws at: i put: char]] timeToRun 10
(Interpreter) ws := WideString new: 100000. char := Character value: 5021. [1 to: 100000 do: [:i | ws at: i put: char]] timeToRun 12 Seems it is instead the becomeForward: call, as each token starts streaming to a ByteString, and thus triggers it :/ A quick hack which will work in your case since all wide tokens usually start with russian character, is replacing (in nextValue and nextQuotedValue) String streamContents: with something like: (stream peek charCode > 255 ifTrue: [WideString] ifFalse: [String]) streamContents: I don't see an easy/elegant way to avoid it in the general case though :( Cheers, Henry