Hi Sven, Thanks for your comments, my matrix includes names, floats and characters, and the matrix isn't squared because I need to build a transposed matrix while reading the CSV, however I will take a look the SqNumberParser. In case anyone is wondering how this is done, this is the code for a matrix with 20 rows: First to make some predefined columns for which I have to repeat values, so I used the Generator ported by Lukas Renggli. matrix := Matrix rows: 29044 columns: 20. firstCol := ( Generator on: [: g | rowCount timesRepeat: [ g yield: 1 ] ] ) upToEnd. matrix atColumn: 1 put: fCol. "... fill remaining 5 columns ..." index := 0. parserResult rowsDo: [: rs | | rowInd colInd | matrix at: ( rowInd := index \\ 20 + 1 ) at: ( colInd := index // 20 + 7 ) put: 'test'. index := index + 1 ]. 1 to: rowCount do: [: rIndex | 1 to: colCount do: [: cIndex | outputFile nextPutAll: ( matrix at: rIndex at: cIndex ) asString; tab ]. outputFile cr ]. Hope it helps someone, Cheers 2010/12/7 Sven Van Caekenberghe <sven@beta9.be>:
Hi Hernán, On 06 Dec 2010, at 20:54, Hernán Morales Durand wrote:
It seems my performance problem involves reading and parsing a "CSV" file
Elements Matrix DhbMatrix 53400 18274 17329 175960 61043 60722 710500 379276 385278
I will check if it's worth to implement a primitive for very fast parsing of CSV files.
I think that instead of going native, it would be worthwhile to try to optimize in Smalltalk first (it certainly is more fun). I thought that this was an interesting problem so I tried writing some code myself, assuming the main problem is getting a CSV matrix in and out of Smalltalk as fast as possible. I simplified further by making the matrix square and containing only Numbers. I also preallocate the matrix and use the fact that I know the row/column sizes. These are my results: Size Elements Read Write 250  62500   1013 7858 500  250000  4185 31007 750  562500  9858 71434 I think this is faster, but it is hard to compare. I am still a bit puzzled as to why the writing is slower than the reading though. The code is available at http://www.squeaksource.com/ADayAtTheBeach.html in the package 'Smalltalk-Hacking-Sven', class NumberMatrix. This is the write loop: writeCsvTo: stream 1 to: size do: [ :row | 1 to: size do: [ :column | column ~= 1 ifTrue: [ stream nextPut: $, ]. stream print: (self at: row at: column) ]. stream nextPut: Character lf ] And this is the read loop: readCsvFrom: stream | numberParser | numberParser := SqNumberParser on: stream. 1 to: size do: [ :row | 1 to: size do: [ :column | self at: row at: column put: numberParser nextNumber. column ~= size ifTrue: [ stream peekFor: $, ] ]. row ~= size ifTrue: [ stream peekFor: Character lf ] ] I am of course cheating a little bit, but should your CSV file be different, I am sure you can adapt the code (for example to deal with quoting). I am also advancing the stream under the SqNumberParser to avoid allocation a new one every time. I think this code generates little garbage. What do you think ? Sven
-- Hernán Morales Information Technology Manager, Institute of Veterinary Genetics. National Scientific and Technical Research Council (CONICET). La Plata (1900), Buenos Aires, Argentina. Telephone: +54 (0221) 421-1799. Internal: 422 Fax: 425-7980 or 421-1799.