Hi Benjamin, On 05 Oct 2014, at 22:37, Benjamin Pollack <benjamin@bitquabit.com> wrote:
On Sun, 05 Oct 2014 10:36:31 -0400, Sven Van Caekenberghe <sven@stfx.eu> wrote:
How come you got WideStrings ? What does the input look like, can you give a partial example ?
I'm guessing I got WideStrings because the file is indeed in UTF-8, with lots of characters outside the lower 128 code points. A sample couple of lines might look like
å ç°åº·å ¸,The National,Sarah McLachlan,å¨æ°å«,Indochine,Rise Against,City and Colour,Cæcilie Norby,El Cumbanchero,Death Letter The Beatles,The Who,Barenaked Ladies,The Doors,Bob Dylan
These are two play lists, one per line; each comma-delimited element is a band on that play list.
The full line for reading and tokenization is just "pathToFile asFileReference contents lines collect: [ :line | (',' split: line) collect: [ :ea | ea trimmed ]]." Based on the profile indicating that a lot of time is lost on things like WideString>>copyFrom:to:, I wasn't optimistic about trying to stream the contents instead of just calling "contents lines", but I admit I didn't try.
--Benjamin
Working with WideStrings is way slower than working with ByteStrings, there is just no way around that. What is especially slow is the automagic switch from ByteString to WideString, for example in a String>>#streamContents: because a #becomeForward: is involved. If that happens for every line or every token, that would be crazy. Apart from that, the tokenisation is not very efficient, #lines is a copy of your whole contents, so is the #split: and #trimmed. The algorithm sounds a bit lazy as well, writing it 'on purpose' with an eye for performance might yield better results. But I guess this is not really an exercise in optimisation. If it is, you should give us the dataset and code (and maybe runnable python code as reference), with some comments. Sven