April 2011 - Pharo-dev - lists.pharo.org

Re: [Pharo-project] Preallocation behavior

by Toon Verwaest

On 04/28/2011 08:35 AM, jannik.laval wrote: > First of all, I spy this source code: > ==== > MessageTally spyOn: > [ 500 timesRepeat: [ > | str | > *str := ''*. > 9000 timesRepeat: [ str := str, 'A' ]]]. > ==== This is what Joel Spolsky called a "Shlemiel the Painter's algorithm" http://www.joelonsoftware.com/articles/fog0000000319.html It's obviously very slow and it's very different from the code below for obvious reasons. Other /languages that won't be named/ just do hacks to allow you to think in terms of string concatenation while it's optimized internally using buffers. This works well to make programs by dummies faster, but it also dumbs down programmers since you forget that stuff actually needs to be done. > > The result appears after *24 961 ms*. > An optimization is to use a Stream. Here is my source code: > === > MessageTally spyOn: > [ 500 timesRepeat: [ > | str | > *str := WriteStream on: (String new)*. > 9000 timesRepeat: [ str nextPut: $A ]]]. > === > > The result appears after *812 ms*, which is a large improvement. > Now, we could optimize again using the preallocation. Here is my > source code: > > ==== > MessageTally spyOn: > [ 500 timesRepeat: [ > | str | > *str := WriteStream on: (String new: 10000)*. > 9000 timesRepeat: [ str nextPutAll: 'A' ]]]. > ==== > > And the result is strange: it makes 2 times slower to display the result. > The result appears after 1656 ms. > > Here is the spy result: > === > - 1656 tallies, 1656 msec. > > **Tree** > -------------------------------- > Process: (40s) 464519168: nil > -------------------------------- > > **Leaves** > *22.9% {380ms} UndefinedObject>>DoIt* > 22.5% {373ms} SmallInteger(Integer)>>timesRepeat: > 22.2% {368ms} WriteStream>>nextPutAll: > === > > There is the call of UndefinedObject>>DoIt which is added and takes time. > Does anyone know what is done during the preallocation ? > Why is it slower than non-preallocation ? > > Thanks for your answers. > Jannik

April 28, 2011

Re: [Pharo-project] Preallocation behavior

by StÃ©phane Ducasse

tx I prefer that. On Apr 28, 2011, at 12:17 PM, Henrik Sperre Johansen wrote: > On 28.04.2011 10:49, StÃ©phane Ducasse wrote: >> Henrik >> >> you lost me >> >>> In the first example, you are making a single string with all A's of size 9000 repeated 500 times. >>> In the second example, you are making 9000 strings with all A's of size 10000 repeated 500 times. >> Why? > My bad, I misread and thought the code was using atAllPut: , like it should :) (that is, if its intent is to initialize a presized collection with default values) > > MessageTally spyOn: > > [ 500 timesRepeat: [ > | str | > str :=String new: 9000. > str atAllPut: $A ]]. > > only takes a couple of milliseconds. > > > As for nextPut: vs nextPutAll:, that is to be expected. > nextPutAll: is optimized for sizes> 1, on my machine it "catches up" when you surpass 4 characters added in each loop. > > Cheers, > Henry > > > > >

April 28, 2011

Re: [Pharo-project] Why we have Behavior >> flushCache?

by Luc Fabresse

2011/4/26 Mariano Martinez Peck <marianopeck(a)gmail.com> > > > On Tue, Apr 26, 2011 at 6:59 PM, Eliot Miranda <eliot.miranda(a)gmail.com>wrote: > >> Mariano, >> >> the point is that it is classes that manage adding methods to classes >> and hence it is their private behavior to flush the VM's method cache. >> > > Ok...I agree, but still it is misleading when we are talking about "method > cache". Because methods are in classes...so I promise everybody that didn't > know and you ask what Behavior >> flushCache does, and everybody will say > that it flushed the cache of the methods of that class, not the whole cache. > I can understand CompiledMethod >> flushCache and even Symbol >> flushCache > ... but this one looks really strange. > > >> You'll see "self flushCache" in Smalltalk-80 v2 images. >> > > that makes more sense indeed > > >> Now of course the VM provides selective cache flushing but back in the >> day adding or removing a method implied flushing the whole cache. >> Personally I don't see that this justifies adding Smalltalk vm yet. I >> would wait until you have a few more use cases. >> > > Sorry but in this case, I disagree. I think the opposite: it is the perfect > moment to reify the VM in the image and start using it. Why? exactly > because right now we have few uses and it is really easy to migrate. In > fact, for Behavior >> flushCache there are only 3 senders. And I don't > pretend to remove them right now, but with the normal deprecation policy. > This is, in Pharo 1.3 we let the messages like Behavior >> flushCache, > SmalltalkImage >> vmVersion, etc. But we let them as deprecated explaining > that now they should use XXX. And of course, inside the image we do fix the > senders. > > In fact, in Pharo 1.3, the method SmalltalkImage >> vm > "Answer the object to query about virtual machine." > > ^self > > already exists :) > > I see the following possible behavior for VM right now (I mean, the > following methods are already in SmalltalkImage): > > - #flushCache > - #buildDate > - #interpreterSourceVersion > - #platformSourceVersion > - #versionLabel > - #extraVMMemory > - #extraVMMemory: > - #gcBiasToGrow: > - #gcBiasToGrowLimit: > - #getVMParameters > - #primitiveGCBiasToGrow: > - #vmParameterAt: > -#vmParameterAt:put: > - #reportCPUandRAM > - #vmStatisticsReportString > - #vmStatisticsShortString > > So....I think they are enough. And the sooner we do this, the easier it > would be to reify the VM in the image. > Yes I agree Mariano. And I also suggest to put meaningful selectors for vmparameters because I can't remember the indexes. e.g: VirtualMachine>>maximumNumberOfSemaphores ^ self vmParameterAt: 49 Luc > > Cheers > > Mariano > > > >> >> On Tue, Apr 26, 2011 at 3:34 AM, Mariano Martinez Peck < >> marianopeck(a)gmail.com> wrote: >> >>> Hi. As far as I can see, >>> >>> Behavior >> flushCache >>> "Tell the interpreter to remove the contents of its method lookup >>> cache, if it has >>> one. Essential. See Object documentation whatIsAPrimitive." >>> >>> <primitive: 89> >>> self primitiveFailed >>> >>> >>> And primitive 89 does nothing in particular with the receiver (the class >>> in this case). In both, InterpreterVM and Cog, the WHOLE cache is flushed, >>> there is NOTHING related to the receiver class. Of course, that's at least >>> what it looks for me (please tell me if I am wrong). >>> So...if this is the case, wouldn't make sense to move it elsewhere? like >>> Smalltalk flushCache or Smalltalk vm flushCache (and it is a good moment to >>> reify the VM). So after we can do: Smalltalk vm version. Smalltlak vm >>> flushCache, Smalltalk vm parameterAt: , etc.... >>> >>> If you don't like doing "Smalltalk vm" then we can create a VM class with >>> all class methods, or a singleton and use #current or a singleton and put it >>> in Smalltalk globals...etc >>> >>> We will need to fix a couple of senders, thus. >>> >>> What do you think? For me is really confusing, and you don't understand >>> it until you see the primitive implementation. >>> >>> Cheers >>> >>> -- >>> Mariano >>> http://marianopeck.wordpress.com >>> >>> >> > > > -- > Mariano > http://marianopeck.wordpress.com > >

April 28, 2011

Re: [Pharo-project] [update 1.3] #13174

by Mariano Martinez Peck

On Thu, Apr 28, 2011 at 10:55 AM, Marcus Denker <marcus.denker(a)inria.fr>wrote: > 13174 > ----- > > Issue 4099: Find and replace doesnt accept with Enter key > Issue 4102: Show commit message only after successfully commiting a new > version > cooool!! > Issue 4108: remove authoringPrototype methods > Issue 4105: Failing test: > Tests.SystemChangeNotification.ChangeHooksTest.testMethodRemovedEvent2 > Issue 4082: nil@nil > Issue 4069: Object>>withArgs:executeMethod: should be removed > we deprecate the specialized version (Prim 189), but keep > #withArgs:executeMethod: for now until > there is a primitive for CompiledMethod... (this is used for > executing DoIts). > For the next step, we will add another Issue later. > > > > -- > Marcus Denker -- http://www.marcusdenker.de > INRIA Lille -- Nord Europe. Team RMoD. > > > -- Mariano http://marianopeck.wordpress.com

April 28, 2011

Re: [Pharo-project] how to deal with string position in relation to cr/crlf

by Toon Verwaest

What is the problem exactly? Do you somewhere rely on exact position in the complete string? What you could do is just keep line + column number since that stays fixed. Just increment newline on the systems newline sequence, and set column back to 0. On Apr 28, 2011 10:48 AM, "Tudor Girba" <tudor.girba(a)gmail.com> wrote: > Hi, > > I have a small problem related to file line endings and storing the token information of PetitParser. > > Sometimes, we parse the sources on Windows and then manipulate the model on Linux or Mac. In this context, if I store the token positions in a string, I encounter problems because CR and LF are considered characters, but the line endings can vary. > > ('abc', Character cr asString , 'd') findString: 'd'. > ==> 5 > > ('abc', Character cr asString , Character lf asString, 'd') findString: 'd'. > ==> 6 > > > How would you approach this problem? > > Cheers, > Doru > > > -- > www.tudorgirba.com > > "Every thing has its own flow." > > > > >

April 28, 2011

[Pharo-project] [update 1.3] #13174

by Marcus Denker

13174 ----- Issue 4099: Find and replace doesnt accept with Enter key Issue 4102: Show commit message only after successfully commiting a new version Issue 4108: remove authoringPrototype methods Issue 4105: Failing test: Tests.SystemChangeNotification.ChangeHooksTest.testMethodRemovedEvent2 Issue 4082: nil@nil Issue 4069: Object>>withArgs:executeMethod: should be removed we deprecate the specialized version (Prim 189), but keep #withArgs:executeMethod: for now until there is a primitive for CompiledMethod... (this is used for executing DoIts). For the next step, we will add another Issue later. -- Marcus Denker -- http://www.marcusdenker.de INRIA Lille -- Nord Europe. Team RMoD.

April 28, 2011

Re: [Pharo-project] Preallocation behavior

by StÃ©phane Ducasse

Henrik you lost me > In the first example, you are making a single string with all A's of size 9000 repeated 500 times. > In the second example, you are making 9000 strings with all A's of size 10000 repeated 500 times. Why? >>> An optimization is to use a Stream. Here is my source code: >>> === >>> MessageTally spyOn: >>> [ 500 timesRepeat: [ >>> | str | >>> *str := WriteStream on: (String new)*. >>> 9000 timesRepeat: [ str nextPut: $A ]]]. >>> === >>> >>> The result appears after *812 ms*, which is a large improvement. >>> Now, we could optimize again using the preallocation. Here is my source >>> code: >>> >>> ==== >>> MessageTally spyOn: >>> [ 500 timesRepeat: [ >>> | str | >>> *str := WriteStream on: (String new: 10000)*. >>> 9000 timesRepeat: [ str nextPutAll: 'A' ]]]. >>> ==== >>> >>> And the result is strange: it makes 2 times slower to display the result. >>> The result appears after 1656 ms. > > In the first example, you are making a single string with all A's of size 9000 repeated 500 times. > In the second example, you are making 9000 strings with all A's of size 10000 repeated 500 times.

April 28, 2011

[Pharo-project] how to deal with string position in relation to cr/crlf

by Tudor Girba

Hi, I have a small problem related to file line endings and storing the token information of PetitParser. Sometimes, we parse the sources on Windows and then manipulate the model on Linux or Mac. In this context, if I store the token positions in a string, I encounter problems because CR and LF are considered characters, but the line endings can vary. ('abc', Character cr asString , 'd') findString: 'd'. ==> 5 ('abc', Character cr asString , Character lf asString, 'd') findString: 'd'. ==> 6 How would you approach this problem? Cheers, Doru -- www.tudorgirba.com "Every thing has its own flow."

April 28, 2011

Re: [Pharo-project] Preallocation behavior

by Andres Valloud

:). It can be improved in a number of ways. Instead of atAllPut:, it should be from:to:put: (so atAllPut: x should be self from: 1 to: self size put: x), it should do 8 or so at:put:s by hand before starting the loop, and it should keep a block of 2048 or so for the iteration (less memory traffic on the CPU that way, read *the same* 8kb or so all the time, write *different* 8kb each time). All of this is thoroughly explained and measured in the book. I forget the minute details right now, but tinkering with those details does make a difference. On 4/28/11 1:00 , Henrik Sperre Johansen wrote: > On 28.04.2011 09:30, Andres Valloud wrote: >> As a side comment, I do not know if an atAllPut: method I wrote back >> in about 2000 or so is still in the image... but if it is not, keep in >> mind that you can use something like replaceFrom:to:with:startingAt: >> using the receiver as the source of data, duplicating the amount of >> data copied each time. Back then, if you had to write the same object >> more than 26 or so times, it was faster to use the duplicating "block >> copy" method. The key is to avoid multiple primitive calls (which >> have expensive overhead compared to what is actually done), and >> replace them with a single primitive call that does a bunch of >> writes. With the duplication method, you could get away with doing >> 9000 writes in no more than 15 primitive calls. I wrote about this in >> much more detail in the Fundamentals volume 2 book. > It is. It's still beautiful :D > >> >> On 4/27/11 23:35 , jannik.laval wrote: >>> Hi all, >>> >>> I am playing with MessageTally, and I have a strange result with >>> prealocation. >>> Here is my example. I am working on a PharoCore1.3, with a VM4.2.5 >>> >>> An optimization is to use a Stream. Here is my source code: >>> === >>> MessageTally spyOn: >>> [ 500 timesRepeat: [ >>> | str | >>> *str := WriteStream on: (String new)*. >>> 9000 timesRepeat: [ str nextPut: $A ]]]. >>> === >>> >>> The result appears after *812 ms*, which is a large improvement. >>> Now, we could optimize again using the preallocation. Here is my source >>> code: >>> >>> ==== >>> MessageTally spyOn: >>> [ 500 timesRepeat: [ >>> | str | >>> *str := WriteStream on: (String new: 10000)*. >>> 9000 timesRepeat: [ str nextPutAll: 'A' ]]]. >>> ==== >>> >>> And the result is strange: it makes 2 times slower to display the >>> result. >>> The result appears after 1656 ms. > > In the first example, you are making a single string with all A's of > size 9000 repeated 500 times. > In the second example, you are making 9000 strings with all A's of size > 10000 repeated 500 times. > > ie. you are not measuring equivalent operations. > > That it only takes twice as long is due to Andres' excellent nextPutAll: > implementation :) > > Cheers, > Henry > . >

April 28, 2011

Re: [Pharo-project] Preallocation behavior

by Henrik Sperre Johansen

On 28.04.2011 09:30, Andres Valloud wrote: > As a side comment, I do not know if an atAllPut: method I wrote back > in about 2000 or so is still in the image... but if it is not, keep in > mind that you can use something like replaceFrom:to:with:startingAt: > using the receiver as the source of data, duplicating the amount of > data copied each time. Back then, if you had to write the same object > more than 26 or so times, it was faster to use the duplicating "block > copy" method. The key is to avoid multiple primitive calls (which > have expensive overhead compared to what is actually done), and > replace them with a single primitive call that does a bunch of > writes. With the duplication method, you could get away with doing > 9000 writes in no more than 15 primitive calls. I wrote about this in > much more detail in the Fundamentals volume 2 book. It is. It's still beautiful :D > > On 4/27/11 23:35 , jannik.laval wrote: >> Hi all, >> >> I am playing with MessageTally, and I have a strange result with >> prealocation. >> Here is my example. I am working on a PharoCore1.3, with a VM4.2.5 >> >> An optimization is to use a Stream. Here is my source code: >> === >> MessageTally spyOn: >> [ 500 timesRepeat: [ >> | str | >> *str := WriteStream on: (String new)*. >> 9000 timesRepeat: [ str nextPut: $A ]]]. >> === >> >> The result appears after *812 ms*, which is a large improvement. >> Now, we could optimize again using the preallocation. Here is my source >> code: >> >> ==== >> MessageTally spyOn: >> [ 500 timesRepeat: [ >> | str | >> *str := WriteStream on: (String new: 10000)*. >> 9000 timesRepeat: [ str nextPutAll: 'A' ]]]. >> ==== >> >> And the result is strange: it makes 2 times slower to display the >> result. >> The result appears after 1656 ms. In the first example, you are making a single string with all A's of size 9000 repeated 500 times. In the second example, you are making 9000 strings with all A's of size 10000 repeated 500 times. ie. you are not measuring equivalent operations. That it only takes twice as long is due to Andres' excellent nextPutAll: implementation :) Cheers, Henry

April 28, 2011