April 2011 - Pharo-dev - lists.pharo.org

Re: [Pharo-project] Memory profiling

by Henrik Sperre Johansen

On 24.04.2011 05:32, Alexandre Bergel wrote: > Hi! > > I have been working on a memory profiler for Pharo. People who attended Smalltalks'10 may remember Mariano Coca's presentation. > > Unfortunately, I am not satisfied with what we can obtain from the execution of a piece of code in Pharo. The amount of memory consumed by a method _significantly varies_ across execution. I have the impression that memory consumption is a less stable metric than execution time. > > For example, I used this as a benchmark: > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > 7 timesRepeat: [ > 5 timesRepeat: [ Smalltalk garbageCollect]. > MemoryProfiler > viewProfiling: [ > 20 timesRepeat: [| view | > 5 timesRepeat: [ Smalltalk garbageCollect]. > Object new. Object new. > 5 timesRepeat: [ Smalltalk garbageCollect]. > > view := MOViewRenderer new. > view nodes: (1 to: 5000). > view root applyLayout] ] > inPackagesMatching: 'Mondrian-*' > ] > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > It opens a simple visualization in Mondrian. I tried to clean as much as I can before executing the code with multiple garbage collect. > > This piece of code return 7 profiles. > In a profile I have for the method MOBoundedShape>>heightFor: : > Number of Young Allocated Bytes: 42,879,008 bytes > Number of Old Allocated Bytes: 6,000,660 bytes > Total Allocated Bytes: 48,879,668 bytes > Number of Increased Memory Bytes: 0 bytes > Number of Incremental GC's: 1,415 > Number of Full GC's: 0 > Number of Tenures of Surving Objects: 164 > > In another profile I have: > Number of Young Allocated Bytes: -7,857,156 bytes > Number of Old Allocated Bytes: 6,008,872 bytes > Total Allocated Bytes: -1,848,284 bytes > Number of Increased Memory Bytes: 0 bytes > Number of Incremental GC's: 1,373 > Number of Full GC's: 0 > Number of Tenures of Surving Objects: 165 > > In another: > Number of Young Allocated Bytes: -11,956,720 bytes > Number of Old Allocated Bytes: 5,315,216 bytes > Total Allocated Bytes: -6,641,504 bytes > Number of Increased Memory Bytes: 0 bytes > Number of Incremental GC's: 1,354 > Number of Full GC's: 0 > Number of Tenures of Surving Objects: 145 > > And another: > Number of Young Allocated Bytes: 59,132,472 bytes > Number of Old Allocated Bytes: 6,177,772 bytes > Total Allocated Bytes: 65,310,244 bytes > Number of Increased Memory Bytes: 0 bytes > Number of Incremental GC's: 1,361 > Number of Full GC's: 0 > Number of Tenures of Surving Objects: 167 > > The metric "Number of Young Allocated Bytes", "Number of Old Allocated Bytes", "Total Allocated Bytes" are highly unpredictable: i.e., two profiles are likely to be completely different. I am not sure why is this so. I use the standard Pharo1.2 One click, without additional parameter to the VM. Do you think that other platforms and languages behave the same? It's not really surprising; since young space allocated size is shrunk by GC, a simple "value after block" - "value before block", for how much was actually allocated will give you rather random results. The amount of surviving objects (ie tenured into old allocated bytes)/tenures though is pretty much the same over the different runs, the differences can probably be explained by higher-priority process activity. > However, the positive point is that it seems that the amount of incremental GC and the number of tenured objects may play an important role in estimating the memory consumption. These metrics are much more stable. > What do you think on this? Does it make sense to say that more a method triggers incremental GC, more it consumes memory? > > What exactly triggers an incremental GC? http://wiki.squeak.org/squeak/1469 "Eden collections are triggered by either running out of eden space, or allocating more than a certain number of objects (currently 4000) since the last collection " Not sure if it's up to date, but I'd guess so. It's hard to build a Profiler for creation of non-transient objects during a run, since the creation and tenuring of created objects usually happens at entirely different places (around 200 incGC's per tenure in my image). In other words, you can't really say that a method is "consuming" memory simply because a tenure happened during it's execution, and there was an increase in old space allocation, as some/many of the surviving objects could have been created 200 incGC's earlier. The amount of incGC's is usually a good indication of how many/how large objects are created during a methods execution though, due to the above mentioned trigger conditions. In the context of determining overhead due to GC, those numbers are usually the interesting ones anyways. Cheers, Henry

April 26, 2011

Re: [Pharo-project] Why we have Behavior >> flushCache?

by Mariano Martinez Peck

On Tue, Apr 26, 2011 at 2:19 PM, Mariano Martinez Peck < marianopeck(a)gmail.com> wrote: > > > On Tue, Apr 26, 2011 at 1:56 PM, Camillo Bruni <camillo.bruni(a)inria.fr>wrote: > >> How about putting such methods on an Interpreter class? >> >> > that's why I was saying by "reify the VM" > > but not Interpreter to avoid collisions with VMMaker. I would call it VirtualMachine- other ideas? > > >> On 2011-04-26, at 14:35, Alexandre Bergel wrote: >> >> > Not using the receiver is indeed misleading. I thought it simply flushes >> the cache for a particular class >> > >> > Alexandre >> > >> > >> > On 26 Apr 2011, at 06:34, Mariano Martinez Peck wrote: >> > >> >> Hi. As far as I can see, >> >> >> >> Behavior >> flushCache >> >> "Tell the interpreter to remove the contents of its method lookup >> cache, if it has >> >> one. Essential. See Object documentation whatIsAPrimitive." >> >> >> >> <primitive: 89> >> >> self primitiveFailed >> >> >> >> >> >> And primitive 89 does nothing in particular with the receiver (the >> class in this case). In both, InterpreterVM and Cog, the WHOLE cache is >> flushed, there is NOTHING related to the receiver class. Of course, that's >> at least what it looks for me (please tell me if I am wrong). >> >> So...if this is the case, wouldn't make sense to move it elsewhere? >> like Smalltalk flushCache or Smalltalk vm flushCache (and it is a good >> moment to reify the VM). So after we can do: Smalltalk vm version. Smalltlak >> vm flushCache, Smalltalk vm parameterAt: , etc.... >> >> >> >> If you don't like doing "Smalltalk vm" then we can create a VM class >> with all class methods, or a singleton and use #current or a singleton and >> put it in Smalltalk globals...etc >> >> >> >> We will need to fix a couple of senders, thus. >> >> >> >> What do you think? For me is really confusing, and you don't >> understand it until you see the primitive implementation. >> >> >> >> Cheers >> >> >> >> -- >> >> Mariano >> >> http://marianopeck.wordpress.com >> >> >> > >> > -- >> > _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: >> > Alexandre Bergel http://www.bergel.eu >> > ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. >> > >> > >> > >> > >> > >> > >> >> >> > > > -- > Mariano > http://marianopeck.wordpress.com > > -- Mariano http://marianopeck.wordpress.com

April 26, 2011

Re: [Pharo-project] Why we have Behavior >> flushCache?

by Mariano Martinez Peck

On Tue, Apr 26, 2011 at 1:56 PM, Camillo Bruni <camillo.bruni(a)inria.fr>wrote: > How about putting such methods on an Interpreter class? > > that's why I was saying by "reify the VM" > On 2011-04-26, at 14:35, Alexandre Bergel wrote: > > > Not using the receiver is indeed misleading. I thought it simply flushes > the cache for a particular class > > > > Alexandre > > > > > > On 26 Apr 2011, at 06:34, Mariano Martinez Peck wrote: > > > >> Hi. As far as I can see, > >> > >> Behavior >> flushCache > >> "Tell the interpreter to remove the contents of its method lookup > cache, if it has > >> one. Essential. See Object documentation whatIsAPrimitive." > >> > >> <primitive: 89> > >> self primitiveFailed > >> > >> > >> And primitive 89 does nothing in particular with the receiver (the class > in this case). In both, InterpreterVM and Cog, the WHOLE cache is flushed, > there is NOTHING related to the receiver class. Of course, that's at least > what it looks for me (please tell me if I am wrong). > >> So...if this is the case, wouldn't make sense to move it elsewhere? > like Smalltalk flushCache or Smalltalk vm flushCache (and it is a good > moment to reify the VM). So after we can do: Smalltalk vm version. Smalltlak > vm flushCache, Smalltalk vm parameterAt: , etc.... > >> > >> If you don't like doing "Smalltalk vm" then we can create a VM class > with all class methods, or a singleton and use #current or a singleton and > put it in Smalltalk globals...etc > >> > >> We will need to fix a couple of senders, thus. > >> > >> What do you think? For me is really confusing, and you don't understand > it until you see the primitive implementation. > >> > >> Cheers > >> > >> -- > >> Mariano > >> http://marianopeck.wordpress.com > >> > > > > -- > > _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: > > Alexandre Bergel http://www.bergel.eu > > ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. > > > > > > > > > > > > > > > -- Mariano http://marianopeck.wordpress.com

April 26, 2011

Re: [Pharo-project] Why we have Behavior >> flushCache?

by Camillo Bruni

How about putting such methods on an Interpreter class? On 2011-04-26, at 14:35, Alexandre Bergel wrote: > Not using the receiver is indeed misleading. I thought it simply flushes the cache for a particular class > > Alexandre > > > On 26 Apr 2011, at 06:34, Mariano Martinez Peck wrote: > >> Hi. As far as I can see, >> >> Behavior >> flushCache >> "Tell the interpreter to remove the contents of its method lookup cache, if it has >> one. Essential. See Object documentation whatIsAPrimitive." >> >> <primitive: 89> >> self primitiveFailed >> >> >> And primitive 89 does nothing in particular with the receiver (the class in this case). In both, InterpreterVM and Cog, the WHOLE cache is flushed, there is NOTHING related to the receiver class. Of course, that's at least what it looks for me (please tell me if I am wrong). >> So...if this is the case, wouldn't make sense to move it elsewhere? like Smalltalk flushCache or Smalltalk vm flushCache (and it is a good moment to reify the VM). So after we can do: Smalltalk vm version. Smalltlak vm flushCache, Smalltalk vm parameterAt: , etc.... >> >> If you don't like doing "Smalltalk vm" then we can create a VM class with all class methods, or a singleton and use #current or a singleton and put it in Smalltalk globals...etc >> >> We will need to fix a couple of senders, thus. >> >> What do you think? For me is really confusing, and you don't understand it until you see the primitive implementation. >> >> Cheers >> >> -- >> Mariano >> http://marianopeck.wordpress.com >> > > -- > _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: > Alexandre Bergel http://www.bergel.eu > ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. > > > > > >

April 26, 2011

Re: [Pharo-project] Online Pharo Documentation

by Camillo Bruni

I am currentlz hacking together a fulltext search index using solr.. once that is running I consider the project to be more or less complete ;). and yeah, then I should get it up and running somewhere :) camillo On 2011-04-26, at 13:00, Bernat Romagosa wrote: > Just tried the demo image, very nice job! > > Maybe we could set up a running instance somewhere and link it from the Pharo website? > > Bernat. > 2011/4/22 laurent laffont <laurent.laffont(a)gmail.com> > That's really nice Camillo. Thank you. I hope to see it online soon ;) > > Laurent. > > On Fri, Apr 22, 2011 at 9:52 PM, Camillo Bruni <camillo.bruni(a)inria.fr> wrote: > Hi everyone, > > During the last month I worked a bit on my pharo web documentation tool. > It's a derivate of ruby's http://yardoc.org/ running on top of seaside. > > A test image is available here: > http://dl.dropbox.com/u/18184943/WebDoc.zip > Open the image then access the documentation in your browser > http://localhost:8080/doc > > There are still some glitches due to the javascript fancyness around. > > Features: > - browse and filter packages > - browse and filter classes of packages > - Class overview with markdown enabled comments > - fully annotated source code with popups for known types and selectors > > Missing: > - fully functional and slightly more intelligent search for methods, classes and packages > - hierarchical display of packages > - bookmarkable links don't work yet. > > Please feel free to add your own wishes, I hope to slowly improve this webdoc to make it really usable. > > v(^_^)v > camillo > >

April 26, 2011

Re: [Pharo-project] Better debugging...

by Toon Verwaest

I'm working on porting this idea already to the Glamour debugger. The idea is basically to use meta-circular interpreters as a way to specify what semantics you want, but for speed reasons you might want to flatten out the interpretation by combining both. This however should all be done behind the scenes, independent of the definition of the debugger. More elaborate debuggers such as the alias debugger can be implemented but there's no good way yet to separate the data spaces involved. That's where me and Erwann are looking at how to maybe use a GC that can run multiple different versions code at the same time while synchronizing the shared data between the versions of the code. For example in the case of the alias debugger, that needs aliases everwhere; but all the global data (classes, methods, ...) is not aliased. To make this work you actually want to have a "copy" of the global data that is aliased, but kept in sync with the original data. Here it starts making a lot of sense to have thread-local heaps + a versioned but synced global heap. Of course that's all still wishful thinking... step by step :) cheers, Toon On 04/25/2011 11:29 PM, Camillo Bruni wrote: > I think a nice way to have a decent debugger is to run the program on top of a changeable interpreter. Since a classical debugger is nothing else but an interpreter with slightly changed semantics. > > The object flow VM does a great job at tracing back stuff.. however it might pose a massive overhead since it collects tons and tons of data. > > In Pinocchio[1] we implemented very nice and simple debuggers by "just" changing the current interpreter to take a user-action into account on each message send. Since this is implemented on top of a metacircular interpreter this was quite simple to achieve. So the whole implementation of the known debugger functionality (step, over...) maybe took an afternoon. Since you have access to the full interpreter protocol it is very easy to intercept different sorts of actions. As described in the linked paper, it was very easy to implement the object-flow behavior in Pinocchio. > > So for me there is only one real way to have a decent debugger. And that means that it is a first-class interpreter whose semantics I can change at will. Thus I can create a specific debugger for my applications. > > But I guess that making the first-class interpreters work right away in Pharo will take a while to do, hence I propose the following additions to the current interpreter to make it partly useable: > > - interception of message sends > - per thread [ok] > - of specific classes [missing] > - of specific instances [missing] > > - interception of instance variable access, read/write > - per thread [missing] > - of specific classes [missing] > - of specific instances [missing] > - of specific instance variables [missing] > > Furthermore I really want to be able to script all of that stuff, eg. provide a simple test block that gets invoked on each message send. Returning true => halt, Returning false => continue execution. > > > [1] http://scg.unibe.ch/archive/papers/Verw10aPinocchio.pdf

April 26, 2011

Re: [Pharo-project] Why we have Behavior >> flushCache?

by Mariano Martinez Peck

On Tue, Apr 26, 2011 at 2:35 PM, Alexandre Bergel <alexandre.bergel(a)me.com>wrote: > Not using the receiver is indeed misleading. I thought it simply flushes > the cache for a particular class > You are not alone :) That's why I am proposing the change. Of course, a nice comment could be enough (even if I muuuuch prefer to have it in elsewhere), but I think it may be a good moment to reify the VM and delegate a couple of things to it. > > Alexandre > > > On 26 Apr 2011, at 06:34, Mariano Martinez Peck wrote: > > > Hi. As far as I can see, > > > > Behavior >> flushCache > > "Tell the interpreter to remove the contents of its method lookup > cache, if it has > > one. Essential. See Object documentation whatIsAPrimitive." > > > > <primitive: 89> > > self primitiveFailed > > > > > > And primitive 89 does nothing in particular with the receiver (the class > in this case). In both, InterpreterVM and Cog, the WHOLE cache is flushed, > there is NOTHING related to the receiver class. Of course, that's at least > what it looks for me (please tell me if I am wrong). > > So...if this is the case, wouldn't make sense to move it elsewhere? like > Smalltalk flushCache or Smalltalk vm flushCache (and it is a good moment to > reify the VM). So after we can do: Smalltalk vm version. Smalltlak vm > flushCache, Smalltalk vm parameterAt: , etc.... > > > > If you don't like doing "Smalltalk vm" then we can create a VM class with > all class methods, or a singleton and use #current or a singleton and put it > in Smalltalk globals...etc > > > > We will need to fix a couple of senders, thus. > > > > What do you think? For me is really confusing, and you don't understand > it until you see the primitive implementation. > > > > Cheers > > > > -- > > Mariano > > http://marianopeck.wordpress.com > > > > -- > _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: > Alexandre Bergel http://www.bergel.eu > ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. > > > > > > > -- Mariano http://marianopeck.wordpress.com

April 26, 2011

Re: [Pharo-project] Why we have Behavior >> flushCache?

by Mariano Martinez Peck

On Tue, Apr 26, 2011 at 12:51 PM, StÃ©phane Ducasse < stephane.ducasse(a)inria.fr> wrote: > what is the cache flushed? > what or when? What ? it flushed all the method lookup cache when? ok, for our refactoring the only thing that matters are the senders in the image. If you read the comment of the method it says "the method cache is flushed on every programming change and garbage collect" However, that seems not to be true. If I do a garbageCollect, Behavior>>flushCache is not sent (not even the primitive), And "every programming change" I don't know what it is. The senders I see in the image are: - Behavior >> superclass: aClass (it does a Object flushCache) - Time class >> benchmarkMillisecondClock (it does a Object flushCache.) - Time class >> benchmarkPrimitiveResponseDelay (it does a Object flushCache.) So, in fact, as much as I can see, all the senders of Behavior>>flushCache are sent to Object and not to a class in particular. Hence, it is one more probe that such method is independent of the receiver. I think that your point is valid and it would be good to do it. > > Stef > > On Apr 26, 2011, at 12:34 PM, Mariano Martinez Peck wrote: > > > Hi. As far as I can see, > > > > Behavior >> flushCache > > "Tell the interpreter to remove the contents of its method lookup > cache, if it has > > one. Essential. See Object documentation whatIsAPrimitive." > > > > <primitive: 89> > > self primitiveFailed > > > > > > And primitive 89 does nothing in particular with the receiver (the class > in this case). In both, InterpreterVM and Cog, the WHOLE cache is flushed, > there is NOTHING related to the receiver class. Of course, that's at least > what it looks for me (please tell me if I am wrong). > > So...if this is the case, wouldn't make sense to move it elsewhere? like > Smalltalk flushCache or Smalltalk vm flushCache (and it is a good moment to > reify the VM). So after we can do: Smalltalk vm version. Smalltlak vm > flushCache, Smalltalk vm parameterAt: , etc.... > > > > If you don't like doing "Smalltalk vm" then we can create a VM class with > all class methods, or a singleton and use #current or a singleton and put it > in Smalltalk globals...etc > > > > We will need to fix a couple of senders, thus. > > > > What do you think? For me is really confusing, and you don't understand > it until you see the primitive implementation. > > > > Cheers > > > > -- > > Mariano > > http://marianopeck.wordpress.com > > > > > -- Mariano http://marianopeck.wordpress.com

April 26, 2011

Re: [Pharo-project] Online Pharo Documentation

by Bernat Romagosa

Just tried the demo image, very nice job! Maybe we could set up a running instance somewhere and link it from the Pharo website? Bernat. 2011/4/22 laurent laffont <laurent.laffont(a)gmail.com> > That's really nice Camillo. Thank you. I hope to see it online soon ;) > > Laurent. > > On Fri, Apr 22, 2011 at 9:52 PM, Camillo Bruni <camillo.bruni(a)inria.fr>wrote: > >> Hi everyone, >> >> During the last month I worked a bit on my pharo web documentation tool. >> It's a derivate of ruby's http://yardoc.org/ running on top of seaside. >> >> A test image is available here: >> http://dl.dropbox.com/u/18184943/WebDoc.zip >> Open the image then access the documentation in your browser >> http://localhost:8080/doc >> >> There are still some glitches due to the javascript fancyness around. >> >> Features: >> - browse and filter packages >> - browse and filter classes of packages >> - Class overview with markdown enabled comments >> - fully annotated source code with popups for known types and >> selectors >> >> Missing: >> - fully functional and slightly more intelligent search for >> methods, classes and packages >> - hierarchical display of packages >> - bookmarkable links don't work yet. >> >> Please feel free to add your own wishes, I hope to slowly improve this >> webdoc to make it really usable. >> >> v(^_^)v >> camillo >> > >

April 26, 2011

Re: [Pharo-project] [Moose-dev] Re: removing versions from configurations

by StÃ©phane Ducasse

>> >> >>> If you want to work with a specific version, work with a specific version. Default is nothing but a baseline. >> Indeed I know that >> >> >>> You did not remove the other baseline, but you removed default. >> >> Yes I know. I did it on purpose. No way to load latest using default. > > But you did not circumvent anything. All I have to do is load the baseline10: and then I get the latest versions of packages :). yes but this is a different thing that load or loadDefault. It requires extra energy. >>> >> Looks like I control RPackage now. > > I am happy you are in control. Me too. I will work a lot on it from thursday. first address some of the flag: I did second understand what cyrille did for the integration with MC write more tests and see where we get. may be implement what we discussed. Stef

April 26, 2011