June 2013 - Pharo-dev - lists.pharo.org

brief explanation of the new class organizer

by Esteban Lorenzano

Hi, So... this is what I made las days (as a conclusion of many days before): The old class organizer was a mess. Basically, it considered all protocols as strings, and for worst it was arranging them into a flat array. While the array stuff is just an internal problem (objects outside doesn't know how are they managed, and that's good), having protocols-as-strings is essentially bad (no need for explanations about that). The new class organizer just do what is more or less obvious: 1) replaces string protocols for first class instances (now instances of Protocol). 2) internally, manages protocols and methods in protocols in a much better and fashion way. (btw, the implementation was made by Ben, thanks!) the real important part is the point 1. Now we can start to assign real behavior to Protocol, For instance, I will create an ExtensionProcol which will contain a direct pointer to it's real package. That means that we do not need weird computations to get a method extension package (and when the tools allows us, we could get rid off the *Blah). I would like also to be able to have "composite" protocols (like "private"+"accesing"), etc. It will allow a refactor of current RPackage implementation, allowing us to simplify the SystemAnnounce mechanism (and speeding up monticello, if my calculous are fine). And like this, there are still a huge room for improvement, I hope many that I do not even thought about :) cheers, Esteban

June 28, 2013

Exploring Heap Size Limits

by Sven Van Caekenberghe

Hi, I wrote a little test to explore the heap size limits of the current Pharo image+VM combination. I did some tests on a 8GB 64-bit Linux machine and the basic conclusion is that using a standard 1GB heap size you can allocate and GC with close to 1GB of actual data. I think that is pretty good news. The machine has 4 effective cores, so could easily run 4 busy images like that. Pushing the limits further, things get weird, up to the point of crashing. Sadly 2, 3 or 4 GB heap size, which should be theoretically possible, fail, thus limiting the amount of data that can be allocated and used. Anyway, my goal is to start the debate around this topic ;-) Here is the test script: t3@coolermaster:smalltalk$ cat memtest.st NonInteractiveTranscript stdout install. ! Transcript show: 'memtest.st'; cr. Smalltalk garbageCollect. Transcript show: SmalltalkImage current vm statisticsReport; cr. ! | count size data timeToRun | count := Smalltalk commandLine arguments first asInteger. size := Smalltalk commandLine arguments second asInteger. Transcript show: ('allocating: {1} times {2} bytes' format: { count. size }); cr. timeToRun := [ data := Array new: count streamContents: [ :out | count timesRepeat: [ out nextPut: (ByteArray new: size) ] ]. ] timeToRun. Transcript show: ('allocated: {1}' format: { (data collect: #size) sum humanReadableSIByteSize }); cr. Transcript show: ('time to run {1}' format: { Duration milliSeconds: timeToRun }); cr. Transcript show: '3 times GC'; cr. timeToRun := [ 3 timesRepeat: [ Smalltalk garbageCollect ] ] timeToRun. Transcript show: ('time to run {1}' format: { Duration milliSeconds: timeToRun }); cr. Transcript show: SmalltalkImage current vm statisticsReport; cr. ! SmalltalkImage current quitPrimitive. And here is a normal result running get.pharo.org/20+vm $ ./pharo Pharo.image memtest.st 512 1024000 memtest.st uptime 0h0m0s memory 25,076,708 bytes old 19,213,624 bytes (76.60000000000001%) young 5,384 bytes (0.0%) used 19,219,008 bytes (76.60000000000001%) free 5,857,700 bytes (23.400000000000002%) GCs 10 (12ms between GCs) full 1 totalling 30ms (25.0% uptime), avg 30.0ms incr 9 totalling 6ms (5.0% uptime), avg 0.7000000000000001ms tenures 0 allocating: 512 times 1024000 bytes allocated: 524.29 MB time to run 0:00:00:02.852 3 times GC time to run 0:00:00:00.196 uptime 0h0m3s memory 546,746,364 bytes old 543,520,476 bytes (99.4%) young 7,184 bytes (0.0%) used 543,527,660 bytes (99.4%) free 3,218,704 bytes (0.6000000000000001%) GCs 356 (9ms between GCs) full 89 totalling 2,743ms (86.4% uptime), avg 30.8ms incr 267 totalling 223ms (7.0% uptime), avg 0.8ms tenures 0 Since last view 346 (9ms between GCs) uptime 3.1s full 88 totalling 2,713ms (88.80000000000001% uptime), avg 30.8ms incr 258 totalling 217ms (7.1000000000000005% uptime), avg 0.8ms tenures 0 It takes about 2.5s to allocate about 512Mb. A GC then takes 30ms. You can play a bit with the 2 parameters to generate different datasets. Not changing any VM parameters, you can successfully do something like $ ./pharo Pharo.image memtest.st 1000 1024000 or $ ./pharo Pharo.image memtest.st 1024000 1000 thus allocating about 1GB. Playing with -mmap gives some weird results and unexpected slowness as well. Consider (the default -mmap is 1024m): $ ./pharo -mmap 1280m Pharo.image memtest.st 500 1024000 memtest.st uptime 0h0m0s memory 25,076,716 bytes old 19,213,480 bytes (76.60000000000001%) young 5,384 bytes (0.0%) used 19,218,864 bytes (76.60000000000001%) free 5,857,852 bytes (23.400000000000002%) GCs 12 (13ms between GCs) full 2 totalling 56ms (37.1% uptime), avg 28.0ms incr 10 totalling 6ms (4.0% uptime), avg 0.6000000000000001ms tenures 0 allocating: 500 times 1024000 bytes allocated: 512.00 MB time to run 0:00:00:13.66 3 times GC time to run 0:00:00:00.192 uptime 0h0m14s memory 533,946,384 bytes old 531,232,232 bytes (99.5%) young 7,644 bytes (0.0%) used 531,239,876 bytes (99.5%) free 2,706,508 bytes (0.5%) GCs 1,011 (14ms between GCs) full 501 totalling 13,819ms (98.7% uptime), avg 27.6ms incr 510 totalling 35ms (0.2% uptime), avg 0.1ms tenures 0 Since last view 999 (14ms between GCs) uptime 13.9s full 499 totalling 13,763ms (99.30000000000001% uptime), avg 27.6ms incr 500 totalling 29ms (0.2% uptime), avg 0.1ms tenures 0 So increasing the heap from 1024 Mb to 1280 Mb slows the allocation down to 14s. Increasing the heap size to 1792 Mb gives the following result: t3@coolermaster:smalltalk$ ./pharo -mmap 1792m Pharo.image memtest.st 1280 1048576 memtest.st uptime 0h0m0s memory 25,076,708 bytes old 19,213,592 bytes (76.60000000000001%) young 5,384 bytes (0.0%) used 19,218,976 bytes (76.60000000000001%) free 5,857,732 bytes (23.400000000000002%) GCs 12 (12ms between GCs) full 2 totalling 55ms (37.9% uptime), avg 27.5ms incr 10 totalling 6ms (4.1000000000000005% uptime), avg 0.6000000000000001ms tenures 0 allocating: 1280 times 1048576 bytes allocated: 1.34 GB time to run 0:00:00:35.258 3 times GC time to run 0:00:00:00.37 uptime 0h0m35s memory -783,340,292 bytes old -786,061,544 bytes (100.30000000000001%) young 7,460 bytes (0.0%) used -786,054,084 bytes (100.30000000000001%) free 2,713,792 bytes (-0.30000000000000004%) GCs 2,572 (14ms between GCs) full 1281 totalling 35,444ms (99.10000000000001% uptime), avg 27.700000000000003ms incr 1291 totalling 93ms (0.30000000000000004% uptime), avg 0.1ms tenures 0 Since last view 2,560 (14ms between GCs) uptime 35.6s full 1279 totalling 35,389ms (99.30000000000001% uptime), avg 27.700000000000003ms incr 1281 totalling 87ms (0.2% uptime), avg 0.1ms tenures 0 The allocation takes a lot longer, but the GC times are stable. I suspect that negative reporting has to do with SmallInteger limitations, but the question is if it is only a cosmetic problem or not. Again, I am not complaining, just exploring/wondering. Sven

June 28, 2013

[ANN] SS3 the canary site

by Tobias Pape

Dear fellow Smalltalkers For me to test new features of SqueakSource3 and for you to have a look on new features and fixed bugs, I have created http://canary.netshed.de/ which is the most recent, say alpha, deployment of SqueakSource3. It contains features that are not yet deployed at other ss3 sites, including the gemstone one. I cannot make any guaranty that code commited to the canary will survive forever; it is a play toy, so go play with it :) Attached a teaser image. Best -Tobias

June 28, 2013

Re: [Pharo-dev] Question about closure compilation

by Sven Van Caekenberghe

On 28 Jun 2013, at 22:42, Marcus Denker <marcus.denker(a)inria.fr> wrote: > > On Jun 28, 2013, at 9:26 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > >> Hi, >> >> Does the closure returned from >> >> | foo | >> foo := #( data 1 2 3 ). >> [ :x | x + 1 ] >> >> close over foo ? >> > No. > >> Since foo is not used, it should not, right ? >> > right. if we get the AST of your method > > > ast := (TT>>#tt) ast. > (ast scope lookupVar: 'foo') isEscaping > > --> false. OK, that seems OK. The reason I was asking was because I wanted to know if this JavaScript bug would hit us as well: http://point.davidglasser.net/2013/06/27/surprising-javascript-memory-leak.… It is slightly more complicated than the basic question of closing over an unreferenced variable though. I just can't figure out a way to quickly test it in Pharo. I think it is worth looking into. > Now the question is what the inspector of the context should show? > > e.g. > > tt > | foo | > foo := #( data 1 2 3 ). > ^[ :x | x + 1. thisContext inspect ] > > > if we now execute: > > TT new tt value: 2 > > the inspector pops up and shows both x and foo. > > The reason is that we need to decide what to show > > 1) the variable that is stored in the context > 2) all *possible* variables that I can write in the block and recompile and it works. > > 1) is this: > ast statements last value scope localTempNames > > 2) is this: > > ast statements last value scope allTemps > > Opal implements is like this because the old Compiler (DebuggerMethodMap) did it like this. > At first I thought this is wrong and I wanted to change it. > But then I thought that it is not that wrong: in the debugger the inspector of the context shows > not just the vars that are in the context, but all that *could* be there. All where I can lookup and > get a result, e.g. > > ast statements last value scope lookupVar: 'foo' > > Thus all those that I *could* write in the code inside the block. > > I am not sure, thoughâ¦ maybe we should change it. > > Marcus > > >> But it seems that the outer context of the block closure knows about foo, although the value seems nil. >> >> And if it closes over foo, foo's value/binding won't be GC-ed so long as there is a reference to the block closure, even though foo is not practically accessible. >> >> Sven > >

June 28, 2013

Re: [Pharo-dev] #30235 is bogus

by StÃ©phane Ducasse

>> Maybe we should revert the update. Having the 3.0 download be not usable for multiple days >> is not good. If you want but I do not have the time to do it now. I do not really think that this is problem since see sven message. >>> Hi guys >>> >>> please do not use the 30235 since we have to find a way to recreate the correct UI thread with >>> the worldState and others. I'm not sure that I will have the time to work on it this week-end >>> since I need to have a pair programmer (igor) to check the thread and other. >>> >>> Stef > > I played a bit with it and the image basically works. > All ReleaseTests pass ;-) > But there is an issue with redrawing when the image comes up, but is seems to go away with a 'Restore display'. yes Now we would like to have a really nice way to do it and do avoid this problem. > > And scrolling in lists while hovering works again ! > > Sven

June 28, 2013

Re: [Pharo-dev] Question about closure compilation

by Marcus Denker

On Jun 28, 2013, at 9:26 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > Hi, > > Does the closure returned from > > | foo | > foo := #( data 1 2 3 ). > [ :x | x + 1 ] > > close over foo ? > No. > Since foo is not used, it should not, right ? > right. if we get the AST of your method ast := (TT>>#tt) ast. (ast scope lookupVar: 'foo') isEscaping --> false. Now the question is what the inspector of the context should show? e.g. tt | foo | foo := #( data 1 2 3 ). ^[ :x | x + 1. thisContext inspect ] if we now execute: TT new tt value: 2 the inspector pops up and shows both x and foo. The reason is that we need to decide what to show 1) the variable that is stored in the context 2) all *possible* variables that I can write in the block and recompile and it works. 1) is this: ast statements last value scope localTempNames 2) is this: ast statements last value scope allTemps Opal implements is like this because the old Compiler (DebuggerMethodMap) did it like this. At first I thought this is wrong and I wanted to change it. But then I thought that it is not that wrong: in the debugger the inspector of the context shows not just the vars that are in the context, but all that *could* be there. All where I can lookup and get a result, e.g. ast statements last value scope lookupVar: 'foo' Thus all those that I *could* write in the code inside the block. I am not sure, thoughâ¦ maybe we should change it. Marcus > But it seems that the outer context of the block closure knows about foo, although the value seems nil. > > And if it closes over foo, foo's value/binding won't be GC-ed so long as there is a reference to the block closure, even though foo is not practically accessible. > > Sven

June 28, 2013

Re: [Pharo-dev] #30235 is bogus

by Camillo Bruni

On 2013-06-28, at 21:44, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > On 28 Jun 2013, at 19:54, Marcus Denker <marcus.denker(a)inria.fr> wrote: > >> Maybe we should revert the update. Having the 3.0 download be not usable for multiple days >> is not good. >> >> Marcus >> >> On Jun 28, 2013, at 7:29 PM, StÃ©phane Ducasse <stephane.ducasse(a)inria.fr> wrote: >> >>> Hi guys >>> >>> please do not use the 30235 since we have to find a way to recreate the correct UI thread with >>> the worldState and others. I'm not sure that I will have the time to work on it this week-end >>> since I need to have a pair programmer (igor) to check the thread and other. >>> >>> Stef > > I played a bit with it and the image basically works. > All ReleaseTests pass ;-) > But there is an issue with redrawing when the image comes up, but is seems to go away with a 'Restore display'. > > And scrolling in lists while hovering works again ! can confirm

June 28, 2013

Re: [Pharo-dev] #30235 is bogus

by Sven Van Caekenberghe

On 28 Jun 2013, at 19:54, Marcus Denker <marcus.denker(a)inria.fr> wrote: > Maybe we should revert the update. Having the 3.0 download be not usable for multiple days > is not good. > > Marcus > > On Jun 28, 2013, at 7:29 PM, StÃ©phane Ducasse <stephane.ducasse(a)inria.fr> wrote: > >> Hi guys >> >> please do not use the 30235 since we have to find a way to recreate the correct UI thread with >> the worldState and others. I'm not sure that I will have the time to work on it this week-end >> since I need to have a pair programmer (igor) to check the thread and other. >> >> Stef I played a bit with it and the image basically works. All ReleaseTests pass ;-) But there is an issue with redrawing when the image comes up, but is seems to go away with a 'Restore display'. And scrolling in lists while hovering works again ! Sven

June 28, 2013

Question about closure compilation

by Sven Van Caekenberghe

Hi, Does the closure returned from | foo | foo := #( data 1 2 3 ). [ :x | x + 1 ] close over foo ? Since foo is not used, it should not, right ? But it seems that the outer context of the block closure knows about foo, although the value seems nil. And if it closes over foo, foo's value/binding won't be GC-ed so long as there is a reference to the block closure, even though foo is not practically accessible. Sven

June 28, 2013

Re: [Pharo-dev] #30235 is bogus

by Marcus Denker

Maybe we should revert the update. Having the 3.0 download be not usable for multiple days is not good. Marcus On Jun 28, 2013, at 7:29 PM, StÃ©phane Ducasse <stephane.ducasse(a)inria.fr> wrote: > Hi guys > > please do not use the 30235 since we have to find a way to recreate the correct UI thread with > the worldState and others. I'm not sure that I will have the time to work on it this week-end > since I need to have a pair programmer (igor) to check the thread and other. > > Stef

June 28, 2013