December 2011 - Pharo-dev - lists.pharo.org

Re: [Pharo-project] Big Image Tests

by Sven Van Caekenberghe

Hendrik, On 13 Dec 2011, at 12:00, Henrik Sperre Johansen wrote: > IIRC, all pointers are sqInts, not uints, so even if compiling with large-address-aware I doubt you will get a 32bit image above 2GB without seeing sporadic failures. Yes, that is what I am seeing: I cannot raise -mmap above 2GB. Does that explain the negative sizes in #vmStatisticsReportString or for a 1GB ByteArray ? Sven

Dec. 13, 2011

Re: [Pharo-project] Startup Preferences

by Tudor Girba

Sounds cool. Doru On Tue, Dec 13, 2011 at 11:37 AM, Benjamin <benjamin.vanryseghem.pharo(a)gmail.com> wrote: > New version: > > Now there is a hierarchy for the setting file > > - first it lloks in the image diretory for st files > - if none is found, it looks into > ~/.config/pharo/${VERSION} > - if none is found it looks into > ~/.config/pharo/general > We could also add a look into > ~/.pharo/ > > > > Gofer new > squeaksource: 'StartupPreferences'; > package: 'StartupPreferences'; > load. > > Ben > > On Dec 1, 2011, at 6:50 PM, StÃ©phane Ducasse wrote: > > me too. > > Stef > > On Dec 1, 2011, at 6:38 PM, Camillo Bruni wrote: > > Yeah I would really love to see this. > > > maybe with a fixed lookup chain > > > 1. Settings from the Image directory/startup.st > > 2. Settings from ~/.pharo/$VERSION/startup.st > > 3. Settings from ~/.pharo/startup.st > > > cami > > > > > -- www.tudorgirba.com "Every thing has its own flow"

Dec. 13, 2011

Re: [Pharo-project] Scat - Public Release and announcement

by Serge Stinckwich

2011/12/13 Bernat Romagosa <tibabenfortlapalanca(a)gmail.com> > Hi all! > > Stef, thanks a lot! Actually we've always tried to rather adapt the > Scratch code to conform to Pharo, better than the other way around. In some > cases this wasn't possible, but we've tried to keep Scat from touching core > Pharo code as much as possible. > > Francois, check out this shot for a preview of what our idea is (not > functional at all, just a sketch!) http://i.imgur.com/ApJGd.png > > Serge, the thing is that the code in Scratch is terribly bad... > maintaining it is close to impossible. The project was coded by several > different students with different coding styles and barely any Smalltalk > experience. They took advantage of bugs in the old Squeak version used to > build Scratch, and mixed view and controllers so hard one can barely > distinguish them at all. We've made our best to get what we could to work > in Pharo, but we never tried to make the code better. > You are definitively right Bernat, this is a good example that even in Smalltalk, this is possible to write nightmare code. As they planned to do the next version in flash, i guess they will even worst ;-) Thank you your good work ! -- Serge Stinckwich UMI UMMISCO 209 (IRD/UPMC), Hanoi, Vietnam Matsuno Laboratory, Kyoto University, Japan (until 12/2011) http://www.mechatronics.me.kyoto-u.ac.jp/ Every DSL ends up being Smalltalk http://doesnotunderstand.org/

Dec. 13, 2011

Re: [Pharo-project] Big Image Tests

by Henrik Sperre Johansen

On 13.12.2011 00:07, Sven Van Caekenberghe wrote: > Today I tried to find out if it would be possible to allocate a large amount of memory in a VM. > > I used the official 32-bit VM (http://gforge.inria.fr/frs/download.php/29042/CogVM-Unix-13307.zip) and a Pharo 1.3 (http://gforge.inria.fr/frs/download.php/29273/Pharo-1.3-13315.zip) derived image on Ubuntu 11.04 (x86_64) Amazon AWS EC2 m1.large instance (about 8Gb RAM) and the following bash and Smalltalk scripts: > > run.sh > > #!/bin/bash > script_home=$(dirname $0) > script_home=$(cd $script_home&& pwd) > image=$script_home/pharo-t3.image > script=$script_home/run.st > args=$* > vm=$script_home/bin/CogVM > options="-mmap 1536m -vm-display-null -vm-sound-null" > #echo $vm $options $image $script $args > $vm $options $image $script $args > > run.st > > | args size monolithic object | > > FileStream stdout wantsLineEndConversion: true. > > Smalltalk garbageCollect. > > FileStream stdout > nextPutAll: 'Running run.st with args: '; > nextPutAll: Smalltalk commandLine arguments printString; > cr; > nextPutAll: Smalltalk vmStatisticsReportString. > > args := Smalltalk commandLine arguments. > size := args first asInteger. > monolithic := (args size> 1 and: [ args second = 'monolithic' ]). > > monolithic > ifTrue: [ > object := ByteArray ofSize: size * 1024 * 1024 ] > ifFalse: [ > object := Array ofSize: size * 1024. > 1 to: size * 1024 do: [ :each | > object at: each put: (ByteArray ofSize: 1024) ] ]. > > Smalltalk garbageCollect. > > FileStream stdout > nextPutAll: 'Allocation of a size '; > print: object size; > space; > print: object class; > nextPutAll: ' OK'; > cr; > nextPutAll: Smalltalk vmStatisticsReportString. > > Smalltalk quitPrimitive. > > These were the results trying to allocate 1GB: > > $ ./run.sh 1024 > Running run.st with args: #('1024') > uptime 0h0m0s > memory 49,103,224 bytes > old 36,625,732 bytes (74.60000000000001%) > young 9,812 bytes (0.0%) > used 36,635,544 bytes (74.60000000000001%) > free 12,467,680 bytes (25.400000000000002%) > GCs 8 (90ms between GCs) > full 2 totalling 212ms (29.400000000000002% uptime), avg 106.0ms > incr 6 totalling 10ms (1.4000000000000001% uptime), avg 1.7000000000000002ms > tenures 0 > Allocation of a size 1048576 Array OK > uptime 0h0m26s > memory -1,007,414,928 bytes > old -1,020,337,428 bytes (101.30000000000001%) > young 5,888 bytes (0.0%) > used -1,020,331,540 bytes (101.30000000000001%) > free 12,916,612 bytes (-1.3%) > GCs 590 (44ms between GCs) > full 62 totalling 14,735ms (56.6% uptime), avg 237.70000000000002ms > incr 528 totalling 10,225ms (39.300000000000004% uptime), avg 19.400000000000002ms > tenures 519 (avg 1 GCs/tenure) > Since last view 582 (44ms between GCs) > uptime 25.3s > full 60 totalling 14,523ms (57.400000000000006% uptime), avg 242.10000000000002ms > incr 522 totalling 10,215ms (40.300000000000004% uptime), avg 19.6ms > tenures 519 (avg 1 GCs/tenure) > > $ ./run.sh 1024 monolithic > Running run.st with args: #('1024' 'monolithic') > uptime 0h0m0s > memory 49,103,216 bytes > old 36,627,976 bytes (74.60000000000001%) > young 9,844 bytes (0.0%) > used 36,637,820 bytes (74.60000000000001%) > free 12,465,396 bytes (25.400000000000002%) > GCs 8 (46ms between GCs) > full 2 totalling 208ms (56.1% uptime), avg 104.0ms > incr 6 totalling 10ms (2.7% uptime), avg 1.7000000000000002ms > tenures 0 > Allocation of a size -1073741824 ByteArray OK > uptime 0h0m2s > memory -1,020,427,928 bytes > old -1,037,112,380 bytes (101.60000000000001%) > young 5,952 bytes (0.0%) > used -1,037,106,428 bytes (101.60000000000001%) > free 16,678,500 bytes (-1.6%) > GCs 13 (177ms between GCs) > full 4 totalling 725ms (31.5% uptime), avg 181.3ms > incr 9 totalling 14ms (0.6000000000000001% uptime), avg 1.6ms > tenures 0 > Since last view 5 (386ms between GCs) > uptime 1.9000000000000001s > full 2 totalling 517ms (26.8% uptime), avg 258.5ms > incr 3 totalling 4ms (0.2% uptime), avg 1.3ms > tenures 0 > > Basically it seems to work. > > What I find particulary strange is that #vmStatisticsReportString starts reporting negative sizes, also the size of the 1GB ByteArray is reported as negative. Why ? > > Tomorrow I might try to find out how much further I can take this ? How close to 4GB ? > > Anybody else tried this ? > > Sven > IIRC, all pointers are sqInts, not uints, so even if compiling with large-address-aware I doubt you will get a 32bit image above 2GB without seeing sporadic failures. Cheers, Henry

Dec. 13, 2011

Re: [Pharo-project] Cleaning up SmartRefStream use

by Henrik Sperre Johansen

On 11.12.2011 19:50, Schwab,Wilhelm K wrote: > When I last looked at SmartReferenceStream, it did two things that > were bad: > > > (2) reshape code was placed in SRS itself, not in the class that is > changing shape. The thing that is being deserialized (which is when > problems become apparent) should be in charge of the change. It > simplifies both packaging (fewer loose methods to consider, and > reduced opportunity for name collisions) and it also helps with using > proxies for special circumstances. > I'm not sure I understand/agree with what you are saying. What has been serialized is (almost) always an older version than what you are loading into, in which case I can't really see how it can answer the question "How do I turn into the current version?" correctly. Maybe there needs to be a split? Serialized version knows how to install itself as older versions (incase that's what you load into), while the image takes care if version in memory is newer than serialized? TBH, I also feel it's better to put the "import old versions" functionality as methods on the class itself rather than have an omnipotent materializer with special methods for installing all versions of all known classes in the system, I mean, exactly in the serializer/materializer do you intend to put such logic for custom user classes? Cheers, Henry

Dec. 13, 2011

Re: [Pharo-project] IdentitySet but using #hash rather than #identityHash ?

by Michael Roberts

Ok I see. Have you measured the collisions on a very big graph? What stats do you get? Cheers Mike On 13 Dec 2011, at 08:37, Mariano Martinez Peck <marianopeck(a)gmail.com> wrote: > > > On Tue, Dec 13, 2011 at 8:43 AM, Michael Roberts <mike(a)mjr104.co.uk> wrote: > Hi Mariano, when I read this thread I was a bit confused that you wanted an IdentitySet that used #hash. From that statement it sounded like you just wanted a Set. This would allow any object to define its own hash and importantly what equality means with #=. So if you want to delegate that to the object use a Set. > > > No, I cannot use a Set because I cannot have repetitions. This that I am asking is what we do in Fuel serializer while traversing the object graph. Each analyzed object is put in an IdentityDictionary (but the question is the same for IdentitySet) as key. To avoid cycles, I need to put each object only once. Since graphs can be very big, such dict could have lots of objects. > > However as the thread has gone on perhaps you want the identity relationship but you just wanted a bigger identity hash space? > > Yes, exactly. I were thinking if there could be a way (maybe...that's why I am asking) of improve its performance considering that I could have much more objects than 2^13. In other words, I wanted to see if I could avoid colisions. > > > IdentitySets are fast because they bypass any delegation. Once you have seen the object in your traversal (common usage pattern) that's it. You grab it's identity which is a pretty low level thing to do. > > Ok...it is a tradeoff. If I use #identityHash it is fast because there is no delegation and it is almost an immediate primitive. But I gues it will be slow if there are lots of colisions. Not using #identityHash but something else maybe could decrease maybe the amount of colisions, but maybe with the delegation it will gets slower. > > I will try with Levente idea of what it is done in SystemTracer: use as a hash the identityHash of the object mixed with the identityHash of its class. Maybe that decreases the colisions and at the same time I don't pay delegation (#class is special bytecode, so nothing, and ok..there are 2 sends to #identityhash but I don't thinnk it is that much). > > Anyway, thanks for the interesting post, I always learn :) > > > As for what happens with collections who knows. Depends. Relying on identity set semantics for a collection is easy. Set semantics is not so. Remember that both hash and equals are important to know if the set already contains the element. Depending on the collection implementation both of these could be composite in terms of the parts. Who knows where you end or how long it takes. I.e. if it is a function of the size of the collection and further collections are composite.... > > > yes... > > Cheers > Mike > > > > On Tuesday, December 13, 2011, Carlo <snoobabk(a)yahoo.ie> wrote: > > Hi Mariano > > I'm no expert either ;) > > Without having access to the exact code it would look like either you have a collection that references itself (which would break all collection implementations) or maybe the tests have just slowed down to the point where you think it's 'crashed'. > > Do you have anymore info or perhaps which methods you changed on IdentitySet? > > Cheers > > Carlo > > On 13 Dec 2011, at 1:57 AM, Mariano Martinez Peck wrote: > > > > > > On Tue, Dec 13, 2011 at 12:32 AM, Mariano Martinez Peck <marianopeck(a)gmail.com> wrote: > >> > >> > >> On Mon, Dec 12, 2011 at 1:56 PM, Carlo <snoobabk(a)yahoo.ie> wrote: > >>> > >>> Hi > >>> Wouldn't the fact that you use hash cause potential loops now? e.g. collection refers to another object that refers to first collection. --> aCollection>>hash references an item which causes this current collection's hash to be called again? > >> > >> Hi Carlo. I am still newbie with Collections but I think I am having exactly that problem. During my tests, it loops in Collection >> #hash when sending #hash to its elements. > >> Sorry, but I couldn't undertand what is the cause of the problem? why it doesn't work while it does using #identityHash? could you elaborate? > >> > > > > Well, now I understood, and I understand also why it doesn't happen with #identityHash. But what happens then with regular Dictionaries using #hash? why it doesn't happen there? > > > >> > >> thanks > >> > >>> > >>> identityHash is deterministic in this case. > >>> Does this help? > >>> Cheers > >>> Carlo > >>> On 12 Dec 2011, at 10:58 AM, Mariano Martinez Peck wrote: > >>> Hi guys. I hope this is not a very stupid question. Background: in Fuel we have a IdentityDictionary where we put each of the objects we find while traversing the graph. We need to use IdentitySet because we cannot have repetitions (and to avoid loops) so we NEED to use #==. In such dictionary we put ALL objects of the graph, so it can be very big. Since IdentitySet uses #identityHash, it means it will be using those ONLY 12 bits in the object header. It means that we have 2^12 = 4096 different values. > >>> > >>> Question: having explained the previous, I wanted to be able to use #hash rather than #identityHash since several classes implement #hash and hence I thought that using #hash I could have less colisions and hence a better performance. I tried to make a subclass of IdentitySet that uses #hash rather than #identityHash but my image freezes. I also tried something like: > >>> > >>> set := PluggableSet new. > >>> set hashBlock: [ :elem | elem hash ]. > >>> set equalBlock: [ :a :b | a == b ]. > >>> > >>> But it doesn't work either. I works with simple tests in a workspace but when I run the full tests of Fuel, it enters in a loop in the method #hash of Collection.. > >>> > >>> Anyway, my question is, should that work? if not, what is the exact reason? > >>> > >>> Thanks in advance, > >>> > >>> -- > >>> Mariano > >>> http://marianopeck.wordpress.com > >>> > >>> > >> > >> > >> > >> -- > >> Mariano > >> http://marianopeck.wordpress.com > >> > > > > > > > > -- > > Mariano > > http://marianopeck.wordpress.com > > > > > > > > > > -- > Mariano > http://marianopeck.wordpress.com >

Dec. 13, 2011

Re: [Pharo-project] Progress bar

by Henrik Sperre Johansen

On 13.12.2011 11:01, StÃ©phane Ducasse wrote: > Hi > > I was looking how we could improve the situation of the progress bar. > We could "simply" remove some annoying updates (I did some first analysis because I got tired of this). > I will try to continue so that we have less updates all the times. > > Now what I see is that there is also an architectural problem. > I will try to explain it: each method in different classes (change set, MCloader, FileStream), can be used in > different scenarios and depending on the path you want or not to raise feedback. > A bad solution would be to duplicate the method and pass a boolean to indicate whether the feedback should be > raised or not. > > Now I have the impression that the architecture should be: > - the different part raised events (but this is not clear how) > - the top level (the part launching) can absorb and aggregate event to give feedback. > I do not know if it makes sense. Do you know how this is done in other systems. > > Stef Makes sense to me. My suggestion: Replace with Announcing progress has been made. (e.g. StartedActivity, ProgressedActivity, EndedActivity ) The places you want visual feedback (ie when invoked from a GUI), install an listener to the announcements (A ProgressMorph?) which displays the progress bar(s). Cheers, Henry

Dec. 13, 2011

Re: [Pharo-project] Startup Preferences

by Benjamin

New version: Now there is a hierarchy for the setting file - first it lloks in the image diretory for st files - if none is found, it looks into ~/.config/pharo/${VERSION} - if none is found it looks into ~/.config/pharo/general We could also add a look into ~/.pharo/ Gofer new squeaksource: 'StartupPreferences'; package: 'StartupPreferences'; load. Ben On Dec 1, 2011, at 6:50 PM, StÃ©phane Ducasse wrote: > me too. > > Stef > > On Dec 1, 2011, at 6:38 PM, Camillo Bruni wrote: > >> Yeah I would really love to see this. >> >> maybe with a fixed lookup chain >> >> 1. Settings from the Image directory/startup.st >> 2. Settings from ~/.pharo/$VERSION/startup.st >> 3. Settings from ~/.pharo/startup.st >> >> cami >> > >

Dec. 13, 2011

[Pharo-project] Progress bar

by StÃ©phane Ducasse

Hi I was looking how we could improve the situation of the progress bar. We could "simply" remove some annoying updates (I did some first analysis because I got tired of this). I will try to continue so that we have less updates all the times. Now what I see is that there is also an architectural problem. I will try to explain it: each method in different classes (change set, MCloader, FileStream), can be used in different scenarios and depending on the path you want or not to raise feedback. A bad solution would be to duplicate the method and pass a boolean to indicate whether the feedback should be raised or not. Now I have the impression that the architecture should be: - the different part raised events (but this is not clear how) - the top level (the part launching) can absorb and aggregate event to give feedback. I do not know if it makes sense. Do you know how this is done in other systems. Stef

Dec. 13, 2011

Re: [Pharo-project] Scat - Public Release and announcement

by Bernat Romagosa

Hi all! Stef, thanks a lot! Actually we've always tried to rather adapt the Scratch code to conform to Pharo, better than the other way around. In some cases this wasn't possible, but we've tried to keep Scat from touching core Pharo code as much as possible. Francois, check out this shot for a preview of what our idea is (not functional at all, just a sketch!) http://i.imgur.com/ApJGd.png Serge, the thing is that the code in Scratch is terribly bad... maintaining it is close to impossible. The project was coded by several different students with different coding styles and barely any Smalltalk experience. They took advantage of bugs in the old Squeak version used to build Scratch, and mixed view and controllers so hard one can barely distinguish them at all. We've made our best to get what we could to work in Pharo, but we never tried to make the code better. What we mean when we say we will not maintain it anymore is that we are not going to perform major fixes to the code any further. We will not waste more time trying to make importing/exporting Scratch files better, or rack our brains to get the sound recorder to work. But if somebody is using the Scat code for some project and needs a small fix which we can help with, we will of course offer our help! After all this time we think the only good approach to enhance the code in Scratch would be to erase it all, rethink it all over and do it right from the beginning, which is the approach the BYOB guys are taking in Berkeley with their Snap project, and the results are so far pretty impressive: http://chirp.scratchr.org/dl/experimental/JsMorphic/nasciturus.html So, what are we to do with all the time and effort we put into Scat? Well, we do have a lot of useful stuff ported to Pharo! We can capture webcam images, we have a graphic block execution engine, a paint editor, a sound engine, etc. So not all efforts have been in vain! Cheers! 2011/12/13 Serge Stinckwich <serge.stinckwich(a)gmail.com> > 2011/12/13 Bernat Romagosa <tibabenfortlapalanca(a)gmail.com>: > > Hi list, > > > > Joan GÃ¼ell and I have been working on Scat -the Scratch port to > Pharo- quite > > a lot recently. We wanted to push it to a point where we could release > the > > first reasonably working version, which we've just managed today. > > Great work Joan & Bernat ! > > > Please take a look at the release announcement (which should be read in a > > bit of a humorous tone), in which we explain what's up with the project > and > > what future is left for it. > > > > For the tl;dr version, please read: > > > > Many things are working, many aren't, but the state of the project > allows us > > to move into more interesting stuff based on the graphic programming > > metaphor. > > Scat is a time-consuming, brain-smashing, life sucking-project. > > We no longer will be maintaining this massive unmaintainable code. > > Suddenly: free time for other projects! > > Why do you say that you will not maintain the project anymore ? > Do you mean this is better to rebuild something better ? > > Any news from Scratch about a relicensing of the core ? > > Regards, > -- > Serge Stinckwich > UMI UMMISCO 209 (IRD/UPMC), Hanoi, Vietnam > Matsuno Laboratory, Kyoto University, Japan (until 12/2011) > http://www.mechatronics.me.kyoto-u.ac.jp/ > Every DSL ends up being Smalltalk > http://doesnotunderstand.org/ > > -- Bernat Romagosa.

Dec. 13, 2011