September 2010 - Pharo-dev - lists.pharo.org

Re: [Pharo-project] One order of magnitude faster sets/dicts?!

by StÃ©phane Ducasse

tx levente Stef >>> This mail is quite long as I copied it from a conversation with Toon Verwaest. Toon and Camillo re-implemented hashed collections for Pinocchio (http://scg.unibe.ch/research/pinocchio) Their benchmarks have shown that the new implementation is significantly faster than the one in Pharo 1.1. The source code is available from www.squeaksource.com/p.html (they haven't set a license, but I'm pretty sure they'll release their code as MIT). >>> I post this here to find somebody knowledgeable in the area to take a look at the code and figure out whether this implementation would be interesting for Pharo... >>> Cheers, >>> Adrian >>> citing Toon: >>> We ran some benchmarks on our data structures (It's in comparison with and benchmarked in standard Pharo 1.1). We just add / remove integers within a certain range thus always perfectly distributed making it ideal for Pharo dictionaries. >>> The numbers show how long a certain benchmark took on average in 20 runs of the benchmark and its standard deviation over those runs. A single benchmark run generally consists of many individual operations on the collection in question. For example: >>> benchIncludes >>> 1 to: set size * 2 do: [ :i| >>> set includes: i ] >>> where the set has size 10000. Evaluating this once results in a single run. So the number at the top of the benchmark is the sum of all averages with the average stdev. >>> The results: >>> PDictionary: 15.8 +/-1.7 >>> Do 0.472 +/-0.079 >>> RemoveKey 0.010 +/-0.021 >>> AtPut 0.947 +/-0.02 >>> AtIfAbsentPut 1.64 +/-0.96 >>> AtPutExisting 0.198 +/-0.011 >>> KeysAndValuesDo 0.502 +/-0.09 >>> IncludesKey 0.365 +/-0.024 >>> AtPutNew 4.2 +/-1.3 >>> Includes 7.40 +/-0.33 >>> Dictionary: 138.2 +/-4.7 >>> Do 0.830 +/-0.098 >>> RemoveKey 10.292 +/-0.077 >>> AtPut 0.957 +/-0.044 >>> AtIfAbsentPut 1.52 +/-0.95 >>> AtPutExisting 0.203 +/-0.011 >>> KeysAndValuesDo 0.850 +/-0.096 >>> IncludesKey 80.25 +/-0.25 >>> AtPutNew 3.5 +/-1.3 >>> Includes 39.8 +/-4.4 >>> PSet: 1.767 +/-0.057 >>> Remove 0.008 +/-0.018 >>> Add 0.845 +/-0.039 >>> AddExisting 0.310 +/-0.021 >>> AddNew 0.240 +/-0.021 >>> Includes 0.365 +/-0.024 >>> Set: 70.9 +/-1.2 >>> Remove 7.737 +/-0.051 >>> Add 0.755 +/-0.022 >>> AddExisting 0.305 +/-0.015 >>> AddNew 2.473 +/-0.03 >>> Includes 59.6 +/-1.2 >>> Obviously the very slight differences have to be taken with a grain of salt, but whenever it's more than 10x faster it's quite unlikely that it's due to some random runtime variation that might totally change in another run. >> >> I'm a bit skeptic, because 10x improvement is pretty hard to get if the benchmark is not flawed and the code doesn't use VM support. > > Okay, I found the benchmark code, and it's flawed. The distribution of the keys is not uniform. Actually it's far from uniform, it's just 1..dictSize. > Anyway I ran the benchmarks in Squeak 4.2 alpha and got the following results: > > PBDictionary: 0.1249 +/-0.0067 > RemoveKey 6.7e-500 +/-4.6e-5 > AtPutNew 0.0648 +/-0.0048 > Do 0.00160 +/-0.0001 > AtPutExisting 0.003300000000000002 +/-0.00018 > AtIfAbsentPut 0.0192 +/-0.0035 > AtPut 0.01557000000000001 +/-0.00028 > IncludesKey 0.0180 +/-0.0029 > KeysAndValuesDo 0.00223 +/-0.0001200000000000001 > Includes 0.000133 +/-6.3e-5 > > PBSTDictionary: 0.2204 +/-0.0063 > RemoveKey 0.07203 +/-0.00032 > AtPutNew 0.0434 +/-0.0055 > Do 0.00 +/-0.0 > AtPutExisting 0.00260 +/-0.00013 > AtIfAbsentPut 0.01193 +/-0.0002 > AtPut 0.0147 +/-0.0031 > IncludesKey 0.01157000000000001 +/-0.0002400000000000001 > KeysAndValuesDo 0.002200 +/-7.4e-5 > Includes 0.05990000000000003 +/-0.00019 > > The PDictionary is only better at 2 benchmarks: > Includes (not IncludesKey!) and RemoveKey which are both rarely used. In all other tests Squeak's Dictionary implementation is faster in this flawed benchmark. > > > Levente > >> >> I wonder what the #pPrimitive:plugin: pragmas stand for in PDictionary's >> #at:ifAbsent: and #at:put:. >> >> Where can I find the benchmark code? >> >>> An important thing for our datastructures is that they don't degrade since we don't have colliding hashes; but we didn't really test that yet (although it becomes obvious in some of the benchmark results such as removing of elements and testing presence of elements). >>>> Is there anything special that one would neet to consider when replacing the old implementation in Pharo with yours? >>> What doesn't happen at the moment is shrinking dictionaries after they have grown significantly. I wouldn't know at what point to do so either; nor do I think Dictionary does that? >> >> The current HashedCollections don't shrink. Remove is a rarely used operation. >> >>> There is a parameter indicating how many elements can be added before it switches from a SmallDictionary version to a dictionary with buckets. This is set to 20 by default, and I have no idea if it makes sense to change it; I didn't really profile what the optimal number is. Maybe it makes sense to make that a constant in the code rather than an instance variable to save a bit of space and time ... >>> We have our own PHashedCollection subclassing from Collection; so the biggest part of the hierarchy is compatible with what Pharo has. Mostly while porting Pinocchio-specific stuff will have to be removed; mostly annotations related to Pinocchio Primitives and exporting directives. This is all straightforward though. >>> The current "ratio" of hashes to elements is set to 500%. So basically when you have 32 buckets it will only grow when it contains 160 elements (or key-value pairs), so 5 elements per bucket on average. This seems to not make it slower which is understandable since SmallDictionary is pretty fast for small amounts of elements; but this ratio might have to be tweaked depending on the kind of objects and is currently a parameter. The advantage of using 500% is that you have A LOT less garbage. >>> Oh, and our dictionaries do use a lot less garbage, since Pharo's dictionaries use association objects for each key-value pair. We don't, so we never generate garbage when you remove an object. >>> Our Sets on the other hand do use a bit more memory because of the bucket scheme. Sets don't use associations so we don't have an edge there :) Only performance-wise in that case. >> >> One could implement the current dictionaries without associations, that's just less object-oriented. That would generate even less "garbage". >> >>> The hashes are masked with the amount of buckets - 1, since the buckets are always a power of 2. >>> This implies that the lower bits of the hash are the most significant ones. This does not work together with the current identityHash / IdentityDictionary since the 12bit hash is shifted to the left by 18 (or so). This would make all the lower 18 bits 0 and all objects thus ending up in the first bucket. >>> That is a very important thing to note enough. However, I also think it's a bit strange to just bitshift the identityHash by 18. You don't win any significant numbers with it ... you have 12bit hashes; no way to improve that (without changing the image format). Since we overfill our dictionaries however this works quite ok with those bad hashes; and we'll only start wasting a bit of space after you have added 20480 elements to the dictionary (you have 4096 different hashes with 12 bits, * 5). >> >> Let's imagine that we want to put 10000 objects into a IdentitySet. Each object has it's identityHash in the 0..4095 range. The hash function is very simple: hash \\ capacity + 1. Therefore the values of the hash function will all be in 1..4096, even if capacity is ~15000. This causes a lot of collisions (the table will be consist of a few very long chains degrading performance to unacceptable leves). These collisions are avoided by the shift. >> >> >> Levente >> >>> _______________________________________________ >>> Pharo-project mailing list >>> Pharo-project(a)lists.gforge.inria.fr >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >> _______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Sept. 14, 2010

Re: [Pharo-project] Strange behavior with TestRunner

by Guillermo Polito

Mmm, so we should change -the tests to decouple them a bit from the method dictionary, or -the Run coverage :). On Tue, Sep 14, 2010 at 9:19 AM, Alexandre Bergel <alexandre(a)bergel.eu>wrote: > You probably rely on what has to be found in a method dictionary. "Run > Coverage" replaces compiled methods by a plain object, in which run:with:in: > is executed [*]. If in your test you expect to find a compile method, then > you may surprises. > > Cheers, > Alexandre > > [*] > http://www.iam.unibe.ch/~scg/Archive/Papers/Bergel06bRDLPrototyping.pdf<http://www.iam.unibe.ch/%7Escg/Archive/Papers/Bergel06bRDLPrototyping.pdf> > > > On 14 Sep 2010, at 14:14, Noury Bouraqadi wrote: > > > On the pharo 1.2-12141, I run the SocketStreamTest. All green. > > But, when I try the "Run Coverage" all turn red. > > > > Any hint? > > > > Noury @ ESUG 2010 - Barcelona :-) > > --- > > 18th International Smalltalk Joint Conference, September 13 -17 2010 > Barcelona, Spain. > > http://www.esug.org/Conferences/2010 > > > > > > > > > > _______________________________________________ > > Pharo-project mailing list > > Pharo-project(a)lists.gforge.inria.fr > > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > -- > _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: > Alexandre Bergel http://www.bergel.eu > ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. > > > > > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

Sept. 14, 2010

[Pharo-project] % of unused objects?

by Alexandre Bergel

Hi Mariano, I also would like to know what is the % of unused objects in my applications. How can I do it? Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

Sept. 14, 2010

Re: [Pharo-project] Automatically (OB?) run class methods?

by Mariano Martinez Peck

On Tue, Sep 14, 2010 at 12:04 PM, Alexandre Bergel <alexandre(a)bergel.eu>wrote: > Hi Mariano, > > Before I propose the test integration in system browser a few years ago, > people used a lot comments like "self debug: #testFoo". > Adding the command cmd-T greatly reduced these kind of comment. > > Probably a kind of "send" like method will help reduce these kind of > comment. > > As Lukas said, I personally often select the method body. But it happens > that the method body is larger than the code pane, which requires too much > time to select. > > At least a cmd+a and then a cmd+d anyway, I wasn't ware of cmd+d :) Maybe adding that to the menu in the case of class side methods? These will help to remove those kind of comments > Cheers, > Alexandre > > > On 14 Sep 2010, at 11:38, Mariano Martinez Peck wrote: > > > HI guys. I tend to see these pattern everywhere: > > > > openLoadAndQueryWorkspace > > "self openLoadAndQueryWorkspace" > > (Workspace new > > acceptContents: ' > > â¦â¦ > > > > > > Having to have that kind of thing from my point of view, reveals a > weakness in our IDE. Furthermore, when we rename the method, the comment is > not refactored. > > > > Now I was thinking if it would be interesting or not, to be able to > directly run class side methods. It would be nice: > > > > - To have an entry in the context (right click) menu, only when selecting > a class method, to directly evaluate it. Maybe we can call it "Evaluate it" > or "Send message" or "Do it" â¦ I don't know. > > > > - To have a shortcut for that. Thenâ¦we are inside that method for > example, and with a shortcut we can evaluate itâ¦no need to go up to the name > and evaluate the comment. > > > > What do you think? > > > > Cheers > > _______________________________________________ > > Pharo-project mailing list > > Pharo-project(a)lists.gforge.inria.fr > > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > -- > _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: > Alexandre Bergel http://www.bergel.eu > ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. > > > > > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

Sept. 14, 2010

Re: [Pharo-project] Strange behavior with TestRunner

by Alexandre Bergel

You probably rely on what has to be found in a method dictionary. "Run Coverage" replaces compiled methods by a plain object, in which run:with:in: is executed [*]. If in your test you expect to find a compile method, then you may surprises. Cheers, Alexandre [*] http://www.iam.unibe.ch/~scg/Archive/Papers/Bergel06bRDLPrototyping.pdf On 14 Sep 2010, at 14:14, Noury Bouraqadi wrote: > On the pharo 1.2-12141, I run the SocketStreamTest. All green. > But, when I try the "Run Coverage" all turn red. > > Any hint? > > Noury @ ESUG 2010 - Barcelona :-) > --- > 18th International Smalltalk Joint Conference, September 13 -17 2010 Barcelona, Spain. > http://www.esug.org/Conferences/2010 > > > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

Sept. 14, 2010

[Pharo-project] Strange behavior with TestRunner

by Noury Bouraqadi

On the pharo 1.2-12141, I run the SocketStreamTest. All green. But, when I try the "Run Coverage" all turn red. Any hint? Noury @ ESUG 2010 - Barcelona :-) --- 18th International Smalltalk Joint Conference, September 13 -17 2010 Barcelona, Spain. http://www.esug.org/Conferences/2010

Sept. 14, 2010

Re: [Pharo-project] Settings for Shout syntax highlighting

by Simon Denier

Following some hints from Alain, a new version (it just removes a method extension) Settings-Shout-simon_denier.2 in PharoInbox On 13 sept. 2010, at 11:00, Simon Denier wrote: > Hi > > A new package Settings-Shout in the Pharo Inbox for issue 1611 > > SLICE-Issue-1611-ShoutSettings-simon_denier.1 > > You can now set how (for example), instance variables, selector patterns, class references appears in the Shout pane. > One point is that Shout comes with a list of hundred token types which can be individually customized, some of which I have no idea what they represent (externalCallTypePointerIndicator?). To keep things manageable, I defined some groups of logically related token types (like the gorup 'selector patterns' for token types patternKeyword, patternBinary, patternUnary). > > "Class SHGroupStyle manages settings for the Shout syntax highlighting. Token types are organized in logical groups which will share the same style. Currently, only color and emphasis can be edited through Shout settings. Text font and size are managed through the more general appearance setting. Changing a style setting on a SHGroupStyle automatically applies the style. > > Groups are defined in the class-side method initializeGroups. Alternatively, one can set its own style table using Shout tokens by calling #customStyleTable (see SHTextStylerST80 class>>defaultStyleTable for the format)." > > > Help needed: > 1) code review > 2) review group definition, because I made some groups based partly on my intuition, partly on styling difference in the current difference: there may be a better partition of token types. > 3) review description for group, setting organization... > > > Thanks to Alain for helping with settings and especially polymorph widgets. > > -- > Simon > > > -- Simon

Sept. 14, 2010

Re: [Pharo-project] DemoMode should not change Fonts only sizes?

by Serge Stinckwich

2010/9/14 Mariano Martinez Peck <marianopeck(a)gmail.com>: > Hi Alain, I was trying the DemoMode before my talk at ESUG and I thought > that it would be better if this guy only changes the size of the fonts. I am > not sure if changing the fonts is good. > > For example, right now it is using FreeType with DejaVu. Since Pharo1.1 dev > doesn't use them, there is no folder "Fonts" with themâ¦..thus they are not > found and ugly fonts are rendered. Yes i already talk with Alain about this problem. One option was to include severals sizes of bitmap DejaVu font. BTW, we fix with Alain a problem of browser switches rendering with the demo font during the Smalltallk Camp here in Barcelona. -- Serge Stinckwich UMI UMMISCO 209 (IRD/UPMC), Hanoi, Vietnam Every DSL ends up being Smalltalk http://doesnotunderstand.org/

Sept. 14, 2010

[Pharo-project] Some thoughts...

by Alexandre Bergel

Hi, I really would like to see projects like Opal and Ocean happening. Even though I may not have the time to directly contribute to them, they are important and I will use them. I am struggling these days to instrument the whole Pharo image and this is not easy. I am eagerly waiting for Opal. I got burned in the past with networks. So, please continue... Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

Sept. 14, 2010

[Pharo-project] Thanks Marcus for the presentation of Pharo

by Alexandre Bergel

I like very much the way you described the role of Pharo for doing Research. Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

Sept. 14, 2010