October 2010 - Pharo-dev - lists.pharo.org

[Pharo-project] Issue 3061: ZipArchiveTest should be removed Compression package

by NicolÃ¡s Paez

Code and Test should be in different packages, to execute your applications you don't need tests. Test should have their own package. http://code.google.com/p/pharo/issues/detail?id=3061 Regards, Saludos! Nico. blog: nicopaez.wordpress.com

Oct. 5, 2010

Re: [Pharo-project] Another fixes to finalization (was Re: [update 1.2] #12161 - #12172)

by StÃ©phane Ducasse

>>> > > Moreover, case (b) comes from the inability to dynamically add extra > slot(s) to object, > which you can easily do in Self or JavaScript, but can't do in smalltalk. > > > > It can be done better, with extra VM features, but without those, it's pretty close to best possible solution IMHO. > There are already two different classes for a) and b): WeakRegistry and WeakKeyDictionary (and subclasses). Igor Levente is right. Having dynamic slots to object is nearly a paradigm shift. So let us focus on what we can do now. Now is there a technical problem to apply the solution implemented by levente in Squeak? Because if not this is what we should do. Stef

Oct. 5, 2010

Re: [Pharo-project] Ocean (was Re: Less plugins, more Smalltalk code! (was: Re: Pharo 1.1.1 OneClickCogVm))

by StÃ©phane Ducasse

On Oct 4, 2010, at 10:23 PM, Levente Uzonyi wrote: > On Mon, 4 Oct 2010, Sven Van Caekenberghe wrote: > >> Noury, >> >> One of the other aspects that I find important is elementary efficiency. >> >> A lot of stream related code in Smalltalk is not efficient, the actual data is copied around like crazy, turning it from a stream to a collection and then back into a stream multiple times. For Zinc HTTP Components I made a lot of effort to make it possible to read data from a socket stream in true streaming fashion (instead of just returning a byte array). Now, the idea was then that for example JPEGReadWriter>>#nextImage would work transparaently on that raw stream. Sadly, the code in JPEGReadWriter and friends just reads everything into an array before it starts to work ! >> >> Doing a #nextPutAll: <some byte array> should really try never to copy the array. Similary, a #next: <count> into: <some byte array> should similary write directly into the given array. What I see in the current SocketStream is <censored/> ;-) > > This is why we implemented our own SocketStream in PostgresV3. what is the license? Stef > >> >> But OK, maybe this is easier said than done. The step from Smalltalk to system call should also not copy if that is possible, but then FFI and GC come into play. > > I wonder if there's a language, that provides socket access via FFI. > > > Levente > >> >> Anyway, good luck, I am willing to try using Ocean as soon as it becomes available. >> >> Sven >> >> On 04 Oct 2010, at 10:59, Noury Bouraqadi wrote: >> >>> >>> On 3 oct. 2010, at 16:28, Schwab,Wilhelm K wrote: >>> >>>> For Ocean to succeed, you either have to work really hard to make non-blocking calls (I'm thinking of SSL sockets), or you need the ability to call functions on OS threads. The calls need to block their calling Process and let the other Processes run as though nothing is happening. >>>> >>> You're right. >>> Another option is having a multi-threaded VM. Eliot seem to work on it. >>> So far we're working on cleaning up the design of basic functionalities before going on. >>> Then, we'll try to find a workaround for non-blocking calls. >>> All ideas are welcome :-) >>> >>> BTW, some people (you included) proposed to help for the Ocean project. >>> Luc and I want to thank you and we are happy that others are willing to help. >>> We thought about the best way to integrate others help. >>> Our conclusion is that we should first do alone (luc and me) the reference design, before integrating other developers. >>> So, we all can work on solid foudations. >>> >>> Noury >>> >>> >>> >>> _______________________________________________ >>> Pharo-project mailing list >>> Pharo-project(a)lists.gforge.inria.fr >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >> >> _______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Oct. 5, 2010

Re: [Pharo-project] Xml Parser

by Levente Uzonyi

On Mon, 4 Oct 2010, Guillermo Polito wrote: > I looked after it in squeaksource and I didn't find it... Can some one > enlight me? http://www.squeaksource.com/XMLSupport.html Levente > > Cheers! >

Oct. 5, 2010

Re: [Pharo-project] Xml Parser

by Schwab,Wilhelm K

Well, I *thought* I could, until I noted where it sits in the list of packages in the image I just built - after Magritte. That had me wondering whether it came in with Seaside (not impossible), but it is also just in front of SIXX: ftp://swikis.ddo.jp/SIXX/squeak/SIXX20091115.sar which can be found at http://www.mars.dti.ne.jp/~umejava/smalltalk/sixx/index.html and I do explicitly load the .sar. I don't see anything else special that I do to load an XML parser, so absent a better answer, try installing SIXX and see if you end up with XML classes. Bill ________________________________________ From: pharo-project-bounces(a)lists.gforge.inria.fr [pharo-project-bounces(a)lists.gforge.inria.fr] On Behalf Of Guillermo Polito [guillermopolito(a)gmail.com] Sent: Monday, October 04, 2010 8:01 PM To: Pharo-project(a)lists.gforge.inria.fr Subject: [Pharo-project] Xml Parser I looked after it in squeaksource and I didn't find it... Can some one enlight me? Cheers!

Oct. 5, 2010

[Pharo-project] Xml Parser

by Guillermo Polito

I looked after it in squeaksource and I didn't find it... Can some one enlight me? Cheers!

Oct. 5, 2010

Re: [Pharo-project] Another fixes to finalization (was Re: [update 1.2] #12161 - #12172)

by Levente Uzonyi

On Tue, 5 Oct 2010, Igor Stasenko wrote: > 2010/10/4 Levente Uzonyi <leves(a)elte.hu>: > On Mon, 4 Oct 2010, Igor Stasenko wrote: > >> On 4 October 2010 22:51, Henrik Sperre Johansen > > <henrik.s.johansen(a)veloxit.no> wrote: >> >> Â On 04.10.2010 21:47, Igor Stasenko wrote: >>> >>> On 4 October 2010 22:09, StÃ©phane Ducasse<stephane.ducasse(a)inria.fr> >>> Â wrote: >>>> >>>> so let us know in the bug entry what is the conclusion :) >>>> >>> I think someone should verify my benchmarks i.e. >>> >>> [ self loadsomething ] timeToRun >>> >>> before and after patch. >>> >>> And conclusion is better be written by Henrik, because he's having >>> concerns about speed, >>> while i don't. :) >> >> I already have: >> >> http://code.google.com/p/pharo/issues/detail?id=1628#c13 >> > emm.. wait. > WeakKeyDict should not delete associations with nil-ed keys > automatically, because otherwise > you won't be able to use it in weak finalization scheme. > > There is two distinct use cases of weak-key dicts: > a) for weak finalization > b) for attaching some extra info(value) per object(key), which can be > automatically discarded once object become garbage > > so, while in case (b) you can mercilessly kill/reuse associations with > nil keys, once they discovered > in case (a) you should preserve them until there is explicit request > from outside to finalize values. > > The need in (a), apparently prohibits from using (b) in most efficient > manner. > So, i think, the solution would be to introduce a specialized weak-key > dicts, which can work much better for (b). > > > Squeak's implementation works well in both cases. > > Trust me, scanning a whole dict to detect nil keys is not what can be called 'well' :) My point is, that these two scenarios should use different classes/implementors each optimized for own case. Moreover, case (b) comes from the inability to dynamically add extra slot(s) to object, which you can easily do in Self or JavaScript, but can't do in smalltalk. It can be done better, with extra VM features, but without those, it's pretty close to best possible solution IMHO. There are already two different classes for a) and b): WeakRegistry and WeakKeyDictionary (and subclasses). Levente > Levente > >> Cheers, >> Henry >> >> _______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > > > -- -- Best regards, Igor Stasenko AKA sig. _______________________________________________ Pharo-project mailing list Pharo-project(a)lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Oct. 4, 2010

Re: [Pharo-project] Another fixes to finalization (was Re: [update 1.2] #12161 - #12172)

by Henrik Sperre Johansen

On 05.10.2010 00:25, Levente Uzonyi wrote: > On Mon, 4 Oct 2010, Henrik Sperre Johansen wrote: > >> On 04.10.2010 22:26, Levente Uzonyi wrote: >>> On Mon, 4 Oct 2010, Igor Stasenko wrote: >>> >>>> On 4 October 2010 22:51, Henrik Sperre Johansen >>> <henrik.s.johansen(a)veloxit.no> wrote: >>>> On 04.10.2010 21:47, Igor Stasenko wrote: >>>>> >>>>> On 4 October 2010 22:09, StÃ©phane Ducasse<stephane.ducasse(a)inria.fr> >>>>> wrote: >>>>>> >>>>>> so let us know in the bug entry what is the conclusion :) >>>>>> >>>>> I think someone should verify my benchmarks i.e. >>>>> >>>>> [ self loadsomething ] timeToRun >>>>> >>>>> before and after patch. >>>>> >>>>> And conclusion is better be written by Henrik, because he's having >>>>> concerns about speed, >>>>> while i don't. :) >>>> >>>> I already have: >>>> >>>> http://code.google.com/p/pharo/issues/detail?id=1628#c13 >>>> >>> emm.. wait. >>> WeakKeyDict should not delete associations with nil-ed keys >>> automatically, because otherwise >>> you won't be able to use it in weak finalization scheme. >> It could in 1.0, without anyone noticing. >> #rehash truncated duplicate nil keys, thus if it was triggered (f.ex. >> by a manual removal, or adding enough to cause growth) in a thread >> with priority higher than finalization process (ie ran after GC but >> before finalization thread), the nil keys could be truncated before >> finalization being run. An unlikely scenario, I admit, which I guess >> is why noone encountered it. > > That's not true. People experienced the issues (leaking Sockets), but > didn't find the cause of the problem, because it was hardly > reproducible. Creating a new Socket could trigger #grow in the > WeakRegistry's dictionary which could trigger a GC when the new array > was allocated. For example ConnectionQueue's #listenLoop uses > #highIOPriority (70) and creates new sockets. Hum, ok. Didn't know that, must've missed a thread or something :) > >> #grow suffered the same problem. (well, actually, it didn't add any >> of the nil keyed associations) >>> >>> There is two distinct use cases of weak-key dicts: >>> a) for weak finalization >>> b) for attaching some extra info(value) per object(key), which can be >>> automatically discarded once object become garbage >>> >>> so, while in case (b) you can mercilessly kill/reuse associations with >>> nil keys, once they discovered >>> in case (a) you should preserve them until there is explicit request >>> from outside to finalize values. >> Yes, that's the point I've been trying to make these past couple of >> months :) >>> >>> The need in (a), apparently prohibits from using (b) in most >>> efficient manner. >>> So, i think, the solution would be to introduce a specialized weak-key >>> dicts, which can work much better for (b). >> The difference between a and b, and the fact Pharo 1.1's >> implementation severly screws with the performance if b) (and >> honestly, isn't that good for a) either) is the point I've been >> trying to make for months now... >> >> In Pharo 1.1 we've moved to the other extreme compared to 1.0, nil >> keys are never removed. b) always works, but a) HAS to be registered >> to work satisfactory at all. > > Um, no. Associations with nil keys are lost during #rehash. Whoops, had the impression it was fixed, not sure how I got that. So, in short, the 1.1 implementation keeps the worst aspects of 1.0, and adds a few new ones. > >> It was changed to do an extra step of keeping finalized assocs in a >> special state where they can be simply replaced without rehashing >> after finalization instead of doing rehash, but that's not really the >> major result of the change. >> >>> >>> >>> Squeak's implementation works well in both cases. >>> >>> >>> Levente >> The way I read it (correct me if I'm wrong), the solution in Squeak >> was adding a finalizer inst var to WeakKeyDictionary, that way you >> can distinguish the two cases and handle them appropriately. (in >> addition there were speed improvements for rehashing, growing, etc.) > > The original goal of the finalizer was to make finalization run in > O(n) time instead of O(n^2). But it's also handy for doing proper > finalization. And there are other improvements, like #remove: triggers > the finalizer and cleans up the slots in the removed element's > "chain". So both addition (via #grow) and removal of elements can free > slots held by GC'd keys. <3 > > The implementation in Pharo 1.1 still has an O(n^2) issue. Not expired > associations with nil keys, are added to the front of the array during > #grow (see #noCheckAdd:). The free slot for the association is found > by a linear search started from the first slot in the array (OMG). If > there are k such elements, then it takes k * (k + 1) / 2 = O(k^2) > time. Since k can be n if the dictionary is not registered to the > finalization process, this means O(n^2) time: Yeah, I've mentioned that before and didn't want to repeat myself ;) Cheers, Henry

Oct. 4, 2010

Re: [Pharo-project] Another fixes to finalization (was Re: [update 1.2] #12161 - #12172)

by Levente Uzonyi

On Mon, 4 Oct 2010, Henrik Sperre Johansen wrote: > On 04.10.2010 22:26, Levente Uzonyi wrote: >> On Mon, 4 Oct 2010, Igor Stasenko wrote: >> >>> On 4 October 2010 22:51, Henrik Sperre Johansen >> <henrik.s.johansen(a)veloxit.no> wrote: >>> On 04.10.2010 21:47, Igor Stasenko wrote: >>>> >>>> On 4 October 2010 22:09, StÃ©phane Ducasse<stephane.ducasse(a)inria.fr> >>>> wrote: >>>>> >>>>> so let us know in the bug entry what is the conclusion :) >>>>> >>>> I think someone should verify my benchmarks i.e. >>>> >>>> [ self loadsomething ] timeToRun >>>> >>>> before and after patch. >>>> >>>> And conclusion is better be written by Henrik, because he's having >>>> concerns about speed, >>>> while i don't. :) >>> >>> I already have: >>> >>> http://code.google.com/p/pharo/issues/detail?id=1628#c13 >>> >> emm.. wait. >> WeakKeyDict should not delete associations with nil-ed keys >> automatically, because otherwise >> you won't be able to use it in weak finalization scheme. > It could in 1.0, without anyone noticing. > #rehash truncated duplicate nil keys, thus if it was triggered (f.ex. by a > manual removal, or adding enough to cause growth) in a thread with priority > higher than finalization process (ie ran after GC but before finalization > thread), the nil keys could be truncated before finalization being run. An > unlikely scenario, I admit, which I guess is why noone encountered it. That's not true. People experienced the issues (leaking Sockets), but didn't find the cause of the problem, because it was hardly reproducible. Creating a new Socket could trigger #grow in the WeakRegistry's dictionary which could trigger a GC when the new array was allocated. For example ConnectionQueue's #listenLoop uses #highIOPriority (70) and creates new sockets. > #grow suffered the same problem. (well, actually, it didn't add any of the > nil keyed associations) >> >> There is two distinct use cases of weak-key dicts: >> a) for weak finalization >> b) for attaching some extra info(value) per object(key), which can be >> automatically discarded once object become garbage >> >> so, while in case (b) you can mercilessly kill/reuse associations with >> nil keys, once they discovered >> in case (a) you should preserve them until there is explicit request >> from outside to finalize values. > Yes, that's the point I've been trying to make these past couple of months :) >> >> The need in (a), apparently prohibits from using (b) in most efficient >> manner. >> So, i think, the solution would be to introduce a specialized weak-key >> dicts, which can work much better for (b). > The difference between a and b, and the fact Pharo 1.1's implementation > severly screws with the performance if b) (and honestly, isn't that good for > a) either) is the point I've been trying to make for months now... > > In Pharo 1.1 we've moved to the other extreme compared to 1.0, nil keys are > never removed. b) always works, but a) HAS to be registered to work > satisfactory at all. Um, no. Associations with nil keys are lost during #rehash. objects := (1 to: 1000) collect: [ :each | Object new ]. dictionary := WeakKeyDictionary new. objects do: [ :each | dictionary at: each put: 1 ]. objects := nil. Smalltalk garbageCollect. counter := 0. dictionary allAssociationsDo: [ :each | counter := counter + 1 ]. self assert: counter = 1000. dictionary rehash. counter := 0. dictionary allAssociationsDo: [ :each | counter := counter + 1 ]. self assert: counter = 1000 "This will fail, because counter = 0." > It was changed to do an extra step of keeping finalized assocs in a special > state where they can be simply replaced without rehashing after finalization > instead of doing rehash, but that's not really the major result of the > change. > >> >> >> Squeak's implementation works well in both cases. >> >> >> Levente > The way I read it (correct me if I'm wrong), the solution in Squeak was > adding a finalizer inst var to WeakKeyDictionary, that way you can > distinguish the two cases and handle them appropriately. (in addition there > were speed improvements for rehashing, growing, etc.) The original goal of the finalizer was to make finalization run in O(n) time instead of O(n^2). But it's also handy for doing proper finalization. And there are other improvements, like #remove: triggers the finalizer and cleans up the slots in the removed element's "chain". So both addition (via #grow) and removal of elements can free slots held by GC'd keys. The implementation in Pharo 1.1 still has an O(n^2) issue. Not expired associations with nil keys, are added to the front of the array during #grow (see #noCheckAdd:). The free slot for the association is found by a linear search started from the first slot in the array (OMG). If there are k such elements, then it takes k * (k + 1) / 2 = O(k^2) time. Since k can be n if the dictionary is not registered to the finalization process, this means O(n^2) time: objects := (1 to: 10000) collect: [ :each | Object new ]. dictionary := WeakKeyDictionary new. objects do: [ :each | dictionary at: each put: 1 ]. objects := nil. Smalltalk garbageCollect. counter := 0. dictionary allAssociationsDo: [ :each | counter := counter + 1 ]. self assert: counter = 10000. [ dictionary grow ] timeToRun "===> 1830" Levente > > I looked for another solution which would keep WeakKeyDictionary oblivious to > whether or not it was being finalized or not and could not find one, thus I > think it is worth porting. > Keeping the (pardon my fren.. err, norwegian) POS that is currectly in 1.1, > or reverting to the 1.0 version for 1.2, is not a good alternative. > > Cheers, > Henry > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

Oct. 4, 2010

Re: [Pharo-project] Ocean (was Re: Less plugins, more Smalltalk code! (was: Re: Pharo 1.1.1 OneClickCogVm))

by Schwab,Wilhelm K

Sig, Yes and no. OS threads can get expensive, and some (admittedly dated) literature shows an edge for asynchronous sockets and green threads over OS threads. That said, the blocking call+OS thread model is a lot easier to understand than are non-block calls, *especially* for OpenSSL. On balance I agree that it would be a good thing for us to do, not necessarily that it is the preferred course in general. So let's say we go for the (in Dolphin speak) overlapped calls. Does NativeBoost do them now? Bill ________________________________________ From: pharo-project-bounces(a)lists.gforge.inria.fr [pharo-project-bounces(a)lists.gforge.inria.fr] On Behalf Of Igor Stasenko [siguctua(a)gmail.com] Sent: Monday, October 04, 2010 5:24 PM To: Pharo-project(a)lists.gforge.inria.fr Subject: Re: [Pharo-project] Ocean (was Re: Less plugins, more Smalltalk code! (was: Re: Pharo 1.1.1 OneClickCogVm)) I think that this project would be good test-bed for testing a non-blocking FFI callouts. On 4 October 2010 23:23, Levente Uzonyi <leves(a)elte.hu> wrote: > On Mon, 4 Oct 2010, Sven Van Caekenberghe wrote: > >> Noury, >> >> One of the other aspects that I find important is elementary efficiency. >> >> A lot of stream related code in Smalltalk is not efficient, the actual >> data is copied around like crazy, turning it from a stream to a collection >> and then back into a stream multiple times. For Zinc HTTP Components I made >> a lot of effort to make it possible to read data from a socket stream in >> true streaming fashion (instead of just returning a byte array). Now, the >> idea was then that for example JPEGReadWriter>>#nextImage would work >> transparaently on that raw stream. Sadly, the code in JPEGReadWriter and >> friends just reads everything into an array before it starts to work ! >> >> Doing a #nextPutAll: <some byte array> should really try never to copy the >> array. Similary, a #next: <count> into: <some byte array> should similary >> write directly into the given array. What I see in the current SocketStream >> is <censored/> ;-) > > This is why we implemented our own SocketStream in PostgresV3. > >> >> But OK, maybe this is easier said than done. The step from Smalltalk to >> system call should also not copy if that is possible, but then FFI and GC >> come into play. > > I wonder if there's a language, that provides socket access via FFI. > > > Levente > >> >> Anyway, good luck, I am willing to try using Ocean as soon as it becomes >> available. >> >> Sven >> >> On 04 Oct 2010, at 10:59, Noury Bouraqadi wrote: >> >>> >>> On 3 oct. 2010, at 16:28, Schwab,Wilhelm K wrote: >>> >>>> For Ocean to succeed, you either have to work really hard to make >>>> non-blocking calls (I'm thinking of SSL sockets), or you need the ability to >>>> call functions on OS threads. The calls need to block their calling Process >>>> and let the other Processes run as though nothing is happening. >>>> >>> You're right. >>> Another option is having a multi-threaded VM. Eliot seem to work on it. >>> So far we're working on cleaning up the design of basic functionalities >>> before going on. >>> Then, we'll try to find a workaround for non-blocking calls. >>> All ideas are welcome :-) >>> >>> BTW, some people (you included) proposed to help for the Ocean project. >>> Luc and I want to thank you and we are happy that others are willing to >>> help. >>> We thought about the best way to integrate others help. >>> Our conclusion is that we should first do alone (luc and me) the >>> reference design, before integrating other developers. >>> So, we all can work on solid foudations. >>> >>> Noury >>> >>> >>> >>> _______________________________________________ >>> Pharo-project mailing list >>> Pharo-project(a)lists.gforge.inria.fr >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >> >> _______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > -- Best regards, Igor Stasenko AKA sig. _______________________________________________ Pharo-project mailing list Pharo-project(a)lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Oct. 4, 2010