April 2017 - Pharo-users - lists.pharo.org

Re: [Pharo-users] why is adding instance variables so slow?

by Denis Kudriashov

2017-04-14 11:09 GMT+02:00 tesonep(a)gmail.com <tesonep(a)gmail.com>: > Hi, I think the problem was not clearly explained. This is the scenario > that is problematic. > This scenario does not happen in the new Spur implementation of > forwarding, but I think is happening in the old one. > > 1. You have, let's say 3 instances of ClassA. > 2. You add a new instance variable to ClassA. It produces > 2a. A new ClassAv2 is created with the instances variables of ClassA > and the newone > 2b. 3 Instances of ClassAv2 are created > 2c. The values of the instance variables of ClassA are copied to the > ones in ClassAv2 (the ones missing are left in nil). > 2d. The 3 instances of ClassA are becomed forward to the 3 instances of > ClassAv2 > 2e. The ClassA is becomed forward ClassAv2 > I still not see why my example not reflects all these steps. I checked also scenario with class becoming: c1 := Class1 new. c2 := Class2 new. c1 becomeForward: c2. Class1 becomeForward: Class2. Class2 allInstances "=>#(aClass2)" It also works in Cog

April 14, 2017

Re: [Pharo-users] Pharo 6 snap install

by Luke Gorrie

On 14 April 2017 at 08:53, Stephane Ducasse <stepharo.self(a)gmail.com> wrote: > We always try to have a major release around mid/end of april > Once the dust settles, I think it would be wonderful to have this invariant: - The latest Pharo VM source release is compatible with the latest Pharo image. This would make it easy for me to help nix users. My job will be to package the latest source release of the VM, period, and I will know that users can use this to run the latest image. I don't have much scope to screw things up and make people sad e.g. by shipping a bad commit from the master branch. (Of course it makes life harder for you upstream Pharo maintainers. Now updating the VM source release is a blocker for releasing a new image. Could be that this is too constraining in practice, I dunno. I'm just a guy who wants to setup a stable Pharo that I can use to build an application :-))

April 14, 2017

Re: [Pharo-users] why is adding instance variables so slow?

by Denis Kudriashov

2017-04-14 10:51 GMT+02:00 tesonep(a)gmail.com <tesonep(a)gmail.com>: > Hi Denis, > in the script you are not generating instances of Class1, the problem > is with the instances. > But it generates: c1 := Class1 new. Or what kind of generation you mean?

April 14, 2017

Re: [Pharo-users] why is adding instance variables so slow?

by Igor Stasenko

i don't see much difference comparing to old implementation of #becomeForward: the only issue is that #allInstances could report same instance twice first, it will find an old instance (that is forwards to new one) and so, add it to results array, and then walking the heap further will find a new version of same instance. That's why , it think #allInstances should use IdentitySet-behavior to mitigate such problem, and always look if there's already same object captured to avoid reporting it twice. Or else we can declare it as a feature and warn users of #allInstances that they could have duplications, so in case if it important to them, they could always do 'allInstances asIdentitySet' But again, this is soooo orthogonal to adding instance variables to class... maybe we should rename the topic instead and speak about #allInstances behavior? :) On 14 April 2017 at 12:09, tesonep(a)gmail.com <tesonep(a)gmail.com> wrote: > Hi, I think the problem was not clearly explained. This is the scenario > that is problematic. > This scenario does not happen in the new Spur implementation of > forwarding, but I think is happening in the old one. > > 1. You have, let's say 3 instances of ClassA. > 2. You add a new instance variable to ClassA. It produces > 2a. A new ClassAv2 is created with the instances variables of ClassA > and the newone > 2b. 3 Instances of ClassAv2 are created > 2c. The values of the instance variables of ClassA are copied to the > ones in ClassAv2 (the ones missing are left in nil). > 2d. The 3 instances of ClassA are becomed forward to the 3 instances of > ClassAv2 > 2e. The ClassA is becomed forward ClassAv2 > > 3. You add a new instance variable to ClassAv2. It produces > 3a. A new ClassAv3 is created with the instances variables of ClassAv2 > and the newone > 3b. 3 Instances of ClassAv3 are created > 3c. The values of the instance variables of ClassAv2 are copied to the > ones in ClassAv3 (the ones missing are left in nil). > 3d. The 3 instances of ClassAv2 are becomed forward to the 3 instances > of ClassAv3 > 3e. The ClassAv2 is becomedFormeward ClassAv3 > > 4. All the instances of ClassAV3 have the correct format and everything > works. > > What is the problem: > =============== > > - When you do the first add instance variable, the old instances (the one > from ClassA) which are smaller (has 1 instance variable less) > have its class changed (after you perform a become of ClassA to ClassAv2). > So if you try to use them everything will explode, because you will trying > to access an instance variable that does not exists. > These instances are not referenced by anyone, however if you perform a > ClassAv2>>allInstances you will find them. So if you modify the class > adding two variables, one after another the second time > you will be accessing the invalid instances. > > > Considering the differences in the Become implementation > ============================================ > > However, the main difference is the implementation of the become forward. > Let's start with the new implementation, as it has not problems. > > When you do a become forward, from object a to b, the primitive replaces > the object a with a forwarder to b. > When this forwarded is accessed the references to it are rewrited. > If the objects are the same size (not this scenario) the object b replaces > object a. It does not produces an error because the old. > In the become forward the old instances are not keeped. > > In the old implementation the whole image is scanned, changing the > references to the old instances, replacing with references to the new > instances. > The old instances are not removed, just kept there to let the GC do its > work. > Again if the objects are the same size there is special behavior. > > I hope know the problem is better explained > > Cheers, > Pablo > > > > On Fri, Apr 14, 2017 at 10:50 AM, Igor Stasenko <siguctua(a)gmail.com> > wrote: > >> >> >> On 14 April 2017 at 10:19, Stephane Ducasse <stepharo.self(a)gmail.com> >> wrote: >> >>> But I do not get how doing that would handle the old instances? >>> Because you want to migrate the old instances. >>> >>> >> +1 >> there are no such thing as 'bad zombies', if they are there, it means you >> either don't care about migrating data >> or again, don't care about doing #becomeForward-ing them properly. >> In any case i don't see how GC could help to fix these issues. You either >> have consistency or don't have it, >> and GC cannot do anything magical to fix it. >> >> >> >>> Stef >>> >>> On Wed, Apr 12, 2017 at 1:26 PM, Denis Kudriashov <dionisiydk(a)gmail.com> >>> wrote: >>> >>>> >>>> 2017-04-12 13:17 GMT+02:00 Guillermo Polito <guillermopolito(a)gmail.com> >>>> : >>>> >>>>> 1) each instance of A is becomed into its corresponding instance of >>>>> A' >>>>> 2) finally we become class A into A' >>>>> This step will make that old instances of A now have: >>>>> - the old format >>>>> - but point to the new class A >>>>> >>>> >>>> step 1) ensures that there are no instances of class A anymore. >>>> Check following script: >>>> >>>> c1 := Class1 new. >>>> c2 := Class2 new. >>>> c1 becomeForward: c2. >>>> Class1 allInstances "=> #()". >>>> >>>> >>>> And full migration is executed in high priority uninterrupted process >>>> to ensure that between 1) and 2) nobody will instantiate Class1 >>>> >>>> >>> >> >> >> -- >> Best regards, >> Igor Stasenko. >> > > > > -- > Pablo Tesone. > tesonep(a)gmail.com > -- Best regards, Igor Stasenko.

April 14, 2017

Re: [Pharo-users] Traits, method flatten

by Hilaire

Give a try to filein the correct .st source file of the Traits+Classes package. It does not get it right :( Pharo5 get it right when loading the .st source file though. Still investigating... Le 14/04/2017 Ã 11:04, Hilaire a Ã©crit : > - Test on Squeak, not sure how stable it is there... -- Dr. Geo http://drgeo.eu

April 14, 2017

Re: [Pharo-users] Export MongoDB to sql format

by Igor Stasenko

On 13 April 2017 at 22:23, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: > > On 13 Apr 2017, at 19:26, Asbath Sama biyalou via Pharo-users < > pharo-users(a)lists.pharo.org> wrote: > > > *From: *Asbath Sama biyalou <asamabiyalou(a)yahoo.com> > *Subject: **Export MongoDB to sql format* > *Date: *13 April 2017 at 19:26:25 GMT+2 > *To: *Pharo users users <pharo-users(a)lists.pharo.org> > > > Hello Everyone. > > I think this is not the right place for this question. But I am doing a > work with Pharo. > > I use MongoDB with Voyage. But I am asked to export my database in sql > format. > > I want to know if there is a way to export mongoDB in sql. > > > there is not. > a nosql database will have problems to export its data as sql. > of course it can be done (as everything), but it is a manual task :S > > This sort of questions most strangest ones , that i always wonder.. Like people who buying microwave asking: cool , now how i could fit a coal burner to it? Why???? I mean if you need your traditional oven, they why even bother to buy a microvawe in a first place? cheers, > Esteban > > > Thanks. > > > Asbath > > -- Best regards, Igor Stasenko.

April 14, 2017

Re: [Pharo-users] why is adding instance variables so slow?

by tesonep＠gmail.com

Hi, I think the problem was not clearly explained. This is the scenario that is problematic. This scenario does not happen in the new Spur implementation of forwarding, but I think is happening in the old one. 1. You have, let's say 3 instances of ClassA. 2. You add a new instance variable to ClassA. It produces 2a. A new ClassAv2 is created with the instances variables of ClassA and the newone 2b. 3 Instances of ClassAv2 are created 2c. The values of the instance variables of ClassA are copied to the ones in ClassAv2 (the ones missing are left in nil). 2d. The 3 instances of ClassA are becomed forward to the 3 instances of ClassAv2 2e. The ClassA is becomed forward ClassAv2 3. You add a new instance variable to ClassAv2. It produces 3a. A new ClassAv3 is created with the instances variables of ClassAv2 and the newone 3b. 3 Instances of ClassAv3 are created 3c. The values of the instance variables of ClassAv2 are copied to the ones in ClassAv3 (the ones missing are left in nil). 3d. The 3 instances of ClassAv2 are becomed forward to the 3 instances of ClassAv3 3e. The ClassAv2 is becomedFormeward ClassAv3 4. All the instances of ClassAV3 have the correct format and everything works. What is the problem: =============== - When you do the first add instance variable, the old instances (the one from ClassA) which are smaller (has 1 instance variable less) have its class changed (after you perform a become of ClassA to ClassAv2). So if you try to use them everything will explode, because you will trying to access an instance variable that does not exists. These instances are not referenced by anyone, however if you perform a ClassAv2>>allInstances you will find them. So if you modify the class adding two variables, one after another the second time you will be accessing the invalid instances. Considering the differences in the Become implementation ============================================ However, the main difference is the implementation of the become forward. Let's start with the new implementation, as it has not problems. When you do a become forward, from object a to b, the primitive replaces the object a with a forwarder to b. When this forwarded is accessed the references to it are rewrited. If the objects are the same size (not this scenario) the object b replaces object a. It does not produces an error because the old. In the become forward the old instances are not keeped. In the old implementation the whole image is scanned, changing the references to the old instances, replacing with references to the new instances. The old instances are not removed, just kept there to let the GC do its work. Again if the objects are the same size there is special behavior. I hope know the problem is better explained Cheers, Pablo On Fri, Apr 14, 2017 at 10:50 AM, Igor Stasenko <siguctua(a)gmail.com> wrote: > > > On 14 April 2017 at 10:19, Stephane Ducasse <stepharo.self(a)gmail.com> > wrote: > >> But I do not get how doing that would handle the old instances? >> Because you want to migrate the old instances. >> >> > +1 > there are no such thing as 'bad zombies', if they are there, it means you > either don't care about migrating data > or again, don't care about doing #becomeForward-ing them properly. > In any case i don't see how GC could help to fix these issues. You either > have consistency or don't have it, > and GC cannot do anything magical to fix it. > > > >> Stef >> >> On Wed, Apr 12, 2017 at 1:26 PM, Denis Kudriashov <dionisiydk(a)gmail.com> >> wrote: >> >>> >>> 2017-04-12 13:17 GMT+02:00 Guillermo Polito <guillermopolito(a)gmail.com>: >>> >>>> 1) each instance of A is becomed into its corresponding instance of A' >>>> 2) finally we become class A into A' >>>> This step will make that old instances of A now have: >>>> - the old format >>>> - but point to the new class A >>>> >>> >>> step 1) ensures that there are no instances of class A anymore. >>> Check following script: >>> >>> c1 := Class1 new. >>> c2 := Class2 new. >>> c1 becomeForward: c2. >>> Class1 allInstances "=> #()". >>> >>> >>> And full migration is executed in high priority uninterrupted process to >>> ensure that between 1) and 2) nobody will instantiate Class1 >>> >>> >> > > > -- > Best regards, > Igor Stasenko. > -- Pablo Tesone. tesonep(a)gmail.com

April 14, 2017

Re: [Pharo-users] Traits, method flatten

by Hilaire

On a restored Traits logic of my package, I made another attempt to persist this time with filetree://. The net result is the same, the Traits logic is lost in the classes using the Traits: all traits methods are duplicated !! To sum up, package saved with the following methods: - *.mcz package with the traits got broken. The .st source file is ~207kB of mostly duplicated methods - filetree package got broken too - Brave old fileout handles it just fine :) with Pharo5 (Pharo4 has UTF8 characters issues) . Source file is about 40KB Solutions: - Get it fixed. The problem is there since Pharo3 or 4... Not sure where to look at. - Escape from Traits :( - Test on Squeak, not sure how stable it is there... - Use fileout + CVS to persist code. Hilaire Le 13/04/2017 Ã 09:44, Hilaire a Ã©crit : > May be an important note, FileOut is handling just right the Traits logic. > > Hilaire > > > Le 13/04/2017 Ã 08:53, Nicolai Hess a Ã©crit : >> I am interested, and I think I already looked at this issue, I vaguely >> remember, that this has >> something to do with the determination of the origin of a method (trait >> methods that are from another trait >> are wrongly recognized as (trait)local methods). >> -- Dr. Geo http://drgeo.eu

April 14, 2017

Re: [Pharo-users] why is adding instance variables so slow?

by Igor Stasenko

On 12 April 2017 at 14:17, Guillermo Polito <guillermopolito(a)gmail.com> wrote: > > > On Wed, Apr 12, 2017 at 11:35 AM, Denis Kudriashov <dionisiydk(a)gmail.com> > wrote: > >> >> 2017-04-12 10:55 GMT+02:00 Guillermo Polito <guillermopolito(a)gmail.com>: >> >>> PharoClassInstaller>>migrateClasses: old to: new using: >>>> anInstanceModification >>>> instanceModification := anInstanceModification. >>>> old ifEmpty: [ ^ self ]. >>>> [ >>>> 1 to: old size do: [ :index | >>>> self updateClass: (old at: index) to: (new at: index)]. >>>> old elementsForwardIdentityTo: new. >>>> " Garbage collect away the zombie instances left behind in garbage >>>> memory in #updateInstancesFrom: " >>>> " If we don't clean up this garbage, a second update would revive them >>>> with a wrong layout! " >>>> " (newClass rather than oldClass, since they are now both newClass) " >>>> Smalltalk garbageCollect. >>>> ] valueUnpreemptively >>>> >>>> Commenting garbage collection here increases performance 10 times. >>>> Then commenting class update loop increases performance 3 times more. >>>> But this loop is required. It adopts all instances of old class to new one. >>>> And time here spent in #allInstances method. >>>> >>>> Can we remove manual garbage collection here? Why it is needed? >>>> >>> >>> Well, there is the comment that explains it and makes pretty good sense. >>> >> >> But is does not explain why these bad zombies exist. We investigates >> possible reasons and could not reproduce them. We will try remove garbage >> collection here in Pharo 7 >> > > No, this will break stuff! I'll try to explain what does it mean by zombie > instances to make some sense: > > - Imagine that you have class A + 10 instances of A. > > - We add an instance variable to A. > - this means the class builder will generate class A' that is the new > version of A. > - then, it migrates all instances of A to class A'. > This migration is not magic: > - 10 new instances of A' are created > - the state is migrated from the instances of A to A' > - each instance of A is becomed into its corresponding instance of > A' > - finally we become class A into A' > This step will make that old instances of A now have: > - the old format > - but point to the new class A > > If we do not garbage collect, this means that doing > > A allInstances > > will return not only the new 10 instances of A, but the old instances of > A'. > And that will break LOOOTS of stuff. > if you run #allInstances and in between you will trigger adding instance var & GC etc etc.. you'll have everything broken.. because there are things didn't meant to work in certain scenarios. IIRC allInstances is highly dependns on NOT having full GC while doing it, and that's why all loops that doing it is highly conservative & cautious about creating new objects while iterating over heap. That's the nature how #allInstance works, and you could have a tons of issues with it regardless , if you do full GC manually, or it triggered by VM itself. So, this is nothing to do with migrating instances of class. -- Best regards, Igor Stasenko.

April 14, 2017

Re: [Pharo-users] why is adding instance variables so slow?

by tesonep＠gmail.com

Hi Denis, in the script you are not generating instances of Class1, the problem is with the instances. Cheers, Pablo On Fri, Apr 14, 2017 at 10:04 AM, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > 2017-04-12 13:29 GMT+02:00 tesonep(a)gmail.com <tesonep(a)gmail.com>: > >> Hi, >> in non spur, the only improve that I think it can be made safely is >> moving the garbage collect operation to the migration of instances side, >> only executing the garbage collect if there are instances. >> >> In spur the garbage collect it is not needed anymore, because the way the >> become is working is not the same. >> > > I checked on prespur cogVM: my script produced same result as on spur. > -- Pablo Tesone. tesonep(a)gmail.com

April 14, 2017