- Pharo-users - lists.pharo.org

Re: [Pharo-users] why is adding instance variables so slow?

by tesonep＠gmail.com

Hi, I think the problem was not clearly explained. This is the scenario that is problematic. This scenario does not happen in the new Spur implementation of forwarding, but I think is happening in the old one. 1. You have, let's say 3 instances of ClassA. 2. You add a new instance variable to ClassA. It produces 2a. A new ClassAv2 is created with the instances variables of ClassA and the newone 2b. 3 Instances of ClassAv2 are created 2c. The values of the instance variables of ClassA are copied to the ones in ClassAv2 (the ones missing are left in nil). 2d. The 3 instances of ClassA are becomed forward to the 3 instances of ClassAv2 2e. The ClassA is becomed forward ClassAv2 3. You add a new instance variable to ClassAv2. It produces 3a. A new ClassAv3 is created with the instances variables of ClassAv2 and the newone 3b. 3 Instances of ClassAv3 are created 3c. The values of the instance variables of ClassAv2 are copied to the ones in ClassAv3 (the ones missing are left in nil). 3d. The 3 instances of ClassAv2 are becomed forward to the 3 instances of ClassAv3 3e. The ClassAv2 is becomedFormeward ClassAv3 4. All the instances of ClassAV3 have the correct format and everything works. What is the problem: =============== - When you do the first add instance variable, the old instances (the one from ClassA) which are smaller (has 1 instance variable less) have its class changed (after you perform a become of ClassA to ClassAv2). So if you try to use them everything will explode, because you will trying to access an instance variable that does not exists. These instances are not referenced by anyone, however if you perform a ClassAv2>>allInstances you will find them. So if you modify the class adding two variables, one after another the second time you will be accessing the invalid instances. Considering the differences in the Become implementation ============================================ However, the main difference is the implementation of the become forward. Let's start with the new implementation, as it has not problems. When you do a become forward, from object a to b, the primitive replaces the object a with a forwarder to b. When this forwarded is accessed the references to it are rewrited. If the objects are the same size (not this scenario) the object b replaces object a. It does not produces an error because the old. In the become forward the old instances are not keeped. In the old implementation the whole image is scanned, changing the references to the old instances, replacing with references to the new instances. The old instances are not removed, just kept there to let the GC do its work. Again if the objects are the same size there is special behavior. I hope know the problem is better explained Cheers, Pablo On Fri, Apr 14, 2017 at 10:50 AM, Igor Stasenko <siguctua(a)gmail.com> wrote: > > > On 14 April 2017 at 10:19, Stephane Ducasse <stepharo.self(a)gmail.com> > wrote: > >> But I do not get how doing that would handle the old instances? >> Because you want to migrate the old instances. >> >> > +1 > there are no such thing as 'bad zombies', if they are there, it means you > either don't care about migrating data > or again, don't care about doing #becomeForward-ing them properly. > In any case i don't see how GC could help to fix these issues. You either > have consistency or don't have it, > and GC cannot do anything magical to fix it. > > > >> Stef >> >> On Wed, Apr 12, 2017 at 1:26 PM, Denis Kudriashov <dionisiydk(a)gmail.com> >> wrote: >> >>> >>> 2017-04-12 13:17 GMT+02:00 Guillermo Polito <guillermopolito(a)gmail.com>: >>> >>>> 1) each instance of A is becomed into its corresponding instance of A' >>>> 2) finally we become class A into A' >>>> This step will make that old instances of A now have: >>>> - the old format >>>> - but point to the new class A >>>> >>> >>> step 1) ensures that there are no instances of class A anymore. >>> Check following script: >>> >>> c1 := Class1 new. >>> c2 := Class2 new. >>> c1 becomeForward: c2. >>> Class1 allInstances "=> #()". >>> >>> >>> And full migration is executed in high priority uninterrupted process to >>> ensure that between 1) and 2) nobody will instantiate Class1 >>> >>> >> > > > -- > Best regards, > Igor Stasenko. > -- Pablo Tesone. tesonep(a)gmail.com

April 14, 2017

Re: [Pharo-users] Traits, method flatten

by Hilaire

On a restored Traits logic of my package, I made another attempt to persist this time with filetree://. The net result is the same, the Traits logic is lost in the classes using the Traits: all traits methods are duplicated !! To sum up, package saved with the following methods: - *.mcz package with the traits got broken. The .st source file is ~207kB of mostly duplicated methods - filetree package got broken too - Brave old fileout handles it just fine :) with Pharo5 (Pharo4 has UTF8 characters issues) . Source file is about 40KB Solutions: - Get it fixed. The problem is there since Pharo3 or 4... Not sure where to look at. - Escape from Traits :( - Test on Squeak, not sure how stable it is there... - Use fileout + CVS to persist code. Hilaire Le 13/04/2017 Ã 09:44, Hilaire a Ã©crit : > May be an important note, FileOut is handling just right the Traits logic. > > Hilaire > > > Le 13/04/2017 Ã 08:53, Nicolai Hess a Ã©crit : >> I am interested, and I think I already looked at this issue, I vaguely >> remember, that this has >> something to do with the determination of the origin of a method (trait >> methods that are from another trait >> are wrongly recognized as (trait)local methods). >> -- Dr. Geo http://drgeo.eu

April 14, 2017

Re: [Pharo-users] why is adding instance variables so slow?

by Igor Stasenko

On 12 April 2017 at 14:17, Guillermo Polito <guillermopolito(a)gmail.com> wrote: > > > On Wed, Apr 12, 2017 at 11:35 AM, Denis Kudriashov <dionisiydk(a)gmail.com> > wrote: > >> >> 2017-04-12 10:55 GMT+02:00 Guillermo Polito <guillermopolito(a)gmail.com>: >> >>> PharoClassInstaller>>migrateClasses: old to: new using: >>>> anInstanceModification >>>> instanceModification := anInstanceModification. >>>> old ifEmpty: [ ^ self ]. >>>> [ >>>> 1 to: old size do: [ :index | >>>> self updateClass: (old at: index) to: (new at: index)]. >>>> old elementsForwardIdentityTo: new. >>>> " Garbage collect away the zombie instances left behind in garbage >>>> memory in #updateInstancesFrom: " >>>> " If we don't clean up this garbage, a second update would revive them >>>> with a wrong layout! " >>>> " (newClass rather than oldClass, since they are now both newClass) " >>>> Smalltalk garbageCollect. >>>> ] valueUnpreemptively >>>> >>>> Commenting garbage collection here increases performance 10 times. >>>> Then commenting class update loop increases performance 3 times more. >>>> But this loop is required. It adopts all instances of old class to new one. >>>> And time here spent in #allInstances method. >>>> >>>> Can we remove manual garbage collection here? Why it is needed? >>>> >>> >>> Well, there is the comment that explains it and makes pretty good sense. >>> >> >> But is does not explain why these bad zombies exist. We investigates >> possible reasons and could not reproduce them. We will try remove garbage >> collection here in Pharo 7 >> > > No, this will break stuff! I'll try to explain what does it mean by zombie > instances to make some sense: > > - Imagine that you have class A + 10 instances of A. > > - We add an instance variable to A. > - this means the class builder will generate class A' that is the new > version of A. > - then, it migrates all instances of A to class A'. > This migration is not magic: > - 10 new instances of A' are created > - the state is migrated from the instances of A to A' > - each instance of A is becomed into its corresponding instance of > A' > - finally we become class A into A' > This step will make that old instances of A now have: > - the old format > - but point to the new class A > > If we do not garbage collect, this means that doing > > A allInstances > > will return not only the new 10 instances of A, but the old instances of > A'. > And that will break LOOOTS of stuff. > if you run #allInstances and in between you will trigger adding instance var & GC etc etc.. you'll have everything broken.. because there are things didn't meant to work in certain scenarios. IIRC allInstances is highly dependns on NOT having full GC while doing it, and that's why all loops that doing it is highly conservative & cautious about creating new objects while iterating over heap. That's the nature how #allInstance works, and you could have a tons of issues with it regardless , if you do full GC manually, or it triggered by VM itself. So, this is nothing to do with migrating instances of class. -- Best regards, Igor Stasenko.

April 14, 2017

Re: [Pharo-users] why is adding instance variables so slow?

by tesonep＠gmail.com

Hi Denis, in the script you are not generating instances of Class1, the problem is with the instances. Cheers, Pablo On Fri, Apr 14, 2017 at 10:04 AM, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > 2017-04-12 13:29 GMT+02:00 tesonep(a)gmail.com <tesonep(a)gmail.com>: > >> Hi, >> in non spur, the only improve that I think it can be made safely is >> moving the garbage collect operation to the migration of instances side, >> only executing the garbage collect if there are instances. >> >> In spur the garbage collect it is not needed anymore, because the way the >> become is working is not the same. >> > > I checked on prespur cogVM: my script produced same result as on spur. > -- Pablo Tesone. tesonep(a)gmail.com

April 14, 2017

Re: [Pharo-users] why is adding instance variables so slow?

by Igor Stasenko

On 14 April 2017 at 10:19, Stephane Ducasse <stepharo.self(a)gmail.com> wrote: > But I do not get how doing that would handle the old instances? > Because you want to migrate the old instances. > > +1 there are no such thing as 'bad zombies', if they are there, it means you either don't care about migrating data or again, don't care about doing #becomeForward-ing them properly. In any case i don't see how GC could help to fix these issues. You either have consistency or don't have it, and GC cannot do anything magical to fix it. > Stef > > On Wed, Apr 12, 2017 at 1:26 PM, Denis Kudriashov <dionisiydk(a)gmail.com> > wrote: > >> >> 2017-04-12 13:17 GMT+02:00 Guillermo Polito <guillermopolito(a)gmail.com>: >> >>> 1) each instance of A is becomed into its corresponding instance of A' >>> 2) finally we become class A into A' >>> This step will make that old instances of A now have: >>> - the old format >>> - but point to the new class A >>> >> >> step 1) ensures that there are no instances of class A anymore. >> Check following script: >> >> c1 := Class1 new. >> c2 := Class2 new. >> c1 becomeForward: c2. >> Class1 allInstances "=> #()". >> >> >> And full migration is executed in high priority uninterrupted process to >> ensure that between 1) and 2) nobody will instantiate Class1 >> >> > -- Best regards, Igor Stasenko.

April 14, 2017

Re: [Pharo-users] why is adding instance variables so slow?

by Denis Kudriashov

2017-04-12 13:29 GMT+02:00 tesonep(a)gmail.com <tesonep(a)gmail.com>: > Hi, > in non spur, the only improve that I think it can be made safely is > moving the garbage collect operation to the migration of instances side, > only executing the garbage collect if there are instances. > > In spur the garbage collect it is not needed anymore, because the way the > become is working is not the same. > I checked on prespur cogVM: my script produced same result as on spur.

April 14, 2017

Re: [Pharo-users] why is adding instance variables so slow?

by Denis Kudriashov

2017-04-14 9:19 GMT+02:00 Stephane Ducasse <stepharo.self(a)gmail.com>: > But I do not get how doing that would handle the old instances? > Because you want to migrate the old instances. > But my example shows that old instances not exist after becomeFormard operation. Or what you mean?

April 14, 2017

Re: [Pharo-users] why is adding instance variables so slow?

by Stephane Ducasse

But I do not get how doing that would handle the old instances? Because you want to migrate the old instances. Stef On Wed, Apr 12, 2017 at 1:26 PM, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > 2017-04-12 13:17 GMT+02:00 Guillermo Polito <guillermopolito(a)gmail.com>: > >> 1) each instance of A is becomed into its corresponding instance of A' >> 2) finally we become class A into A' >> This step will make that old instances of A now have: >> - the old format >> - but point to the new class A >> > > step 1) ensures that there are no instances of class A anymore. > Check following script: > > c1 := Class1 new. > c2 := Class2 new. > c1 becomeForward: c2. > Class1 allInstances "=> #()". > > > And full migration is executed in high priority uninterrupted process to > ensure that between 1) and 2) nobody will instantiate Class1 > >

April 14, 2017

Re: [Pharo-users] Pharo 6 snap install

by Stephane Ducasse

We always try to have a major release around mid/end of april and this is in addition we have pharodays + several people moving to new houses :). And deadline for projects on top. On Thu, Apr 13, 2017 at 12:39 PM, Luke Gorrie <luke(a)snabb.co> wrote: > On 13 April 2017 at 12:20, Stephane Ducasse <stepharo.self(a)gmail.com> > wrote: > >> We will do that. >> > > Thank you! I will wait :). > > Now we are under stress for the release :) >> And PharoDays. Great conference! >> > > Thank you for explaining. I am new around here ("oh, are you working on a > release now?") so it helps me to have this kind of hint about short-term > issues (e.g. busy period) vs long term ones (e.g. no longer making source > releases of the VM for some reason.) > > >

April 14, 2017

Re: [Pharo-users] Methods & Classes using the command line.

by Peter Uhnak

This should be similar to having the code in playground and executing it at once. I guess it's more forgiving. What I sometimes do to work around it is to store the created class into variable, or use symbol and asClass cast e.g. cls := Object subclass: #TestClass. cls compile:'meth'. or Object subclass: #TestClass. #TestClass asClass compile:'meth'. Peter On Thu, Apr 13, 2017 at 07:39:24PM +0200, Guillermo Polito wrote: > Hi, > > On Thu, Apr 13, 2017 at 6:57 PM, nacho <0800nacho(a)gmail.com> wrote: > > > Hi pharoers, > > > > If I do the following in a minimal Pharo image, using the command line: > > > > ./Pharo --headless Pharo.image eval "Object subclass: #TestClass > > instanceVariableNames: 'one' classVariableNames: '' package: 'IS-Test'." > > > > I get the class created. However, if I try to pass the following argument > > to > > eval: > > > > "Object subclass: #TestClass instanceVariableNames: 'one' > > classVariableNames: '' package: 'IS-Test'. TestClass compile:'one ^one' > > classified: 'accessor'." > > > > I get an UndefinedObject>>DoIt (TestClass is Undeclared). > > > > The problem is that the entire code is compiled at the same time. So, when > the expression is compiled, the class TestClass does not exist yet. And > what you see is a compiler warning. > > However, as soon as the class is created, the class is available to the > already compiled code and should work. > > Moreover: if the TestClass was not created, you would have got an exception > "nil does not understand compile:classified:" ;) > > > > > > At first I thought it could be that it takes sometime to compile the class, > > so added some delay: > > > > "Object subclass: #TestClass instanceVariableNames: 'one' > > classVariableNames: '' package: 'IS-Test'. (Delay forSeconds: 5) wait. > > TestClass compile:'one ^one' classified: 'accessor'." > > > > But again: UndefinedObject>>DoIt (TestClass is Undeclared). > > > > I even tried to do: > > > > "Object subclass: #TestClass instanceVariableNames: 'one' > > classVariableNames: '' package: 'IS-Test'. Smalltalk saveAndQuit." > > > > and after that add only a method by passing: > > > > "TestClass compile:'one ^one' classified: 'accessor'. Smalltalk > > saveAndQuit." > > > > This is strange because, the class is created and you're saving the > image... If I try: > > ./pharo Pharo.image eval "Object subclass: #TestClass > instanceVariableNames: 'one' > classVariableNames: '' package: 'IS-Test'. TestClass compile:'one ^one' > classified: 'accessor'. TestClass" > > (note that I just aded *TestClass * at the end) I get the following output: > > => > UndefinedObject>>DoIt (TestClass is Undeclared) > > UndefinedObject>>DoIt (TestClass is Undeclared) > TestClass > > That is: two compiler warnings. + the class that is printed at the end. > > So it's working well on my end, delta the confusing warnings. > > > > But again: UndefinedObject>>DoIt (TestClass is Undeclared) > > > > How do I add a method to a class using the command line? > > Thanks in advance > > Nacho > > > > > > > > ----- > > Nacho > > Smalltalker apprentice. > > Buenos Aires, Argentina. > > -- > > View this message in context: http://forum.world.st/Methods- > > Classes-using-the-command-line-tp4941977.html > > Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. > > > >

April 13, 2017