why is adding instance variables so slow?
Hi, does anyone know why adding instance variables is so slow? I did some quick benchmarks (see below), resulting in more than order of magnitude speed difference between having it in the class definition and adding it later. In fact it is still much faster to delete the class and then recreate it with the instance variables in the d efinition. For four arguments it is till 8x faster to delete the class four times and recreate it then just add the variable. Unfortunately I cannot just trash the classes (they have methods and inheritance). So the question is: why is it so slow? can I somehow improve the performance? Thanks, Peter Benchmarks: [ cls := Object subclass: #Some1. cls removeFromSystem. ] bench. "'91 per second'" [ cls := Object subclass: #Some2 instanceVariableNames: 'variable'. cls removeFromSystem ] bench. "'90 per second'" [ cls := Object subclass: #Some3. cls addInstVarNamed: 'variable'. cls removeFromSystem. ] bench. "'7 per second'" [ cls := Object subclass: #Some4. cls removeFromSystem. cls := Object subclass: #Some4 instanceVariableNames: 'variable'. cls removeFromSystem. ] bench. "'43 per second'" [ cls := Object subclass: #Some3. cls addInstVarNamed: 'variable1'. cls addInstVarNamed: 'variable2'. cls addInstVarNamed: 'variable3'. cls addInstVarNamed: 'variable4'. cls removeFromSystem. ] bench. "'2 per second'" [ cls := Object subclass: #Some4. cls removeFromSystem. cls := Object subclass: #Some4 instanceVariableNames: 'variable1 variable2 variable3 variable4'. cls removeFromSystem. ] bench. "'44 per second'" [ cls := Object subclass: #Some5. cls removeFromSystem. cls := Object subclass: #Some5 instanceVariableNames: 'variable1'. cls removeFromSystem. cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2'. cls removeFromSystem. cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2 variable3'. cls removeFromSystem. cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2 variable3 variable4'. cls removeFromSystem. ] bench. "'17.269 per second'"
Hi. I profiled a bit and found problem: PharoClassInstaller>>migrateClasses: old to: new using: anInstanceModification instanceModification := anInstanceModification. old ifEmpty: [ ^ self ]. [ 1 to: old size do: [ :index | self updateClass: (old at: index) to: (new at: index)]. old elementsForwardIdentityTo: new. " Garbage collect away the zombie instances left behind in garbage memory in #updateInstancesFrom: " " If we don't clean up this garbage, a second update would revive them with a wrong layout! " " (newClass rather than oldClass, since they are now both newClass) " Smalltalk garbageCollect. ] valueUnpreemptively Commenting garbage collection here increases performance 10 times. Then commenting class update loop increases performance 3 times more. But this loop is required. It adopts all instances of old class to new one. And time here spent in #allInstances method. Can we remove manual garbage collection here? Why it is needed? 2017-04-12 9:34 GMT+02:00 Peter Uhnak <i.uhnak@gmail.com>:
Hi,
does anyone know why adding instance variables is so slow?
I did some quick benchmarks (see below), resulting in more than order of magnitude speed difference between having it in the class definition and adding it later.
In fact it is still much faster to delete the class and then recreate it with the instance variables in the d efinition. For four arguments it is till 8x faster to delete the class four times and recreate it then just add the variable. Unfortunately I cannot just trash the classes (they have methods and inheritance).
So the question is: why is it so slow? can I somehow improve the performance?
Thanks, Peter
Benchmarks:
[ cls := Object subclass: #Some1. cls removeFromSystem. ] bench. "'91 per second'"
[ cls := Object subclass: #Some2 instanceVariableNames: 'variable'. cls removeFromSystem ] bench. "'90 per second'"
[ cls := Object subclass: #Some3. cls addInstVarNamed: 'variable'. cls removeFromSystem. ] bench. "'7 per second'"
[ cls := Object subclass: #Some4. cls removeFromSystem. cls := Object subclass: #Some4 instanceVariableNames: 'variable'. cls removeFromSystem. ] bench. "'43 per second'"
[ cls := Object subclass: #Some3. cls addInstVarNamed: 'variable1'. cls addInstVarNamed: 'variable2'. cls addInstVarNamed: 'variable3'. cls addInstVarNamed: 'variable4'. cls removeFromSystem. ] bench. "'2 per second'"
[ cls := Object subclass: #Some4. cls removeFromSystem. cls := Object subclass: #Some4 instanceVariableNames: 'variable1 variable2 variable3 variable4'. cls removeFromSystem. ] bench. "'44 per second'"
[ cls := Object subclass: #Some5. cls removeFromSystem. cls := Object subclass: #Some5 instanceVariableNames: 'variable1'. cls removeFromSystem. cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2'. cls removeFromSystem. cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2 variable3'. cls removeFromSystem. cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2 variable3 variable4'. cls removeFromSystem. ] bench. "'17.269 per second'"
On Wed, Apr 12, 2017 at 10:51 AM, Denis Kudriashov <dionisiydk@gmail.com> wrote:
Hi.
I profiled a bit and found problem:
PharoClassInstaller>>migrateClasses: old to: new using: anInstanceModification instanceModification := anInstanceModification. old ifEmpty: [ ^ self ]. [ 1 to: old size do: [ :index | self updateClass: (old at: index) to: (new at: index)]. old elementsForwardIdentityTo: new. " Garbage collect away the zombie instances left behind in garbage memory in #updateInstancesFrom: " " If we don't clean up this garbage, a second update would revive them with a wrong layout! " " (newClass rather than oldClass, since they are now both newClass) " Smalltalk garbageCollect. ] valueUnpreemptively
Commenting garbage collection here increases performance 10 times. Then commenting class update loop increases performance 3 times more. But this loop is required. It adopts all instances of old class to new one. And time here spent in #allInstances method.
Can we remove manual garbage collection here? Why it is needed?
Well, there is the comment that explains it and makes pretty good sense.
2017-04-12 9:34 GMT+02:00 Peter Uhnak <i.uhnak@gmail.com>:
Hi,
does anyone know why adding instance variables is so slow?
I did some quick benchmarks (see below), resulting in more than order of magnitude speed difference between having it in the class definition and adding it later.
In fact it is still much faster to delete the class and then recreate it with the instance variables in the d efinition. For four arguments it is till 8x faster to delete the class four times and recreate it then just add the variable. Unfortunately I cannot just trash the classes (they have methods and inheritance).
So the question is: why is it so slow? can I somehow improve the performance?
Thanks, Peter
Benchmarks:
[ cls := Object subclass: #Some1. cls removeFromSystem. ] bench. "'91 per second'"
[ cls := Object subclass: #Some2 instanceVariableNames: 'variable'. cls removeFromSystem ] bench. "'90 per second'"
[ cls := Object subclass: #Some3. cls addInstVarNamed: 'variable'. cls removeFromSystem. ] bench. "'7 per second'"
[ cls := Object subclass: #Some4. cls removeFromSystem. cls := Object subclass: #Some4 instanceVariableNames: 'variable'. cls removeFromSystem. ] bench. "'43 per second'"
[ cls := Object subclass: #Some3. cls addInstVarNamed: 'variable1'. cls addInstVarNamed: 'variable2'. cls addInstVarNamed: 'variable3'. cls addInstVarNamed: 'variable4'. cls removeFromSystem. ] bench. "'2 per second'"
[ cls := Object subclass: #Some4. cls removeFromSystem. cls := Object subclass: #Some4 instanceVariableNames: 'variable1 variable2 variable3 variable4'. cls removeFromSystem. ] bench. "'44 per second'"
[ cls := Object subclass: #Some5. cls removeFromSystem. cls := Object subclass: #Some5 instanceVariableNames: 'variable1'. cls removeFromSystem. cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2'. cls removeFromSystem. cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2 variable3'. cls removeFromSystem. cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2 variable3 variable4'. cls removeFromSystem. ] bench. "'17.269 per second'"
2017-04-12 10:55 GMT+02:00 Guillermo Polito <guillermopolito@gmail.com>:
PharoClassInstaller>>migrateClasses: old to: new using:
anInstanceModification instanceModification := anInstanceModification. old ifEmpty: [ ^ self ]. [ 1 to: old size do: [ :index | self updateClass: (old at: index) to: (new at: index)]. old elementsForwardIdentityTo: new. " Garbage collect away the zombie instances left behind in garbage memory in #updateInstancesFrom: " " If we don't clean up this garbage, a second update would revive them with a wrong layout! " " (newClass rather than oldClass, since they are now both newClass) " Smalltalk garbageCollect. ] valueUnpreemptively
Commenting garbage collection here increases performance 10 times. Then commenting class update loop increases performance 3 times more. But this loop is required. It adopts all instances of old class to new one. And time here spent in #allInstances method.
Can we remove manual garbage collection here? Why it is needed?
Well, there is the comment that explains it and makes pretty good sense.
But is does not explain why these bad zombies exist. We investigates possible reasons and could not reproduce them. We will try remove garbage collection here in Pharo 7
On Wed, Apr 12, 2017 at 11:35 AM, Denis Kudriashov <dionisiydk@gmail.com> wrote:
2017-04-12 10:55 GMT+02:00 Guillermo Polito <guillermopolito@gmail.com>:
PharoClassInstaller>>migrateClasses: old to: new using:
anInstanceModification instanceModification := anInstanceModification. old ifEmpty: [ ^ self ]. [ 1 to: old size do: [ :index | self updateClass: (old at: index) to: (new at: index)]. old elementsForwardIdentityTo: new. " Garbage collect away the zombie instances left behind in garbage memory in #updateInstancesFrom: " " If we don't clean up this garbage, a second update would revive them with a wrong layout! " " (newClass rather than oldClass, since they are now both newClass) " Smalltalk garbageCollect. ] valueUnpreemptively
Commenting garbage collection here increases performance 10 times. Then commenting class update loop increases performance 3 times more. But this loop is required. It adopts all instances of old class to new one. And time here spent in #allInstances method.
Can we remove manual garbage collection here? Why it is needed?
Well, there is the comment that explains it and makes pretty good sense.
But is does not explain why these bad zombies exist. We investigates possible reasons and could not reproduce them. We will try remove garbage collection here in Pharo 7
No, this will break stuff! I'll try to explain what does it mean by zombie instances to make some sense: - Imagine that you have class A + 10 instances of A. - We add an instance variable to A. - this means the class builder will generate class A' that is the new version of A. - then, it migrates all instances of A to class A'. This migration is not magic: - 10 new instances of A' are created - the state is migrated from the instances of A to A' - each instance of A is becomed into its corresponding instance of A' - finally we become class A into A' This step will make that old instances of A now have: - the old format - but point to the new class A If we do not garbage collect, this means that doing A allInstances will return not only the new 10 instances of A, but the old instances of A'. And that will break LOOOTS of stuff.
2017-04-12 13:17 GMT+02:00 Guillermo Polito <guillermopolito@gmail.com>:
1) each instance of A is becomed into its corresponding instance of A' 2) finally we become class A into A' This step will make that old instances of A now have: - the old format - but point to the new class A
step 1) ensures that there are no instances of class A anymore. Check following script: c1 := Class1 new. c2 := Class2 new. c1 becomeForward: c2. Class1 allInstances "=> #()". And full migration is executed in high priority uninterrupted process to ensure that between 1) and 2) nobody will instantiate Class1
But I do not get how doing that would handle the old instances? Because you want to migrate the old instances. Stef On Wed, Apr 12, 2017 at 1:26 PM, Denis Kudriashov <dionisiydk@gmail.com> wrote:
2017-04-12 13:17 GMT+02:00 Guillermo Polito <guillermopolito@gmail.com>:
1) each instance of A is becomed into its corresponding instance of A' 2) finally we become class A into A' This step will make that old instances of A now have: - the old format - but point to the new class A
step 1) ensures that there are no instances of class A anymore. Check following script:
c1 := Class1 new. c2 := Class2 new. c1 becomeForward: c2. Class1 allInstances "=> #()".
And full migration is executed in high priority uninterrupted process to ensure that between 1) and 2) nobody will instantiate Class1
On 14 April 2017 at 10:19, Stephane Ducasse <stepharo.self@gmail.com> wrote:
But I do not get how doing that would handle the old instances? Because you want to migrate the old instances.
+1 there are no such thing as 'bad zombies', if they are there, it means you either don't care about migrating data or again, don't care about doing #becomeForward-ing them properly. In any case i don't see how GC could help to fix these issues. You either have consistency or don't have it, and GC cannot do anything magical to fix it.
Stef
On Wed, Apr 12, 2017 at 1:26 PM, Denis Kudriashov <dionisiydk@gmail.com> wrote:
2017-04-12 13:17 GMT+02:00 Guillermo Polito <guillermopolito@gmail.com>:
1) each instance of A is becomed into its corresponding instance of A' 2) finally we become class A into A' This step will make that old instances of A now have: - the old format - but point to the new class A
step 1) ensures that there are no instances of class A anymore. Check following script:
c1 := Class1 new. c2 := Class2 new. c1 becomeForward: c2. Class1 allInstances "=> #()".
And full migration is executed in high priority uninterrupted process to ensure that between 1) and 2) nobody will instantiate Class1
-- Best regards, Igor Stasenko.
Hi, I think the problem was not clearly explained. This is the scenario that is problematic. This scenario does not happen in the new Spur implementation of forwarding, but I think is happening in the old one. 1. You have, let's say 3 instances of ClassA. 2. You add a new instance variable to ClassA. It produces 2a. A new ClassAv2 is created with the instances variables of ClassA and the newone 2b. 3 Instances of ClassAv2 are created 2c. The values of the instance variables of ClassA are copied to the ones in ClassAv2 (the ones missing are left in nil). 2d. The 3 instances of ClassA are becomed forward to the 3 instances of ClassAv2 2e. The ClassA is becomed forward ClassAv2 3. You add a new instance variable to ClassAv2. It produces 3a. A new ClassAv3 is created with the instances variables of ClassAv2 and the newone 3b. 3 Instances of ClassAv3 are created 3c. The values of the instance variables of ClassAv2 are copied to the ones in ClassAv3 (the ones missing are left in nil). 3d. The 3 instances of ClassAv2 are becomed forward to the 3 instances of ClassAv3 3e. The ClassAv2 is becomedFormeward ClassAv3 4. All the instances of ClassAV3 have the correct format and everything works. What is the problem: =============== - When you do the first add instance variable, the old instances (the one from ClassA) which are smaller (has 1 instance variable less) have its class changed (after you perform a become of ClassA to ClassAv2). So if you try to use them everything will explode, because you will trying to access an instance variable that does not exists. These instances are not referenced by anyone, however if you perform a ClassAv2>>allInstances you will find them. So if you modify the class adding two variables, one after another the second time you will be accessing the invalid instances. Considering the differences in the Become implementation ============================================ However, the main difference is the implementation of the become forward. Let's start with the new implementation, as it has not problems. When you do a become forward, from object a to b, the primitive replaces the object a with a forwarder to b. When this forwarded is accessed the references to it are rewrited. If the objects are the same size (not this scenario) the object b replaces object a. It does not produces an error because the old. In the become forward the old instances are not keeped. In the old implementation the whole image is scanned, changing the references to the old instances, replacing with references to the new instances. The old instances are not removed, just kept there to let the GC do its work. Again if the objects are the same size there is special behavior. I hope know the problem is better explained Cheers, Pablo On Fri, Apr 14, 2017 at 10:50 AM, Igor Stasenko <siguctua@gmail.com> wrote:
On 14 April 2017 at 10:19, Stephane Ducasse <stepharo.self@gmail.com> wrote:
But I do not get how doing that would handle the old instances? Because you want to migrate the old instances.
+1 there are no such thing as 'bad zombies', if they are there, it means you either don't care about migrating data or again, don't care about doing #becomeForward-ing them properly. In any case i don't see how GC could help to fix these issues. You either have consistency or don't have it, and GC cannot do anything magical to fix it.
Stef
On Wed, Apr 12, 2017 at 1:26 PM, Denis Kudriashov <dionisiydk@gmail.com> wrote:
2017-04-12 13:17 GMT+02:00 Guillermo Polito <guillermopolito@gmail.com>:
1) each instance of A is becomed into its corresponding instance of A' 2) finally we become class A into A' This step will make that old instances of A now have: - the old format - but point to the new class A
step 1) ensures that there are no instances of class A anymore. Check following script:
c1 := Class1 new. c2 := Class2 new. c1 becomeForward: c2. Class1 allInstances "=> #()".
And full migration is executed in high priority uninterrupted process to ensure that between 1) and 2) nobody will instantiate Class1
-- Best regards, Igor Stasenko.
-- Pablo Tesone. tesonep@gmail.com
i don't see much difference comparing to old implementation of #becomeForward: the only issue is that #allInstances could report same instance twice first, it will find an old instance (that is forwards to new one) and so, add it to results array, and then walking the heap further will find a new version of same instance. That's why , it think #allInstances should use IdentitySet-behavior to mitigate such problem, and always look if there's already same object captured to avoid reporting it twice. Or else we can declare it as a feature and warn users of #allInstances that they could have duplications, so in case if it important to them, they could always do 'allInstances asIdentitySet' But again, this is soooo orthogonal to adding instance variables to class... maybe we should rename the topic instead and speak about #allInstances behavior? :) On 14 April 2017 at 12:09, tesonep@gmail.com <tesonep@gmail.com> wrote:
Hi, I think the problem was not clearly explained. This is the scenario that is problematic. This scenario does not happen in the new Spur implementation of forwarding, but I think is happening in the old one.
1. You have, let's say 3 instances of ClassA. 2. You add a new instance variable to ClassA. It produces 2a. A new ClassAv2 is created with the instances variables of ClassA and the newone 2b. 3 Instances of ClassAv2 are created 2c. The values of the instance variables of ClassA are copied to the ones in ClassAv2 (the ones missing are left in nil). 2d. The 3 instances of ClassA are becomed forward to the 3 instances of ClassAv2 2e. The ClassA is becomed forward ClassAv2
3. You add a new instance variable to ClassAv2. It produces 3a. A new ClassAv3 is created with the instances variables of ClassAv2 and the newone 3b. 3 Instances of ClassAv3 are created 3c. The values of the instance variables of ClassAv2 are copied to the ones in ClassAv3 (the ones missing are left in nil). 3d. The 3 instances of ClassAv2 are becomed forward to the 3 instances of ClassAv3 3e. The ClassAv2 is becomedFormeward ClassAv3
4. All the instances of ClassAV3 have the correct format and everything works.
What is the problem: ===============
- When you do the first add instance variable, the old instances (the one from ClassA) which are smaller (has 1 instance variable less) have its class changed (after you perform a become of ClassA to ClassAv2). So if you try to use them everything will explode, because you will trying to access an instance variable that does not exists. These instances are not referenced by anyone, however if you perform a ClassAv2>>allInstances you will find them. So if you modify the class adding two variables, one after another the second time you will be accessing the invalid instances.
Considering the differences in the Become implementation ============================================
However, the main difference is the implementation of the become forward. Let's start with the new implementation, as it has not problems.
When you do a become forward, from object a to b, the primitive replaces the object a with a forwarder to b. When this forwarded is accessed the references to it are rewrited. If the objects are the same size (not this scenario) the object b replaces object a. It does not produces an error because the old. In the become forward the old instances are not keeped.
In the old implementation the whole image is scanned, changing the references to the old instances, replacing with references to the new instances. The old instances are not removed, just kept there to let the GC do its work. Again if the objects are the same size there is special behavior.
I hope know the problem is better explained
Cheers, Pablo
On Fri, Apr 14, 2017 at 10:50 AM, Igor Stasenko <siguctua@gmail.com> wrote:
On 14 April 2017 at 10:19, Stephane Ducasse <stepharo.self@gmail.com> wrote:
But I do not get how doing that would handle the old instances? Because you want to migrate the old instances.
+1 there are no such thing as 'bad zombies', if they are there, it means you either don't care about migrating data or again, don't care about doing #becomeForward-ing them properly. In any case i don't see how GC could help to fix these issues. You either have consistency or don't have it, and GC cannot do anything magical to fix it.
Stef
On Wed, Apr 12, 2017 at 1:26 PM, Denis Kudriashov <dionisiydk@gmail.com> wrote:
2017-04-12 13:17 GMT+02:00 Guillermo Polito <guillermopolito@gmail.com> :
1) each instance of A is becomed into its corresponding instance of A' 2) finally we become class A into A' This step will make that old instances of A now have: - the old format - but point to the new class A
step 1) ensures that there are no instances of class A anymore. Check following script:
c1 := Class1 new. c2 := Class2 new. c1 becomeForward: c2. Class1 allInstances "=> #()".
And full migration is executed in high priority uninterrupted process to ensure that between 1) and 2) nobody will instantiate Class1
-- Best regards, Igor Stasenko.
-- Pablo Tesone. tesonep@gmail.com
-- Best regards, Igor Stasenko.
2017-04-14 11:09 GMT+02:00 tesonep@gmail.com <tesonep@gmail.com>:
Hi, I think the problem was not clearly explained. This is the scenario that is problematic. This scenario does not happen in the new Spur implementation of forwarding, but I think is happening in the old one.
1. You have, let's say 3 instances of ClassA. 2. You add a new instance variable to ClassA. It produces 2a. A new ClassAv2 is created with the instances variables of ClassA and the newone 2b. 3 Instances of ClassAv2 are created 2c. The values of the instance variables of ClassA are copied to the ones in ClassAv2 (the ones missing are left in nil). 2d. The 3 instances of ClassA are becomed forward to the 3 instances of ClassAv2 2e. The ClassA is becomed forward ClassAv2
I still not see why my example not reflects all these steps. I checked also scenario with class becoming: c1 := Class1 new. c2 := Class2 new. c1 becomeForward: c2. Class1 becomeForward: Class2. Class2 allInstances "=>#(aClass2)" It also works in Cog
Hi, Please letâs move this discussion on pharo-dev. And keep going :) Cheers, Doru
On Apr 14, 2017, at 12:09 PM, Denis Kudriashov <dionisiydk@gmail.com> wrote:
2017-04-14 11:09 GMT+02:00 tesonep@gmail.com <tesonep@gmail.com>: Hi, I think the problem was not clearly explained. This is the scenario that is problematic. This scenario does not happen in the new Spur implementation of forwarding, but I think is happening in the old one.
1. You have, let's say 3 instances of ClassA. 2. You add a new instance variable to ClassA. It produces 2a. A new ClassAv2 is created with the instances variables of ClassA and the newone 2b. 3 Instances of ClassAv2 are created 2c. The values of the instance variables of ClassA are copied to the ones in ClassAv2 (the ones missing are left in nil). 2d. The 3 instances of ClassA are becomed forward to the 3 instances of ClassAv2 2e. The ClassA is becomed forward ClassAv2
I still not see why my example not reflects all these steps. I checked also scenario with class becoming:
c1 := Class1 new. c2 := Class2 new. c1 becomeForward: c2. Class1 becomeForward: Class2. Class2 allInstances "=>#(aClass2)"
It also works in Cog
-- www.tudorgirba.com www.feenk.com "What we can governs what we wish."
Hi, in non spur, the only improve that I think it can be made safely is moving the garbage collect operation to the migration of instances side, only executing the garbage collect if there are instances. In spur the garbage collect it is not needed anymore, because the way the become is working is not the same. In the old implementation, the old instances where in the image until they are removed by the garbage collector. So you can access them with the allInstances. In the new implementation, the old instances are marked as a forwarded to the new instances. So, there is no need for the garbage collect. This should be tried but I think it can work. Cheers, Pablo On Wed, Apr 12, 2017 at 1:17 PM, Guillermo Polito <guillermopolito@gmail.com
wrote:
On Wed, Apr 12, 2017 at 11:35 AM, Denis Kudriashov <dionisiydk@gmail.com> wrote:
2017-04-12 10:55 GMT+02:00 Guillermo Polito <guillermopolito@gmail.com>:
PharoClassInstaller>>migrateClasses: old to: new using:
anInstanceModification instanceModification := anInstanceModification. old ifEmpty: [ ^ self ]. [ 1 to: old size do: [ :index | self updateClass: (old at: index) to: (new at: index)]. old elementsForwardIdentityTo: new. " Garbage collect away the zombie instances left behind in garbage memory in #updateInstancesFrom: " " If we don't clean up this garbage, a second update would revive them with a wrong layout! " " (newClass rather than oldClass, since they are now both newClass) " Smalltalk garbageCollect. ] valueUnpreemptively
Commenting garbage collection here increases performance 10 times. Then commenting class update loop increases performance 3 times more. But this loop is required. It adopts all instances of old class to new one. And time here spent in #allInstances method.
Can we remove manual garbage collection here? Why it is needed?
Well, there is the comment that explains it and makes pretty good sense.
But is does not explain why these bad zombies exist. We investigates possible reasons and could not reproduce them. We will try remove garbage collection here in Pharo 7
No, this will break stuff! I'll try to explain what does it mean by zombie instances to make some sense:
- Imagine that you have class A + 10 instances of A.
- We add an instance variable to A. - this means the class builder will generate class A' that is the new version of A. - then, it migrates all instances of A to class A'. This migration is not magic: - 10 new instances of A' are created - the state is migrated from the instances of A to A' - each instance of A is becomed into its corresponding instance of A' - finally we become class A into A' This step will make that old instances of A now have: - the old format - but point to the new class A
If we do not garbage collect, this means that doing
A allInstances
will return not only the new 10 instances of A, but the old instances of A'. And that will break LOOOTS of stuff.
-- Pablo Tesone. tesonep@gmail.com
2017-04-12 13:29 GMT+02:00 tesonep@gmail.com <tesonep@gmail.com>:
Hi, in non spur, the only improve that I think it can be made safely is moving the garbage collect operation to the migration of instances side, only executing the garbage collect if there are instances.
In spur the garbage collect it is not needed anymore, because the way the become is working is not the same.
I checked on prespur cogVM: my script produced same result as on spur.
Hi Denis, in the script you are not generating instances of Class1, the problem is with the instances. Cheers, Pablo On Fri, Apr 14, 2017 at 10:04 AM, Denis Kudriashov <dionisiydk@gmail.com> wrote:
2017-04-12 13:29 GMT+02:00 tesonep@gmail.com <tesonep@gmail.com>:
Hi, in non spur, the only improve that I think it can be made safely is moving the garbage collect operation to the migration of instances side, only executing the garbage collect if there are instances.
In spur the garbage collect it is not needed anymore, because the way the become is working is not the same.
I checked on prespur cogVM: my script produced same result as on spur.
-- Pablo Tesone. tesonep@gmail.com
On 12 April 2017 at 14:17, Guillermo Polito <guillermopolito@gmail.com> wrote:
On Wed, Apr 12, 2017 at 11:35 AM, Denis Kudriashov <dionisiydk@gmail.com> wrote:
2017-04-12 10:55 GMT+02:00 Guillermo Polito <guillermopolito@gmail.com>:
PharoClassInstaller>>migrateClasses: old to: new using:
anInstanceModification instanceModification := anInstanceModification. old ifEmpty: [ ^ self ]. [ 1 to: old size do: [ :index | self updateClass: (old at: index) to: (new at: index)]. old elementsForwardIdentityTo: new. " Garbage collect away the zombie instances left behind in garbage memory in #updateInstancesFrom: " " If we don't clean up this garbage, a second update would revive them with a wrong layout! " " (newClass rather than oldClass, since they are now both newClass) " Smalltalk garbageCollect. ] valueUnpreemptively
Commenting garbage collection here increases performance 10 times. Then commenting class update loop increases performance 3 times more. But this loop is required. It adopts all instances of old class to new one. And time here spent in #allInstances method.
Can we remove manual garbage collection here? Why it is needed?
Well, there is the comment that explains it and makes pretty good sense.
But is does not explain why these bad zombies exist. We investigates possible reasons and could not reproduce them. We will try remove garbage collection here in Pharo 7
No, this will break stuff! I'll try to explain what does it mean by zombie instances to make some sense:
- Imagine that you have class A + 10 instances of A.
- We add an instance variable to A. - this means the class builder will generate class A' that is the new version of A. - then, it migrates all instances of A to class A'. This migration is not magic: - 10 new instances of A' are created - the state is migrated from the instances of A to A' - each instance of A is becomed into its corresponding instance of A' - finally we become class A into A' This step will make that old instances of A now have: - the old format - but point to the new class A
If we do not garbage collect, this means that doing
A allInstances
will return not only the new 10 instances of A, but the old instances of A'. And that will break LOOOTS of stuff.
if you run #allInstances and in between you will trigger adding instance var & GC etc etc.. you'll have everything broken.. because there are things didn't meant to work in certain scenarios. IIRC allInstances is highly dependns on NOT having full GC while doing it, and that's why all loops that doing it is highly conservative & cautious about creating new objects while iterating over heap. That's the nature how #allInstance works, and you could have a tons of issues with it regardless , if you do full GC manually, or it triggered by VM itself. So, this is nothing to do with migrating instances of class. -- Best regards, Igor Stasenko.
participants (7)
-
Denis Kudriashov -
Guillermo Polito -
Igor Stasenko -
Peter Uhnak -
Stephane Ducasse -
tesonep＠gmail.com -
Tudor Girba