Re: [Pharo-project] [update 1.2] #12161 - #12172
Igor I was thinking to rollback now because we cannot do anything. Is is ok for you? And after we can include everything. Stef On Oct 1, 2010, at 3:08 PM, Igor Stasenko wrote:
On 1 October 2010 15:47, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
Igor Can you have a look because right now in 12174 I cannot do any update anymore so - either we fix it now or we have to rollbakc to 12161.
I will take a look
Thanks.
On Oct 1, 2010, at 12:49 PM, Igor Stasenko wrote:
2010/10/1 Levente Uzonyi <leves@elte.hu>:
On Fri, 1 Oct 2010, Igor Stasenko wrote:
On 1 October 2010 12:06, Levente Uzonyi <leves@elte.hu> wrote: On Fri, 1 Oct 2010, Marcus Denker wrote:
Issue 3002: A new finalization code ready for integration in Pharo Issue 3026: after 12172, sync with repository and reload
If you didn't fix the migration code, then WeakFinalizationRegistry is still in the system. Details here:
http://lists.squeakfoundation.org/pipermail/squeak-dev/2010-September/154009...
You mean this not work: Smalltalk forgetClass: #WeakFinalizationRegistry logged: false.
should be Smalltalk forgetClass: WeakFinalizationRegistry logged: false.
right?
Yes.
Thanks for noticing.
Levente
Levente
-- Marcus Denker -- http://www.marcusdenker.de INRIA Lille -- Nord Europe. Team RMoD.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On 1 October 2010 16:15, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
Igor
I was thinking to rollback now because we cannot do anything. Is is ok for you? And after we can include everything.
Sure, no problem. It should not behave like that. There could be some very little speed degradation, but not like you describing. I found that after update 12172->12174 a FinalizationDependents in WeakArray held 4 weak registries before, but after update it also holds a WeakIdentityKeyDictionary with 9387 elements. This is something from monticello cache, but it screwed somehow.. I were able to delete it from weakdependents,manually, like in this script: arr := (WeakArray classPool at: #FinalizationDependents ). arr withIndexDo: [:e :i | e class == WeakRegistry ifFalse: [ arr at: i put: nil ]] After that, everything vent back to normal. This dictionary comes from MCMethodDefinition Definitions class var. So you can simply do: MCMethodDefinition shutDown to stop CPU hogging. But i wonder, if this is just one-time issue, or it will repeat again, after update. During writing this, i found that you already rewinded update, so i was unable to check if it will start consuming CPU on new code loaded, or its just some tension between MC and my migration procedure, which can be healed by MCMethodDefinition shutDown.
Stef
On Oct 1, 2010, at 3:08 PM, Igor Stasenko wrote:
On 1 October 2010 15:47, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
Igor Can you have a look because right now in 12174 I cannot do any update anymore so     - either we fix it now     or we have to rollbakc to 12161.
I will take a look
Thanks.
On Oct 1, 2010, at 12:49 PM, Igor Stasenko wrote:
2010/10/1 Levente Uzonyi <leves@elte.hu>:
On Fri, 1 Oct 2010, Igor Stasenko wrote:
On 1 October 2010 12:06, Levente Uzonyi <leves@elte.hu> wrote: On Fri, 1 Oct 2010, Marcus Denker wrote:
Issue 3002: Â Â A new finalization code ready for integration in Pharo Issue 3026: after 12172, sync with repository and reload
If you didn't fix the migration code, then WeakFinalizationRegistry is still in the system. Details here:
http://lists.squeakfoundation.org/pipermail/squeak-dev/2010-September/154009...
You mean this not work: Â Â Â Â Smalltalk forgetClass: #WeakFinalizationRegistry logged: false.
should be     Smalltalk forgetClass: WeakFinalizationRegistry logged: false.
right?
Yes.
Thanks for noticing.
Levente
Levente
-- Marcus Denker  -- http://www.marcusdenker.de INRIA Lille -- Nord Europe. Team RMoD.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
I found the bug.. This was in WeakKeyDictionary>>expiredValuesDo: Please, try doing the same with this patch, and see if problem gone. See attachment. -- Best regards, Igor Stasenko AKA sig.
On Fri, 1 Oct 2010, Igor Stasenko wrote:
On 1 October 2010 16:15, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote: Igor
I was thinking to rollback now because we cannot do anything. Is is ok for you? And after we can include everything.
Sure, no problem. It should not behave like that. There could be some very little speed degradation, but not like you describing. I found that after update 12172->12174 a FinalizationDependents in WeakArray held 4 weak registries before, but after update it also holds a WeakIdentityKeyDictionary with 9387 elements. This is something from monticello cache, but it screwed somehow.. I were able to delete it from weakdependents,manually, like in this script: arr := (WeakArray classPool at: #FinalizationDependents ). arr withIndexDo: [:e :i | e class == WeakRegistry ifFalse: [ arr at: i put: nil ]] After that, everything vent back to normal. This dictionary comes from MCMethodDefinition Definitions class var. So you can simply do: MCMethodDefinition shutDown to stop CPU hogging. But i wonder, if this is just one-time issue, or it will repeat again, after update. During writing this, i found that you already rewinded update, so i was unable to check if it will start consuming CPU on new code loaded, or its just some tension between MC and my migration procedure, which can be healed by MCMethodDefinition shutDown. IIRC the WeakKeyDictionary is Pharo doesn't work (doesn't free slots or so) if it's not registered to the finalization process, but Henrik will correct me if I'm wrong. Levente
Stef
On Oct 1, 2010, at 3:08 PM, Igor Stasenko wrote:
On 1 October 2010 15:47, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
Igor Can you have a look because right now in 12174 I cannot do any update anymore so     - either we fix it now     or we have to rollbakc to 12161.
I will take a look
Thanks.
On Oct 1, 2010, at 12:49 PM, Igor Stasenko wrote:
2010/10/1 Levente Uzonyi <leves@elte.hu>:
On Fri, 1 Oct 2010, Igor Stasenko wrote:
On 1 October 2010 12:06, Levente Uzonyi <leves@elte.hu> wrote: On Fri, 1 Oct 2010, Marcus Denker wrote:
Issue 3002: Â Â A new finalization code ready for integration in Pharo Issue 3026: after 12172, sync with repository and reload
If you didn't fix the migration code, then WeakFinalizationRegistry is still in the system. Details here:
http://lists.squeakfoundation.org/pipermail/squeak-dev/2010-September/154009...
You mean this not work: Â Â Â Â Smalltalk forgetClass: #WeakFinalizationRegistry logged: false.
should be     Smalltalk forgetClass: WeakFinalizationRegistry logged: false.
right?
Yes.
Thanks for noticing.
Levente
Levente
-- Marcus Denker  -- http://www.marcusdenker.de INRIA Lille -- Nord Europe. Team RMoD.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig. _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
2010/10/1 Levente Uzonyi <leves@elte.hu>:
On Fri, 1 Oct 2010, Igor Stasenko wrote:
On 1 October 2010 16:15, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote: Igor
I was thinking to rollback now because we cannot do anything. Is is ok for you? And after we can include everything.
Sure, no problem. It should not behave like that. There could be some very little speed degradation, but not like you describing.
I found that after update 12172->12174 a FinalizationDependents in WeakArray held 4 weak registries before, but after update it also holds a WeakIdentityKeyDictionary with 9387 elements.
This is something from monticello cache, but it screwed somehow.. I were able to delete it from weakdependents,manually, Â like in this script:
arr := (WeakArray classPool at: #FinalizationDependents ). arr withIndexDo: [:e :i |     e class == WeakRegistry ifFalse: [ arr at: i put: nil     ]]
After that, everything vent back to normal.
This dictionary comes from MCMethodDefinition Definitions class var. So you can simply do:
MCMethodDefinition shutDown
to stop CPU hogging.
But i wonder, if this is just one-time issue, or it will repeat again, after update.
During writing this, i found that you already rewinded update, so i was unable to check if it will start consuming CPU on new code loaded, or its just some tension between MC and my migration procedure, which can be healed by MCMethodDefinition shutDown.
IIRC the WeakKeyDictionary is Pharo doesn't work (doesn't free slots or so) if it's not registered to the finalization process, but Henrik will correct me if I'm wrong.
It marks associations whose keys became nil as expired ones (association key: association). So, then associations get reused when new entries added. But i tricked myself by using #allAssociationsDo: , which were wiping out associations, before sending #finalize to their values... so, no finalization were run. Somehow, this lead to blowing up an MC method definitions cache.
Levente
Stef
On Oct 1, 2010, at 3:08 PM, Igor Stasenko wrote:
On 1 October 2010 15:47, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
Igor Can you have a look because right now in 12174 I cannot do any update anymore so     - either we fix it now     or we have to rollbakc to 12161.
I will take a look
Thanks.
On Oct 1, 2010, at 12:49 PM, Igor Stasenko wrote:
2010/10/1 Levente Uzonyi <leves@elte.hu>:
On Fri, 1 Oct 2010, Igor Stasenko wrote:
On 1 October 2010 12:06, Levente Uzonyi <leves@elte.hu> wrote: On Fri, 1 Oct 2010, Marcus Denker wrote:
Issue 3002: Â Â A new finalization code ready for integration in Pharo Issue 3026: after 12172, sync with repository and reload
If you didn't fix the migration code, then WeakFinalizationRegistry is still in the system. Details here:
http://lists.squeakfoundation.org/pipermail/squeak-dev/2010-September/154009...
You mean this not work: Â Â Â Â Smalltalk forgetClass: #WeakFinalizationRegistry logged: false.
should be     Smalltalk forgetClass: WeakFinalizationRegistry logged: false.
right?
Yes.
Thanks for noticing.
Levente
Levente
-- Marcus Denker  -- http://www.marcusdenker.de INRIA Lille -- Nord Europe. Team RMoD.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
Oh crap.. the problem is not in that. MC produces huge weak-key dictionary during updates for instance 12159 -> 12162 produces dictionary with 14k entries, which sits in #FinalizationDependents array and scanned after each GC. And during scanning, its using block closures (and each push of block closure consuming memory), so it big enough to trigger another GC. And its self-supporting process of wasting CPU cycles just for nothing... Here some numbers: (note, i din't installed new finalization code yet) MCMethodDefinition cachedDefinitions size 14987 [MCMethodDefinition cachedDefinitions finalizeValues ] timeToRun 10 so, it spends 10 ms dueing each GC cycle for scanning it. So, the quest is to avoid using block closures during scanning it, so it won't produce garbage. -- Best regards, Igor Stasenko AKA sig.
Here what happens: Original method WeakKeyDictionary>>finalizeValues WeakKeyDictionary>>finalizeValues "Clear all associations with nil keys" array do:[:assoc| 1 to: array size do:[:i | | assoc | assoc := array at: i. assoc ifNotNil:[ assoc expire ifTrue:[ expired := expired + 1. tally := tally - 1. ]. ]. ]. And i noticed that after recompilation of this method, it starts spending much more time to scan 14k entries: [MCMethodDefinition cachedDefinitions finalizeValues ] timeToRun 167 (my guess that this method was never recompiled with new Eliot's closures). Now, if i change it to: finalizeValues "Clear all associations with nil keys" 1 to: array size do:[:i | | assoc | assoc := array at: i. array do:[:assoc| assoc ifNotNil:[ assoc expire ifTrue:[ expired := expired + 1. tally := tally - 1. ]. ]. ]. so, #to:do: is inlined by compiler, then numbers is much better: [MCMethodDefinition cachedDefinitions finalizeValues ] timeToRun 4 Just try it yourself, if you not believe. -- Best regards, Igor Stasenko AKA sig.
participants (3)
-
Igor Stasenko -
Levente Uzonyi -
Stéphane Ducasse