[Pharo-project] invalid sources for unregistered CompiledMethod
Most CompiledMethod are registered in a methodDictionary. But some are not. (CompiledMethod allInstances reject: [:e | e methodClass ~~ nil and: [e selector ~~ nil and: [e == (e methodClass methodDictionary at: e selector ifAbsent: [nil])]]]) size This is often because a BlockClosure has been registered somewhere in a registry (a ServiceRegistry for example). The objects pointers chain is something like: a SeviceRegistry -> a ServiceAction -> a BlockClosure -> a MethodContext -> a CompiledMethod... Another source of MethodContext instances is a concurrent Process running in the image. If ever recompiled, a new instance of the CompiledMethod is installed in the methodDictionary. But the old CompiledMethod is still there, being pointed to by above chains. The old CompiledMethod has a source code pointer pointing to sourceCode somewhere in the change Log, and so far it's fine. But what if you compress the change log ? The registered CompiledMethod all get a valid new pointer to the new source location. But the unregistered ones retain their old pointer now pointing randomly in the change file. This is bad. (CompiledMethod allInstances select: [:e | e selector = #id:text:button:description:action: and: [(e methodClass methodDictionary at: e selector ifAbsent: [nil]) ~~ e]]) anyOne getSource Now if you try to decompile the method, either there is an Error like here, or unfortunately the random sourceCode is valid a valid chunk of Smalltalk and you open the door to potentially strange things... It could be misleading if you try to understand what the hell this SeviceAction could be doing. If you try to decompile the BlockClosure, and getSource does not point to valid code, it doesn't work, you get nil... This is another bug, because #decompile should revert to homeMethod decompile decompileString... I propose that #condenseChanges and the like, anihilate source code pointers of unregistered CompiledMethod. This should solve the issue. What do you think ?
Ah, I forgot to say that if you recompile the methodClass, it won't trigger recompilation of those unregistered methods. Some actions like removing an inst var can then trigger very bad things when you try to execute those unregistered methods, like corrupting memory. A security in the class builder would be to: - mutate all unregistered CompiledMethod pointing to this methodClass before the oldClass becomeForward: newClass Then, a security in the VM should prevent those CompiledMethod mutant to be executed. (or maybe the mutant could simply be modified to send a message like self mustBeRecompiled). What do you think of this one ? 2009/11/6 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
Most CompiledMethod are registered in a methodDictionary. But some are not.
(CompiledMethod allInstances reject: [:e | e methodClass ~~ nil and: [e selector ~~ nil and: [e == (e methodClass methodDictionary at: e selector ifAbsent: [nil])]]]) size
This is often because a BlockClosure has been registered somewhere in a registry (a ServiceRegistry for example). The objects pointers chain is something like: a SeviceRegistry -> a ServiceAction -> a BlockClosure -> a MethodContext -> a CompiledMethod...
Another source of MethodContext instances is a concurrent Process running in the image.
If ever recompiled, a new instance of the CompiledMethod is installed in the methodDictionary. But the old CompiledMethod is still there, being pointed to by above chains. The old CompiledMethod has a source code pointer pointing to sourceCode somewhere in the change Log, and so far it's fine.
But what if you compress the change log ? The registered CompiledMethod all get a valid new pointer to the new source location. But the unregistered ones retain their old pointer now pointing randomly in the change file. This is bad.
(CompiledMethod allInstances select: [:e | e selector = #id:text:button:description:action: and: [(e methodClass methodDictionary at: e selector ifAbsent: [nil]) ~~ e]]) anyOne getSource
Now if you try to decompile the method, either there is an Error like here, or unfortunately the random sourceCode is valid a valid chunk of Smalltalk and you open the door to potentially strange things... It could be misleading if you try to understand what the hell this SeviceAction could be doing.
If you try to decompile the BlockClosure, and getSource does not point to valid code, it doesn't work, you get nil... This is another bug, because #decompile should revert to homeMethod decompile decompileString...
I propose that #condenseChanges and the like, anihilate source code pointers of unregistered CompiledMethod. This should solve the issue. What do you think ?
2009/11/7 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
Ah, I forgot to say that if you recompile the methodClass, it won't trigger recompilation of those unregistered methods. Some actions like removing an inst var can then trigger very bad things when you try to execute those unregistered methods, like corrupting memory.
A security in the class builder would be to: - mutate all unregistered CompiledMethod pointing to this methodClass  before the oldClass becomeForward: newClass
Then, a security in the VM should prevent those CompiledMethod mutant to be executed. (or maybe the mutant could simply be modified to send a message like self mustBeRecompiled).
What do you think of this one ?
I would just throw an error and prevent (oldClass becomeForward: newClass) before any unregistered methods is removed from system, or .. skip mutating (oldClass becomeForward: newClass) if such methods hanging around, to avoid possible problems, and let user to deal with obsolete classes. The less magic we have, the more predicatable system will be.
2009/11/6 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
Most CompiledMethod are registered in a methodDictionary. But some are not.
(CompiledMethod allInstances reject: [:e | e methodClass ~~ nil and: [e selector ~~ nil and: [e == (e methodClass methodDictionary at: e selector ifAbsent: [nil])]]]) size
This is often because a BlockClosure has been registered somewhere in a registry (a ServiceRegistry for example). The objects pointers chain is something like: a SeviceRegistry -> a ServiceAction -> a BlockClosure -> a MethodContext -> a CompiledMethod...
Another source of MethodContext instances is a concurrent Process running in the image.
If ever recompiled, a new instance of the CompiledMethod is installed in the methodDictionary. But the old CompiledMethod is still there, being pointed to by above chains. The old CompiledMethod has a source code pointer pointing to sourceCode somewhere in the change Log, and so far it's fine.
But what if you compress the change log ? The registered CompiledMethod all get a valid new pointer to the new source location. But the unregistered ones retain their old pointer now pointing randomly in the change file. This is bad.
(CompiledMethod allInstances select: [:e | e selector = #id:text:button:description:action: and: [(e methodClass methodDictionary at: e selector ifAbsent: [nil]) ~~ e]]) anyOne getSource
Now if you try to decompile the method, either there is an Error like here, or unfortunately the random sourceCode is valid a valid chunk of Smalltalk and you open the door to potentially strange things... It could be misleading if you try to understand what the hell this SeviceAction could be doing.
If you try to decompile the BlockClosure, and getSource does not point to valid code, it doesn't work, you get nil... This is another bug, because #decompile should revert to homeMethod decompile decompileString...
I propose that #condenseChanges and the like, anihilate source code pointers of unregistered CompiledMethod. This should solve the issue. What do you think ?
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
2009/11/6 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:
Most CompiledMethod are registered in a methodDictionary. But some are not.
(CompiledMethod allInstances reject: [:e | e methodClass ~~ nil and: [e selector ~~ nil and: [e == (e methodClass methodDictionary at: e selector ifAbsent: [nil])]]]) size
This is often because a BlockClosure has been registered somewhere in a registry (a ServiceRegistry for example). The objects pointers chain is something like: a SeviceRegistry -> a ServiceAction -> a BlockClosure -> a MethodContext -> a CompiledMethod...
Another source of MethodContext instances is a concurrent Process running in the image.
If ever recompiled, a new instance of the CompiledMethod is installed in the methodDictionary. But the old CompiledMethod is still there, being pointed to by above chains. The old CompiledMethod has a source code pointer pointing to sourceCode somewhere in the change Log, and so far it's fine.
But what if you compress the change log ? The registered CompiledMethod all get a valid new pointer to the new source location. But the unregistered ones retain their old pointer now pointing randomly in the change file. This is bad.
(CompiledMethod allInstances select: [:e | e selector = #id:text:button:description:action: and: [(e methodClass methodDictionary at: e selector ifAbsent: [nil]) ~~ e]]) anyOne getSource
Now if you try to decompile the method, either there is an Error like here, or unfortunately the random sourceCode is valid a valid chunk of Smalltalk and you open the door to potentially strange things... It could be misleading if you try to understand what the hell this SeviceAction could be doing.
If you try to decompile the BlockClosure, and getSource does not point to valid code, it doesn't work, you get nil... This is another bug, because #decompile should revert to homeMethod decompile decompileString...
I propose that #condenseChanges and the like, anihilate source code pointers of unregistered CompiledMethod. This should solve the issue. What do you think ?
Seems like a right way to go.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
Nicolas Cellier wrote:
But what if you compress the change log ?
Perhaps it's time to finally get rid of source pointers and just keep the source code as objects in the image, referenced directly from the CompiledMethod. External sources were necessary 25 years ago when an image without sources consumed 3/4 of the total RAM of the machine, but it's not like that anymore. Regards, -Martin
Perhaps it's time to finally get rid of source pointers and just keep the source code as objects in the image, referenced directly from the CompiledMethod.
+1 That would not only fix that bug, but dozens of other serious bugs in the system. Lukas -- Lukas Renggli http://www.lukas-renggli.ch
On Nov 7, 2009, at 6:05 PM, Martin McClure wrote:
Nicolas Cellier wrote:
But what if you compress the change log ?
Perhaps it's time to finally get rid of source pointers and just keep the source code as objects in the image, referenced directly from the CompiledMethod.
External sources were necessary 25 years ago when an image without sources consumed 3/4 of the total RAM of the machine, but it's not like that anymore.
+100 I did that once as an experiment: searching for strings in all the code gets *fast*. Marcus
2009/11/7 Marcus Denker <denker@acm.org>:
On Nov 7, 2009, at 6:05 PM, Martin McClure wrote:
Nicolas Cellier wrote:
But what if you compress the change log ?
Perhaps it's time to finally get rid of source pointers and just keep the source code as objects in the image, referenced directly from the CompiledMethod.
External sources were necessary 25 years ago when an image without sources consumed 3/4 of the total RAM of the machine, but it's not like that anymore.
+100
I did that once as an experiment: searching for strings in all the code gets *fast*.
    Marcus
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
I'd say +16M This is what i got when condensing sources: 16249263 2009-10-19 20:25 PharoV10.sources
On Nov 7, 2009, at 7:13 PM, Nicolas Cellier wrote:
Perhaps it's time to finally get rid of source pointers and just keep the source code as objects in the image, referenced directly from the CompiledMethod.
External sources were necessary 25 years ago when an image without sources consumed 3/4 of the total RAM of the machine, but it's not like that anymore.
+100
I did that once as an experiment: searching for strings in all the code gets *fast*.
I'd say +16M
This is what i got when condensing sources: 16249263 2009-10-19 20:25 PharoV10.sources
yes, but nevertheless, this is how much percent of the Ram of your machine? Two notes: - We could compress the source-strings. That should lead to something like 5MB increase. - We should invest into a general mechanism of swapping in unused objects on demand. It's fun that there is a special case for the source, but all the other resources happily live in the image (bitmaps, sounds... in general objects not part of the working set, e.g. I guess 80% of all compiled methods...). Marcus
Yes, in "normal" use of Pharo, holding the sources in memory would only consume a fraction of the available RAM. Some applications, however, have several hundred images running at the same time on a server. In this case the picture looks very different again. I think Marcus' idea of swapping in/out ressources is very interesting. We could get rid of the ancient sources handling but still have a low, or even lower, memory footprint. Cheers, Adrian On Nov 7, 2009, at 19:21 , Marcus Denker wrote:
On Nov 7, 2009, at 7:13 PM, Nicolas Cellier wrote:
Perhaps it's time to finally get rid of source pointers and just keep the source code as objects in the image, referenced directly from the CompiledMethod.
External sources were necessary 25 years ago when an image without sources consumed 3/4 of the total RAM of the machine, but it's not like that anymore.
+100
I did that once as an experiment: searching for strings in all the code gets *fast*.
I'd say +16M
This is what i got when condensing sources: 16249263 2009-10-19 20:25 PharoV10.sources
yes, but nevertheless, this is how much percent of the Ram of your machine?
Two notes:
- We could compress the source-strings. That should lead to something like 5MB increase.
- We should invest into a general mechanism of swapping in unused objects on demand. It's fun that there is a special case for the source, but all the other resources happily live in the image (bitmaps, sounds... in general objects not part of the working set, e.g. I guess 80% of all compiled methods...).
Marcus
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Adrian Lienhard wrote:
Yes, in "normal" use of Pharo, holding the sources in memory would only consume a fraction of the available RAM. Some applications, however, have several hundred images running at the same time on a server. In this case the picture looks very different again.
It would be easy to strip all sources from production images, but the simple way of doing that would mean any debugging would be done on decompiled code.
I think Marcus' idea of swapping in/out ressources is very interesting. We could get rid of the ancient sources handling but still have a low, or even lower, memory footprint.
This would be even better. ImageSegments are a start in this direction. -Martin
On Nov 7, 2009, at 8:30 PM, Martin McClure wrote:
Adrian Lienhard wrote:
Yes, in "normal" use of Pharo, holding the sources in memory would only consume a fraction of the available RAM. Some applications, however, have several hundred images running at the same time on a server. In this case the picture looks very different again.
It would be easy to strip all sources from production images, but the simple way of doing that would mean any debugging would be done on decompiled code.
I think Marcus' idea of swapping in/out ressources is very interesting. We could get rid of the ancient sources handling but still have a low, or even lower, memory footprint.
This would be even better. ImageSegments are a start in this direction.
yes this is why I always wanted to have tests to make sure that we do not break them. Stef
-Martin
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
2009/11/7 Stéphane Ducasse <stephane.ducasse@inria.fr>:
On Nov 7, 2009, at 8:30 PM, Martin McClure wrote:
Adrian Lienhard wrote:
Yes, in "normal" use of Pharo, holding the sources in memory would only consume a fraction of the available RAM. Some applications, however, have several hundred images running at the same time on a server. In this case the picture looks very different again.
It would be easy to strip all sources from production images, but the simple way of doing that would mean any debugging would be done on decompiled code.
I think Marcus' idea of swapping in/out ressources is very interesting. We could get rid of the ancient sources handling but still have a low, or even lower, memory footprint.
This would be even better. ImageSegments are a start in this direction.
yes this is why I always wanted to have tests to make sure that we do not break them.
Stef
-Martin
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Until these great ideas take form, find a workaround attached.
2009/11/7 Adrian Lienhard <adi@netstyle.ch>
Yes, in "normal" use of Pharo, holding the sources in memory would only consume a fraction of the available RAM. Some applications, however, have several hundred images running at the same time on a server. In this case the picture looks very different again.
There are scenarios where one would want to build a Smalltalk inside another, both sharing the same VM (i.e. testing intra-image communication or simulating a big-bang process), but of course, several hundred images running each one with its own VM is a more common case, and none of these is a "normal" use of Smalltalk currently. Cheers, Hernán
El sáb, 07-11-2009 a las 19:33 +0100, Adrian Lienhard escribió:
Yes, in "normal" use of Pharo, holding the sources in memory would only consume a fraction of the available RAM. Some applications, however, have several hundred images running at the same time on a server. In this case the picture looks very different again.
I think that the pros of not having a separate sources files surpases the cons of having a couple more MB per image. Also, in deployment, a core image will be used that it is very striped already so the sources are even smaller and the overhead in the image is better. I vote for +1 on having sources inside the image as other resources (bitmaps, fonts, etc). And, if the sources can be stripped also from the image for deployment, the additional MB aren't an issue either.
I think Marcus' idea of swapping in/out ressources is very interesting. We could get rid of the ancient sources handling but still have a low, or even lower, memory footprint.
Cheers, Adrian
On Nov 7, 2009, at 19:21 , Marcus Denker wrote:
On Nov 7, 2009, at 7:13 PM, Nicolas Cellier wrote:
Perhaps it's time to finally get rid of source pointers and just keep the source code as objects in the image, referenced directly from the CompiledMethod.
External sources were necessary 25 years ago when an image without sources consumed 3/4 of the total RAM of the machine, but it's not like that anymore.
+100
I did that once as an experiment: searching for strings in all the code gets *fast*.
I'd say +16M
This is what i got when condensing sources: 16249263 2009-10-19 20:25 PharoV10.sources
yes, but nevertheless, this is how much percent of the Ram of your machine?
Two notes:
- We could compress the source-strings. That should lead to something like 5MB increase.
- We should invest into a general mechanism of swapping in unused objects on demand. It's fun that there is a special case for the source, but all the other resources happily live in the image (bitmaps, sounds... in general objects not part of the working set, e.g. I guess 80% of all compiled methods...).
Marcus
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Miguel Cobá http://miguel.leugim.com.mx
participants (9)
-
Adrian Lienhard -
Hernán Morales Durand -
Igor Stasenko -
Lukas Renggli -
Marcus Denker -
Martin McClure -
Miguel Enrique Cobá Martinez -
Nicolas Cellier -
Stéphane Ducasse