[Pharo-project] Loading MCZ freezes, but loading the source.st works?
Hi: I got here a Monticello package that does not properly load using the Metacello configuration. The image freezes and cmd+. does not pop up a debugger anymore. However, if I extract the source.st contained in the package and file it in manually, it does not freeze the image. I belief that I once saw with appropriate Transcript>>#show that something caused an infinite loop. There are dependencies in overrides I do on kernel classes. They seem to be properly resolved in the source.st file. Is there a way to force Monticello/Metacello to fall back on the included source instead of using the binary snapshot? (I guess that is going wrong for some reason). Thanks Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
On Mon, Dec 5, 2011 at 4:21 PM, Stefan Marr <pharo@stefan-marr.de> wrote:
Hi:
I got here a Monticello package that does not properly load using the Metacello configuration. The image freezes and cmd+. does not pop up a debugger anymore.
However, if I extract the source.st contained in the package and file it in manually, it does not freeze the image.
I belief that I once saw with appropriate Transcript>>#show that something caused an infinite loop. There are dependencies in overrides I do on kernel classes. They seem to be properly resolved in the source.st file.
It is likely you broke DataStream, and hence #loadDefinitions or friends is looping.
Is there a way to force Monticello/Metacello to fall back on the included source instead of using the binary snapshot? (I guess that is going wrong for some reason).
The binary snapshot is only for the definitions. The source is always get from the .sources.
Thanks Stefan
-- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
-- Mariano http://marianopeck.wordpress.com
On 06.12.2011 15:55, Mariano Martinez Peck wrote:
On Mon, Dec 5, 2011 at 4:21 PM, Stefan Marr <pharo@stefan-marr.de <mailto:pharo@stefan-marr.de>> wrote:
Hi:
I got here a Monticello package that does not properly load using the Metacello configuration. The image freezes and cmd+. does not pop up a debugger anymore.
Is there a way to force Monticello/Metacello to fall back on the included source instead of using the binary snapshot? (I guess that is going wrong for some reason).
Not very elegant, but you can delete the .bin from the .mcz file, and it should fall back to importing from sources.
The binary snapshot is only for the definitions. The source is always get from the .sources.
I'm not sure what you mean, but the source compiled when you install an MCDefinition, is taken from the .bin if that was the source of it. (its source instvar is a string, not a pointer to source location in .sources file). Which is why it's safe to do non-ascii code in mcz as long as you can read from the .bin (As WideStrings are decoded properly), otherwise the .sources will be read as latin1, though they weren't written as such if any of the methods contained non-latin1, iirc. Cheers, Henry
On Tue, Dec 6, 2011 at 4:30 PM, Henrik Sperre Johansen < henrik.s.johansen@veloxit.no> wrote:
On 06.12.2011 15:55, Mariano Martinez Peck wrote:
On Mon, Dec 5, 2011 at 4:21 PM, Stefan Marr <pharo@stefan-marr.de> wrote:
Hi:
I got here a Monticello package that does not properly load using the Metacello configuration. The image freezes and cmd+. does not pop up a debugger anymore.
Is there a way to force Monticello/Metacello to fall back on the included source instead of using the binary snapshot? (I guess that is going wrong for some reason).
Not very elegant, but you can delete the .bin from the .mcz file, and it should fall back to importing from sources.
The binary snapshot is only for the definitions. The source is always get from the .sources.
I'm not sure what you mean, but the source compiled when you install an MCDefinition, is taken from the .bin if that was the source of it. (its source instvar is a string, not a pointer to source location in .sources file).
It seems I was wrong. All I wanted to say is that the final representation of code (classes, compiled methods, etc) are compiled from source code. The snapshot does not include a "binary represenation of the code". The code is always needed. What I thought is that it was always from sources.st, but you say it can be from the bin as well.
Which is why it's safe to do non-ascii code in mcz as long as you can read from the .bin (As WideStrings are decoded properly), otherwise the .sources will be read as latin1, though they weren't written as such if any of the methods contained non-latin1, iirc.
Cheers, Henry
-- Mariano http://marianopeck.wordpress.com
On 06.12.2011 16:35, Mariano Martinez Peck wrote:
On Tue, Dec 6, 2011 at 4:30 PM, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no <mailto:henrik.s.johansen@veloxit.no>> wrote:
On 06.12.2011 15:55, Mariano Martinez Peck wrote:
On Mon, Dec 5, 2011 at 4:21 PM, Stefan Marr <pharo@stefan-marr.de <mailto:pharo@stefan-marr.de>> wrote:
Hi:
I got here a Monticello package that does not properly load using the Metacello configuration. The image freezes and cmd+. does not pop up a debugger anymore.
Is there a way to force Monticello/Metacello to fall back on the included source instead of using the binary snapshot? (I guess that is going wrong for some reason).
Not very elegant, but you can delete the .bin from the .mcz file, and it should fall back to importing from sources.
The binary snapshot is only for the definitions. The source is always get from the .sources.
I'm not sure what you mean, but the source compiled when you install an MCDefinition, is taken from the .bin if that was the source of it. (its source instvar is a string, not a pointer to source location in .sources file).
It seems I was wrong. All I wanted to say is that the final representation of code (classes, compiled methods, etc) are compiled from source code. The snapshot does not include a "binary represenation of the code". The code is always needed. What I thought is that it was always from sources.st <http://sources.st> , but you say it can be from the bin as well.
Yup! OT: Which, as you surely know, is also why a more radical rethinking/restructuring than the neat experiment Tobias Pape posted recently is needed if one wanted to take full advantage of a Fuel monticello format :) Eliminate the compilation step, and mcz loading would probably be FAST! Cheers, Henry
On Tue, Dec 6, 2011 at 4:44 PM, Henrik Sperre Johansen < henrik.s.johansen@veloxit.no> wrote:
On 06.12.2011 16:35, Mariano Martinez Peck wrote:
On Tue, Dec 6, 2011 at 4:30 PM, Henrik Sperre Johansen < henrik.s.johansen@veloxit.no> wrote:
On 06.12.2011 15:55, Mariano Martinez Peck wrote:
On Mon, Dec 5, 2011 at 4:21 PM, Stefan Marr <pharo@stefan-marr.de>wrote:
Hi:
I got here a Monticello package that does not properly load using the Metacello configuration. The image freezes and cmd+. does not pop up a debugger anymore.
Is there a way to force Monticello/Metacello to fall back on the included source instead of using the binary snapshot? (I guess that is going wrong for some reason).
Not very elegant, but you can delete the .bin from the .mcz file, and it should fall back to importing from sources.
The binary snapshot is only for the definitions. The source is always get from the .sources.
I'm not sure what you mean, but the source compiled when you install an MCDefinition, is taken from the .bin if that was the source of it. (its source instvar is a string, not a pointer to source location in .sources file).
It seems I was wrong. All I wanted to say is that the final representation of code (classes, compiled methods, etc) are compiled from source code. The snapshot does not include a "binary represenation of the code". The code is always needed. What I thought is that it was always from sources.st , but you say it can be from the bin as well.
Yup!
OT: Which, as you surely know, is also why a more radical rethinking/restructuring than the neat experiment Tobias Pape posted recently is needed if one wanted to take full advantage of a Fuel monticello format :) Eliminate the compilation step, and mcz loading would probably be FAST!
Yes. With the experiment of Tobias, the #writeSnapshot: and #loadDefinitions took only 1% of the whole process.....so... Now, for that radical change you mention, we have started to think a little bit: 1) we need to store the source code anyway. This is because we would need to recompile all CompiledMethod (because of instance variable accessing bytecodes) when the superclass of a class which is in the image changes it shape (hence, instance variables moved their index). For this purpose, we thought we can easily store the source code in the method trailer and serialize that. The thing is at materialization. What do we once we have materialized? we keep the sources in teh CM or we put them in the .changes file and we put back a .changes pointer in the trailer??? should a load of a package always store in .changes ? 2) Since we materialize classes without class builder, and this monster does everything together, we need to analyze what has to be done exactly when materializing a package for example: notify class creation/modification, validations (what happens if the class already exist, etc.), etc, etc. This is the most difficult part for us, since understanding what the classbuilder does is complicated.... Our progress is in FuelPackageLoader... Cheers -- Mariano http://marianopeck.wordpress.com
Yes. With the experiment of Tobias, the #writeSnapshot: and #loadDefinitions took only 1% of the whole process.....soâ¦
the problem is - classbuilder - logging - ⦠so we should go step by step
Now, for that radical change you mention, we have started to think a little bit:
1) we need to store the source code anyway. This is because we would need to recompile all CompiledMethod (because of instance variable accessing bytecodes) when the superclass of a class which is in the image changes it shape (hence, instance variables moved their index). For this purpose, we thought we can easily store the source code in the method trailer and serialize that. The thing is at materialization. What do we once we have materialized? we keep the sources in teh CM or we put them in the .changes file and we put back a .changes pointer in the trailer??? should a load of a package always store in .changes ?
2) Since we materialize classes without class builder, and this monster does everything together, we need to analyze what has to be done exactly when materializing a package for example: notify class creation/modification, validations (what happens if the class already exist, etc.), etc, etc. This is the most difficult part for us, since understanding what the classbuilder does is complicated....
Our progress is in FuelPackageLoader...
Cheers
-- Mariano http://marianopeck.wordpress.com
On Tue, 6 Dec 2011, Henrik Sperre Johansen wrote:
On 06.12.2011 16:35, Mariano Martinez Peck wrote:
On Tue, Dec 6, 2011 at 4:30 PM, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no <mailto:henrik.s.johansen@veloxit.no>> wrote:
On 06.12.2011 15:55, Mariano Martinez Peck wrote:
On Mon, Dec 5, 2011 at 4:21 PM, Stefan Marr <pharo@stefan-marr.de <mailto:pharo@stefan-marr.de>> wrote:
Hi:
I got here a Monticello package that does not properly load using the Metacello configuration. The image freezes and cmd+. does not pop up a debugger anymore.
Is there a way to force Monticello/Metacello to fall back on the included source instead of using the binary snapshot? (I guess that is going wrong for some reason).
Not very elegant, but you can delete the .bin from the .mcz file, and it should fall back to importing from sources.
The binary snapshot is only for the definitions. The source is always get from the .sources.
I'm not sure what you mean, but the source compiled when you install an MCDefinition, is taken from the .bin if that was the source of it. (its source instvar is a string, not a pointer to source location in .sources file).
It seems I was wrong. All I wanted to say is that the final representation of code (classes, compiled methods, etc) are compiled from source code. The snapshot does not include a "binary represenation of the code". The code is always needed. What I thought is that it was always from sources.st <http://sources.st> , but you say it can be from the bin as well.
Yup!
OT: Which, as you surely know, is also why a more radical rethinking/restructuring than the neat experiment Tobias Pape posted recently is needed if one wanted to take full advantage of a Fuel monticello format :) Eliminate the compilation step, and mcz loading would probably be FAST!
5% faster? :) Levente
Cheers, Henry
OT: Which, as you surely know, is also why a more radical rethinking/restructuring than the neat experiment Tobias Pape posted recently is needed if one wanted to take full advantage of a Fuel monticello format :) Eliminate the compilation step, and mcz loading would probably be FAST!
Now you see why I originally proposed tristan, then martin, then mariano to work on a fast binary serializer :)
Hi: On 06 Dec 2011, at 15:55, Mariano Martinez Peck wrote:
On Mon, Dec 5, 2011 at 4:21 PM, Stefan Marr <pharo@stefan-marr.de> wrote:
I got here a Monticello package that does not properly load using the Metacello configuration. The image freezes and cmd+. does not pop up a debugger anymore.
However, if I extract the source.st contained in the package and file it in manually, it does not freeze the image.
It is likely you broke DataStream, and hence #loadDefinitions or friends is looping.
I debugged the problem a bit. Stepping through the whole loading step. #loadDefinitions is already executed during opening and showing the package in the browser. It needs it, I think, for instance to show the commit message. And that is not the problem. Stepping through MCFileRepositoryInspector>>#load reveals that the image freezes on installing the method additions. (methodAdditions do: [:each | each installMethod].) There are for instance changes in Behavior and TPureBehavior. In general there is a dependency between changes, and a wrong order can easily break things. So, I checked whether the order in methodAdditions actually reflects what is in the change set. And the order is not consistent. So, what ever the approach is to decide on the order in the source.st, it is more stable/reliable in this case than what is in the binary snapshot. Any ideas to make these things consistent? Thanks Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
On 07.12.2011 11:02, Stefan Marr wrote:
Hi:
On 06 Dec 2011, at 15:55, Mariano Martinez Peck wrote:
On Mon, Dec 5, 2011 at 4:21 PM, Stefan Marr<pharo@stefan-marr.de> wrote:
I got here a Monticello package that does not properly load using the Metacello configuration. The image freezes and cmd+. does not pop up a debugger anymore.
However, if I extract the source.st contained in the package and file it in manually, it does not freeze the image. It is likely you broke DataStream, and hence #loadDefinitions or friends is looping. I debugged the problem a bit. Stepping through the whole loading step.
#loadDefinitions is already executed during opening and showing the package in the browser. It needs it, I think, for instance to show the commit message.
And that is not the problem.
Stepping through MCFileRepositoryInspector>>#load reveals that the image freezes on installing the method additions. (methodAdditions do: [:each | each installMethod].)
There are for instance changes in Behavior and TPureBehavior. In general there is a dependency between changes, and a wrong order can easily break things.
So, I checked whether the order in methodAdditions actually reflects what is in the change set. And the order is not consistent.
So, what ever the approach is to decide on the order in the source.st, it is more stable/reliable in this case than what is in the binary snapshot.
Any ideas to make these things consistent?
Thanks Stefan
I can't claim to know exactly, but I seem to remember it does batches of new, then changed, then removed methods. Other than that, I think they happen "seemingly" at random. There are many sorts of internal dependencies, so other than the old workaround of making multiple .mcz's without internal dependencies, one kind of needs an insight into what exactly was at fault here to see if anything could be improved of the loading order in the general case, which does not simultaneously break something else... Cheers, Henry
Yes Yesterday we looked at basicLoad implementation and compared with the squeak one. But igor' conclusions was that this is the same but we should relook at it.
I can't claim to know exactly, but I seem to remember it does batches of new, then changed, then removed methods. Other than that, I think they happen "seemingly" at random. There are many sorts of internal dependencies, so other than the old workaround of making multiple .mcz's without internal dependencies, one kind of needs an insight into what exactly was at fault here to see if anything could be improved of the loading order in the general case, which does not simultaneously break something else...
Cheers, Henry
On 07 Dec 2011, at 12:34, Henrik Sperre Johansen wrote:
I can't claim to know exactly, but I seem to remember it does batches of new, then changed, then removed methods. Other than that, I think they happen "seemingly" at random. There are many sorts of internal dependencies, so other than the old workaround of making multiple .mcz's without internal dependencies, one kind of needs an insight into what exactly was at fault here to see if anything could be improved of the loading order in the general case, which does not simultaneously break something else...
For me the question is: is the fileout/source.st order in which changes are applied more stable/less prone to such errors? And in my example, it is. But I haven't looked into the implementation of either. Anyway, it might not be 'stable', but I can remove a single override on Symbol>>#precedence and now it loads. Would there be any way other than splitting up packages? Because from a semantic perspective, it just does not make sense to split these things up. The typical separation of core system and overrides does also not help, since here, the dependency is between the overrides. Thanks Stefan
Cheers, Henry
-- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
participants (5)
-
Henrik Sperre Johansen -
Levente Uzonyi -
Mariano Martinez Peck -
Stefan Marr -
Stéphane Ducasse