Re: [Pharo-project] Fuel - class loading issue when superclass changed inst vars
On Mar 6, 2012, at 12:03 PM, Henrik Johansen wrote:
Parcels - Check a hash based on class layout vs equivalent hash for the stored method's class (stored with the method). If different: 1) check for existence of a backwards-compatability reader method, hand the old instance to it, and expect the old instance to be #become'd into something current, 2) raise an error if no such method exists.
Now, the main problem with this scheme is it's left as an exercise to the user to come up with a procedure to ensure said backwards-compat reader method stays up to date as additional changes are made.
Is it a big problem? Depends on your tests, and programmer diligence at any given day.
Added inst vars are not a big deal, as if you forget them, there will usually be a nil #DNU somewhere down the line. (or you use lazy initialization, and everything works as expected even for existing instances)
Removals/reorderings are a bigger issue, as the detection of failure (inst vars in wrong slots) are often far removed from the source of problems (forgetting to update the reader method), If seldomly used, it may even go unnoticed until the instance is next saved, at which time you're in real trouble (especially if saved alongside newly created ones, in which case there is no consistency).
In general I think it's an ok scheme, but would like to see a solution more resilient to user-error. How to achieve that is an interesting topic, which I haven't yet found time to think through as thoroughly as I had wished/intended some months ago. :(
Errr, BOSS, not Parcels. My bad. Cheers, Henry
On Tue, Mar 6, 2012 at 3:05 AM, Henrik Johansen < henrik.s.johansen@veloxit.no> wrote:
On Mar 6, 2012, at 12:03 PM, Henrik Johansen wrote:
Parcels - Check a hash based on class layout vs equivalent hash for the stored method's class (stored with the method). If different: 1) check for existence of a backwards-compatability reader method, hand the old instance to it, and expect the old instance to be #become'd into something current, 2) raise an error if no such method exists.
Now, the main problem with this scheme is it's left as an exercise to the user to come up with a procedure to ensure said backwards-compat reader method stays up to date as additional changes are made.
Is it a big problem? Depends on your tests, and programmer diligence at any given day.
Added inst vars are not a big deal, as if you forget them, there will usually be a nil #DNU somewhere down the line. (or you use lazy initialization, and everything works as expected even for existing instances)
Removals/reorderings are a bigger issue, as the detection of failure (inst vars in wrong slots) are often far removed from the source of problems (forgetting to update the reader method), If seldomly used, it may even go unnoticed until the instance is next saved, at which time you're in real trouble (especially if saved alongside newly created ones, in which case there is no consistency).
In general I think it's an ok scheme, but would like to see a solution more resilient to user-error. How to achieve that is an interesting topic, which I haven't yet found time to think through as thoroughly as I had wished/intended some months ago. :(
Errr, BOSS, not Parcels.
Right. What Parcels do is shape-change instances *and* rescan methods to fix up inst var offsets. When a parcel is saved the "signature" of classes of instances there-in are saved so that the parcel contains all the inst var names of the class and its superclass. If on load the superclass chain has a different set of inst vars then lost inst vars are omitted and added inst vars are nil in the materialized instances. If a Parcel contains a full class (not just instances) that has methods, and the superclass chains inst vars have changed then these methods are "rescanned" (disassembled, and reassembled, not using the compiler) and inst var offsets are fixed up, missing inst vars getting changed into undeclared variable references (I think; at least this is what *should* happen; its what happens when one removes an inst var that is still referenced from methods). What's missing in the parcel scheme is any hook to allow the user to process shape-changing instances with the values of lost inst vars in hand. Presumably there will be cases when those values are essential to any schema migration. Personally I would want to decouple schema migration and add it as a post-processing step, which would imply that the materializer would offer a service (materialize in a special mode) where it constructed a dictionary from materialized and shape-changed instance to in parcel state (e.g. an Array of inst var values as they occurred in the parcel). Then the materialized instance could be migrated after the fact, with the default behaviour being analogous to what the system does now on class redefinition (values of deleted inst vars are lost, aded inst vars are nil). This reminds me of a bug with the ClassBuilder/RefactoringBrowser combination. If one refactors pushing an inst var up to a superclass or down to subclasses, the values of te inst var in instances are lost because the class change is in fact done as two changes, a deletion followed by an addition. This again could be fixed in a wrapper, building a dictionary from instance to values, performing the set of class changes, and then restoring the inst var state from the dictionary.
My bad.
Cheers, Henry
-- best, Eliot
Hi! Mariano, yes, I forgot to say that B is subclass of A. Henry, thanks a lot for sharing your knowledge about BOSS. Following my example, the "backwards-compatability reader method" has to be defined in A, right? Or do you specify it as a parameter to the class loader? I would like to have automatic tolerance to simple layout changes like this. Eliot, thanks for the correction. Rescan the methods seems to be a nice solution, and has not dependency with compiler. The migration service you describe makes me think, because we have recently added migration support to Fuel; it works like: aMaterializer migrateClassNamed: #OldPoint toClass: NewPoint variables: {('x' -> 'posX'). ('y' -> 'posY')}. On one hand, in simple cases it should be more efficient than creating a dictionary for each instance and then become them. On the other hand, in more complex cases it's not enough and the dictionary could be a more uniform solution. For example, if NewPoint has just a variable named "coordinates", which points to an array with x and y as elements. MartÃn On Tue, Mar 6, 2012 at 3:27 PM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
On Tue, Mar 6, 2012 at 3:05 AM, Henrik Johansen < henrik.s.johansen@veloxit.no> wrote:
On Mar 6, 2012, at 12:03 PM, Henrik Johansen wrote:
Parcels - Check a hash based on class layout vs equivalent hash for the stored method's class (stored with the method). If different: 1) check for existence of a backwards-compatability reader method, hand the old instance to it, and expect the old instance to be #become'd into something current, 2) raise an error if no such method exists.
Now, the main problem with this scheme is it's left as an exercise to the user to come up with a procedure to ensure said backwards-compat reader method stays up to date as additional changes are made.
Is it a big problem? Depends on your tests, and programmer diligence at any given day.
Added inst vars are not a big deal, as if you forget them, there will usually be a nil #DNU somewhere down the line. (or you use lazy initialization, and everything works as expected even for existing instances)
Removals/reorderings are a bigger issue, as the detection of failure (inst vars in wrong slots) are often far removed from the source of problems (forgetting to update the reader method), If seldomly used, it may even go unnoticed until the instance is next saved, at which time you're in real trouble (especially if saved alongside newly created ones, in which case there is no consistency).
In general I think it's an ok scheme, but would like to see a solution more resilient to user-error. How to achieve that is an interesting topic, which I haven't yet found time to think through as thoroughly as I had wished/intended some months ago. :(
Errr, BOSS, not Parcels.
Right. What Parcels do is shape-change instances *and* rescan methods to fix up inst var offsets. When a parcel is saved the "signature" of classes of instances there-in are saved so that the parcel contains all the inst var names of the class and its superclass. If on load the superclass chain has a different set of inst vars then lost inst vars are omitted and added inst vars are nil in the materialized instances. If a Parcel contains a full class (not just instances) that has methods, and the superclass chains inst vars have changed then these methods are "rescanned" (disassembled, and reassembled, not using the compiler) and inst var offsets are fixed up, missing inst vars getting changed into undeclared variable references (I think; at least this is what *should* happen; its what happens when one removes an inst var that is still referenced from methods).
What's missing in the parcel scheme is any hook to allow the user to process shape-changing instances with the values of lost inst vars in hand. Presumably there will be cases when those values are essential to any schema migration. Personally I would want to decouple schema migration and add it as a post-processing step, which would imply that the materializer would offer a service (materialize in a special mode) where it constructed a dictionary from materialized and shape-changed instance to in parcel state (e.g. an Array of inst var values as they occurred in the parcel). Then the materialized instance could be migrated after the fact, with the default behaviour being analogous to what the system does now on class redefinition (values of deleted inst vars are lost, aded inst vars are nil).
This reminds me of a bug with the ClassBuilder/RefactoringBrowser combination. If one refactors pushing an inst var up to a superclass or down to subclasses, the values of te inst var in instances are lost because the class change is in fact done as two changes, a deletion followed by an addition. This again could be fixed in a wrapper, building a dictionary from instance to values, performing the set of class changes, and then restoring the inst var state from the dictionary.
My bad.
Cheers, Henry
-- best, Eliot
On 6 March 2012 18:27, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Tue, Mar 6, 2012 at 3:05 AM, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
On Mar 6, 2012, at 12:03 PM, Henrik Johansen wrote:
Parcels - Check a hash based on class layout vs equivalent hash for the stored method's class (stored with the method). If different: 1) check for existence of a backwards-compatability reader method, hand the old instance to it, and expect the old instance to be #become'd into something current, 2) raise an error if no such method exists.
Now, the main problem with this scheme is it's left as an exercise to the user to come up with a procedure to ensure said backwards-compat reader method stays up to date as additional changes are made.
Is it a big problem? Depends on your tests, and programmer diligence at any given day.
Added inst vars are not a big deal, as if you forget them, there will usually be a nil #DNU somewhere down the line. (or you use lazy initialization, and everything works as expected even for existing instances)
Removals/reorderings are a bigger issue, as the detection of failure (inst vars in wrong slots) are often far removed from the source of problems (forgetting to update the reader method), If seldomly used, it may even go unnoticed until the instance is next saved, at which time you're in real trouble (especially if saved alongside newly created ones, in which case there is no consistency).
In general I think it's an ok scheme, but would like to see a solution more resilient to user-error. How to achieve that is an interesting topic, which I haven't yet found time to think through as thoroughly as I had wished/intended some months ago. :(
Errr, BOSS, not Parcels.
Right. What Parcels do is shape-change instances *and* rescan methods to fix up inst var offsets.  When a parcel is saved the "signature" of classes of instances there-in are saved so that the parcel contains all the inst var names of the class and its superclass.  If on load the superclass chain has a different set of inst vars then lost inst vars are omitted and added inst vars are nil in the materialized instances.  If a Parcel contains a full class (not just instances) that has methods, and the superclass chains inst vars have changed  then these methods are "rescanned" (disassembled, and reassembled, not using the compiler) and inst var offsets are fixed up, missing inst vars getting changed into undeclared variable references (I think; at least this is what *should* happen; its what happens when one removes an inst var that is still referenced from methods).
What's missing in the parcel scheme is any hook to allow the user to process shape-changing instances with the values of lost inst vars in hand. Â Presumably there will be cases when those values are essential to any schema migration. Â Personally I would want to decouple schema migration and add it as a post-processing step, which would imply that the materializer would offer a service (materialize in a special mode) where it constructed a dictionary from materialized and shape-changed instance to in parcel state (e.g. an Array of inst var values as they occurred in the parcel). Â Then the materialized instance could be migrated after the fact, with the default behaviour being analogous to what the system does now on class redefinition (values of deleted inst vars are lost, aded inst vars are nil).
This sounds a bit like CLOS' UPDATE-INSTANCE-FOR-REDEFINED-CLASS [1], which is a generic function (for our purposes just think "something a bit like a method") that you define and is invoked by the machinery involved in changing/redefining classes. A colleague told me about this part of CLOS the other day when discussing class-based OO: it's the sort of thing we need in at least the Browser, so one can change the shape of existing instances and ensure the new class invariants are maintained. (For instance, for "purely functional" objects, you really don't want to use reflection to initialise new instvars from outside the object.) So you'd change the class definition, supply a way of handling the schema migration, and then the image would change the class definition and run the migration. frank [1] http://clhs.lisp.se/Body/f_upda_1.htm
This reminds me of a bug with the ClassBuilder/RefactoringBrowser combination. Â If one refactors pushing an inst var up to a superclass or down to subclasses, the values of te inst var in instances are lost because the class change is in fact done as two changes, a deletion followed by an addition. Â This again could be fixed in a wrapper, building a dictionary from instance to values, performing the set of class changes, and then restoring the inst var state from the dictionary.
My bad.
Cheers, Henry
-- best, Eliot
Hi Frank, On Tue, Mar 6, 2012 at 1:49 PM, Frank Shearar <frank.shearar@gmail.com>wrote:
On 6 March 2012 18:27, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Tue, Mar 6, 2012 at 3:05 AM, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
On Mar 6, 2012, at 12:03 PM, Henrik Johansen wrote:
Parcels - Check a hash based on class layout vs equivalent hash for the stored method's class (stored with the method). If different: 1) check for existence of a backwards-compatability reader method,
hand
the old instance to it, and expect the old instance to be #become'd into something current, 2) raise an error if no such method exists.
Now, the main problem with this scheme is it's left as an exercise to the user to come up with a procedure to ensure said backwards-compat reader method stays up to date as additional changes are made.
Is it a big problem? Depends on your tests, and programmer diligence at any given day.
Added inst vars are not a big deal, as if you forget them, there will usually be a nil #DNU somewhere down the line. (or you use lazy initialization, and everything works as expected even for existing instances)
Removals/reorderings are a bigger issue, as the detection of failure (inst vars in wrong slots) are often far removed from the source of problems (forgetting to update the reader method), If seldomly used, it may even go unnoticed until the instance is next saved, at which time you're in real trouble (especially if saved alongside newly created ones, in which case there is no consistency).
In general I think it's an ok scheme, but would like to see a solution more resilient to user-error. How to achieve that is an interesting topic, which I haven't yet found time to think through as thoroughly as I had wished/intended some months ago. :(
Errr, BOSS, not Parcels.
Right. What Parcels do is shape-change instances *and* rescan methods to fix up inst var offsets. When a parcel is saved the "signature" of classes of instances there-in are saved so that the parcel contains all the inst var names of the class and its superclass. If on load the superclass chain has a different set of inst vars then lost inst vars are omitted and added inst vars are nil in the materialized instances. If a Parcel contains a full class (not just instances) that has methods, and the superclass chains inst vars have changed then these methods are "rescanned" (disassembled, and reassembled, not using the compiler) and inst var offsets are fixed up, missing inst vars getting changed into undeclared variable references (I think; at least this is what *should* happen; its what happens when one removes an inst var that is still referenced from methods).
What's missing in the parcel scheme is any hook to allow the user to process shape-changing instances with the values of lost inst vars in hand. Presumably there will be cases when those values are essential to any schema migration. Personally I would want to decouple schema migration and add it as a post-processing step, which would imply that the materializer would offer a service (materialize in a special mode) where it constructed a dictionary from materialized and shape-changed instance to in parcel state (e.g. an Array of inst var values as they occurred in the parcel). Then the materialized instance could be migrated after the fact, with the default behaviour being analogous to what the system does now on class redefinition (values of deleted inst vars are lost, aded inst vars are nil).
This sounds a bit like CLOS' UPDATE-INSTANCE-FOR-REDEFINED-CLASS [1], which is a generic function (for our purposes just think "something a bit like a method") that you define and is invoked by the machinery involved in changing/redefining classes.
A colleague told me about this part of CLOS the other day when discussing class-based OO: it's the sort of thing we need in at least the Browser, so one can change the shape of existing instances and ensure the new class invariants are maintained.
But Smalltalk has done without this for 40 years. It has supported existing instance redefinition but no-one has felt the need to provide a scheme for redefinition in the IDE. In any case it is relatively trivial to script it, e.g.: | instanceState | instanceState := IdentityDictionary new. myClass allInstancesDo: [:i| instanceState at: i put: ((1 to: myClass instSize) collect: [:ivi| i instVarAt: ivi])]. myClass superclass subclass: myClass name instanceVariableNames: myClass instVarNamesString, ' theExtraInstVar' [...]. myClass compile: sourceForupdateToNewFormatFrom. myClass allInstancesDo: [:i| i updateToNewFormatFrom: (instanceState at: i] So if and when you need it you can roll your own (and I've never needed it; but I have wanted the refactoring browser to preserve inst vars on push up/push dwn). However, collections of persistent objects are a different kettle of fish. It is much more likely to need instance migration on load, especially for library objects that the user is merely using and whose implementation has evolved over time (e.g. UI elements such as morphs).
(For instance, for "purely functional" objects, you really don't want to use reflection to initialise new instvars from outside the object.)
But the class builder uses an unprotected interface (instVarAt: and instVarAt:put:). So this turns out not to be an issue.
So you'd change the class definition, supply a way of handling the schema migration, and then the image would change the class definition and run the migration.
frank
[1] http://clhs.lisp.se/Body/f_upda_1.htm
This reminds me of a bug with the ClassBuilder/RefactoringBrowser combination. If one refactors pushing an inst var up to a superclass or down to subclasses, the values of te inst var in instances are lost because the class change is in fact done as two changes, a deletion followed by an addition. This again could be fixed in a wrapper, building a dictionary from instance to values, performing the set of class changes, and then restoring the inst var state from the dictionary.
My bad.
Cheers, Henry
-- best, Eliot
-- best, Eliot
On 6 March 2012 23:22, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Frank,
On Tue, Mar 6, 2012 at 1:49 PM, Frank Shearar <frank.shearar@gmail.com> wrote:
On 6 March 2012 18:27, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Tue, Mar 6, 2012 at 3:05 AM, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
On Mar 6, 2012, at 12:03 PM, Henrik Johansen wrote:
Parcels - Check a hash based on class layout vs equivalent hash for the stored method's class (stored with the method). If different: 1) check for existence of a backwards-compatability reader method, hand the old instance to it, and expect the old instance to be #become'd into something current, 2) raise an error if no such method exists.
Now, the main problem with this scheme is it's left as an exercise to the user to come up with a procedure to ensure said backwards-compat reader method stays up to date as additional changes are made.
Is it a big problem? Depends on your tests, and programmer diligence at any given day.
Added inst vars are not a big deal, as if you forget them, there will usually be a nil #DNU somewhere down the line. (or you use lazy initialization, and everything works as expected even for existing instances)
Removals/reorderings are a bigger issue, as the detection of failure (inst vars in wrong slots) are often far removed from the source of problems (forgetting to update the reader method), If seldomly used, it may even go unnoticed until the instance is next saved, at which time you're in real trouble (especially if saved alongside newly created ones, in which case there is no consistency).
In general I think it's an ok scheme, but would like to see a solution more resilient to user-error. How to achieve that is an interesting topic, which I haven't yet found time to think through as thoroughly as I had wished/intended some months ago. :(
Errr, BOSS, not Parcels.
Right. What Parcels do is shape-change instances *and* rescan methods to fix up inst var offsets.  When a parcel is saved the "signature" of classes of instances there-in are saved so that the parcel contains all the inst var names of the class and its superclass.  If on load the superclass chain has a different set of inst vars then lost inst vars are omitted and added inst vars are nil in the materialized instances.  If a Parcel contains a full class (not just instances) that has methods, and the superclass chains inst vars have changed  then these methods are "rescanned" (disassembled, and reassembled, not using the compiler) and inst var offsets are fixed up, missing inst vars getting changed into undeclared variable references (I think; at least this is what *should* happen; its what happens when one removes an inst var that is still referenced from methods).
What's missing in the parcel scheme is any hook to allow the user to process shape-changing instances with the values of lost inst vars in hand. Â Presumably there will be cases when those values are essential to any schema migration. Â Personally I would want to decouple schema migration and add it as a post-processing step, which would imply that the materializer would offer a service (materialize in a special mode) where it constructed a dictionary from materialized and shape-changed instance to in parcel state (e.g. an Array of inst var values as they occurred in the parcel). Â Then the materialized instance could be migrated after the fact, with the default behaviour being analogous to what the system does now on class redefinition (values of deleted inst vars are lost, aded inst vars are nil).
This sounds a bit like CLOS' Â UPDATE-INSTANCE-FOR-REDEFINED-CLASS [1], which is a generic function (for our purposes just think "something a bit like a method") that you define and is invoked by the machinery involved in changing/redefining classes.
A colleague told me about this part of CLOS the other day when discussing class-based OO: it's the sort of thing we need in at least the Browser, so one can change the shape of existing instances and ensure the new class invariants are maintained.
But Smalltalk has done without this for 40 years. Â It has supported existing instance redefinition but no-one has felt the need to provide a scheme for redefinition in the IDE. Â In any case it is relatively trivial to script it, e.g.:
| instanceState | instanceState := IdentityDictionary new. myClass allInstancesDo: [:i| instanceState at: i put: ((1 to: myClass instSize) collect: [:ivi| i instVarAt: ivi])]. myClass superclass subclass: myClass name instanceVariableNames: myClass instVarNamesString, ' theExtraInstVar' [...]. myClass compile: sourceForupdateToNewFormatFrom. myClass allInstancesDo: [:i| i updateToNewFormatFrom: (instanceState at: i]
So if and when you need it you can roll your own (and I've never needed it; but I have wanted the refactoring browser to preserve inst vars on push up/push dwn).
However, collections of persistent objects are a different kettle of fish. Â It is much more likely to need instance migration on load, especially for library objects that the user is merely using and whose implementation has evolved over time (e.g. UI elements such as morphs).
Well, as long as we're staying in same image, i agree there's little , if none, needed to support sophisticated/automated object migration schemes. And i understand why smalltalk lived well without it for 40 years. But if we start talking about distributed systems, where objects need to be persisted/migrated from system to system, this argument may not hold. Anyways, i think there is no simple (silver bullet) solution to automate migration process, because it involves humans (as developers), who making the changes and system simply cannot predict all side effects of it. Having a system which is clever enough to do an auto-migration correctly is the same as having a system clever enough to do auto-refactoring. These two things are closely related, and i hardly believe we will have self-improving systems in nearest future.
(For instance, for "purely functional" objects, you really don't want to use reflection to initialise new instvars from outside the object.)
But the class builder uses an unprotected interface (instVarAt: and instVarAt:put:). Â So this turns out not to be an issue.
-- Best regards, Igor Stasenko.
On 6 March 2012 22:22, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Frank,
On Tue, Mar 6, 2012 at 1:49 PM, Frank Shearar <frank.shearar@gmail.com> wrote:
On 6 March 2012 18:27, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Tue, Mar 6, 2012 at 3:05 AM, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
On Mar 6, 2012, at 12:03 PM, Henrik Johansen wrote:
Parcels - Check a hash based on class layout vs equivalent hash for the stored method's class (stored with the method). If different: 1) check for existence of a backwards-compatability reader method, hand the old instance to it, and expect the old instance to be #become'd into something current, 2) raise an error if no such method exists.
Now, the main problem with this scheme is it's left as an exercise to the user to come up with a procedure to ensure said backwards-compat reader method stays up to date as additional changes are made.
Is it a big problem? Depends on your tests, and programmer diligence at any given day.
Added inst vars are not a big deal, as if you forget them, there will usually be a nil #DNU somewhere down the line. (or you use lazy initialization, and everything works as expected even for existing instances)
Removals/reorderings are a bigger issue, as the detection of failure (inst vars in wrong slots) are often far removed from the source of problems (forgetting to update the reader method), If seldomly used, it may even go unnoticed until the instance is next saved, at which time you're in real trouble (especially if saved alongside newly created ones, in which case there is no consistency).
In general I think it's an ok scheme, but would like to see a solution more resilient to user-error. How to achieve that is an interesting topic, which I haven't yet found time to think through as thoroughly as I had wished/intended some months ago. :(
Errr, BOSS, not Parcels.
Right. What Parcels do is shape-change instances *and* rescan methods to fix up inst var offsets.  When a parcel is saved the "signature" of classes of instances there-in are saved so that the parcel contains all the inst var names of the class and its superclass.  If on load the superclass chain has a different set of inst vars then lost inst vars are omitted and added inst vars are nil in the materialized instances.  If a Parcel contains a full class (not just instances) that has methods, and the superclass chains inst vars have changed  then these methods are "rescanned" (disassembled, and reassembled, not using the compiler) and inst var offsets are fixed up, missing inst vars getting changed into undeclared variable references (I think; at least this is what *should* happen; its what happens when one removes an inst var that is still referenced from methods).
What's missing in the parcel scheme is any hook to allow the user to process shape-changing instances with the values of lost inst vars in hand. Â Presumably there will be cases when those values are essential to any schema migration. Â Personally I would want to decouple schema migration and add it as a post-processing step, which would imply that the materializer would offer a service (materialize in a special mode) where it constructed a dictionary from materialized and shape-changed instance to in parcel state (e.g. an Array of inst var values as they occurred in the parcel). Â Then the materialized instance could be migrated after the fact, with the default behaviour being analogous to what the system does now on class redefinition (values of deleted inst vars are lost, aded inst vars are nil).
This sounds a bit like CLOS' Â UPDATE-INSTANCE-FOR-REDEFINED-CLASS [1], which is a generic function (for our purposes just think "something a bit like a method") that you define and is invoked by the machinery involved in changing/redefining classes.
A colleague told me about this part of CLOS the other day when discussing class-based OO: it's the sort of thing we need in at least the Browser, so one can change the shape of existing instances and ensure the new class invariants are maintained.
But Smalltalk has done without this for 40 years. Â It has supported existing instance redefinition but no-one has felt the need to provide a scheme for redefinition in the IDE. Â In any case it is relatively trivial to script it, e.g.:
| instanceState | instanceState := IdentityDictionary new. myClass allInstancesDo: [:i| instanceState at: i put: ((1 to: myClass instSize) collect: [:ivi| i instVarAt: ivi])]. myClass superclass subclass: myClass name instanceVariableNames: myClass instVarNamesString, ' theExtraInstVar' [...]. myClass compile: sourceForupdateToNewFormatFrom. myClass allInstancesDo: [:i| i updateToNewFormatFrom: (instanceState at: i]
So if and when you need it you can roll your own (and I've never needed it; but I have wanted the refactoring browser to preserve inst vars on push up/push dwn).
Indeed, it is trivial. Using #allInstancesDo: is exactly what I used when my colleague and I were discussing the issue. The only difference is that _I_ did the migration, not my tools.
However, collections of persistent objects are a different kettle of fish. Â It is much more likely to need instance migration on load, especially for library objects that the user is merely using and whose implementation has evolved over time (e.g. UI elements such as morphs).
(For instance, for "purely functional" objects, you really don't want to use reflection to initialise new instvars from outside the object.)
But the class builder uses an unprotected interface (instVarAt: and instVarAt:put:). Â So this turns out not to be an issue.
That's exactly to what I was referring: I lump #instVarAt: in the reflection bucket. I realise its _utility_, but it's strong medicine. Which is pretty much why you're pushing mirrors, right? So that some things - the class builder, as an example - can make use of such a (powerful and dangerous) tool while some random library can't. frank
So you'd change the class definition, supply a way of handling the schema migration, and then the image would change the class definition and run the migration.
frank
[1] http://clhs.lisp.se/Body/f_upda_1.htm
This reminds me of a bug with the ClassBuilder/RefactoringBrowser combination. Â If one refactors pushing an inst var up to a superclass or down to subclasses, the values of te inst var in instances are lost because the class change is in fact done as two changes, a deletion followed by an addition. Â This again could be fixed in a wrapper, building a dictionary from instance to values, performing the set of class changes, and then restoring the inst var state from the dictionary.
My bad.
Cheers, Henry
-- best, Eliot
-- best, Eliot
I was in another live a CLOS mop expert so I know this hook. Now we should really build a new class builder (and with first class slot) and a nice protocol so that we can think about such issues. This is on my roadmap, I'm just too busy and slow. Stef
On Wed, Mar 7, 2012 at 2:10 AM, Frank Shearar <frank.shearar@gmail.com>wrote:
On 6 March 2012 22:22, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Frank,
On Tue, Mar 6, 2012 at 1:49 PM, Frank Shearar <frank.shearar@gmail.com> wrote:
On 6 March 2012 18:27, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Tue, Mar 6, 2012 at 3:05 AM, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
On Mar 6, 2012, at 12:03 PM, Henrik Johansen wrote:
Parcels - Check a hash based on class layout vs equivalent hash for the stored method's class (stored with the method). If different: 1) check for existence of a backwards-compatability reader method, hand the old instance to it, and expect the old instance to be #become'd into something current, 2) raise an error if no such method exists.
Now, the main problem with this scheme is it's left as an exercise
to
the user to come up with a procedure to ensure said backwards-compat reader method stays up to date as additional changes are made.
Is it a big problem? Depends on your tests, and programmer diligence at any given day.
Added inst vars are not a big deal, as if you forget them, there will usually be a nil #DNU somewhere down the line. (or you use lazy initialization, and everything works as expected even for existing instances)
Removals/reorderings are a bigger issue, as the detection of failure (inst vars in wrong slots) are often far removed from the source of problems (forgetting to update the reader method), If seldomly used, it may even go unnoticed until the instance is next saved, at which time you're in real trouble (especially if saved alongside newly created ones, in which case there is no consistency).
In general I think it's an ok scheme, but would like to see a solution more resilient to user-error. How to achieve that is an interesting topic, which I haven't yet found time to think through as thoroughly as I had wished/intended some months ago. :(
Errr, BOSS, not Parcels.
Right. What Parcels do is shape-change instances *and* rescan methods to fix up inst var offsets. When a parcel is saved the "signature" of classes of instances there-in are saved so that the parcel contains all the inst var names of the class and its superclass. If on load the superclass chain has a different set of inst vars then lost inst vars are omitted and added inst vars are nil in the materialized instances. If a Parcel contains a full class (not just instances) that has methods, and the superclass chains inst vars have changed then these methods are "rescanned" (disassembled, and reassembled, not using the compiler) and inst var offsets are fixed up, missing inst vars getting changed into undeclared variable references (I think; at least this is what *should* happen; its what happens when one removes an inst var that is still referenced from methods).
What's missing in the parcel scheme is any hook to allow the user to process shape-changing instances with the values of lost inst vars in hand. Presumably there will be cases when those values are essential to any schema migration. Personally I would want to decouple schema migration and add it as a post-processing step, which would imply that the materializer would offer a service (materialize in a special mode) where it constructed a dictionary from materialized and shape-changed instance to in parcel state (e.g. an Array of inst var values as they occurred in the parcel). Then the materialized instance could be migrated after the fact, with the default behaviour being analogous to what the system does now on class redefinition (values of deleted inst vars are lost, aded inst vars are nil).
This sounds a bit like CLOS' UPDATE-INSTANCE-FOR-REDEFINED-CLASS [1], which is a generic function (for our purposes just think "something a bit like a method") that you define and is invoked by the machinery involved in changing/redefining classes.
A colleague told me about this part of CLOS the other day when discussing class-based OO: it's the sort of thing we need in at least the Browser, so one can change the shape of existing instances and ensure the new class invariants are maintained.
But Smalltalk has done without this for 40 years. It has supported existing instance redefinition but no-one has felt the need to provide a scheme for redefinition in the IDE. In any case it is relatively trivial to script it, e.g.:
| instanceState | instanceState := IdentityDictionary new. myClass allInstancesDo: [:i| instanceState at: i put: ((1 to: myClass instSize) collect: [:ivi| i instVarAt: ivi])]. myClass superclass subclass: myClass name instanceVariableNames: myClass instVarNamesString, ' theExtraInstVar' [...]. myClass compile: sourceForupdateToNewFormatFrom. myClass allInstancesDo: [:i| i updateToNewFormatFrom: (instanceState at: i]
So if and when you need it you can roll your own (and I've never needed it; but I have wanted the refactoring browser to preserve inst vars on push up/push dwn).
Indeed, it is trivial. Using #allInstancesDo: is exactly what I used when my colleague and I were discussing the issue. The only difference is that _I_ did the migration, not my tools.
However, collections of persistent objects are a different kettle of fish. It is much more likely to need instance migration on load, especially for library objects that the user is merely using and whose implementation has evolved over time (e.g. UI elements such as morphs).
(For instance, for "purely functional" objects, you really don't want to use reflection to initialise new instvars from outside the object.)
But the class builder uses an unprotected interface (instVarAt: and instVarAt:put:). So this turns out not to be an issue.
That's exactly to what I was referring: I lump #instVarAt: in the reflection bucket. I realise its _utility_, but it's strong medicine. Which is pretty much why you're pushing mirrors, right? So that some things - the class builder, as an example - can make use of such a (powerful and dangerous) tool while some random library can't.
Certainly one can't have security and unfettered reflection, and hence mirrors are a way of providing reflection safely. But the complications a full mirror framework introduce shouldn't be underestimated. And how mirrors are made available safely is something I don't yet fully understand. In an IDE setting they're pretty much freely available so for me I'm not convinced of the utility. Deployment is another thing altogether. However, albeit a small point, I do think that light-weight mirror methods like object:instVarAt:put: are the interface the ClassBuilder should use, not instVarAt: on instances. The latter breaks with proxies, and the change from one to the other is pretty trivial.
frank
So you'd change the class definition, supply a way of handling the schema migration, and then the image would change the class definition and run the migration.
frank
[1] http://clhs.lisp.se/Body/f_upda_1.htm
This reminds me of a bug with the ClassBuilder/RefactoringBrowser combination. If one refactors pushing an inst var up to a superclass or down to subclasses, the values of te inst var in instances are lost because the class change is in fact done as two changes, a deletion followed by an addition. This again could be fixed in a wrapper, building a dictionary from instance to values, performing the set of class changes, and then restoring the inst var state from the dictionary.
My bad.
Cheers, Henry
-- best, Eliot
-- best, Eliot
-- best, Eliot
participants (6)
-
Eliot Miranda -
Frank Shearar -
Henrik Johansen -
Igor Stasenko -
Martin Dias -
Stéphane Ducasse