- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Fuel materialization on spur

by Mariano Martinez Peck

On Wed, Dec 16, 2015 at 10:17 AM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > OK, thanks for the explanation, good to hear it. > > (But, just nitpicking, it still does not say officially 'you can safely > save your data in Fuel format and rest assured that you will always be able > to read it back'). > > No, I can't say that (at least myself). It does mean it's not possible, it simply means that it will take some time to make it work and so if nobody does it, it won't work. However, note that the efforts seem to have been much smaller than years ago. For example, with very little changes, it's likely we could have manage automatically from Pharo 3.0 until now. > (Also, if there are compatibility problems between Pharo versions > (understandably Pharo's doing), how can there be compatibility with other > Smalltalk implementation, not that I expect any, but again, what is the > official word ?) > There is none compatibility with other Smalltalk versions in the sense that a serialized stream in one st could be materialized in another st. However, note that what we DO support is that Fuel works in every possible version of Pharo and in a wide range of latest squeak images. > > > On 16 Dec 2015, at 14:10, Mariano Martinez Peck <marianopeck(a)gmail.com> > wrote: > > > > Hi guys, > > > > I would like to say a couple of things: > > > > 1) Fuel format: Sven is right in the sense that Fuel has problems of > migrating to another Fuel version. However.... there is some explanation > needed here. Previously (years ago), a given Fuel version was not > compatible with previous versions because we were still changing/improving > what we call Fuel format, that is, basically, how objects are written and > read from the stream. We have stopped with those optimization/changes years > ago. So the "Fuel format" has been stable so far for a couple of years. > > > > 2) Fuel gets in the internals: In Pharo 4.0 Fuel was not compatible > with Fuel from 3.0. I don't remember the exact detail but it was a change > in Pharo's Date (or DateAndTime I cannot remember). Now the version that > should work in 5.0 changes the Characater serialization. What I want to > make clear here is that it's NOT that we are changing the fuel format 1) as > years ago when Fuel as still evolving from this point of view. It's Pharo > changes that make our incompatibility. But that doesn't mean we cannot fix > it. > > > > 3) Yes, compromises for speed: I think that both examples that break > compatibility in 4.0 (Date / DateAndTime change) and in 5.0 with Spur > (Character) is because we target speed. We do have special clusters for > both of those cases (Date and Character), and so, we get into it's > internals for performance reasons. If the internals change, then we break. > This is why a text serializer for example may simply not break, because it > continues using text representation that might not have been affected by > the underlaying internal representation. > > > > We will try to do a Pharo 40 - Pharo 5.0 (or spur actually) > compatibility. > > > > Best, > > > > > > On Wed, Dec 16, 2015 at 9:49 AM, Sven Van Caekenberghe <sven(a)stfx.eu> > wrote: > > > > > On 16 Dec 2015, at 13:46, Denis Kudriashov <dionisiydk(a)gmail.com> > wrote: > > > > > > Hi. > > > > > > 2015-12-16 13:33 GMT+01:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > > > > On 16 Dec 2015, at 13:24, ClÃ©ment Bera <bera.clement(a)gmail.com> > wrote: > > > > > > > > I don't know if SqueakV3 - Spur32 support is needed, however, I > think it is really important to be able to serialize and materialize > objects between 32 bits spur images and 64bits spur images. > > > > > > I don't know. > > > > > > Yes it would be nice, but since Fuel cannot even exchange data between > different version itself (If I understand the situation correctly), why > should that have to work ? > > > > > > It is needed to use fuel as communication format between services > which run on different platforms. For example you can exchange data between > 64bits server and 32bits android phone. > > > > What you want might not be the same to what it was designed for. > > > > I might be wrong, but that is how I understood it from discussion in the > past when moving data from one fuel version to another. > > > > Let's see what the Fuel guys have to say about this. > > > > > > > > -- > > Mariano > > http://marianopeck.wordpress.com > > > -- Mariano http://marianopeck.wordpress.com

Dec. 16, 2015

Re: [Pharo-dev] Fuel materialization on spur

by Sven Van Caekenberghe

OK, thanks for the explanation, good to hear it. (But, just nitpicking, it still does not say officially 'you can safely save your data in Fuel format and rest assured that you will always be able to read it back'). (Also, if there are compatibility problems between Pharo versions (understandably Pharo's doing), how can there be compatibility with other Smalltalk implementation, not that I expect any, but again, what is the official word ?) > On 16 Dec 2015, at 14:10, Mariano Martinez Peck <marianopeck(a)gmail.com> wrote: > > Hi guys, > > I would like to say a couple of things: > > 1) Fuel format: Sven is right in the sense that Fuel has problems of migrating to another Fuel version. However.... there is some explanation needed here. Previously (years ago), a given Fuel version was not compatible with previous versions because we were still changing/improving what we call Fuel format, that is, basically, how objects are written and read from the stream. We have stopped with those optimization/changes years ago. So the "Fuel format" has been stable so far for a couple of years. > > 2) Fuel gets in the internals: In Pharo 4.0 Fuel was not compatible with Fuel from 3.0. I don't remember the exact detail but it was a change in Pharo's Date (or DateAndTime I cannot remember). Now the version that should work in 5.0 changes the Characater serialization. What I want to make clear here is that it's NOT that we are changing the fuel format 1) as years ago when Fuel as still evolving from this point of view. It's Pharo changes that make our incompatibility. But that doesn't mean we cannot fix it. > > 3) Yes, compromises for speed: I think that both examples that break compatibility in 4.0 (Date / DateAndTime change) and in 5.0 with Spur (Character) is because we target speed. We do have special clusters for both of those cases (Date and Character), and so, we get into it's internals for performance reasons. If the internals change, then we break. This is why a text serializer for example may simply not break, because it continues using text representation that might not have been affected by the underlaying internal representation. > > We will try to do a Pharo 40 - Pharo 5.0 (or spur actually) compatibility. > > Best, > > > On Wed, Dec 16, 2015 at 9:49 AM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > On 16 Dec 2015, at 13:46, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > > > Hi. > > > > 2015-12-16 13:33 GMT+01:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > > > On 16 Dec 2015, at 13:24, ClÃ©ment Bera <bera.clement(a)gmail.com> wrote: > > > > > > I don't know if SqueakV3 - Spur32 support is needed, however, I think it is really important to be able to serialize and materialize objects between 32 bits spur images and 64bits spur images. > > > > I don't know. > > > > Yes it would be nice, but since Fuel cannot even exchange data between different version itself (If I understand the situation correctly), why should that have to work ? > > > > It is needed to use fuel as communication format between services which run on different platforms. For example you can exchange data between 64bits server and 32bits android phone. > > What you want might not be the same to what it was designed for. > > I might be wrong, but that is how I understood it from discussion in the past when moving data from one fuel version to another. > > Let's see what the Fuel guys have to say about this. > > > > -- > Mariano > http://marianopeck.wordpress.com

Dec. 16, 2015

Re: [Pharo-dev] Fuel materialization on spur

by Mariano Martinez Peck

Hi guys, I would like to say a couple of things: 1) *Fuel format*: Sven is right in the sense that Fuel has problems of migrating to another Fuel version. However.... there is some explanation needed here. Previously (years ago), a given Fuel version was not compatible with previous versions because we were still changing/improving what we call Fuel format, that is, basically, how objects are written and read from the stream. We have stopped with those optimization/changes years ago. So the "Fuel format" has been stable so far for a couple of years. 2) *Fuel gets in the internals*: In Pharo 4.0 Fuel was not compatible with Fuel from 3.0. I don't remember the exact detail but it was a change in Pharo's Date (or DateAndTime I cannot remember). Now the version that should work in 5.0 changes the Characater serialization. What I want to make clear here is that it's NOT that we are changing the fuel format 1) as years ago when Fuel as still evolving from this point of view. It's Pharo changes that make our incompatibility. But that doesn't mean we cannot fix it. 3) *Yes, compromises for speed*: I think that both examples that break compatibility in 4.0 (Date / DateAndTime change) and in 5.0 with Spur (Character) is because we target speed. We do have special clusters for both of those cases (Date and Character), and so, we get into it's internals for performance reasons. If the internals change, then we break. This is why a text serializer for example may simply not break, because it continues using text representation that might not have been affected by the underlaying internal representation. We will try to do a Pharo 40 - Pharo 5.0 (or spur actually) compatibility. Best, On Wed, Dec 16, 2015 at 9:49 AM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > On 16 Dec 2015, at 13:46, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > > > Hi. > > > > 2015-12-16 13:33 GMT+01:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > > > On 16 Dec 2015, at 13:24, ClÃ©ment Bera <bera.clement(a)gmail.com> wrote: > > > > > > I don't know if SqueakV3 - Spur32 support is needed, however, I think > it is really important to be able to serialize and materialize objects > between 32 bits spur images and 64bits spur images. > > > > I don't know. > > > > Yes it would be nice, but since Fuel cannot even exchange data between > different version itself (If I understand the situation correctly), why > should that have to work ? > > > > It is needed to use fuel as communication format between services which > run on different platforms. For example you can exchange data between > 64bits server and 32bits android phone. > > What you want might not be the same to what it was designed for. > > I might be wrong, but that is how I understood it from discussion in the > past when moving data from one fuel version to another. > > Let's see what the Fuel guys have to say about this. > -- Mariano http://marianopeck.wordpress.com

Dec. 16, 2015

Re: [Pharo-dev] Fuel materialization on spur

by Denis Kudriashov

2015-12-16 13:49 GMT+01:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > I might be wrong, but that is how I understood it from discussion in the > past when moving data from one fuel version to another. Will see what they answer. But supporting different versions of Fuel is not same as supporting single format for any platform

Dec. 16, 2015

[pharo-project/pharo-core]

by GitHub

Branch: refs/tags/50503 Home: https://github.com/pharo-project/pharo-core

Dec. 16, 2015

[pharo-project/pharo-core] 715494: 50503

by GitHub

Branch: refs/heads/5.0 Home: https://github.com/pharo-project/pharo-core Commit: 71549497531da1216508a2948b7284d6bbcf0922 https://github.com/pharo-project/pharo-core/commit/71549497531da1216508a294… Author: Jenkins Build Server <board(a)pharo-project.org> Date: 2015-12-16 (Wed, 16 Dec 2015) Changed paths: R Kernel.package/Behavior.class/instance/accessing instances and variables/allInstances.st R Kernel.package/Behavior.class/instance/accessing/typeOfClass.st R Kernel.package/Behavior.class/instance/enumerating/allInstancesDo_.st R Kernel.package/Behavior.class/instance/private/indexIfCompact.st R Kernel.package/Behavior.class/instance/testing class hierarchy/kindOfSubclass.st R Kernel.package/Behavior.class/instance/testing/instSpec.st R Kernel.package/Behavior.class/instance/testing/isBits.st R Kernel.package/Behavior.class/instance/testing/isBytes.st R Kernel.package/Behavior.class/instance/testing/isVariable.st R ScriptLoader50.package/ScriptLoader.class/instance/pharo - scripts/script50502.st A ScriptLoader50.package/ScriptLoader.class/instance/pharo - scripts/script50503.st R ScriptLoader50.package/ScriptLoader.class/instance/pharo - updates/update50502.st A ScriptLoader50.package/ScriptLoader.class/instance/pharo - updates/update50503.st M ScriptLoader50.package/ScriptLoader.class/instance/public/commentForCurrentUpdate.st R Traits.package/TraitBehavior.class/instance/testing class hierarchy/kindOfSubclass.st R Traits.package/TraitBehavior.class/instance/testing/instSpec.st R Traits.package/TraitBehavior.class/instance/testing/isBits.st R Traits.package/TraitBehavior.class/instance/testing/isBytes.st R Traits.package/TraitBehavior.class/instance/testing/isVariable.st Log Message: ----------- 50503 17237 restore trait methods (spur migration) https://pharo.fogbugz.com/f/cases/17237 http://files.pharo.org/image/50/50503.zip

Dec. 16, 2015

Re: [Pharo-dev] Fuel materialization on spur

by Sven Van Caekenberghe

> On 16 Dec 2015, at 13:46, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > Hi. > > 2015-12-16 13:33 GMT+01:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > > On 16 Dec 2015, at 13:24, ClÃ©ment Bera <bera.clement(a)gmail.com> wrote: > > > > I don't know if SqueakV3 - Spur32 support is needed, however, I think it is really important to be able to serialize and materialize objects between 32 bits spur images and 64bits spur images. > > I don't know. > > Yes it would be nice, but since Fuel cannot even exchange data between different version itself (If I understand the situation correctly), why should that have to work ? > > It is needed to use fuel as communication format between services which run on different platforms. For example you can exchange data between 64bits server and 32bits android phone. What you want might not be the same to what it was designed for. I might be wrong, but that is how I understood it from discussion in the past when moving data from one fuel version to another. Let's see what the Fuel guys have to say about this.

Dec. 16, 2015

Re: [Pharo-dev] Fuel materialization on spur

by Denis Kudriashov

Hi. 2015-12-16 13:33 GMT+01:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > > On 16 Dec 2015, at 13:24, ClÃ©ment Bera <bera.clement(a)gmail.com> wrote: > > > > I don't know if SqueakV3 - Spur32 support is needed, however, I think it > is really important to be able to serialize and materialize objects between > 32 bits spur images and 64bits spur images. > > I don't know. > > Yes it would be nice, but since Fuel cannot even exchange data between > different version itself (If I understand the situation correctly), why > should that have to work ? It is needed to use fuel as communication format between services which run on different platforms. For example you can exchange data between 64bits server and 32bits android phone.

Dec. 16, 2015

Re: [Pharo-dev] Fuel materialization on spur

by Sven Van Caekenberghe

> On 16 Dec 2015, at 13:24, ClÃ©ment Bera <bera.clement(a)gmail.com> wrote: > > I don't know if SqueakV3 - Spur32 support is needed, however, I think it is really important to be able to serialize and materialize objects between 32 bits spur images and 64bits spur images. I don't know. Yes it would be nice, but since Fuel cannot even exchange data between different version itself (If I understand the situation correctly), why should that have to work ? >From my POV FUEL is only compatible with its own version, a sacrifice made to gain performance. IIUC it is not a stable external format for long lived data with backward/forward compatibility. > 2015-12-16 10:34 GMT+01:00 Esteban Lorenzano <estebanlm(a)gmail.com>: > Hi, > >> On 16 Dec 2015, at 08:21, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: >> >> Hi Esteban, >> >> On Dec 15, 2015, at 5:06 AM, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: >> >>> >>>> On 15 Dec 2015, at 13:18, Norbert Hartl <norbert(a)hartl.name> wrote: >>>> >>>> We have a test build for pharo5 that checks if we need to do stuff for future migrations. Since yesterday it fails, of course. >>>> I have a fueled out object graph stored as byte array in a method. The fuel bytes are serialized using a non-spur image. Now when the spur image reads the bytes I get an exception. Has this something to do with immediate characters? Is there a way to fix that or will it just not be possible to exchange objects between spur and non-spur using fuel? >>> >>> I think is not possible, but I will let fuel maintainers to have the last word :) >> >> >> Of course it's possible; in VW we exchanged objects between 32- & 64-but images using a format much like fuel and in 32-bits all floats are arrays of 32-bit bit objects whereas in 64-bits a range of floats are immediate. > > Oh wellâ¦ of course is possible. What I meant is that I think current implementation of Fuel does not support it, just that :) > but again, Iâm not sure: Martin and Mariano can for sure say something more detailed about :) > (problem is Martin is on vacations after getting his PhD and Mariano didnât followed in close the spur migration) > > cheers, > Esteban > >> >> In spur characters are immediate (Max, "primitive" is not the right term) whereas in v3 they're objects with a single isn't var. So at least fuel needs to be modified to read characters specially. One way to architect this is to ask the class (SmallInteger, Character, Float etc) to materialize, and then the class can decide how to represent its instances. The question is whether characters in spur should serialize themselves using the same wire format as v3 or whether fuel should modify its wire format to serialize immediate characters specially and have v3 materialize characters as serialized by spur. The issue is whether it's important to preserve object identity of characters with code >= 256 or not. >> >>> >>> Esteban >>> >>>> >>>> thanks, >>>> >>>> Norbert >>>> >>>> >>>> primitive #value: in Character class failed >>>> Stacktrace >>>> >>>> Character class(Object)>>primitiveFailed: >>>> Character class(Object)>>primitiveFailed >>>> Character class>>value: >>>> Character class>>materializeFrom: >>>> FLHookPrimitiveCluster>>materializeInstanceWith: >>>> FLHookPrimitiveCluster(FLIteratingCluster)>>materializeInstancesStepWith: >>>> FLMaterialization>>clusterInstancesStep >>>> [ self clusterInstancesStep ] in FLMaterialization>>instancesStep >>>> SmallInteger(Integer)>>timesRepeat: >>>> FLMaterialization>>instancesStep >>>> FLMaterialization>>run >>>> [ :aDecoder | >>>> (FLMaterialization with: aDecoder) >>>> run; >>>> yourself ] in FLMaterializer>>setDefaultMaterialization >>>> FLMaterializer>>materializeFrom: >>>> FLMaterializer class>>materializeFromByteArray: >>>> MAPExampleModels class>>readModelFromSelector: >>>> MAPExampleModels class>>readModelNamed: >>>> [ self readModelNamed: aString ] in MAPExampleModels class>>named: >>>> [ self at: key put: aBlock value ] in Dictionary>>at:ifAbsentPut: >>>> Dictionary>>at:ifAbsent: >>>> Dictionary>>at:ifAbsentPut: >>>> MAPExampleModels class>>named: >>>> MAPTest>>model >>>> MAPTest>>tcapModule >>>> MAPTest>>testInsertSubscriberDataBitStringAccess >>>> [ testMethod perform: testMethod selector ] in Given>>produceReturnValueAt: >>>> [ self at: key put: aBlock value ] in Dictionary>>at:ifAbsentPut: >>>> Dictionary>>at:ifAbsent: >>>> Dictionary>>at:ifAbsentPut: >>>> Given>>produceReturnValueAt: >>>> MAPTest(Phexample)>>performTest >>>> >>>> >>>> >>> > >

Dec. 16, 2015

Re: [Pharo-dev] Fuel materialization on spur

by ClÃ©ment Bera

I don't know if SqueakV3 - Spur32 support is needed, however, I think it is really important to be able to serialize and materialize objects between 32 bits spur images and 64bits spur images. 2015-12-16 10:34 GMT+01:00 Esteban Lorenzano <estebanlm(a)gmail.com>: > Hi, > > On 16 Dec 2015, at 08:21, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: > > Hi Esteban, > > On Dec 15, 2015, at 5:06 AM, Esteban Lorenzano <estebanlm(a)gmail.com> > wrote: > > > On 15 Dec 2015, at 13:18, Norbert Hartl <norbert(a)hartl.name> wrote: > > We have a test build for pharo5 that checks if we need to do stuff for > future migrations. Since yesterday it fails, of course. > I have a fueled out object graph stored as byte array in a method. The > fuel bytes are serialized using a non-spur image. Now when the spur image > reads the bytes I get an exception. Has this something to do with immediate > characters? Is there a way to fix that or will it just not be possible to > exchange objects between spur and non-spur using fuel? > > > I think is not possible, but I will let fuel maintainers to have the last > word :) > > > > Of course it's possible; in VW we exchanged objects between 32- & 64-but > images using a format much like fuel and in 32-bits all floats are arrays > of 32-bit bit objects whereas in 64-bits a range of floats are immediate. > > > Oh wellâ¦ of course is possible. What I meant is that I think current > implementation of Fuel does not support it, just that :) > but again, Iâm not sure: Martin and Mariano can for sure say something > more detailed about :) > (problem is Martin is on vacations after getting his PhD and Mariano > didnât followed in close the spur migration) > > cheers, > Esteban > > > In spur characters are immediate (Max, "primitive" is not the right term) > whereas in v3 they're objects with a single isn't var. So at least fuel > needs to be modified to read characters specially. One way to architect > this is to ask the class (SmallInteger, Character, Float etc) to > materialize, and then the class can decide how to represent its instances. > The question is whether characters in spur should serialize themselves > using the same wire format as v3 or whether fuel should modify its wire > format to serialize immediate characters specially and have v3 materialize > characters as serialized by spur. The issue is whether it's important to > preserve object identity of characters with code >= 256 or not. > > > Esteban > > > thanks, > > Norbert > > > primitive #value: in Character class failed > > Stacktrace > > Character class(Object)>>primitiveFailed: > Character class(Object)>>primitiveFailed > Character class>>value: > Character class>>materializeFrom: > FLHookPrimitiveCluster>>materializeInstanceWith: > FLHookPrimitiveCluster(FLIteratingCluster)>>materializeInstancesStepWith: > FLMaterialization>>clusterInstancesStep > [ self clusterInstancesStep ] in FLMaterialization>>instancesStep > SmallInteger(Integer)>>timesRepeat: > FLMaterialization>>instancesStep > FLMaterialization>>run > [ :aDecoder | > (FLMaterialization with: aDecoder) > run; > yourself ] in FLMaterializer>>setDefaultMaterialization > FLMaterializer>>materializeFrom: > FLMaterializer class>>materializeFromByteArray: > MAPExampleModels class>>readModelFromSelector: > MAPExampleModels class>>readModelNamed: > [ self readModelNamed: aString ] in MAPExampleModels class>>named: > [ self at: key put: aBlock value ] in Dictionary>>at:ifAbsentPut: > Dictionary>>at:ifAbsent: > Dictionary>>at:ifAbsentPut: > MAPExampleModels class>>named: > MAPTest>>model > MAPTest>>tcapModule > MAPTest>>testInsertSubscriberDataBitStringAccess > [ testMethod perform: testMethod selector ] in Given>>produceReturnValueAt: > [ self at: key put: aBlock value ] in Dictionary>>at:ifAbsentPut: > Dictionary>>at:ifAbsent: > Dictionary>>at:ifAbsentPut: > Given>>produceReturnValueAt: > MAPTest(Phexample)>>performTest > > > > > >

Dec. 16, 2015