[Pharo-project] When will Fuel file format stabilize?
Hi guys, We are starting to use Fuel to exchange data daily, but without exactly the same version of Fuel loaded, exported data is just too frequently not importable anymore. And exception report is meaningless, you can only guess that Fuel version is to blame. So, it is more that a time to: - stabilize and freeze the file format - put a version info into a file and check it while importing - meaningful error reporting in such cases This is IMHO more than necessary for Fuel to become a production ready serializer and I'd say Fuel is now "old enough" to become such :) Best regards Janko -- Janko Mivšek Svetovalec za informatiko Eranova d.o.o. Ljubljana, Slovenija www.eranova.si tel: 01 514 22 55 faks: 01 514 22 56 gsm: 031 674 565
On Oct 8, 2011, at 5:56 PM, Janko Mivšek wrote:
Hi guys,
We are starting to use Fuel to exchange data daily, but without exactly the same version of Fuel loaded, exported data is just too frequently not importable anymore. And exception report is meaningless, you can only guess that Fuel version is to blame.
So, it is more that a time to:
- stabilize and freeze the file format - put a version info into a file and check it while importing
there is one already.
- meaningful error reporting in such cases
This is IMHO more than necessary for Fuel to become a production ready serializer and I'd say Fuel is now "old enough" to become such :)
Yes. Now what I would love is that even if fuel changes that the evolution of information is taken into account because like that it will be exercised for real.
Best regards Janko
-- Janko Mivšek Svetovalec za informatiko Eranova d.o.o. Ljubljana, Slovenija www.eranova.si tel: 01 514 22 55 faks: 01 514 22 56 gsm: 031 674 565
s
This is IMHO more than necessary for Fuel to become a production ready serializer and I'd say Fuel is now "old enough" to become such :)
Yes. Now what I would love is that even if fuel changes that the evolution of information is taken into account because like that it will be exercised for real.
No, that's impossible, and if posible, it is not worth it. Migrating from an old format to a new one is extremelly complicated and innecessary. The easiest way to solve this is to take the correct version of Fuel, materialize the graph from the stream, load new version of Fuel, and seriaize it again. That the easiest, more secure, and more practical approach I can see. Cheers
Best regards Janko
-- Janko Mivšek Svetovalec za informatiko Eranova d.o.o. Ljubljana, Slovenija www.eranova.si tel: 01 514 22 55 faks: 01 514 22 56 gsm: 031 674 565
-- Mariano http://marianopeck.wordpress.com
Mariano
No, that's impossible, and if posible, it is not worth it. Migrating from an old format to a new one is extremelly complicated and innecessary. The easiest way to solve this is to take the correct version of Fuel, materialize the graph from the stream, load new version of Fuel, and seriaize it again. That the easiest, more secure, and more practical approach I can see.
It would be interesting to see if you can get two different versions in memory :) (a need for a module system) so that you can load with one and save with the other. Stef
On Sun, Oct 9, 2011 at 12:03 AM, Stéphane Ducasse <stephane.ducasse@inria.fr
wrote:
Mariano
No, that's impossible, and if posible, it is not worth it. Migrating from an old format to a new one is extremelly complicated and innecessary. The easiest way to solve this is to take the correct version of Fuel, materialize the graph from the stream, load new version of Fuel, and seriaize it again. That the easiest, more secure, and more practical approach I can see.
It would be interesting to see if you can get two different versions in memory :)
heheh that would be nice :)
(a need for a module system)
yep, so far it is not possible.
so that you can load with one and save with the other.
Stef
-- Mariano http://marianopeck.wordpress.com
I wonder if it would be possible for you to come up with a separate format, whose purpose was only ever for this purpose. It would not need to be efficient in any respect, just something you could write out to file (say) during the process. It would be a shame to rely on the whole graph needing to be in memory. Maybe something very crude and text based? It could even be considered a debugging format, if it was somewhat human readable. If so, you could easily send this over a socket from an old version of a running system to a newer one, without relying on either serialising format. It could be quite handy. cheers, Mike On Sun, Oct 9, 2011 at 10:27 AM, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Sun, Oct 9, 2011 at 12:03 AM, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
Mariano
No, that's impossible, and if posible, it is not worth it. Migrating from an old format to a new one is extremelly complicated and innecessary. The easiest way to solve this is to take the correct version of Fuel, materialize the graph from the stream, load new version of Fuel, and seriaize it again. That the easiest, more secure, and more practical approach I can see.
It would be interesting to see if you can get two different versions in memory :)
heheh that would be nice :)
(a need for a module system)
yep, so far it is not possible.
so that you can load with one and save with the other.
Stef
-- Mariano http://marianopeck.wordpress.com
That's a good idea. Then every version of Fuel would pass an ever growing set of tests designed to ensure that the "backup" format works as intended. Hopefully, the format will eventually stabilize. I might be alone in this, but I see serializers as a very simple object oriented database, aka a "free" file format. Yes, one could do much the same thing with an OODMS, but there were reasons to avoid that path, especially at the time we got started. One needs to cope with schema changes (instance variable layout) in the application classes, but that can be done (Object Arts strikes again). *IF* Fuel is to serve a market that archives data, the format needs to be readable well into the future. Bill ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Michael Roberts [mike@mjr104.co.uk] Sent: Sunday, October 09, 2011 10:30 AM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize? I wonder if it would be possible for you to come up with a separate format, whose purpose was only ever for this purpose. It would not need to be efficient in any respect, just something you could write out to file (say) during the process. It would be a shame to rely on the whole graph needing to be in memory. Maybe something very crude and text based? It could even be considered a debugging format, if it was somewhat human readable. If so, you could easily send this over a socket from an old version of a running system to a newer one, without relying on either serialising format. It could be quite handy. cheers, Mike On Sun, Oct 9, 2011 at 10:27 AM, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Sun, Oct 9, 2011 at 12:03 AM, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
Mariano
No, that's impossible, and if posible, it is not worth it. Migrating from an old format to a new one is extremelly complicated and innecessary. The easiest way to solve this is to take the correct version of Fuel, materialize the graph from the stream, load new version of Fuel, and seriaize it again. That the easiest, more secure, and more practical approach I can see.
It would be interesting to see if you can get two different versions in memory :)
heheh that would be nice :)
(a need for a module system)
yep, so far it is not possible.
so that you can load with one and save with the other.
Stef
-- Mariano http://marianopeck.wordpress.com
On 08/10/11 6:03 PM, Stéphane Ducasse wrote:
Mariano
No, that's impossible, and if posible, it is not worth it. Migrating from an old format to a new one is extremelly complicated and innecessary. The easiest way to solve this is to take the correct version of Fuel, materialize the graph from the stream, load new version of Fuel, and seriaize it again. That the easiest, more secure, and more practical approach I can see.
It would be interesting to see if you can get two different versions in memory :) (a need for a module system) so that you can load with one and save with the other.
For the case where a lot of serialized files need to be converted, having two versions in memory is almost a necessity; otherwise, the Fuel code update would have to be applied for each file being converted. A separate issue is that at some point I wanted to serialize something differently, in two different contexts. In one case, serializing a Pier kernel, I want Fuel to just do its thing. In the other case, when exporting a portion a Pier kernel, I want to fine tune (i.e. map to different objects) the objects being exported/imported. There is a hook to allow the object swap, but the swap would then take place for the other case of serializing to whole kernel (where the desired behaviour was for no such mapping). -- Yanni
Maybe the way to go is to provide a metadescription of the changes done to the format since last version. That way, you can use it to convert your serialized objects from one version to another (even back and forward). That's what I think databases do, if you update your version, you have to update the database. It seems like a less effort approach. Cheers! Pocho. On Mon, Oct 10, 2011 at 1:50 PM, Yanni Chiu <yanni@rogers.com> wrote:
On 08/10/11 6:03 PM, Stéphane Ducasse wrote:
Mariano
No, that's impossible, and if posible, it is not worth it.
Migrating from an old format to a new one is extremelly complicated and innecessary. The easiest way to solve this is to take the correct version of Fuel, materialize the graph from the stream, load new version of Fuel, and seriaize it again. That the easiest, more secure, and more practical approach I can see.
It would be interesting to see if you can get two different versions in memory :) (a need for a module system) so that you can load with one and save with the other.
For the case where a lot of serialized files need to be converted, having two versions in memory is almost a necessity; otherwise, the Fuel code update would have to be applied for each file being converted.
A separate issue is that at some point I wanted to serialize something differently, in two different contexts. In one case, serializing a Pier kernel, I want Fuel to just do its thing. In the other case, when exporting a portion a Pier kernel, I want to fine tune (i.e. map to different objects) the objects being exported/imported. There is a hook to allow the object swap, but the swap would then take place for the other case of serializing to whole kernel (where the desired behaviour was for no such mapping).
-- Yanni
-- Lic. Javier Pimás Ciudad de Buenos Aires
Thanks guys for all the answers and ideas. I really appreaciate all of them. And in fact, this thread would probably be a source of information for Fuel's future work. But so far, nobody (janko?) yet answer my initial question of this thread. So I will copy paste it here and I will be willing to hear the answers: "We cannot. Let me just give you my thoughts: - It is a pity that we cannot make progress just because we break the format. If improving means breaking the format, we will do it. - The main question is WHY you need to update Fuel all the time? which are the reasons behind that? I have only 3 things in mind: a) because Fuel is too slow for you and hence you need a newer/faster version b) because you have changed Pharo version and Fuel doesn't work anymore there with the same version? c) because you found a bug. a) looks unlikely. I think Fuel was fast enough for most scenarios from the beginning even if new versions are faster. b) is unlikely also. For example, the last version (1.6) works out of the box in Pharo 1.2, 1.3 and 1.4. So it means that even between 3 versions of Pharo you don't need to update Fuel. c) we haven't found so far a critical bug which would requires to update Fuel. So...my question is, WHY you need to always use a new version of Fuel and why you cannot stay with a stable/fixed Metacello version? That's why we have metecello configurations. You can stick with version 1.6 (or whatever version) and just migrate when it is really necessary. For example, Francois Stephany is using Fuel for his website, and he is still using Fuel 1.4. He didn't need to update yet, because there aren't bugs and it is fast enough for what he needs. " On Mon, Oct 10, 2011 at 7:24 PM, Javier Pimás <elpochodelagente@gmail.com>wrote:
Maybe the way to go is to provide a metadescription of the changes done to the format since last version. That way, you can use it to convert your serialized objects from one version to another (even back and forward). That's what I think databases do, if you update your version, you have to update the database. It seems like a less effort approach.
Cheers! Pocho.
On Mon, Oct 10, 2011 at 1:50 PM, Yanni Chiu <yanni@rogers.com> wrote:
On 08/10/11 6:03 PM, Stéphane Ducasse wrote:
Mariano
No, that's impossible, and if posible, it is not worth it.
Migrating from an old format to a new one is extremelly complicated and innecessary. The easiest way to solve this is to take the correct version of Fuel, materialize the graph from the stream, load new version of Fuel, and seriaize it again. That the easiest, more secure, and more practical approach I can see.
It would be interesting to see if you can get two different versions in memory :) (a need for a module system) so that you can load with one and save with the other.
For the case where a lot of serialized files need to be converted, having two versions in memory is almost a necessity; otherwise, the Fuel code update would have to be applied for each file being converted.
A separate issue is that at some point I wanted to serialize something differently, in two different contexts. In one case, serializing a Pier kernel, I want Fuel to just do its thing. In the other case, when exporting a portion a Pier kernel, I want to fine tune (i.e. map to different objects) the objects being exported/imported. There is a hook to allow the object swap, but the swap would then take place for the other case of serializing to whole kernel (where the desired behaviour was for no such mapping).
-- Yanni
-- Lic. Javier Pimás Ciudad de Buenos Aires
-- Mariano http://marianopeck.wordpress.com
On Mon, Oct 10, 2011 at 6:50 PM, Yanni Chiu <yanni@rogers.com> wrote:
On 08/10/11 6:03 PM, Stéphane Ducasse wrote:
Mariano
No, that's impossible, and if posible, it is not worth it.
Migrating from an old format to a new one is extremelly complicated and innecessary. The easiest way to solve this is to take the correct version of Fuel, materialize the graph from the stream, load new version of Fuel, and seriaize it again. That the easiest, more secure, and more practical approach I can see.
It would be interesting to see if you can get two different versions in memory :) (a need for a module system) so that you can load with one and save with the other.
For the case where a lot of serialized files need to be converted, having two versions in memory is almost a necessity;
heheheheh how people could live with ReferenceStream for 10 years?
otherwise, the Fuel code update would have to be applied for each file being converted.
but answer my question: why would you need to update Fuel so frequently ?
A separate issue is that at some point I wanted to serialize something differently, in two different contexts. In one case, serializing a Pier kernel, I want Fuel to just do its thing. In the other case, when exporting a portion a Pier kernel, I want to fine tune (i.e. map to different objects) the objects being exported/imported. There is a hook to allow the object swap, but the swap would then take place for the other case of serializing to whole kernel (where the desired behaviour was for no such mapping).
That's an interesting point....Let me see if I understood. Let's say that you want to exclude certain instance variables of a class. What you mean is that you want to apply this hook (or another one) in an specific serialization and not always? because right now with Fuel (and SIXX, and I guess StOMP also) since you put the hooks in methods of the class, they are always taken into account. How would the API for the user should be? you can you specify for an specific serialization run that certain hook of certain object/class must be applied and some should not ? Apart from that, do you have a real example where you need such behavior ? Beause it looks quite complex/rare scenario ;) Thanks guys for the nice thread.
-- Yanni
-- Mariano http://marianopeck.wordpress.com
I concur with Mariano. In production images, you want probably stick with concrete verision(s) of frameworks you using and don't follow the bleeding edge. Because if you do, then you spending time on coping up with latest and greatest versions of these frameworks instead of spending time on developing your project :) -- Best regards, Igor Stasenko.
True, but what about reading data from five years ago? I have ten year old Dolphin STB that I can still load. ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Igor Stasenko [siguctua@gmail.com] Sent: Monday, October 10, 2011 4:59 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize? I concur with Mariano. In production images, you want probably stick with concrete verision(s) of frameworks you using and don't follow the bleeding edge. Because if you do, then you spending time on coping up with latest and greatest versions of these frameworks instead of spending time on developing your project :) -- Best regards, Igor Stasenko.
On 11 October 2011 00:19, Schwab,Wilhelm K <bschwab@anest.ufl.edu> wrote:
True, but what about reading data from five years ago? Â I have ten year old Dolphin STB that I can still load.
to my experience a 10 years old data has very small chances to be relevant :)
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Igor Stasenko [siguctua@gmail.com] Sent: Monday, October 10, 2011 4:59 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize?
I concur with Mariano. In production images, you want probably stick with concrete verision(s) of frameworks you using and don't follow the bleeding edge. Because if you do, then you spending time on coping up with latest and greatest versions of these frameworks instead of spending time on developing your project :)
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
2011/10/10 Igor Stasenko <siguctua@gmail.com>:
On 11 October 2011 00:19, Schwab,Wilhelm K <bschwab@anest.ufl.edu> wrote:
True, but what about reading data from five years ago? Â I have ten year old Dolphin STB that I can still load.
to my experience a 10 years old data has very small chances to be relevant :)
The problem is that if it ever is relevant, you have very small chance to know because you cannot load ;) Nicolas
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Igor Stasenko [siguctua@gmail.com] Sent: Monday, October 10, 2011 4:59 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize?
I concur with Mariano. In production images, you want probably stick with concrete verision(s) of frameworks you using and don't follow the bleeding edge. Because if you do, then you spending time on coping up with latest and greatest versions of these frameworks instead of spending time on developing your project :)
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
The data in question are medical records - they *really* need to load when demanded. ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Nicolas Cellier [nicolas.cellier.aka.nice@gmail.com] Sent: Monday, October 10, 2011 5:30 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize? 2011/10/10 Igor Stasenko <siguctua@gmail.com>:
On 11 October 2011 00:19, Schwab,Wilhelm K <bschwab@anest.ufl.edu> wrote:
True, but what about reading data from five years ago? I have ten year old Dolphin STB that I can still load.
to my experience a 10 years old data has very small chances to be relevant :)
The problem is that if it ever is relevant, you have very small chance to know because you cannot load ;) Nicolas
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Igor Stasenko [siguctua@gmail.com] Sent: Monday, October 10, 2011 4:59 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize?
I concur with Mariano. In production images, you want probably stick with concrete verision(s) of frameworks you using and don't follow the bleeding edge. Because if you do, then you spending time on coping up with latest and greatest versions of these frameworks instead of spending time on developing your project :)
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
Last one (I think<g>): I have an active study with data spanning about six years. I wish I had known then some of the things I know now, but I'm not in a hurry to ditch data simply because they are old - sample size, rare events, and all that good stuff. ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Schwab,Wilhelm K [bschwab@anest.ufl.edu] Sent: Monday, October 10, 2011 6:32 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize? The data in question are medical records - they *really* need to load when demanded. ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Nicolas Cellier [nicolas.cellier.aka.nice@gmail.com] Sent: Monday, October 10, 2011 5:30 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize? 2011/10/10 Igor Stasenko <siguctua@gmail.com>:
On 11 October 2011 00:19, Schwab,Wilhelm K <bschwab@anest.ufl.edu> wrote:
True, but what about reading data from five years ago? I have ten year old Dolphin STB that I can still load.
to my experience a 10 years old data has very small chances to be relevant :)
The problem is that if it ever is relevant, you have very small chance to know because you cannot load ;) Nicolas
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Igor Stasenko [siguctua@gmail.com] Sent: Monday, October 10, 2011 4:59 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize?
I concur with Mariano. In production images, you want probably stick with concrete verision(s) of frameworks you using and don't follow the bleeding edge. Because if you do, then you spending time on coping up with latest and greatest versions of these frameworks instead of spending time on developing your project :)
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
On 10 Oct 2011, at 23:25, Igor Stasenko wrote:
On 11 October 2011 00:19, Schwab,Wilhelm K <bschwab@anest.ufl.edu> wrote:
True, but what about reading data from five years ago? I have ten year old Dolphin STB that I can still load.
to my experience a 10 years old data has very small chances to be relevant :)
You might want to reconsider that: Some countries have laws, that regulate how long and what has to be archived: Switzerland: http://www.itandlaw.ch/html/publikationen/pdf/KMUMagazin04.pdf Germany: http://www.recht-im-internet.de/themen/archivierung.htm 10 years is considered the minimum. But that might not be what Pharo or Fuel is supposed to be used for. -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
You have the idea. If Fuel turns out to meet my expectations, it would be nice to be able to use it for such things. ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Stefan Marr [pharo@stefan-marr.de] Sent: Monday, October 10, 2011 5:57 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize? On 10 Oct 2011, at 23:25, Igor Stasenko wrote:
On 11 October 2011 00:19, Schwab,Wilhelm K <bschwab@anest.ufl.edu> wrote:
True, but what about reading data from five years ago? I have ten year old Dolphin STB that I can still load.
to my experience a 10 years old data has very small chances to be relevant :)
You might want to reconsider that: Some countries have laws, that regulate how long and what has to be archived: Switzerland: http://www.itandlaw.ch/html/publikationen/pdf/KMUMagazin04.pdf Germany: http://www.recht-im-internet.de/themen/archivierung.htm 10 years is considered the minimum. But that might not be what Pharo or Fuel is supposed to be used for. -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
On 11 October 2011 00:57, Stefan Marr <pharo@stefan-marr.de> wrote:
On 10 Oct 2011, at 23:25, Igor Stasenko wrote:
On 11 October 2011 00:19, Schwab,Wilhelm K <bschwab@anest.ufl.edu> wrote:
True, but what about reading data from five years ago? Â I have ten year old Dolphin STB that I can still load.
to my experience a 10 years old data has very small chances to be relevant :)
You might want to reconsider that: Some countries have laws, that regulate how long and what has to be archived:
Switzerland: http://www.itandlaw.ch/html/publikationen/pdf/KMUMagazin04.pdf Germany: http://www.recht-im-internet.de/themen/archivierung.htm
10 years is considered the minimum.
Then how about following idea: help implementing format migration code :) I think what can be done in Fuel is to have a separate class per each binary format version, so when you do format changes, instead of modifying existing class you creating a new one and put new stuff there. Then at a point of detecting incoming data, it could decide which class to use for deserialization.
But that might not be what Pharo or Fuel is supposed to be used for.
-- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: Â +32 2 629 3525
-- Best regards, Igor Stasenko.
On Tue, Oct 11, 2011 at 5:51 AM, Igor Stasenko <siguctua@gmail.com> wrote:
On 11 October 2011 00:57, Stefan Marr <pharo@stefan-marr.de> wrote:
On 10 Oct 2011, at 23:25, Igor Stasenko wrote:
On 11 October 2011 00:19, Schwab,Wilhelm K <bschwab@anest.ufl.edu>
wrote:
True, but what about reading data from five years ago? I have ten year old Dolphin STB that I can still load.
to my experience a 10 years old data has very small chances to be relevant :)
You might want to reconsider that: Some countries have laws, that regulate how long and what has to be archived:
Switzerland: http://www.itandlaw.ch/html/publikationen/pdf/KMUMagazin04.pdf Germany: http://www.recht-im-internet.de/themen/archivierung.htm
10 years is considered the minimum.
Then how about following idea: help implementing format migration code :)
I think what can be done in Fuel is to have a separate class per each binary format version, so when you do format changes, instead of modifying existing class you creating a new one and put new stuff there. Then at a point of detecting incoming data, it could decide which class to use for deserialization.
Yes, but the problem is that in Fuel we have a flatten design where we group the encoding/decoding of objects in "Cluster" classes. So, the format is not only defined by FLSerializer and FLMaterializer but also with its 20 cluster classes. So it is not that easy...
But that might not be what Pharo or Fuel is supposed to be used for.
-- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
-- Best regards, Igor Stasenko.
-- Mariano http://marianopeck.wordpress.com
On Mon, Oct 10, 2011 at 11:19 PM, Schwab,Wilhelm K <bschwab@anest.ufl.edu>wrote:
True, but what about reading data from five years ago?
Well, can you load your 5-year-old code out of the box? No. You will probably need to update its code. The same here. You can always take the same Fuel version and updated (without changing the format) to work in a new Pharo image. Just as you do with your own code. As an example, the last stable version 1.6 of Fuel works if Pharo 1.2, 1.3 and 1.4. I think it is good enough. Cheers
I have ten year old Dolphin STB that I can still load.
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [ pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Igor Stasenko [ siguctua@gmail.com] Sent: Monday, October 10, 2011 4:59 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize?
I concur with Mariano. In production images, you want probably stick with concrete verision(s) of frameworks you using and don't follow the bleeding edge. Because if you do, then you spending time on coping up with latest and greatest versions of these frameworks instead of spending time on developing your project :)
-- Best regards, Igor Stasenko.
-- Mariano http://marianopeck.wordpress.com
On Mon, Oct 10, 2011 at 11:25 PM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Mon, Oct 10, 2011 at 11:19 PM, Schwab,Wilhelm K <bschwab@anest.ufl.edu>wrote:
True, but what about reading data from five years ago?
Well, can you load your 5-year-old code out of the box? No. You will probably need to update its code. The same here. You can always take the same Fuel version and updated (without changing the format) to work in a new Pharo image. Just as you do with your own code. As an example, the last stable version 1.6 of Fuel works if Pharo 1.2, 1.3 and 1.4. I think it is good enough.
Ahhh and if you are so worried about that, then use a proper database strategy than a simple/plain object serializer.
Cheers
I have ten year old Dolphin STB that I can still load.
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [ pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Igor Stasenko [ siguctua@gmail.com] Sent: Monday, October 10, 2011 4:59 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize?
I concur with Mariano. In production images, you want probably stick with concrete verision(s) of frameworks you using and don't follow the bleeding edge. Because if you do, then you spending time on coping up with latest and greatest versions of these frameworks instead of spending time on developing your project :)
-- Best regards, Igor Stasenko.
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
I think that somebody (probably not the fuel team which is full right now) should use a jSON or other formats to externatlise data out of fuel and this version should be use for old mgration. The point of evolution is important because we should be able to load old code in the future. Now let us make sure that fuel is really working well. On Oct 10, 2011, at 11:25 PM, Mariano Martinez Peck wrote:
On Mon, Oct 10, 2011 at 11:19 PM, Schwab,Wilhelm K <bschwab@anest.ufl.edu> wrote: True, but what about reading data from five years ago?
Well, can you load your 5-year-old code out of the box? No. You will probably need to update its code. The same here. You can always take the same Fuel version and updated (without changing the format) to work in a new Pharo image. Just as you do with your own code. As an example, the last stable version 1.6 of Fuel works if Pharo 1.2, 1.3 and 1.4. I think it is good enough.
Cheers
I have ten year old Dolphin STB that I can still load.
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Igor Stasenko [siguctua@gmail.com] Sent: Monday, October 10, 2011 4:59 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize?
I concur with Mariano. In production images, you want probably stick with concrete verision(s) of frameworks you using and don't follow the bleeding edge. Because if you do, then you spending time on coping up with latest and greatest versions of these frameworks instead of spending time on developing your project :)
-- Best regards, Igor Stasenko.
-- Mariano http://marianopeck.wordpress.com
On 10/10/11 2:43 PM, Mariano Martinez Peck wrote:
On Mon, Oct 10, 2011 at 6:50 PM, Yanni Chiu <yanni@rogers.com
For the case where a lot of serialized files need to be converted, having two versions in memory is almost a necessity;
heheheheh how people could live with ReferenceStream for 10 years?
otherwise, the Fuel code update would have to be applied for each file being converted.
but answer my question: why would you need to update Fuel so frequently ?
I don't need to update Fuel frequently. I'm currently "stuck" at some pre-1.6 version of Fuel (some combination of package versions that I managed to get working at the time, using early Fuel releases). I want to migrate to a newer version at some point (to use some new Fuel features), but the hold up is that I would have to migrate about a dozen serialization files - probably manually, since a dozen is roughly at my threshold to automate. However, I expect that the number of files to migrate would be a larger number in the future. At that point I'd automate the: deserialize with old Fuel, update to newer Fuel and serialize again. The migration procedure seems safer to me, if it did not load code in the middle of the process. There's lots of workarounds for this issue, so it's not a high priority.
A separate issue is that at some point I wanted to serialize something differently, in two different contexts. In one case, serializing a Pier kernel, I want Fuel to just do its thing. In the other case, when exporting a portion a Pier kernel, I want to fine tune (i.e. map to different objects) the objects being exported/imported. There is a hook to allow the object swap, but the swap would then take place for the other case of serializing to whole kernel (where the desired behaviour was for no such mapping).
That's an interesting point....Let me see if I understood. Let's say that you want to exclude certain instance variables of a class. What you mean is that you want to apply this hook (or another one) in an specific serialization and not always? because right now with Fuel (and SIXX, and I guess StOMP also) since you put the hooks in methods of the class, they are always taken into account. How would the API for the user should be? you can you specify for an specific serialization run that certain hook of certain object/class must be applied and some should not ?
Yes, you've got the idea. No idea what the API should be for such functionality - probably a serialization context (passed in at the start) that answers configuration parameters to use to pick an object mapping method.
Apart from that, do you have a real example where you need such behavior ? Beause it looks quite complex/rare scenario ;)
My serialization code is not top of mind, but I think the issue I had was that when exporting a portion of a Pier kernel, for importing into another Pier kernel, the security objects should be nil'ed out. However, when using Fuel to serialize the kernel, the security objects must be serialized instead of being nil'ed out. Not sure why I don't currently have a bug - probably, the security is being inherited from the parent structures when they're nil. I'll see the problem once I try to set different security on lower down structures - they'll be lost on restart. -- Yanni
On Tue, Oct 11, 2011 at 2:33 AM, Yanni Chiu <yanni@rogers.com> wrote:
On 10/10/11 2:43 PM, Mariano Martinez Peck wrote:
On Mon, Oct 10, 2011 at 6:50 PM, Yanni Chiu <yanni@rogers.com
For the case where a lot of serialized files need to be converted, having two versions in memory is almost a necessity;
heheheheh how people could live with ReferenceStream for 10 years?
otherwise, the Fuel code update would have to be applied for each file being converted.
but answer my question: why would you need to update Fuel so frequently ?
I don't need to update Fuel frequently. I'm currently "stuck" at some pre-1.6 version of Fuel (some combination of package versions that I managed to get working at the time, using early Fuel releases).
You are an early adopter!!! cool :)
I want to migrate to a newer version at some point (to use some new Fuel features), but the hold up is that I would have to migrate about a dozen serialization files - probably manually, since a dozen is roughly at my threshold to automate. However, I expect that the number of files to migrate would be a larger number in the future. At that point I'd automate the: deserialize with old Fuel, update to newer Fuel and serialize again.
Exactly. That's what I meant. From my point of view it is easier to provide such scripts rather than to find a complex automagic conversion between versions. Imagine something like: FuelMigrator migrateFromFuelVersion: '1.6' to: '1.7 streams: aListOfStreams basically it can use gofer/metacello to load the version of fuel, iterate over each file and serialize/materialize.
The migration procedure seems safer to me, if it did not load code in the middle of the process. There's lots of workarounds for this issue, so it's not a high priority.
A separate issue is that at some point I wanted to serialize
something differently, in two different contexts. In one case, serializing a Pier kernel, I want Fuel to just do its thing. In the other case, when exporting a portion a Pier kernel, I want to fine tune (i.e. map to different objects) the objects being exported/imported. There is a hook to allow the object swap, but the swap would then take place for the other case of serializing to whole kernel (where the desired behaviour was for no such mapping).
That's an interesting point....Let me see if I understood. Let's say that you want to exclude certain instance variables of a class. What you mean is that you want to apply this hook (or another one) in an specific serialization and not always? because right now with Fuel (and SIXX, and I guess StOMP also) since you put the hooks in methods of the class, they are always taken into account. How would the API for the user should be? you can you specify for an specific serialization run that certain hook of certain object/class must be applied and some should not ?
Yes, you've got the idea. No idea what the API should be for such functionality - probably a serialization context (passed in at the start) that answers configuration parameters to use to pick an object mapping method.
Interesting. I will open an issue for this.
Apart from that, do you have a real example where you need such behavior
? Beause it looks quite complex/rare scenario ;)
My serialization code is not top of mind, but I think the issue I had was that when exporting a portion of a Pier kernel, for importing into another Pier kernel, the security objects should be nil'ed out. However, when using Fuel to serialize the kernel, the security objects must be serialized instead of being nil'ed out.
Well...that can be simple workarrounded with a simple if depending on what you want. So depending what you do, you can serialize or not the security objects. This can be done without needing what you say. It may not be the best solution, but a workarround.
Not sure why I don't currently have a bug - probably, the security is being inherited from the parent structures when they're nil. I'll see the problem once I try to set different security on lower down structures - they'll be lost on restart.
Ok. Check: http://rmod.lille.inria.fr/web/pier/software/Fuel/Hooks http://forum.world.st/Any-serializer-can-customize-serialization-strategy-td... Cheers
-- Yanni
-- Mariano http://marianopeck.wordpress.com
On 10/08/2011 10:42 PM, Mariano Martinez Peck wrote:
s
This is IMHO more than necessary for Fuel to become a production ready serializer and I'd say Fuel is now "old enough" to become such :)
Yes. Now what I would love is that even if fuel changes that the evolution of information is taken into account because like that it will be exercised for real.
No, that's impossible, and if posible, it is not worth it. Migrating from an old format to a new one is extremelly complicated and innecessary. The easiest way to solve this is to take the correct version of Fuel, materialize the graph from the stream, load new version of Fuel, and seriaize it again. That the easiest, more secure, and more practical approach I can see.
That is horribly naïve an excludes fuel from a lot of use cases. You can't use fuel for "archiving" objects outside of the image because you will never know whether you will be able to read them in again because the format changes. You will always need to have "live" ones in the image. That means you can't use fuel for anything Monticello related because you may never be able to load those versions in again because the file format has changed in the mean time. Cheers Philippe
Mariano I think that evolution and migration is key if we want to use fuel for loading code. This is why in MCZ you have always the source files. In case of trouble. It is important to use the current situation as case for evolution. I like the idea of igor of having class that represent different versions. Stef On Oct 11, 2011, at 8:55 AM, Philippe Marschall wrote:
On 10/08/2011 10:42 PM, Mariano Martinez Peck wrote:
s
This is IMHO more than necessary for Fuel to become a production ready serializer and I'd say Fuel is now "old enough" to become such :)
Yes. Now what I would love is that even if fuel changes that the evolution of information is taken into account because like that it will be exercised for real.
No, that's impossible, and if posible, it is not worth it. Migrating from an old format to a new one is extremelly complicated and innecessary. The easiest way to solve this is to take the correct version of Fuel, materialize the graph from the stream, load new version of Fuel, and seriaize it again. That the easiest, more secure, and more practical approach I can see.
That is horribly naïve an excludes fuel from a lot of use cases. You can't use fuel for "archiving" objects outside of the image because you will never know whether you will be able to read them in again because the format changes. You will always need to have "live" ones in the image.
That means you can't use fuel for anything Monticello related because you may never be able to load those versions in again because the file format has changed in the mean time.
Cheers Philippe
On Tue, Oct 11, 2011 at 9:10 AM, Stéphane Ducasse <stephane.ducasse@inria.fr
wrote:
Mariano
I think that evolution and migration is key if we want to use fuel for loading code. This is why in MCZ you have always the source files. In case of trouble.
Exactly! and even more, you NEED the code if you are managing code ;) so you need it always, in case of problem or not.
It is important to use the current situation as case for evolution. I like the idea of igor of having class that represent different versions.
We will think about it.
Stef
On Oct 11, 2011, at 8:55 AM, Philippe Marschall wrote:
On 10/08/2011 10:42 PM, Mariano Martinez Peck wrote:
s
This is IMHO more than necessary for Fuel to become a production ready serializer and I'd say Fuel is now "old enough" to become such :)
Yes. Now what I would love is that even if fuel changes that the evolution of information is taken into account because like that it will be exercised for real.
No, that's impossible, and if posible, it is not worth it. Migrating from an old format to a new one is extremelly complicated and innecessary. The easiest way to solve this is to take the correct version of Fuel, materialize the graph from the stream, load new version of Fuel, and seriaize it again. That the easiest, more secure, and more practical approach I can see.
That is horribly naïve an excludes fuel from a lot of use cases. You can't use fuel for "archiving" objects outside of the image because you will never know whether you will be able to read them in again because the format changes. You will always need to have "live" ones in the image.
That means you can't use fuel for anything Monticello related because you may never be able to load those versions in again because the file format has changed in the mean time.
Cheers Philippe
-- Mariano http://marianopeck.wordpress.com
On 11 October 2011 22:52, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Tue, Oct 11, 2011 at 9:10 AM, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
Mariano
I think that evolution and migration is key if we want to use fuel for loading code. This is why in MCZ you have always the source files. In case of trouble.
Exactly! and even more, you NEED the code if you are managing code ;)Â so you need it always, in case of problem or not.
It is important to use the current situation as case for evolution. I like the idea of igor of having class that represent different versions.
We will think about it.
Yes, think. :) The lesson that i learn by staring at VMMaker, events handling in image and other places: never ever spread bit-twiddling logic over multiple places. If you cannot avoid bit-twiddling, make sure to put it into a single place, so rest of the code could be kept clean of it. -- Best regards, Igor Stasenko.
On Tue, Oct 11, 2011 at 8:55 AM, Philippe Marschall < philippe.marschall@netcetera.ch> wrote:
On 10/08/2011 10:42 PM, Mariano Martinez Peck wrote:
s
This is IMHO more than necessary for Fuel to become a production ready serializer and I'd say Fuel is now "old enough" to become such :)
Yes. Now what I would love is that even if fuel changes that the evolution of information is taken into account because like that it will be exercised for real.
No, that's impossible, and if posible, it is not worth it. Migrating from an old format to a new one is extremelly complicated and innecessary. The easiest way to solve this is to take the correct version of Fuel, materialize the graph from the stream, load new version of Fuel, and seriaize it again. That the easiest, more secure, and more practical approach I can see.
That is horribly naïve an excludes fuel from a lot of use cases. You can't use fuel for "archiving" objects outside of the image because you will never know whether you will be able to read them in again because the format changes. You will always need to have "live" ones in the image.
No. That's incorrect. You won't be able to do that ONLY if you update Fuel to a new image that breaks format. You can still continue to use the same version and you will never have that problem. So, again, why you need to update Fuel? Why SmartRefStream does not have this problem? because it hasn't changed in the last 10 years. So..do the same, take an specific Fuel version and keep it for 10 years. Just update it to make it work in Pharo without changing the format and you are done.
That means you can't use fuel for anything Monticello related because you may never be able to load those versions in again because the file format has changed in the mean time.
I guess that in the end, if someone can really do something for Monticello on top of Fuel it will be like 2 years from now, and at some point Fuel format will be stable. And as Stef says...you always have the code there in case of problems.
Cheers Philippe
-- Mariano http://marianopeck.wordpress.com
On 11.10.2011 21:51, Mariano Martinez Peck wrote:
On Tue, Oct 11, 2011 at 8:55 AM, Philippe Marschall <philippe.marschall@netcetera.ch <mailto:philippe.marschall@netcetera.ch>> wrote:
On 10/08/2011 10:42 PM, Mariano Martinez Peck wrote: > s > >>> >>> This is IMHO more than necessary for Fuel to become a production ready >>> serializer and I'd say Fuel is now "old enough" to become such :) >> >> Yes. >> Now what I would love is that even if fuel changes that the evolution of >> information >> is taken into account because like that it will be exercised for real. >> >> > No, that's impossible, and if posible, it is not worth it. Migrating from an > old format to a new one is extremelly complicated and innecessary. The > easiest way to solve this is to take the correct version of Fuel, > materialize the graph from the stream, load new version of Fuel, and > seriaize it again. That the easiest, more secure, and more practical > approach I can see.
That is horribly naïve an excludes fuel from a lot of use cases. You can't use fuel for "archiving" objects outside of the image because you will never know whether you will be able to read them in again because the format changes. You will always need to have "live" ones in the image.
No. That's incorrect. You won't be able to do that ONLY if you update Fuel to a new image that breaks format. You can still continue to use the same version and you will never have that problem. So, again, why you need to update Fuel?
Because it's old software. Bugs may not get fixed. It may not work in newer Pharo versions. I may have dependencies on other libraries that may require a new version of Fuel. You name it.
Why SmartRefStream does not have this problem? because it hasn't changed in the last 10 years. So..do the same, take an specific Fuel version and keep it for 10 years. Just update it to make it work in Pharo without changing the format and you are done.
That means you can't use fuel for anything Monticello related because you may never be able to load those versions in again because the file format has changed in the mean time.
I guess that in the end, if someone can really do something for Monticello on top of Fuel it will be like 2 years from now, and at some point Fuel format will be stable. And as Stef says...you always have the code there in case of problems.
Monticello is just an example for a case where I want to store objects outside of the image. Cheers Philippe
On Sat, Oct 15, 2011 at 11:49 AM, Philippe Marschall <kustos@gmx.net> wrote:
On 11.10.2011 21:51, Mariano Martinez Peck wrote:
On Tue, Oct 11, 2011 at 8:55 AM, Philippe Marschall <philippe.marschall@netcetera.ch <mailto:philippe.marschall@netcetera.ch>> wrote:
On 10/08/2011 10:42 PM, Mariano Martinez Peck wrote: > s > >>> >>> This is IMHO more than necessary for Fuel to become a production ready >>> serializer and I'd say Fuel is now "old enough" to become such :) >> >> Yes. >> Now what I would love is that even if fuel changes that the evolution of >> information >> is taken into account because like that it will be exercised for real. >> >> > No, that's impossible, and if posible, it is not worth it. Migrating from an > old format to a new one is extremelly complicated and innecessary.
The
> easiest way to solve this is to take the correct version of Fuel, > materialize the graph from the stream, load new version of Fuel,
and
> seriaize it again. That the easiest, more secure, and more
practical
> approach I can see.
That is horribly naïve an excludes fuel from a lot of use cases. You can't use fuel for "archiving" objects outside of the image because
you
will never know whether you will be able to read them in again
because
the format changes. You will always need to have "live" ones in the image.
No. That's incorrect. You won't be able to do that ONLY if you update Fuel to a new image that breaks format. You can still continue to use the same version and you will never have that problem. So, again, why you need to update Fuel?
Because it's old software. Bugs may not get fixed. It may not work in newer Pharo versions. I may have dependencies on other libraries that may require a new version of Fuel. You name it.
I understand. What I mean is that depending on the changes and the amount of work, you can just adapt Fuel to new versions of Pharo but without changing its format. I mean....say you were in Fuel 1.4. You don't need to move to Fuel 1.8 just because. You can just try to fix 1.4 to make it work in latest pharo. That's EXACTLY what we do with ReferenceStream and friends. The only difference is that the Pharo team does that because it is in the core. But yes, it may happen that ideed you will require a new version of Fuel.
Why SmartRefStream does not have this problem? because it hasn't changed in the last 10 years. So..do the same, take an specific Fuel version and keep it for 10 years. Just update it to make it work in Pharo without changing the format and you are done.
That means you can't use fuel for anything Monticello related because you may never be able to load those versions in again because the file format has changed in the mean time.
I guess that in the end, if someone can really do something for Monticello on top of Fuel it will be like 2 years from now, and at some point Fuel format will be stable. And as Stef says...you always have the code there in case of problems.
Monticello is just an example for a case where I want to store objects outside of the image.
Well, then you can just wait until we finish. Give us one year more. Instead of doing 1-year-long releases, we like to do 3 months releases. But again, that doesn't mean you have to update...
Cheers Philippe
-- Mariano http://marianopeck.wordpress.com
I like the ideas of Stef and Ben of loading multiple fuel versions at the same time. Do you know something done in that sense? MartÃn On Sat, Oct 15, 2011 at 7:09 AM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Sat, Oct 15, 2011 at 11:49 AM, Philippe Marschall <kustos@gmx.net>wrote:
On 11.10.2011 21:51, Mariano Martinez Peck wrote:
On Tue, Oct 11, 2011 at 8:55 AM, Philippe Marschall <philippe.marschall@netcetera.ch <mailto:philippe.marschall@netcetera.ch>> wrote:
On 10/08/2011 10:42 PM, Mariano Martinez Peck wrote: > s > >>> >>> This is IMHO more than necessary for Fuel to become a production ready >>> serializer and I'd say Fuel is now "old enough" to become such
:)
>> >> Yes. >> Now what I would love is that even if fuel changes that the evolution of >> information >> is taken into account because like that it will be exercised for real. >> >> > No, that's impossible, and if posible, it is not worth it. Migrating from an > old format to a new one is extremelly complicated and innecessary.
The
> easiest way to solve this is to take the correct version of Fuel, > materialize the graph from the stream, load new version of Fuel,
and
> seriaize it again. That the easiest, more secure, and more
practical
> approach I can see.
That is horribly naïve an excludes fuel from a lot of use cases. You can't use fuel for "archiving" objects outside of the image because
you
will never know whether you will be able to read them in again
because
the format changes. You will always need to have "live" ones in the image.
No. That's incorrect. You won't be able to do that ONLY if you update Fuel to a new image that breaks format. You can still continue to use the same version and you will never have that problem. So, again, why you need to update Fuel?
Because it's old software. Bugs may not get fixed. It may not work in newer Pharo versions. I may have dependencies on other libraries that may require a new version of Fuel. You name it.
I understand. What I mean is that depending on the changes and the amount of work, you can just adapt Fuel to new versions of Pharo but without changing its format. I mean....say you were in Fuel 1.4. You don't need to move to Fuel 1.8 just because. You can just try to fix 1.4 to make it work in latest pharo. That's EXACTLY what we do with ReferenceStream and friends. The only difference is that the Pharo team does that because it is in the core. But yes, it may happen that ideed you will require a new version of Fuel.
Why SmartRefStream does not have this problem? because it hasn't changed in the last 10 years. So..do the same, take an specific Fuel version and keep it for 10 years. Just update it to make it work in Pharo without changing the format and you are done.
That means you can't use fuel for anything Monticello related because you may never be able to load those versions in again because the file format has changed in the mean time.
I guess that in the end, if someone can really do something for Monticello on top of Fuel it will be like 2 years from now, and at some point Fuel format will be stable. And as Stef says...you always have the code there in case of problems.
Monticello is just an example for a case where I want to store objects outside of the image.
Well, then you can just wait until we finish. Give us one year more. Instead of doing 1-year-long releases, we like to do 3 months releases. But again, that doesn't mean you have to update...
Cheers Philippe
-- Mariano http://marianopeck.wordpress.com
I should have thought of this earlier: Dolphin's #environment concept will do the job. Instead of hard references to classes or adding a new name space syntax, one can simply use messages to delay binding. I should *look* at Fuel to find an example, but suppose there is a class FuelDWORD. Instead of referencing it, one would do something like Fuel formatDWORD or Fuel environment formatDWORD or ( Fuel version:3 ) formatDWORD etc. There are many ways one could handle the details, but #environment or #version: can return a "system dictionary" (an ordinary class would suffice) that is version specific, allowing FuelDWORD_3 to be used by code that otherwise knows only that it should use given selector to get the class it needs. To develop the different version/environments, one could use the RB to rename classes and then save the new version classes in a Fuel sub-package. Add FuelVersion3 to give out the correct selectors, and a basic facade should be able to make everything go with the various versions in the system at one time. Finally, one could facade the entire serializer, something like (Fuel version:3) save:anObject on:aStream. (Fuel version:3) readFrom:aStream. etc. Does that make sense? Bill ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Martin Dias [tinchodias@gmail.com] Sent: Saturday, October 15, 2011 9:06 AM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize? I like the ideas of Stef and Ben of loading multiple fuel versions at the same time. Do you know something done in that sense? MartÃn On Sat, Oct 15, 2011 at 7:09 AM, Mariano Martinez Peck <marianopeck@gmail.com<mailto:marianopeck@gmail.com>> wrote: On Sat, Oct 15, 2011 at 11:49 AM, Philippe Marschall <kustos@gmx.net<mailto:kustos@gmx.net>> wrote: On 11.10.2011 21:51, Mariano Martinez Peck wrote:
On Tue, Oct 11, 2011 at 8:55 AM, Philippe Marschall <philippe.marschall@netcetera.ch<mailto:philippe.marschall@netcetera.ch> <mailto:philippe.marschall@netcetera.ch<mailto:philippe.marschall@netcetera.ch>>> wrote:
On 10/08/2011 10:42 PM, Mariano Martinez Peck wrote: > s > >>> >>> This is IMHO more than necessary for Fuel to become a production ready >>> serializer and I'd say Fuel is now "old enough" to become such :) >> >> Yes. >> Now what I would love is that even if fuel changes that the evolution of >> information >> is taken into account because like that it will be exercised for real. >> >> > No, that's impossible, and if posible, it is not worth it. Migrating from an > old format to a new one is extremelly complicated and innecessary. The > easiest way to solve this is to take the correct version of Fuel, > materialize the graph from the stream, load new version of Fuel, and > seriaize it again. That the easiest, more secure, and more practical > approach I can see.
That is horribly naïve an excludes fuel from a lot of use cases. You can't use fuel for "archiving" objects outside of the image because you will never know whether you will be able to read them in again because the format changes. You will always need to have "live" ones in the image.
No. That's incorrect. You won't be able to do that ONLY if you update Fuel to a new image that breaks format. You can still continue to use the same version and you will never have that problem. So, again, why you need to update Fuel?
Because it's old software. Bugs may not get fixed. It may not work in newer Pharo versions. I may have dependencies on other libraries that may require a new version of Fuel. You name it. I understand. What I mean is that depending on the changes and the amount of work, you can just adapt Fuel to new versions of Pharo but without changing its format. I mean....say you were in Fuel 1.4. You don't need to move to Fuel 1.8 just because. You can just try to fix 1.4 to make it work in latest pharo. That's EXACTLY what we do with ReferenceStream and friends. The only difference is that the Pharo team does that because it is in the core. But yes, it may happen that ideed you will require a new version of Fuel.
Why SmartRefStream does not have this problem? because it hasn't changed in the last 10 years. So..do the same, take an specific Fuel version and keep it for 10 years. Just update it to make it work in Pharo without changing the format and you are done.
That means you can't use fuel for anything Monticello related because you may never be able to load those versions in again because the file format has changed in the mean time.
I guess that in the end, if someone can really do something for Monticello on top of Fuel it will be like 2 years from now, and at some point Fuel format will be stable. And as Stef says...you always have the code there in case of problems.
Monticello is just an example for a case where I want to store objects outside of the image. Well, then you can just wait until we finish. Give us one year more. Instead of doing 1-year-long releases, we like to do 3 months releases. But again, that doesn't mean you have to update... Cheers Philippe -- Mariano http://marianopeck.wordpress.com
On 15.10.2011 15:06, Martin Dias wrote:
I like the ideas of Stef and Ben of loading multiple fuel versions at the same time. Do you know something done in that sense?
MartÃn
On Sat, Oct 15, 2011 at 7:09 AM, Mariano Martinez Peck <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> wrote:
On Sat, Oct 15, 2011 at 11:49 AM, Philippe Marschall <kustos@gmx.net <mailto:kustos@gmx.net>> wrote:
On 11.10.2011 21:51, Mariano Martinez Peck wrote: > > > On Tue, Oct 11, 2011 at 8:55 AM, Philippe Marschall > <philippe.marschall@netcetera.ch <mailto:philippe.marschall@netcetera.ch> > <mailto:philippe.marschall@netcetera.ch <mailto:philippe.marschall@netcetera.ch>>> wrote: > > On 10/08/2011 10:42 PM, Mariano Martinez Peck wrote: > > s > > > >>> > >>> This is IMHO more than necessary for Fuel to become a production > ready > >>> serializer and I'd say Fuel is now "old enough" to become such :) > >> > >> Yes. > >> Now what I would love is that even if fuel changes that the > evolution of > >> information > >> is taken into account because like that it will be exercised for > real. > >> > >> > > No, that's impossible, and if posible, it is not worth it. > Migrating from an > > old format to a new one is extremelly complicated and innecessary. The > > easiest way to solve this is to take the correct version of Fuel, > > materialize the graph from the stream, load new version of Fuel, and > > seriaize it again. That the easiest, more secure, and more practical > > approach I can see. > > That is horribly naïve an excludes fuel from a lot of use cases. You > can't use fuel for "archiving" objects outside of the image because you > will never know whether you will be able to read them in again because > the format changes. You will always need to have "live" ones in the > image. > > > No. That's incorrect. You won't be able to do that ONLY if you update > Fuel to a new image that breaks format. > You can still continue to use the same version and you will never have > that problem. So, again, why you need to update Fuel?
Because it's old software. Bugs may not get fixed. It may not work in newer Pharo versions. I may have dependencies on other libraries that may require a new version of Fuel. You name it.
I understand. What I mean is that depending on the changes and the amount of work, you can just adapt Fuel to new versions of Pharo but without changing its format. I mean....say you were in Fuel 1.4. You don't need to move to Fuel 1.8 just because. You can just try to fix 1.4 to make it work in latest pharo.
Oh yeah sure, I can always fork Fuel and maintain it myself. That's exactly what I don't don't want to do.
That's EXACTLY what we do with ReferenceStream and friends. The only difference is that the Pharo team does that because it is in the core. But yes, it may happen that ideed you will require a new version of Fuel.
Whatâ½ What's the advantage of Fuel over ReferenceStream again?
> Why SmartRefStream does not have this problem? because it hasn't > changed in the last 10 years. > So..do the same, take an specific Fuel version and keep it for 10 years. > Just update it to make it work in Pharo without changing the format and > you are done. > > > > That means you can't use fuel for anything Monticello related because > you may never be able to load those versions in again because the file > format has changed in the mean time. > > > I guess that in the end, if someone can really do something for > Monticello on top of Fuel it will be like 2 years from now, and at some > point Fuel format will be stable. > And as Stef says...you always have the code there in case of problems.
Monticello is just an example for a case where I want to store objects outside of the image.
Well, then you can just wait until we finish. Give us one year more. Instead of doing 1-year-long releases, we like to do 3 months releases. But again, that doesn't mean you have to update...
So you're saying I shouldn't use Fuel? Cheers Philippe
On Sun, Oct 16, 2011 at 1:20 PM, Philippe Marschall <kustos@gmx.net> wrote:
On 15.10.2011 15:06, Martin Dias wrote:
I like the ideas of Stef and Ben of loading multiple fuel versions at the same time. Do you know something done in that sense?
MartÃn
On Sat, Oct 15, 2011 at 7:09 AM, Mariano Martinez Peck <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> wrote:
On Sat, Oct 15, 2011 at 11:49 AM, Philippe Marschall <kustos@gmx.net <mailto:kustos@gmx.net>> wrote:
On 11.10.2011 21:51, Mariano Martinez Peck wrote: > > > On Tue, Oct 11, 2011 at 8:55 AM, Philippe Marschall > <philippe.marschall@netcetera.ch <mailto:philippe.marschall@netcetera.ch> > <mailto:philippe.marschall@netcetera.ch <mailto:philippe.marschall@netcetera.ch>>> wrote: > > On 10/08/2011 10:42 PM, Mariano Martinez Peck wrote: > > s > > > >>> > >>> This is IMHO more than necessary for Fuel to become a production > ready > >>> serializer and I'd say Fuel is now "old enough" to become such :) > >> > >> Yes. > >> Now what I would love is that even if fuel changes that the > evolution of > >> information > >> is taken into account because like that it will be exercised for > real. > >> > >> > > No, that's impossible, and if posible, it is not worth it. > Migrating from an > > old format to a new one is extremelly complicated and innecessary. The > > easiest way to solve this is to take the correct version of Fuel, > > materialize the graph from the stream, load new version of Fuel, and > > seriaize it again. That the easiest, more secure, and more practical > > approach I can see. > > That is horribly naïve an excludes fuel from a lot of use cases. You > can't use fuel for "archiving" objects outside of the image because you > will never know whether you will be able to read them in again because > the format changes. You will always need to have "live" ones in the > image. > > > No. That's incorrect. You won't be able to do that ONLY if you update > Fuel to a new image that breaks format. > You can still continue to use the same version and you will never have > that problem. So, again, why you need to update Fuel?
Because it's old software. Bugs may not get fixed. It may not work in newer Pharo versions. I may have dependencies on other libraries that may require a new version of Fuel. You name it.
I understand. What I mean is that depending on the changes and the amount of work, you can just adapt Fuel to new versions of Pharo but without changing its format. I mean....say you were in Fuel 1.4. You don't need to move to Fuel 1.8 just because. You can just try to fix 1.4 to make it work in latest pharo.
Oh yeah sure, I can always fork Fuel and maintain it myself. That's exactly what I don't don't want to do.
Well...the thread is starting to be offtopic. I will try once again to explain MY thoguhts: A serializer can evolve in 2 ways: just changing or changing but also making previous versions incompatible. At the same time, a serializer can be in development stage or in maintenance. Fuel is still in development. Even if it is stable in the sense there are no bugs, we are still developing it and we will continue to improve it and change its format. ReferenceSteam is finish, its in maintenance. Now..as an example, Fuel 1.6 is working in Pharo 1.2, 1.3 and 1.4. THat is, 3 mayor versions of Pharo without changing even Fuel version. Isn't that enough for you? do you want to be able to materialize a graph 10 years after with the same version of Fuel? well, then you have to update Fuel to latest version of pharo without changing its format. And that is exactly what Pharo team does with ReferenceStream. Why? because it is just in the image, so if there are methods renamed or removed, ReferneceStream is immediately fixed. What's the difference with Fuel? that it is not in the core, hence not updated by them.
That's EXACTLY what we do with ReferenceStream and friends. The only difference is that the Pharo team does that because it is in the
core.
But yes, it may happen that ideed you will require a new version of Fuel.
Whatâ½ What's the advantage of Fuel over ReferenceStream again?
I don't understand your question, but if you want to know the advantages of Fuel over ReferenceStream, read the Fuel paper.
> Why SmartRefStream does not have this problem? because it
hasn't
> changed in the last 10 years. > So..do the same, take an specific Fuel version and keep it for 10 years. > Just update it to make it work in Pharo without changing the format and > you are done. > > > > That means you can't use fuel for anything Monticello related because > you may never be able to load those versions in again because the file > format has changed in the mean time. > > > I guess that in the end, if someone can really do something for > Monticello on top of Fuel it will be like 2 years from now, and at some > point Fuel format will be stable. > And as Stef says...you always have the code there in case of problems.
Monticello is just an example for a case where I want to store objects outside of the image.
Well, then you can just wait until we finish. Give us one year more. Instead of doing 1-year-long releases, we like to do 3 months releases. But again, that doesn't mean you have to update...
So you're saying I shouldn't use Fuel?
If you plan to serialize now and materialize in 10 years with a new Pharo image and you are not willing to update Fuel to make it work in that Pharo image, then yes, don't use it. Use ReferenceStream and hope that Pharo team keep it updated for 10 years without changing its format.
Cheers Philippe
-- Mariano http://marianopeck.wordpress.com
Mariano I think that the point that Philippe (our national butcher) is trying to make is the following: If by design a serializer takes the perspective that it will be able to load old formats and ensure evolution all the time then you have a really cool serializer. Now why in fuel it is not possible to have FuelMaterializer load: aFile load using the default = current format FuelMaterializer load: aFile asOfVersion: 1.4 Of course this can be much slower but when you want to load data of the past you are just happy that this is working. So would not be possible to milestone formats get the default working fast get old version loading slowler but loading. Because then we get a win win situation. Stef
On Sun, Oct 16, 2011 at 2:03 PM, Stéphane Ducasse <stephane.ducasse@inria.fr
wrote:
Mariano I think that the point that Philippe (our national butcher) is trying to make is the following: If by design a serializer takes the perspective that it will be able to load old formats and ensure evolution all the time then you have a really cool serializer.
Yes, I know. The problem is if it is worth it. The effort of providing such feature could be too much in compare to just load a previous version, materialize, load new version, and serialize again.
Now why in fuel it is not possible to have FuelMaterializer load: aFile
load using the default = current format
FuelMaterializer load: aFile asOfVersion: 1.4 Of course this can be much slower but when you want to load data of the past you are just happy that this is working.
So would not be possible to milestone formats get the default working fast get old version loading slowler but loading.
It sounds easy, but at least from what I can see this is very complicated. Do you know a serializer that support this? Lets clarify with an example: I serialize graph X with Fuel 1.4 in Pharo 1.1. Then I am in Pharo 1.3 with Fuel 1.7 and I want to materialize X. Fuel 1.4 doesn't load anymore in Pharo 1.3, so you have to use Fuel 1.7. So you want to be able to take Fuel 1.7 and materialize X "using version 1.4" without even loading 1.4 (because it doesn't load) ? Thanks
Because then we get a win win situation. Stef
-- Mariano http://marianopeck.wordpress.com
On Sun, Oct 16, 2011 at 3:01 PM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Sun, Oct 16, 2011 at 2:03 PM, Stéphane Ducasse < stephane.ducasse@inria.fr> wrote:
Mariano I think that the point that Philippe (our national butcher) is trying to make is the following: If by design a serializer takes the perspective that it will be able to load old formats and ensure evolution all the time then you have a really cool serializer.
Yes, I know. The problem is if it is worth it. The effort of providing such feature could be too much in compare to just load a previous version, materialize, load new version, and serialize again.
Now why in fuel it is not possible to have FuelMaterializer load: aFile
load using the default = current format
FuelMaterializer load: aFile asOfVersion: 1.4 Of course this can be much slower but when you want to load data of the past you are just happy that this is working.
So would not be possible to milestone formats get the default working fast get old version loading slowler but loading.
It sounds easy, but at least from what I can see this is very complicated. Do you know a serializer that support this? Lets clarify with an example:
(of course, without putting all along the code IFs like (version > 2) ifTrue: [blabla] )
I serialize graph X with Fuel 1.4 in Pharo 1.1. Then I am in Pharo 1.3 with Fuel 1.7 and I want to materialize X. Fuel 1.4 doesn't load anymore in Pharo 1.3, so you have to use Fuel 1.7. So you want to be able to take Fuel 1.7 and materialize X "using version 1.4" without even loading 1.4 (because it doesn't load) ?
Thanks
Because then we get a win win situation. Stef
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
Stef, Mariano, There are obvious advantages to a development phase during which serialized streams are at risk. At some point, I suspect Fuel will have to take the reading of old streams as a nearly sacred obligation. Otherwise, it sounds like a cheap horror movie series: "Fuel II: Your data go in, but they don't come out." I am using humor to make a point, not to ridicule you. I read the paper, and learned a LOT. Please see my post from yesterday referring to Dolphin's #environment. You might be able to simply facade old formats into a coherent backwardly-compatible whole. Without some attention to that, there are going to be projects that will not adopt Fuel. Bill ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Mariano Martinez Peck [marianopeck@gmail.com] Sent: Sunday, October 16, 2011 9:03 AM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize? On Sun, Oct 16, 2011 at 3:01 PM, Mariano Martinez Peck <marianopeck@gmail.com<mailto:marianopeck@gmail.com>> wrote: On Sun, Oct 16, 2011 at 2:03 PM, Stéphane Ducasse <stephane.ducasse@inria.fr<mailto:stephane.ducasse@inria.fr>> wrote: Mariano I think that the point that Philippe (our national butcher) is trying to make is the following: If by design a serializer takes the perspective that it will be able to load old formats and ensure evolution all the time then you have a really cool serializer. Yes, I know. The problem is if it is worth it. The effort of providing such feature could be too much in compare to just load a previous version, materialize, load new version, and serialize again. Now why in fuel it is not possible to have FuelMaterializer load: aFile load using the default = current format FuelMaterializer load: aFile asOfVersion: 1.4 Of course this can be much slower but when you want to load data of the past you are just happy that this is working. So would not be possible to milestone formats get the default working fast get old version loading slowler but loading. It sounds easy, but at least from what I can see this is very complicated. Do you know a serializer that support this? Lets clarify with an example: (of course, without putting all along the code IFs like (version > 2) ifTrue: [blabla] ) I serialize graph X with Fuel 1.4 in Pharo 1.1. Then I am in Pharo 1.3 with Fuel 1.7 and I want to materialize X. Fuel 1.4 doesn't load anymore in Pharo 1.3, so you have to use Fuel 1.7. So you want to be able to take Fuel 1.7 and materialize X "using version 1.4" without even loading 1.4 (because it doesn't load) ? Thanks Because then we get a win win situation. Stef -- Mariano http://marianopeck.wordpress.com -- Mariano http://marianopeck.wordpress.com
On Oct 16, 2011, at 4:05 PM, Schwab,Wilhelm K wrote:
Stef, Mariano,
There are obvious advantages to a development phase during which serialized streams are at risk. At some point, I suspect Fuel will have to take the reading of old streams as a nearly sacred obligation. Otherwise, it sounds like a cheap horror movie series: "Fuel II: Your data go in, but they don't come out."
With Fuel II no zombies :) Fuel III the zombies are dead again :)
I am using humor to make a point, not to ridicule you. I read the paper, and learned a LOT. Please see my post from yesterday referring to Dolphin's #environment. You might be able to simply facade old formats into a coherent backwardly-compatible whole. Without some attention to that, there are going to be projects that will not adopt Fuel.
Bill
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Mariano Martinez Peck [marianopeck@gmail.com] Sent: Sunday, October 16, 2011 9:03 AM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize?
On Sun, Oct 16, 2011 at 3:01 PM, Mariano Martinez Peck <marianopeck@gmail.com<mailto:marianopeck@gmail.com>> wrote:
On Sun, Oct 16, 2011 at 2:03 PM, Stéphane Ducasse <stephane.ducasse@inria.fr<mailto:stephane.ducasse@inria.fr>> wrote: Mariano I think that the point that Philippe (our national butcher) is trying to make is the following: If by design a serializer takes the perspective that it will be able to load old formats and ensure evolution all the time then you have a really cool serializer.
Yes, I know. The problem is if it is worth it. The effort of providing such feature could be too much in compare to just load a previous version, materialize, load new version, and serialize again.
Now why in fuel it is not possible to have FuelMaterializer load: aFile
load using the default = current format
FuelMaterializer load: aFile asOfVersion: 1.4 Of course this can be much slower but when you want to load data of the past you are just happy that this is working.
So would not be possible to milestone formats get the default working fast get old version loading slowler but loading.
It sounds easy, but at least from what I can see this is very complicated. Do you know a serializer that support this? Lets clarify with an example:
(of course, without putting all along the code IFs like (version > 2) ifTrue: [blabla] )
I serialize graph X with Fuel 1.4 in Pharo 1.1. Then I am in Pharo 1.3 with Fuel 1.7 and I want to materialize X. Fuel 1.4 doesn't load anymore in Pharo 1.3, so you have to use Fuel 1.7. So you want to be able to take Fuel 1.7 and materialize X "using version 1.4" without even loading 1.4 (because it doesn't load) ?
Thanks
Because then we get a win win situation. Stef
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
Well...thanks guys. The point is taken. Thanks Philip, Bill and all who gave feedback and suggestions. We are aware of this and we are already discussing with Martin possible solutions and analyzing all the things that has been said in this thread. We will keep you informed. Cheers On Sun, Oct 16, 2011 at 5:14 PM, Stéphane Ducasse <stephane.ducasse@inria.fr
wrote:
On Oct 16, 2011, at 4:05 PM, Schwab,Wilhelm K wrote:
Stef, Mariano,
There are obvious advantages to a development phase during which serialized streams are at risk. At some point, I suspect Fuel will have to take the reading of old streams as a nearly sacred obligation. Otherwise, it sounds like a cheap horror movie series: "Fuel II: Your data go in, but they don't come out."
With Fuel II no zombies :)
Fuel III the zombies are dead again :)
I am using humor to make a point, not to ridicule you. I read the paper, and learned a LOT. Please see my post from yesterday referring to Dolphin's #environment. You might be able to simply facade old formats into a coherent backwardly-compatible whole. Without some attention to that, there are going to be projects that will not adopt Fuel.
Bill
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [ pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Mariano Martinez Peck [marianopeck@gmail.com] Sent: Sunday, October 16, 2011 9:03 AM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize?
On Sun, Oct 16, 2011 at 3:01 PM, Mariano Martinez Peck < marianopeck@gmail.com<mailto:marianopeck@gmail.com>> wrote:
On Sun, Oct 16, 2011 at 2:03 PM, Stéphane Ducasse < stephane.ducasse@inria.fr<mailto:stephane.ducasse@inria.fr>> wrote: Mariano I think that the point that Philippe (our national butcher) is trying to make is the following: If by design a serializer takes the perspective that it will be able to load old formats and ensure evolution all the time then you have a really cool serializer.
Yes, I know. The problem is if it is worth it. The effort of providing such feature could be too much in compare to just load a previous version, materialize, load new version, and serialize again.
Now why in fuel it is not possible to have FuelMaterializer load: aFile
load using the default = current format
FuelMaterializer load: aFile asOfVersion: 1.4 Of course this can be much slower but when you want to load data of the past you are just happy that this is working.
So would not be possible to milestone formats get the default working fast get old version loading slowler but loading.
It sounds easy, but at least from what I can see this is very complicated. Do you know a serializer that support this? Lets clarify with an example:
(of course, without putting all along the code IFs like (version > 2) ifTrue: [blabla] )
I serialize graph X with Fuel 1.4 in Pharo 1.1. Then I am in Pharo 1.3 with Fuel 1.7 and I want to materialize X. Fuel 1.4 doesn't load anymore in Pharo 1.3, so you have to use Fuel 1.7. So you want to be able to take Fuel 1.7 and materialize X "using version 1.4" without even loading 1.4 (because it doesn't load) ?
Thanks
Because then we get a win win situation. Stef
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
My 2 cents: i understand that people want to have backwards compatibility, so they could read even 10 years old data from files. now, it could mean a lot of effort to do that. For instance, if your binary model is spawned along 30 classes, then in order to support 2 versions of it, you need another 30 classes.. 3 versions - 90 classes, and so on. So i am thinking that a least painful way could be to create a migration or "upgrade" scripts which people can use to update their data from old format to new one, and put it in a separate package, to not burden a main development line with cruft. Then it will be like a patch system. This is how i imagine it: suppose that you having 2.0 version installed in your system, but want to load 1.6 version data. you load an additional package(s) titled like: migrating 1.6 -> 1.7 migrating 1.7 -> 1.8 migrating 1.8 -> 2.0 and then run migration scripts for your data until you get to the current version. Once you done, you can unload this stuff and be happy. And i think it is also cheaper for developers: it takes an effort to write a new migration procedure every time you changing a format, but you don't have to keep everything in one place and change a model to be compatible with everything. A migration scripts should do their work, while core model and implementation can be kept clean from cruft. -- Best regards, Igor Stasenko.
Mariano Martinez Peck wrote:
Well...thanks guys. The point is taken. Thanks Philip, Bill and all who gave feedback and suggestions. We are aware of this and we are already discussing with Martin possible solutions and analyzing all the things that has been said in this thread. We will keep you informed.
Cheers Just one thing I hadn't seen discussed. Is Fuel possibly a way to exchange objects/data between users/systems ? In which case it would have to deal with different systems having different versions of Fuel.
cheers, Ben
On Thu, Oct 20, 2011 at 12:54 PM, Ben Coman <btc@openinworld.com> wrote:
Mariano Martinez Peck wrote:
Well...thanks guys. The point is taken. Thanks Philip, Bill and all who gave feedback and suggestions. We are aware of this and we are already discussing with Martin possible solutions and analyzing all the things that has been said in this thread. We will keep you informed.
Cheers
Just one thing I hadn't seen discussed. Is Fuel possibly a way to exchange objects/data between users/systems ?
Yes
In which case it would have to deal with different systems having different versions of Fuel.
Exactly. It doesn't matter whether the different versions of Fuel happens in your own image or in different systems. So yes, it is the same problem. Note this is not only a problem for Fuel, but for any serializer which does not manages full backward compatibility between versions. -- Mariano http://marianopeck.wordpress.com
On Thu, Oct 20, 2011 at 1:00 PM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Thu, Oct 20, 2011 at 12:54 PM, Ben Coman <btc@openinworld.com> wrote:
Mariano Martinez Peck wrote:
Well...thanks guys. The point is taken. Thanks Philip, Bill and all who gave feedback and suggestions. We are aware of this and we are already discussing with Martin possible solutions and analyzing all the things that has been said in this thread. We will keep you informed.
Cheers
Just one thing I hadn't seen discussed. Is Fuel possibly a way to exchange objects/data between users/systems ?
Yes
But it doesn't mean it is the best option. Maybe a XML or JSON serializer is better there...
In which case it would have to deal with different systems having different
versions of Fuel.
Exactly. It doesn't matter whether the different versions of Fuel happens in your own image or in different systems. So yes, it is the same problem. Note this is not only a problem for Fuel, but for any serializer which does not manages full backward compatibility between versions.
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
On 20 October 2011 13:01, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Thu, Oct 20, 2011 at 1:00 PM, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Thu, Oct 20, 2011 at 12:54 PM, Ben Coman <btc@openinworld.com> wrote:
Mariano Martinez Peck wrote:
Well...thanks guys. The point is taken. Thanks Philip, Bill and all who gave feedback and suggestions. We are aware of this and we are already discussing with Martin possible solutions and analyzing all the things that has been said in this thread. We will keep you informed.
Cheers
Just one thing I hadn't seen discussed. Â Is Fuel possibly a way to exchange objects/data between users/systems ?
Yes
But it doesn't mean it is the best option. Maybe a XML or JSON serializer is better there...
If i want speed, i can live with non-portable binary format. If i want portability, compatibility and other blah blah then yeah.. one can always use textual representation of object graph, but don't cry about being slow anymore.
In which case it would have to deal with different systems having different versions of Fuel.
Exactly. It doesn't matter whether the different versions of Fuel happens in your own image or in different systems. So yes, it is the same problem. Note this is not only a problem for Fuel, but for any serializer which does not manages full backward compatibility between versions.
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
-- Best regards, Igor Stasenko.
Mariano, I gave you a pretty tough time :) It appears that you understand that I did so with the best of intentions. Thanks for listening. One problem that I do *not* think you need to solve is that of old software talking to new software. Someone wanting forward compatibility is probably going to use XML or something similar to get a timeless protocol that will be of necessity very simple. A good serializer is hard to write, and forward-compatibility is probably too much to ask, particularly when efficiency is a stated goal. A backwardly-compatible Fuel will be most welcome and useful. Bill ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Mariano Martinez Peck [marianopeck@gmail.com] Sent: Thursday, October 20, 2011 7:00 AM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize? On Thu, Oct 20, 2011 at 12:54 PM, Ben Coman <btc@openinworld.com<mailto:btc@openinworld.com>> wrote: Mariano Martinez Peck wrote: Well...thanks guys. The point is taken. Thanks Philip, Bill and all who gave feedback and suggestions. We are aware of this and we are already discussing with Martin possible solutions and analyzing all the things that has been said in this thread. We will keep you informed. Cheers Just one thing I hadn't seen discussed. Is Fuel possibly a way to exchange objects/data between users/systems ? Yes In which case it would have to deal with different systems having different versions of Fuel. Exactly. It doesn't matter whether the different versions of Fuel happens in your own image or in different systems. So yes, it is the same problem. Note this is not only a problem for Fuel, but for any serializer which does not manages full backward compatibility between versions. -- Mariano http://marianopeck.wordpress.com
Hi I followed with interest the discussion but without many interventions. I want to explain an idea I have for supporting backward-compatibility without neither stabilizing the format nor getting the current code dirty. Maybe it's a bit crazy but I think it can be a practical solution. I would like to know your opinions. We can offer: - Maintain a *compatibility package* for each released version, that can co-exist in the same image. For example Fuel1_5, Fuel1_6, etc. Obviously, the current version always has the nice clean name (Fuel). - Provide a facade that works like "Fuel load: aFile asOfVersion: '1.4'" who interacts with the compatibility packages. To generate each *compatibility package*, we rename several elements of the original Fuel released package, this way: * Extension methods: Append at beggining of each extension method selector a prefix. For example: #fuelAccept: --> #fuel1_6_fuelAccept: * Classes: Append to each one some version identifier. For example: FLSerializer --> FLSerializer_1_6 * Methods: Fix each reference to a class or extension method selector THAT BELONGS to the package. For example: "FLSerializer new" --> "FLSerializer_1_6 new". I think it could be not that hard to make such renames automatically with a small tool. Then, we can simultaneously maintain each *compatibility package* (Fuel1_5, Fuel1_6, etc.) when Pharo evolves: Our Jenkins will run tests for all the compatibility Fuel packages. For example... Does Fuel1_7 stop working in Pharo1.5? then (if it's possible) we fix it, save the monticello version and mark it in our ConfigurationOfFuel as "this is the stable monticello version of fuel 1.7 in pharo 1.5". Now that I see how many lines I took to explain it, I think it's a bit insane. I hope to read some opinions! Bests, MartÃn On Thu, Oct 20, 2011 at 6:00 PM, Schwab,Wilhelm K <bschwab@anest.ufl.edu>wrote:
Mariano,
I gave you a pretty tough time :) It appears that you understand that I did so with the best of intentions. Thanks for listening.
One problem that I do *not* think you need to solve is that of old software talking to new software. Someone wanting forward compatibility is probably going to use XML or something similar to get a timeless protocol that will be of necessity very simple. A good serializer is hard to write, and forward-compatibility is probably too much to ask, particularly when efficiency is a stated goal.
A backwardly-compatible Fuel will be most welcome and useful.
Bill
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [ pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Mariano Martinez Peck [marianopeck@gmail.com] Sent: Thursday, October 20, 2011 7:00 AM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] When will Fuel file format stabilize?
On Thu, Oct 20, 2011 at 12:54 PM, Ben Coman <btc@openinworld.com<mailto: btc@openinworld.com>> wrote: Mariano Martinez Peck wrote: Well...thanks guys. The point is taken. Thanks Philip, Bill and all who gave feedback and suggestions. We are aware of this and we are already discussing with Martin possible solutions and analyzing all the things that has been said in this thread. We will keep you informed.
Cheers Just one thing I hadn't seen discussed. Is Fuel possibly a way to exchange objects/data between users/systems ?
Yes
In which case it would have to deal with different systems having different versions of Fuel.
Exactly. It doesn't matter whether the different versions of Fuel happens in your own image or in different systems. So yes, it is the same problem. Note this is not only a problem for Fuel, but for any serializer which does not manages full backward compatibility between versions.
-- Mariano http://marianopeck.wordpress.com
It sounds easy, but at least from what I can see this is very complicated. Do you know a serializer that support this? Lets clarify with an example:
(of course, without putting all along the code IFs like (version > 2) ifTrue: [blabla] )
of course we are smalltalkers no? :) It means that for the slow version you have to get a format representation explicit.
I serialize graph X with Fuel 1.4 in Pharo 1.1. Then I am in Pharo 1.3 with Fuel 1.7 and I want to materialize X. Fuel 1.4 doesn't load anymore in Pharo 1.3, so you have to use Fuel 1.7. So you want to be able to take Fuel 1.7 and materialize X "using version 1.4" without even loading 1.4 (because it doesn't load) ?
Thanks
Because then we get a win win situation. Stef
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
Yes, I know. The problem is if it is worth it. The effort of providing such feature could be too much in compare to just load a previous version, materialize, load new version, and serialize again.
for a serializer I think that it is worth it. I once used a backup system which used a private format for my archives and I lost them since it was discontinued....
Now why in fuel it is not possible to have FuelMaterializer load: aFile
load using the default = current format
FuelMaterializer load: aFile asOfVersion: 1.4 Of course this can be much slower but when you want to load data of the past you are just happy that this is working.
So would not be possible to milestone formats get the default working fast get old version loading slowler but loading.
It sounds easy, but at least from what I can see this is very complicated. Do you know a serializer that support this? Lets clarify with an example:
I serialize graph X with Fuel 1.4 in Pharo 1.1. Then I am in Pharo 1.3 with Fuel 1.7 and I want to materialize X. Fuel 1.4 doesn't load anymore in Pharo 1.3, so you have to use Fuel 1.7. So you want to be able to take Fuel 1.7 and materialize X "using version 1.4" without even loading 1.4 (because it doesn't load) ?
Yes what I want to say is that if evolution is the key then you have to be explicit about milestones and version and in the end this is excellent. But of course you get a cost and effort. But this means that after that there is no more other serializer. I think that it is also a nice way to get meta about the format. Stef
Thanks
Because then we get a win win situation. Stef
-- Mariano http://marianopeck.wordpress.com
On 10/16/2011 02:03 PM, Stéphane Ducasse wrote:
Mariano I think that the point that Philippe (our national butcher) is trying to make is the following: If by design a serializer takes the perspective that it will be able to load old formats and ensure evolution all the time then you have a really cool serializer.
Now why in fuel it is not possible to have FuelMaterializer load: aFile
load using the default = current format
FuelMaterializer load: aFile asOfVersion: 1.4 Of course this can be much slower but when you want to load data of the past you are just happy that this is working.
Not exactly but almost. Put the version in the format and automatically load it using the right class (it has already been written, just don't delete it). This also allows to find out whether the format is too new. Not being able to serialize in an older format would be fine as long as you can still load them. Cheers Philippe
2011/10/8 Janko Mivšek <janko.mivsek@eranova.si>
Hi guys,
We are starting to use Fuel to exchange data daily, but without exactly the same version of Fuel loaded, exported data is just too frequently not importable anymore.
Yes, that's true. We are changing its format all the time.
And exception report is meaningless, you can only guess that Fuel version is to blame.
Yes, thanks for reporting this. In fact, we have already fix it yesterday ;) Now, if you try to materialize with a different version, you will have a FLBadVersion exception. Check the tests FLVersionTest and FLSignatureTest for more details.
So, it is more that a time to:
- stabilize and freeze the file format
We cannot. Let me just give you my thoughts: - It is a pity that we cannot make progress just because we break the format. If improving means breaking the format, we will do it. - The main question is WHY you need to update Fuel all the time? which are the reasons behind that? I have only 3 things in mind: a) because Fuel is too slow for you and hence you need a newer/faster version b) because you have changed Pharo version and Fuel doesn't work anymore there with the same version? c) because you found a bug. a) looks unlikely. I think Fuel was fast enough for most scenarios from the beginning even if new versions are faster. b) is unlikely also. For example, the last version (1.6) works out of the box in Pharo 1.2, 1.3 and 1.4. So it means that even between 3 versions of Pharo you don't need to update Fuel. c) we haven't found so far a critical bug which would requires to update Fuel. So...my question is, WHY you need to always use a new version of Fuel and why you cannot stay with a stable/fixed Metacello version? That's why we have metecello configurations. You can stick with version 1.6 (or whatever version) and just migrate when it is really necessary. For example, Francois Stephany is using Fuel for his website, and he is still using Fuel 1.4. He didn't need to update yet, because there aren't bugs and it is fast enough for what he needs. - put a version info into a file and check it while importing
Done. Thanks for the suggestion.
- meaningful error reporting in such cases
Done also :)
This is IMHO more than necessary for Fuel to become a production ready
I totally agree.
serializer and I'd say Fuel is now "old enough" to become such :)
heheheh yes. It is already more than year year old hhaa. BTW...yesterday we have finished with Martin the version 1.7 which we will release probably monday or tuesday. Cheers
Best regards Janko
-- Janko Mivšek Svetovalec za informatiko Eranova d.o.o. Ljubljana, Slovenija www.eranova.si tel: 01 514 22 55 faks: 01 514 22 56 gsm: 031 674 565
-- Mariano http://marianopeck.wordpress.com
participants (15)
-
Ben Coman -
btc＠openInWorld.com
-
Igor Stasenko -
Janko Mivšek -
Javier Pimás -
Mariano Martinez Peck -
Martin Dias -
Michael Roberts -
Nicolas Cellier -
Philippe Marschall -
Philippe Marschall -
Schwab,Wilhelm K -
Stefan Marr -
Stéphane Ducasse -
Yanni Chiu