Just curious... why don't you use MSE? Peter On Wed, May 17, 2017 at 12:44:58PM +0200, Sven Van Caekenberghe wrote:

Hi Cyril,

This can be done quite easily in STON, I will make you a small example and add it to the unit tests.

Sven

...

On 17 May 2017, at 12:19, Cyril Ferlicot D. <cyril.ferlicot@gmail.com> wrote:

Hi!

At Synectique sometimes we use Ston to export models that contains some MooseEntities.

Moose entities been complex, we usually export only the moose name of the entity then we retrieve the entity during the import. But we did not find any "easy" way to do this in Ston.

I used voyage and it has a system like this. Example:

Amount class>>mongoCurrency <mongoDescription>

^ VOMongoToOneDescription new attributeName: 'currency'; accessor: (MAPluggableAccessor read: [ :amount | amount currency abbreviation ] write: [ :amount :value | amount currency: (Currency fromAbbreviation: value) ]); yourself

With this Voyage know how to read/write the instance variable "currency".

I think it would be cool to be able to easily customize import/export variables with Ston. Maybe it is already possible but I missed the way to do it?

I would like your thoughts about that. :)

-- Cyril Ferlicot https://ferlicot.fr

http://www.synectique.eu 2 rue Jacques Prévert 01, 59650 Villeneuve d'ascq France

On 17 May 2017, at 12:44, Sven Van Caekenberghe <sven@stfx.eu> wrote:

Hi Cyril,

This can be done quite easily in STON, I will make you a small example and add it to the unit tests.

=== Name: STON-Tests-SvenVanCaekenberghe.79 Author: SvenVanCaekenberghe Time: 17 May 2017, 1:43:03.944446 pm UUID: fb472e87-180c-0d00-b559-fec60aee7555 Ancestors: STON-Tests-SvenVanCaekenberghe.78 Add STONWriteReadTests>>#testKnownObjects as an example of how to serialize only an ID and resolve that ID upon materialization Add STONTestKnownObject with custom #stonOn: and class side #fromSton: === You basically (over)write 2 methods, instance side "protocol: ston-core" stonOn: stonWriter "We only write out our id" stonWriter writeObject: self listSingleton: self id asString and class side "protocol: ston-core" fromSton: stonReader ^ self fromId: stonReader parseListSingleton The #fromId: methods retrieves an existing instance. HTH, Sven

Sven

...

On 17 May 2017, at 12:19, Cyril Ferlicot D. <cyril.ferlicot@gmail.com> wrote:

Hi!

At Synectique sometimes we use Ston to export models that contains some MooseEntities.

Moose entities been complex, we usually export only the moose name of the entity then we retrieve the entity during the import. But we did not find any "easy" way to do this in Ston.

I used voyage and it has a system like this. Example:

Amount class>>mongoCurrency <mongoDescription>

^ VOMongoToOneDescription new attributeName: 'currency'; accessor: (MAPluggableAccessor read: [ :amount | amount currency abbreviation ] write: [ :amount :value | amount currency: (Currency fromAbbreviation: value) ]); yourself

With this Voyage know how to read/write the instance variable "currency".

I think it would be cool to be able to easily customize import/export variables with Ston. Maybe it is already possible but I missed the way to do it?

I would like your thoughts about that. :)

-- Cyril Ferlicot https://ferlicot.fr

http://www.synectique.eu 2 rue Jacques Prévert 01, 59650 Villeneuve d'ascq France

On 17 May 2017, at 14:15, Cyril Ferlicot D. <cyril.ferlicot@gmail.com> wrote:

On 17/05/2017 13:46, Sven Van Caekenberghe wrote:

...

...

On 17 May 2017, at 12:44, Sven Van Caekenberghe <sven@stfx.eu> wrote:

Hi Cyril,

This can be done quite easily in STON, I will make you a small example and add it to the unit tests.

=== Name: STON-Tests-SvenVanCaekenberghe.79 Author: SvenVanCaekenberghe Time: 17 May 2017, 1:43:03.944446 pm UUID: fb472e87-180c-0d00-b559-fec60aee7555 Ancestors: STON-Tests-SvenVanCaekenberghe.78

Add STONWriteReadTests>>#testKnownObjects as an example of how to serialize only an ID and resolve that ID upon materialization

Add STONTestKnownObject with custom #stonOn: and class side #fromSton: ===

You basically (over)write 2 methods, instance side

"protocol: ston-core" stonOn: stonWriter "We only write out our id"

stonWriter writeObject: self listSingleton: self id asString

and class side

"protocol: ston-core" fromSton: stonReader ^ self fromId: stonReader parseListSingleton

The #fromId: methods retrieves an existing instance.

HTH,

Sven

Hi Sven,

I do not want to keep all the KnowObjects in the memory. One of the goal is to export from an image and to load from another.

The following does not work:

| obj string| obj := STONTestKnownObject new description: 'something else'; yourself. string := STON toStringPretty: obj. STONTestKnownObject resetKnownObjects. self assert: (STON fromString: string) description = 'something else'

Of course that does not work, it is your job to do something similar that fits your use case. You certainly do not have to cache everything. On the other had, if you only serialise part of an object, or just one reference (id, url, name) like in the example, how will you recreate the rest, unless you have access to it ? And that is what your version of #fromId: should do. I can't understand how your 'to export from an image and to load from another' can possibly work, unless there is some other way of sharing the data. But I probably don't understand you, I known nothing about Moose models.

What is the simple may to customize only one or two variable(s)?

-- Cyril Ferlicot https://ferlicot.fr

http://www.synectique.eu 2 rue Jacques Prévert 01, 59650 Villeneuve d'ascq France

On 17 May 2017, at 14:38, Cyril Ferlicot D. <cyril.ferlicot@gmail.com> wrote:

On 17/05/2017 14:22, Sven Van Caekenberghe wrote:

...

Of course that does not work, it is your job to do something similar that fits your use case. You certainly do not have to cache everything.

On the other had, if you only serialise part of an object, or just one reference (id, url, name) like in the example, how will you recreate the rest, unless you have access to it ? And that is what your version of #fromId: should do.

I can't understand how your 'to export from an image and to load from another' can possibly work, unless there is some other way of sharing the data.

But I probably don't understand you, I known nothing about Moose models.

What I would like to know is if there is a simple way to customize the export of only one or two of the variables. And let the other variables be exported as usual.

The use of #stonAllInstVarNames works in limiting what you write out (there is even a unit test based on STONTestUser3). But you seem to want to do something else, you want to write the others ones out in a different way. What I tried to explain is that that is generally not needed, if you customise the value of #variableToCustomise (that is one level deeper). This is what I did in the example. Since you can't or won't explain your use case better, I really can't help, can I ?

I think I can do something like (I did not test but the spirit is here):

myObject class>>stonAllInstVarNames ^ super stonAllInstVarNames \ { #variableToCustomise }

myObject>>stonOn: stonWriter

stonWriter writeObject: self streamMap: [ :dictionary | self class stonAllInstVarNames do: [ :each | (self instVarNamed: each) ifNotNil: [ :value | dictionary at: each asSymbol put: value ] ifNil: [ self stonShouldWriteNilInstVars ifTrue: [ dictionary at: each asSymbol put: nil ] ] ]. dictionary at: #variableToCustomize put: (self instVarNamed: #variableToCustomize) asString. ]

myObject class>>fromSton: stonReader

| obj | obj := super fromSton: stonReader. obj variableToCustomize: (UUID fromString: obj variableToCustomize) ^ obj

But this imply to rewrite inside #stonOn: a part of the code used in the super stonOn: and I don't really like this in general.

So I wanted to know if there was a hook to customize certain variables while exporting the other as usual.

-- Cyril Ferlicot https://ferlicot.fr

http://www.synectique.eu 2 rue Jacques Prévert 01, 59650 Villeneuve d'ascq France

On 17 May 2017, at 15:11, Cyril Ferlicot D. <cyril.ferlicot@gmail.com> wrote:

On 17/05/2017 14:55, Sven Van Caekenberghe wrote:

...

The use of #stonAllInstVarNames works in limiting what you write out (there is even a unit test based on STONTestUser3).

But you seem to want to do something else, you want to write the others ones out in a different way.

What I tried to explain is that that is generally not needed, if you customise the value of #variableToCustomise (that is one level deeper). This is what I did in the example.

Since you can't or won't explain your use case better, I really can't help, can I ?

I don't really have the time right now but later I'll try to write a simple example.

OK. But don't go too far in writing actual code. I want to understand the use case at a high level (like how you started), then I can try to find how/if STON can do what you want. Right now, you are already too deep into a possible solution.

-- Cyril Ferlicot https://ferlicot.fr

http://www.synectique.eu 2 rue Jacques Prévert 01, 59650 Villeneuve d'ascq France

On 17 May 2017, at 15:47, Cyril Ferlicot D. <cyril.ferlicot@gmail.com> wrote:

On 17/05/2017 15:17, Sven Van Caekenberghe wrote:

...

OK.

But don't go too far in writing actual code. I want to understand the use case at a high level (like how you started), then I can try to find how/if STON can do what you want. Right now, you are already too deep into a possible solution.

If I have to give a really simple use case.

Imagine an invoice.

The invoice has the name of the customer and the currency used as a class.

During the export of the invoice you do not want to export the currency as a class but as a String.

So if I have an Invoice instance like this:

username: 'user' currency: Currency[Euro]

I want to transform the currency to a string during the export. (and keep the username as strings).

Result:

Invoice { #username : 'user', #currency : 'eur' }

Instead of:

Invoice { #username : 'user', #currency : Currency { #unit: 'Euro' } }

Then during the import I want to transform the currency into a class again. (For example `Currency fromAbbreviation: stonInput`).

OK, that is an understandable example. But why exactly do you want Currency to be serialised differently ? You don't want too many instances ? You want all instances to be #== ? Is Currency too big ? You want to allow humans to edit the STON file ? What ? Why ? You can easily make it Currency['Euro'] or even Currency[#Euro] like it is already done for a number of built-in classes. The last example that I gave solved the 'too many instances', make them #== issue. If it is size, that got covered too, but you must solve the problem of how the receiving end will resolve the reference then. But since that is all no good, there must still be another requirement.

-- Cyril Ferlicot https://ferlicot.fr

http://www.synectique.eu 2 rue Jacques Prévert 01, 59650 Villeneuve d'ascq France

On 17 May 2017, at 16:15, Cyril Ferlicot D. <cyril.ferlicot@gmail.com> wrote:

On 17/05/2017 16:02, Sven Van Caekenberghe wrote:

...

OK, that is an understandable example.

But why exactly do you want Currency to be serialised differently ? You don't want too many instances ? You want all instances to be #== ? Is Currency too big ? You want to allow humans to edit the STON file ? What ? Why ?

You can easily make it Currency['Euro'] or even Currency[#Euro] like it is already done for a number of built-in classes.

The last example that I gave solved the 'too many instances', make them #== issue.

If it is size, that got covered too, but you must solve the problem of how the receiving end will resolve the reference then.

But since that is all no good, there must still be another requirement.

Here it is a simple example to show what I want.

The real use case is that in Moose, entities are really connected to their model, and if one of our classes contains an entity it would produce a really really really big ston file since it would export millions of entities.

But there is a way to retrieve an entity of this model from its moose name.

Thus, I would like to export only the moose name of the entities, then get the corresponding entity from a Model the user would have load before during the ston reading.

OK. So there is some existing method that gives you the entity given its name ? MooseEntity class>>fromEntityName: 'mooseEntityName' Right ? So why can't you use the example that I gave you earlier, replace the UUID string with the mooseEntityName string, and call the above method instead of the #fromId: in my example ? In STON, you will then see MooseEntity['mooseEntityName'] as value.

...

-- Cyril Ferlicot https://ferlicot.fr

http://www.synectique.eu 2 rue Jacques Prévert 01, 59650 Villeneuve d'ascq France

On 17 May 2017, at 16:27, Cyril Ferlicot D. <cyril.ferlicot@gmail.com> wrote:

On 17/05/2017 16:25, Sven Van Caekenberghe wrote:

...

OK.

So there is some existing method that gives you the entity given its name ?

MooseEntity class>>fromEntityName: 'mooseEntityName'

Right ?

So why can't you use the example that I gave you earlier, replace the UUID string with the mooseEntityName string, and call the above method instead of the #fromId: in my example ?

In STON, you will then see

MooseEntity['mooseEntityName']

as value.

But it would not export/import all the other variables no?

You change the representation of MooseEntity to the above short form, then you just use MooseEntity instances inside other objects. All other instance variables of MooseEntity will be skipped. Just like Date, Time, or DateAndTime have short representations, that you see when you include these as values inside other objects. We must be talking about something different because I really don't see the problem.

...

-- Cyril Ferlicot https://ferlicot.fr

http://www.synectique.eu 2 rue Jacques Prévert 01, 59650 Villeneuve d'ascq France

Hi, Thanks for this interesting conversation. I had some similar problem with Grafoscopio notebooks being to big when I run the default serializer because it puts data about the Text object and some other info about runs. What I did was to flat the tree to contain only the info I need. I solved this with two methods (listed below). Maybe this can be helpful: GrafoscopioNotebook>>exportAsSton: aNotebook on: aFileStream | stonPrettyString | aNotebook flatten. stonPrettyString := String streamContents: [ :stream | (STON writer on: stream) newLine: String crlf; prettyPrint: true; keepNewLines: true; nextPut: aNotebook children]. aFileStream nextPutAll: stonPrettyString GrafoscopioNode>>flatten "I traverse the tree looking for node bodies containing 'Text' objects and transform them to their string content, so space is saved and storage format is DVCS friendly while serializing them to STON" (self preorderTraversal) do: [ :eachNode | (eachNode body class = Text) ifTrue: [eachNode body: (eachNode body asString)]] Cheers, Offray On 17/05/17 09:15, Cyril Ferlicot D. wrote:

On 17/05/2017 16:02, Sven Van Caekenberghe wrote:

...

OK, that is an understandable example.

But why exactly do you want Currency to be serialised differently ? You don't want too many instances ? You want all instances to be #== ? Is Currency too big ? You want to allow humans to edit the STON file ? What ? Why ?

You can easily make it Currency['Euro'] or even Currency[#Euro] like it is already done for a number of built-in classes.

The last example that I gave solved the 'too many instances', make them #== issue.

If it is size, that got covered too, but you must solve the problem of how the receiving end will resolve the reference then.

But since that is all no good, there must still be another requirement. Here it is a simple example to show what I want.

The real use case is that in Moose, entities are really connected to their model, and if one of our classes contains an entity it would produce a really really really big ston file since it would export millions of entities.

But there is a way to retrieve an entity of this model from its moose name.

Thus, I would like to export only the moose name of the entities, then get the corresponding entity from a Model the user would have load before during the ston reading.

...

On 19 May 2017, at 10:39, Offray Vladimir Luna Cárdenas <offray.luna@mutabit.com> wrote:

Hi,

Thanks for this interesting conversation. I had some similar problem with Grafoscopio notebooks being to big when I run the default serializer because it puts data about the Text object and some other info about runs. What I did was to flat the tree to contain only the info I need. I solved this with two methods (listed below). Maybe this can be helpful:

GrafoscopioNotebook>>exportAsSton: aNotebook on: aFileStream | stonPrettyString | aNotebook flatten. stonPrettyString := String streamContents: [ :stream | (STON writer on: stream) newLine: String crlf; prettyPrint: true; keepNewLines: true; nextPut: aNotebook children]. aFileStream nextPutAll: stonPrettyString

Any reason why you first write to a string and then directly put that string on aFileStream ? You can directly write STON on aFileStream, which is more memory efficient, as in (STON writer on: aFileStream) If speed is important, it is best to wrap a ZnBufferedWriteStream around it.

GrafoscopioNode>>flatten "I traverse the tree looking for node bodies containing 'Text' objects and transform them to their string content, so space is saved and storage format is DVCS friendly while serializing them to STON"

(self preorderTraversal) do: [ :eachNode | (eachNode body class = Text) ifTrue: [eachNode body: (eachNode body asString)]]

Cheers,

Offray

On 17/05/17 09:15, Cyril Ferlicot D. wrote:

...

On 17/05/2017 16:02, Sven Van Caekenberghe wrote:

...

OK, that is an understandable example.

But why exactly do you want Currency to be serialised differently ? You don't want too many instances ? You want all instances to be #== ? Is Currency too big ? You want to allow humans to edit the STON file ? What ? Why ?

You can easily make it Currency['Euro'] or even Currency[#Euro] like it is already done for a number of built-in classes.

The last example that I gave solved the 'too many instances', make them #== issue.

If it is size, that got covered too, but you must solve the problem of how the receiving end will resolve the reference then.

But since that is all no good, there must still be another requirement. Here it is a simple example to show what I want.

The real use case is that in Moose, entities are really connected to their model, and if one of our classes contains an entity it would produce a really really really big ston file since it would export millions of entities.

But there is a way to retrieve an entity of this model from its moose name.

Thus, I would like to export only the moose name of the entities, then get the corresponding entity from a Model the user would have load before during the ston reading.

...