STON updates/improvements
Hi, I have consolidated all repositories where STON code lives so that they are now all in sync, and in sync with changes from Pharo 7. http://ss3.gemtalksystems.com/ss/STON/ http://smalltalkhub.com/mc/SvenVanCaekenberghe/STON/main/ https://github.com/svenvc/ston There are 2 CI builds https://travis-ci.org/svenvc/ston https://ci.inria.fr/pharo-contribution/job/STON/ The project's format will remain FileTree (until Tonel is supported on other Smalltalk implementations). Last week I applied a couple of changes that I wanted to apply for a long time. Traits are no longer used in the implementation (which should help porting to other Smalltalk implementations). More specifically the following are used (again) - Class>>#stonOn: - ClassDescription>>#stonContainsSubObjects - Metaclass>>#stonName - Metaclass>>#stonOn: instead of - TClass>>#stonOn: - TClassDescription>>#stonContainsSubObjects - TApplyingOnClassSide>>#stonName - TApplyingOnClassSide>>#stonOn: use #instanceSide instead of #theNonMetaClass in MetaClass>>#stonOn: Reorganised the packages with sub tags. Add support for Fraction and ScaledDecimal literals (not in JSON mode). Previously a float conversion meant there was a loss of information. Change the representation of Date to include a timezone offset (since the current Date implementation is sensitive to this). 2018-10-11Z 2018-10-11+00:00 2018-10-11+02:00 A missing timezone offset is interpreted as being the local timezone offset. Add more special representations for common value style objects. One of STON's goals is to be a human readable and human editable representation of an object graph while remaining 100% correct (not losing information). The following were added for this reason: MimeType and URL using ZnMimeType and ZnUrl respectively as simplified values. MimeType['application/json'] URL['https://pharo.org'] Color Color[#red] Color{#red:1.0,#green:0.0,#blue:0.0,#alpha:0.4} FileReferences to the DiskFileSystem (effectively normal files) FILE['/tmp/foo.txt'] Here is an example of how much difference that makes. Given the following Dictionary { #background->Color red. #workdir->'/tmp/pharo/work/' asFileReference. #home->'https://pharo.org/experimental' asUrl. #datatype->'application/json' asMIMEType } asDictionary Which contains real objects as its values. was serialised by STON BEFORE the changes as { #datatype : ZnMimeType { #main : 'application', #sub : 'json' }, #background : Color { #rgb : 1072693248, #cachedDepth : 32, #cachedBitPattern : Bitmap [ 4294901760 ], #alpha : 255 }, #home : ZnUrl { #scheme : #https, #host : 'pharo.org', #segments : OrderedCollection [ 'experimental' ] }, #workdir : FileReference { #filesystem : FileSystem { #store : MacStore { #maxFileNameLength : 255 } }, #path : AbsolutePath [ 'tmp', 'pharo', 'work' ] } } which is 100% correct, but not very human friendly. Now, AFTER the changes, the STON serialisation looks as follows: { #datatype : MimeType [ 'application/json' ], #background : Color [ #red ], #home : URL [ 'https://pharo.org/experimental' ], #workdir : FILE [ '/tmp/pharo/work' ] } which is a huge improvement, IMO. What do you think ? Comments, feedback, remarks ? Any other suggestions for other object that could use this treatment ? Sven
Thanks Sven :) On Fri, Oct 12, 2018 at 11:38 AM Sven Van Caekenberghe <sven@stfx.eu> wrote:
Now, AFTER the changes, the STON serialisation looks as follows:
{ #datatype : MimeType [ 'application/json' ], #background : Color [ #red ], #home : URL [ 'https://pharo.org/experimental' ], #workdir : FILE [ '/tmp/pharo/work' ] }
which is a huge improvement, IMO.
Yes, I've suffered the things you explain before :)
What do you think ? Comments, feedback, remarks ? Any other suggestions for other object that could use this treatment ?
Question: is it a new mechanism or it is based on the existing mapping mechanisms? Can we override it/extend it?
On 12 Oct 2018, at 11:49, Guillermo Polito <guillermopolito@gmail.com> wrote:
Thanks Sven :)
On Fri, Oct 12, 2018 at 11:38 AM Sven Van Caekenberghe <sven@stfx.eu> wrote: Now, AFTER the changes, the STON serialisation looks as follows:
{ #datatype : MimeType [ 'application/json' ], #background : Color [ #red ], #home : URL [ 'https://pharo.org/experimental' ], #workdir : FILE [ '/tmp/pharo/work' ] }
which is a huge improvement, IMO.
Yes, I've suffered the things you explain before :)
What do you think ? Comments, feedback, remarks ? Any other suggestions for other object that could use this treatment ?
Question: is it a new mechanism or it is based on the existing mapping mechanisms? Can we override it/extend it?
This is using the existing mechanism (with specific #stonName's that differ from the class name). Several class already used this mechanism (the Chronology classes for example). In the case of FileReference it is a bit more of a hack: FileReferences to other filesystems than disk are serialised as before. Furthermore I had to use FILE instead of File since that class already exists. On other platforms, other concrete classes might handle the same tags. It wouldn't hurt if others thought a bit about this too. In any case, I intend to work a bit on documentation later on.
On ven. 12 oct. 2018 at 11:38, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
I have consolidated all repositories where STON code lives so that they are now all in sync, and in sync with changes from Pharo 7.
http://ss3.gemtalksystems.com/ss/STON/ http://smalltalkhub.com/mc/SvenVanCaekenberghe/STON/main/ https://github.com/svenvc/ston
There are 2 CI builds
https://travis-ci.org/svenvc/ston https://ci.inria.fr/pharo-contribution/job/STON/
The project's format will remain FileTree (until Tonel is supported on other Smalltalk implementations).
Last week I applied a couple of changes that I wanted to apply for a long time.
Traits are no longer used in the implementation (which should help porting to other Smalltalk implementations).
More specifically the following are used (again)
- Class>>#stonOn: - ClassDescription>>#stonContainsSubObjects - Metaclass>>#stonName - Metaclass>>#stonOn:
instead of
- TClass>>#stonOn: - TClassDescription>>#stonContainsSubObjects - TApplyingOnClassSide>>#stonName - TApplyingOnClassSide>>#stonOn:
use #instanceSide instead of #theNonMetaClass in MetaClass>>#stonOn:
Reorganised the packages with sub tags.
Add support for Fraction and ScaledDecimal literals (not in JSON mode). Previously a float conversion meant there was a loss of information.
Change the representation of Date to include a timezone offset (since the current Date implementation is sensitive to this).
2018-10-11Z 2018-10-11+00:00 2018-10-11+02:00
A missing timezone offset is interpreted as being the local timezone offset.
Add more special representations for common value style objects. One of STON's goals is to be a human readable and human editable representation of an object graph while remaining 100% correct (not losing information). The following were added for this reason:
MimeType and URL using ZnMimeType and ZnUrl respectively as simplified values.
MimeType['application/json'] URL['https://pharo.org']
Color
Color[#red] Color{#red:1.0,#green:0.0,#blue:0.0,#alpha:0.4}
FileReferences to the DiskFileSystem (effectively normal files)
FILE['/tmp/foo.txt']
Here is an example of how much difference that makes. Given the following Dictionary
{ #background->Color red. #workdir->'/tmp/pharo/work/' asFileReference. #home->'https://pharo.org/experimental' asUrl. #datatype->'application/json' asMIMEType } asDictionary
Which contains real objects as its values.
was serialised by STON BEFORE the changes as
{ #datatype : ZnMimeType { #main : 'application', #sub : 'json' }, #background : Color { #rgb : 1072693248, #cachedDepth : 32, #cachedBitPattern : Bitmap [ 4294901760 ], #alpha : 255 }, #home : ZnUrl { #scheme : #https, #host : 'pharo.org', #segments : OrderedCollection [ 'experimental' ] }, #workdir : FileReference { #filesystem : FileSystem { #store : MacStore { #maxFileNameLength : 255 } }, #path : AbsolutePath [ 'tmp', 'pharo', 'work' ] } }
which is 100% correct, but not very human friendly.
Now, AFTER the changes, the STON serialisation looks as follows:
{ #datatype : MimeType [ 'application/json' ], #background : Color [ #red ], #home : URL [ 'https://pharo.org/experimental' ], #workdir : FILE [ '/tmp/pharo/work' ] }
which is a huge improvement, IMO.
What do you think ? Comments, feedback, remarks ? Any other suggestions for other object that could use this treatment ?
This is great thank you! In some projects I had some hacks for the export of FileReferences in a readable way, it is great to know have a readable export by default in ston.
Sven
-- Cyril Ferlicot https://ferlicot.fr
Sven, You rule. Definitely nice changes and thanks for the consolidation. Best, Phil On Fri, Oct 12, 2018, 11:38 Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
I have consolidated all repositories where STON code lives so that they are now all in sync, and in sync with changes from Pharo 7.
http://ss3.gemtalksystems.com/ss/STON/ http://smalltalkhub.com/mc/SvenVanCaekenberghe/STON/main/ https://github.com/svenvc/ston
There are 2 CI builds
https://travis-ci.org/svenvc/ston https://ci.inria.fr/pharo-contribution/job/STON/
The project's format will remain FileTree (until Tonel is supported on other Smalltalk implementations).
Last week I applied a couple of changes that I wanted to apply for a long time.
Traits are no longer used in the implementation (which should help porting to other Smalltalk implementations).
More specifically the following are used (again)
- Class>>#stonOn: - ClassDescription>>#stonContainsSubObjects - Metaclass>>#stonName - Metaclass>>#stonOn:
instead of
- TClass>>#stonOn: - TClassDescription>>#stonContainsSubObjects - TApplyingOnClassSide>>#stonName - TApplyingOnClassSide>>#stonOn:
use #instanceSide instead of #theNonMetaClass in MetaClass>>#stonOn:
Reorganised the packages with sub tags.
Add support for Fraction and ScaledDecimal literals (not in JSON mode). Previously a float conversion meant there was a loss of information.
Change the representation of Date to include a timezone offset (since the current Date implementation is sensitive to this).
2018-10-11Z 2018-10-11+00:00 2018-10-11+02:00
A missing timezone offset is interpreted as being the local timezone offset.
Add more special representations for common value style objects. One of STON's goals is to be a human readable and human editable representation of an object graph while remaining 100% correct (not losing information). The following were added for this reason:
MimeType and URL using ZnMimeType and ZnUrl respectively as simplified values.
MimeType['application/json'] URL['https://pharo.org']
Color
Color[#red] Color{#red:1.0,#green:0.0,#blue:0.0,#alpha:0.4}
FileReferences to the DiskFileSystem (effectively normal files)
FILE['/tmp/foo.txt']
Here is an example of how much difference that makes. Given the following Dictionary
{ #background->Color red. #workdir->'/tmp/pharo/work/' asFileReference. #home->'https://pharo.org/experimental' asUrl. #datatype->'application/json' asMIMEType } asDictionary
Which contains real objects as its values.
was serialised by STON BEFORE the changes as
{ #datatype : ZnMimeType { #main : 'application', #sub : 'json' }, #background : Color { #rgb : 1072693248, #cachedDepth : 32, #cachedBitPattern : Bitmap [ 4294901760 ], #alpha : 255 }, #home : ZnUrl { #scheme : #https, #host : 'pharo.org', #segments : OrderedCollection [ 'experimental' ] }, #workdir : FileReference { #filesystem : FileSystem { #store : MacStore { #maxFileNameLength : 255 } }, #path : AbsolutePath [ 'tmp', 'pharo', 'work' ] } }
which is 100% correct, but not very human friendly.
Now, AFTER the changes, the STON serialisation looks as follows:
{ #datatype : MimeType [ 'application/json' ], #background : Color [ #red ], #home : URL [ 'https://pharo.org/experimental' ], #workdir : FILE [ '/tmp/pharo/work' ] }
which is a huge improvement, IMO.
What do you think ? Comments, feedback, remarks ? Any other suggestions for other object that could use this treatment ?
Sven
Sounds great! As for feedback, I've ported a slightly old version to VAST semi-manually (which was a minimal amount of work, thank you!), from before Traits were in use. Most fit neatly into a compatibility layer, only a few small things in STON-Core stood out as things I ended up fixing inline, which could be suitable for changing in the Pharo canonical version: STONReader >> #parseNamedInstVarsFor: instVarNamed:put: takes a string as arg in VA (at least in VA, strings and symbols are not interchangeable), so had to add an asString. Pharo code would continue to work with this in base. In STONReaderError >> #messageText, format: isn't very portable, and can be somewhat cumbersome to port. I ended up replacing it with expandMacrosWithArguments:, which (afaik) works in multiple dialects (VA, Pharo, and VW,that I know of); 'At character <1s>: <2s>' expandMacrosWithArguments: #('12' 'Test') Cheers, Henry -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
Porting the newest version (0.96 from ss3), a few comments: - Date in other dialects (and VA specifically) do not contain tz offset - the roundtrip will thus necessarily be lossy. I chose to truncate the TZ info on import, and write without any when storing. In other words, if unknown, treat the dates as if they are in local tz from the PoV of STON. This will obviously cause some discrepancies if transferring dates from one dialect to another, but AFAICT, that's already the case when doing Pharo -> Pharo transfer between systems in different timezones with tz info omitted. - STONFileReference is a subclass of FileReference, this could/should be Object, right? - STONFileReference >> #stonName and the other new implentors return a string, while the existing implentors return a symbol. Not a big deal for Pharo, but for others, it would be nice if they were consistent. - Is STONCStyleCommentsSkipStream >> #readInto:startingAt:count: in use, but untested? It contains a reference to self encoder which seems to not be implemented. - ScaledDecimals have different internal representations in different dialects*. numerator/denominator aren't always implemented in terms of interpretation as a fraction; explicitly converting to fraction before printing numerator / denominator in STONWriter >> #writeScaledDecimal: would be of help in keeping the code cross-platform compatible. By contrast, implementing newFromNumber:scale: for STONReader is much less likely to cause clashes with existing functionality. - From a cross-dialect perspective, ZnMimeType, ZnUrl, and (to an extent) FileReference extensions feel like they don't really belong in core, but I have no better suggestion of how to package them in an easily loadable manner :/ The stonName definition split used for STONFileReference seems like a good thing from this PoV; maybe the same could be done for MimeType and Url? Then you can load core, ignore the extensions on missing classes, but still have classes in the image implying you might want to map the custom core STON tags. * ScaledDecimals will thus also experience precision loss (like Dates) when doing dialect-roundtrips. For instance, in VAST, it stores an integer with a fixed precision and scale; 3/7s3 will come back as 429/1000s3 Cheers, Henry -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
On 16 Oct 2018, at 19:03, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Porting the newest version (0.96 from ss3), a few comments:
- Date in other dialects (and VA specifically) do not contain tz offset - the roundtrip will thus necessarily be lossy. I chose to truncate the TZ info on import, and write without any when storing. In other words, if unknown, treat the dates as if they are in local tz from the PoV of STON.
If you have a Date without a TZ (or better an implicit one), that would be one strategy: write out without TZ info, drop all TZ info on read. Another option would be to add a Z (for UTC), but that would also not be 100% correct. It feels as if a TZ context is necessary after all.
This will obviously cause some discrepancies if transferring dates from one dialect to another, but AFAICT, that's already the case when doing Pharo -> Pharo transfer between systems in different timezones with tz info omitted.
These are interesting questions that I have to think more about. The first goal of STON is to be 100% when going from Smalltalk objects to STON and back. Being cross platform or cross version is nice and important, but not a main goal, nor is it solvable I think.
- STONFileReference is a subclass of FileReference, this could/should be Object, right?
Uhh, no, since it acts as a facade that creates FileReferences. But I can see how that would be hard to port, I guess this has to be implemented differently in your case.
- STONFileReference >> #stonName and the other new implentors return a string, while the existing implentors return a symbol. Not a big deal for Pharo, but for others, it would be nice if they were consistent.
Yes, I'll check and try to make it consistent, you are right about that.
- Is STONCStyleCommentsSkipStream >> #readInto:startingAt:count: in use, but untested? It contains a reference to self encoder which seems to not be implemented.
A copy/paste error, I am sure. I'll check.
- ScaledDecimals have different internal representations in different dialects*. numerator/denominator aren't always implemented in terms of interpretation as a fraction; explicitly converting to fraction before printing numerator / denominator in STONWriter >> #writeScaledDecimal: would be of help in keeping the code cross-platform compatible. By contrast, implementing newFromNumber:scale: for STONReader is much less likely to cause clashes with existing functionality.
- From a cross-dialect perspective, ZnMimeType, ZnUrl, and (to an extent) FileReference extensions feel like they don't really belong in core, but I have no better suggestion of how to package them in an easily loadable manner :/ The stonName definition split used for STONFileReference seems like a good thing from this PoV; maybe the same could be done for MimeType and Url? Then you can load core, ignore the extensions on missing classes, but still have classes in the image implying you might want to map the custom core STON tags.
Again, if a dialect does not implement certain classes/types, they have to be skipped. In terms of code organisation, separate packages could help. An the other hand, there could be dummy objects that just hold the external representation. I.e. if there is no URL class, just wrap the string/text. This would maintain some compatibility/portability. Thx again for the feedback !
* ScaledDecimals will thus also experience precision loss (like Dates) when doing dialect-roundtrips. For instance, in VAST, it stores an integer with a fixed precision and scale; 3/7s3 will come back as 429/1000s3
Cheers, Henry
-- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
On Tue, Oct 16, 2018 at 9:28 PM Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 16 Oct 2018, at 19:03, Henrik Sperre Johansen < henrik.s.johansen@veloxit.no> wrote:
- STONFileReference is a subclass of FileReference, this could/should be Object, right?
Uhh, no, since it acts as a facade that creates FileReferences. But I can see how that would be hard to port, I guess this has to be implemented differently in your case.
I must be missing something, it seemed to me neither FileReference >> #stonOn: nor STONFileReference class >> #fromSton: actually required STONFileReference to be a subclass of FileReference... Cheers, Henry
On 18 Oct 2018, at 10:59, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
On Tue, Oct 16, 2018 at 9:28 PM Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 16 Oct 2018, at 19:03, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
- STONFileReference is a subclass of FileReference, this could/should be Object, right?
Uhh, no, since it acts as a facade that creates FileReferences. But I can see how that would be hard to port, I guess this has to be implemented differently in your case.
I must be missing something, it seemed to me neither FileReference >> #stonOn: nor STONFileReference class >> #fromSton: actually required STONFileReference to be a subclass of FileReference...
Yes, you are right, but by making it a subclass, you can infer that it generates something that is compatible with its superclass, that was the idea. Anyway, it is an implementation artefact, so it is not that important. If you handle FILE on a platform that does not have FileReference, then you won't need it. You would then add an #stonName equal to FILE somewhere else.
Cheers, Henry
Hi Henrik, I implemented some of your suggestions and did some cleanups: === Name: STON-Core-SvenVanCaekenberghe.97 Author: SvenVanCaekenberghe Time: 23 October 2018, 3:32:00.595497 pm UUID: a627a421-4735-0d00-8c11-99b80be56389 Ancestors: STON-Core-SvenVanCaekenberghe.96 make #stonName a Symbol everywhere and add comments to STONReader>>#lookupClass: add an #asString conversion in STONReader>>#parseNamedInstVarsFor: to ease porting fix STONCStyleCommentsSkipStream>>#readInto:startingAt:count: and add #testBlockReading change the implementation of STONReaderError>>#messageText to ease porting === Name: STON-Tests-SvenVanCaekenberghe.90 Author: SvenVanCaekenberghe Time: 23 October 2018, 3:32:18.123909 pm UUID: 309faf22-4735-0d00-8c12-16b60be56389 Ancestors: STON-Tests-SvenVanCaekenberghe.89 make #stonName a Symbol everywhere and add comments to STONReader>>#lookupClass: add an #asString conversion in STONReader>>#parseNamedInstVarsFor: to ease porting fix STONCStyleCommentsSkipStream>>#readInto:startingAt:count: and add #testBlockReading change the implementation of STONReaderError>>#messageText to ease porting === Or https://github.com/svenvc/ston/commit/90a96aeb7042822f370e6f4cda6f1aed784576... I still do not clearly understand the problems with ScaledDecial. If the implementation is different on the other platform, the writing and reading might need different implementations, but the general syntax, <nominator>/<denominator>[s<scale>] can remain the same, no ? Sven
On 18 Oct 2018, at 11:03, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 18 Oct 2018, at 10:59, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
On Tue, Oct 16, 2018 at 9:28 PM Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 16 Oct 2018, at 19:03, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
- STONFileReference is a subclass of FileReference, this could/should be Object, right?
Uhh, no, since it acts as a facade that creates FileReferences. But I can see how that would be hard to port, I guess this has to be implemented differently in your case.
I must be missing something, it seemed to me neither FileReference >> #stonOn: nor STONFileReference class >> #fromSton: actually required STONFileReference to be a subclass of FileReference...
Yes, you are right, but by making it a subclass, you can infer that it generates something that is compatible with its superclass, that was the idea.
Anyway, it is an implementation artefact, so it is not that important.
If you handle FILE on a platform that does not have FileReference, then you won't need it. You would then add an #stonName equal to FILE somewhere else.
Cheers, Henry
Hi Henrik,
On 16 Oct 2018, at 15:16, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Sounds great!
As for feedback, I've ported a slightly old version to VAST semi-manually (which was a minimal amount of work, thank you!), from before Traits were in use.
Thx for working on the code base and porting to VAST.
Most fit neatly into a compatibility layer, only a few small things in STON-Core stood out as things I ended up fixing inline, which could be suitable for changing in the Pharo canonical version:
STONReader >> #parseNamedInstVarsFor: instVarNamed:put: takes a string as arg in VA (at least in VA, strings and symbols are not interchangeable), so had to add an asString. Pharo code would continue to work with this in base.
Hmm, yes, I will have a look to see how we solved this for NeoJSON with Mariano.
In STONReaderError >> #messageText, format: isn't very portable, and can be somewhat cumbersome to port. I ended up replacing it with expandMacrosWithArguments:, which (afaik) works in multiple dialects (VA, Pharo, and VW,that I know of); 'At character <1s>: <2s>' expandMacrosWithArguments: #('12' 'Test')
OK, I'll note this and have a look later on. Sven
Cheers, Henry
-- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
participants (5)
-
Cyril Ferlicot -
Guillermo Polito -
Henrik Sperre Johansen -
phil＠highoctane.be -
Sven Van Caekenberghe