Fraction and ScaledDecimal should be not be serialized as Float in STON format
Hello, I realised that in the current implementation, when a number is serialised by STON it is either as an integer literal or as a float literal. There is a risk to loose precision, especially if what you serialise is a ScaledDecimal or a fraction. I propose a simple change to fix this: 1. Add STONWriter>>#isInJsonMode method which returns true if the STONWriter wants to write JSON 2. Add: Fraction>>#stonOn: stonWriter stonWriter isInJsonMode ifTrue: [ ^ super stonOn: stonWriter ]. stonWriter writeObject: self streamMap: [ :dictionary | dictionary at: #numerator put: numerator. dictionary at: #denominator put: denominator ] 3. Add: ScaledDecimal>>#stonOn: stonWriter stonWriter isInJsonMode ifTrue: [ ^ super stonOn: stonWriter ]. stonWriter writeObject: self streamMap: [ :dictionary | dictionary at: #numerator put: numerator. dictionary at: #denominator put: denominator. dictionary at: #scale put: scale ] This change has a main drawback, storing fractions and scaled decimal will result in a huge overhead. Maybe for ScaledDecimal we can use Pharoâs literal (e.g. 1.12s2). For fractions I donât know what literal could be used? Or maybe the overhead is fine? Cheers, Julien --- Julien Delplanque Doctorant à lâUniversité de Lille http://juliendelplanque.be/phd.html Equipe Rmod, Inria Bâtiment B 40, Avenue Halley 59650 Villeneuve d'Ascq Numéro de téléphone: +333 59 35 86 40
Hi Julien, Good and interesting point. Your summary is correct: STON, inheriting from JSON so to speak, only knowns about integer and float numbers. All other Smalltalk numbers get converted, which results in a loss of type and precision. That might not be a perfect situation, but nobody complained so far. If we would change it, efficiency has to be taken into account: speed/memory efficiency as well as human typing efficiency/complexity. Compatibility, especially between different Smalltalk and other object languages, is also an aspect. Apart from the current approach, the full STON approach could be, for example for 1/3 and 1.5s2 Fraction { #numerator:1, #denominator:3 } ScaledDecimal { #numerator:3, #denominator:2, #scale:2} For some types, STON uses shorter notations, either using positional arguments Point [ 1, 2 ] Fraction [ 1, 3 ] ScaledFraction [ 3, 2, 2 ] Or a string argument, when there is a clear external representation DateAndTime [ '2018-09-18T16:39:10.325129+02:00' ] Fraction [ '1/3' ] ScaledFraction [ '3/2s2' ] Of course, we could also extend the basic number parser and allow the native (stored) notation 1/3 3/2s2 Note how that last one should use the more exact #storeOn: notation, not the #printOn: I'll have to think about this a bit more, as I can really not choose right now. Sven
On 18 Sep 2018, at 10:34, Julien <julien.delplanque@inria.fr> wrote:
Hello,
I realised that in the current implementation, when a number is serialised by STON it is either as an integer literal or as a float literal.
There is a risk to loose precision, especially if what you serialise is a ScaledDecimal or a fraction.
I propose a simple change to fix this:
1. Add STONWriter>>#isInJsonMode method which returns true if the STONWriter wants to write JSON
2. Add:
Fraction>>#stonOn: stonWriter stonWriter isInJsonMode ifTrue: [ ^ super stonOn: stonWriter ].
stonWriter writeObject: self streamMap: [ :dictionary | dictionary at: #numerator put: numerator. dictionary at: #denominator put: denominator ]
3. Add:
ScaledDecimal>>#stonOn: stonWriter stonWriter isInJsonMode ifTrue: [ ^ super stonOn: stonWriter ].
stonWriter writeObject: self streamMap: [ :dictionary | dictionary at: #numerator put: numerator. dictionary at: #denominator put: denominator. dictionary at: #scale put: scale ]
This change has a main drawback, storing fractions and scaled decimal will result in a huge overhead.
Maybe for ScaledDecimal we can use Pharoâs literal (e.g. 1.12s2).
For fractions I donât know what literal could be used?
Or maybe the overhead is fine?
Cheers,
Julien
--- Julien Delplanque Doctorant à lâUniversité de Lille http://juliendelplanque.be/phd.html Equipe Rmod, Inria Bâtiment B 40, Avenue Halley 59650 Villeneuve d'Ascq Numéro de téléphone: +333 59 35 86 40
On 18 Sep 2018, at 16:45, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Of course, we could also extend the basic number parser and allow the native (stored) notation
1/3 3/2s2
Note how that last one should use the more exact #storeOn: notation, not the #printOn:
I'll have to think about this a bit more, as I can really not choose right now.
I am still not sure if we should do this or not, but here is a POC implementation of the above. It implements the following syntax extension (see the class comment of STON for the rest). number int int denominator int denominator scale int frac int exp int frac exp denominator / digits scale s digits STONWriter>>#writeFraction: fraction jsonMode ifTrue: [ self writeFloat: fraction asFloat ] ifFalse: [ writeStream print: fraction numerator; nextPut: $/; print: fraction denominator ] STONWriter>>#writeScaledDecimal: scaledDecimal jsonMode ifTrue: [ self writeFloat: scaledDecimal asFloat ] ifFalse: [ writeStream print: scaledDecimal numerator; nextPut: $/; print: scaledDecimal denominator; nextPut: $s; print: scaledDecimal scale ] Fraction>>#stonOn: stonWriter stonWriter writeFraction: self ScaledDecimal>>#stonOn: stonWriter stonWriter writeScaledDecimal: self STONReader>>parseNumber | negated number | negated := readStream peekFor: $-. number := self parseNumberInteger. (readStream peekFor: $/) ifTrue: [ number := Fraction numerator: number denominator: self parseNumberInteger. (readStream peekFor: $s) ifTrue: [ number := ScaledDecimal newFromNumber: number scale: self parseNumberInteger ] ] ifFalse: [ (readStream peekFor: $.) ifTrue: [ number := number + self parseNumberFraction ]. ((readStream peekFor: $e) or: [ readStream peekFor: $E ]) ifTrue: [ number := number * self parseNumberExponent ] ]. negated ifTrue: [ number := number negated ]. self consumeWhitespace. ^ number Sven
I decided to add it for real: https://github.com/svenvc/ston/commit/9c83e3cc2f00cab83e57f2e10a139d6ecef3cb... add support for Fraction and ScaledDecimal as direct numeric literals (in STON mode, not in JSON mode) with units tests
On 25 Sep 2018, at 14:13, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 18 Sep 2018, at 16:45, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Of course, we could also extend the basic number parser and allow the native (stored) notation
1/3 3/2s2
Note how that last one should use the more exact #storeOn: notation, not the #printOn:
I'll have to think about this a bit more, as I can really not choose right now.
I am still not sure if we should do this or not, but here is a POC implementation of the above.
It implements the following syntax extension (see the class comment of STON for the rest).
number int int denominator int denominator scale int frac int exp int frac exp
denominator / digits
scale s digits
STONWriter>>#writeFraction: fraction jsonMode ifTrue: [ self writeFloat: fraction asFloat ] ifFalse: [ writeStream print: fraction numerator; nextPut: $/; print: fraction denominator ]
STONWriter>>#writeScaledDecimal: scaledDecimal jsonMode ifTrue: [ self writeFloat: scaledDecimal asFloat ] ifFalse: [ writeStream print: scaledDecimal numerator; nextPut: $/; print: scaledDecimal denominator; nextPut: $s; print: scaledDecimal scale ]
Fraction>>#stonOn: stonWriter stonWriter writeFraction: self
ScaledDecimal>>#stonOn: stonWriter stonWriter writeScaledDecimal: self
STONReader>>parseNumber | negated number | negated := readStream peekFor: $-. number := self parseNumberInteger. (readStream peekFor: $/) ifTrue: [ number := Fraction numerator: number denominator: self parseNumberInteger. (readStream peekFor: $s) ifTrue: [ number := ScaledDecimal newFromNumber: number scale: self parseNumberInteger ] ] ifFalse: [ (readStream peekFor: $.) ifTrue: [ number := number + self parseNumberFraction ]. ((readStream peekFor: $e) or: [ readStream peekFor: $E ]) ifTrue: [ number := number * self parseNumberExponent ] ]. negated ifTrue: [ number := number negated ]. self consumeWhitespace. ^ number
Sven
This is cool because indeed scaledDecimal are nice objects :) On Tue, Oct 9, 2018 at 9:23 PM Sven Van Caekenberghe <sven@stfx.eu> wrote:
I decided to add it for real:
https://github.com/svenvc/ston/commit/9c83e3cc2f00cab83e57f2e10a139d6ecef3cb...
add support for Fraction and ScaledDecimal as direct numeric literals (in STON mode, not in JSON mode) with units tests
On 25 Sep 2018, at 14:13, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 18 Sep 2018, at 16:45, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Of course, we could also extend the basic number parser and allow the native (stored) notation
1/3 3/2s2
Note how that last one should use the more exact #storeOn: notation, not the #printOn:
I'll have to think about this a bit more, as I can really not choose right now.
I am still not sure if we should do this or not, but here is a POC implementation of the above.
It implements the following syntax extension (see the class comment of STON for the rest).
number int int denominator int denominator scale int frac int exp int frac exp
denominator / digits
scale s digits
STONWriter>>#writeFraction: fraction jsonMode ifTrue: [ self writeFloat: fraction asFloat ] ifFalse: [ writeStream print: fraction numerator; nextPut: $/; print: fraction denominator ]
STONWriter>>#writeScaledDecimal: scaledDecimal jsonMode ifTrue: [ self writeFloat: scaledDecimal asFloat ] ifFalse: [ writeStream print: scaledDecimal numerator; nextPut: $/; print: scaledDecimal denominator; nextPut: $s; print: scaledDecimal scale ]
Fraction>>#stonOn: stonWriter stonWriter writeFraction: self
ScaledDecimal>>#stonOn: stonWriter stonWriter writeScaledDecimal: self
STONReader>>parseNumber | negated number | negated := readStream peekFor: $-. number := self parseNumberInteger. (readStream peekFor: $/) ifTrue: [ number := Fraction numerator: number denominator: self parseNumberInteger. (readStream peekFor: $s) ifTrue: [ number := ScaledDecimal newFromNumber: number scale: self parseNumberInteger ] ] ifFalse: [ (readStream peekFor: $.) ifTrue: [ number := number + self parseNumberFraction ]. ((readStream peekFor: $e) or: [ readStream peekFor: $E ]) ifTrue: [ number := number * self parseNumberExponent ] ]. negated ifTrue: [ number := number negated ]. self consumeWhitespace. ^ number
Sven
Sorry, I forgot to answer. This change is indeed really nice. Thank you for it! Cheers, Julien --- Julien Delplanque Doctorant à lâUniversité de Lille http://juliendelplanque.be/phd.html Equipe Rmod, Inria Bâtiment B 40, Avenue Halley 59650 Villeneuve d'Ascq Numéro de téléphone: +333 59 35 86 40
Le 9 oct. 2018 à 21:22, Sven Van Caekenberghe <sven@stfx.eu> a écrit :
I decided to add it for real:
https://github.com/svenvc/ston/commit/9c83e3cc2f00cab83e57f2e10a139d6ecef3cb...
add support for Fraction and ScaledDecimal as direct numeric literals (in STON mode, not in JSON mode) with units tests
On 25 Sep 2018, at 14:13, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 18 Sep 2018, at 16:45, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Of course, we could also extend the basic number parser and allow the native (stored) notation
1/3 3/2s2
Note how that last one should use the more exact #storeOn: notation, not the #printOn:
I'll have to think about this a bit more, as I can really not choose right now.
I am still not sure if we should do this or not, but here is a POC implementation of the above.
It implements the following syntax extension (see the class comment of STON for the rest).
number int int denominator int denominator scale int frac int exp int frac exp
denominator / digits
scale s digits
STONWriter>>#writeFraction: fraction jsonMode ifTrue: [ self writeFloat: fraction asFloat ] ifFalse: [ writeStream print: fraction numerator; nextPut: $/; print: fraction denominator ]
STONWriter>>#writeScaledDecimal: scaledDecimal jsonMode ifTrue: [ self writeFloat: scaledDecimal asFloat ] ifFalse: [ writeStream print: scaledDecimal numerator; nextPut: $/; print: scaledDecimal denominator; nextPut: $s; print: scaledDecimal scale ]
Fraction>>#stonOn: stonWriter stonWriter writeFraction: self
ScaledDecimal>>#stonOn: stonWriter stonWriter writeScaledDecimal: self
STONReader>>parseNumber | negated number | negated := readStream peekFor: $-. number := self parseNumberInteger. (readStream peekFor: $/) ifTrue: [ number := Fraction numerator: number denominator: self parseNumberInteger. (readStream peekFor: $s) ifTrue: [ number := ScaledDecimal newFromNumber: number scale: self parseNumberInteger ] ] ifFalse: [ (readStream peekFor: $.) ifTrue: [ number := number + self parseNumberFraction ]. ((readStream peekFor: $e) or: [ readStream peekFor: $E ]) ifTrue: [ number := number * self parseNumberExponent ] ]. negated ifTrue: [ number := number negated ]. self consumeWhitespace. ^ number
Sven
participants (3)
-
Julien -
Stephane Ducasse -
Sven Van Caekenberghe