- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Fraction and ScaledDecimal should be not be serialized as Float in STON format

by Sven Van Caekenberghe

Hi Julien, Good and interesting point. Your summary is correct: STON, inheriting from JSON so to speak, only knowns about integer and float numbers. All other Smalltalk numbers get converted, which results in a loss of type and precision. That might not be a perfect situation, but nobody complained so far. If we would change it, efficiency has to be taken into account: speed/memory efficiency as well as human typing efficiency/complexity. Compatibility, especially between different Smalltalk and other object languages, is also an aspect. Apart from the current approach, the full STON approach could be, for example for 1/3 and 1.5s2 Fraction { #numerator:1, #denominator:3 } ScaledDecimal { #numerator:3, #denominator:2, #scale:2} For some types, STON uses shorter notations, either using positional arguments Point [ 1, 2 ] Fraction [ 1, 3 ] ScaledFraction [ 3, 2, 2 ] Or a string argument, when there is a clear external representation DateAndTime [ '2018-09-18T16:39:10.325129+02:00' ] Fraction [ '1/3' ] ScaledFraction [ '3/2s2' ] Of course, we could also extend the basic number parser and allow the native (stored) notation 1/3 3/2s2 Note how that last one should use the more exact #storeOn: notation, not the #printOn: I'll have to think about this a bit more, as I can really not choose right now. Sven > On 18 Sep 2018, at 10:34, Julien <julien.delplanque(a)inria.fr> wrote: > > Hello, > > I realised that in the current implementation, when a number is serialised by STON it is either as an integer literal or as a float literal. > > There is a risk to loose precision, especially if what you serialise is a ScaledDecimal or a fraction. > > I propose a simple change to fix this: > > 1. Add STONWriter>>#isInJsonMode method which returns true if the STONWriter wants to write JSON > > 2. Add: > > Fraction>>#stonOn: stonWriter > stonWriter isInJsonMode > ifTrue: [ ^ super stonOn: stonWriter ]. > > stonWriter writeObject: self streamMap: [ :dictionary | > dictionary at: #numerator put: numerator. > dictionary at: #denominator put: denominator ] > > 3. Add: > > ScaledDecimal>>#stonOn: stonWriter > stonWriter isInJsonMode > ifTrue: [ ^ super stonOn: stonWriter ]. > > stonWriter writeObject: self streamMap: [ :dictionary | > dictionary at: #numerator put: numerator. > dictionary at: #denominator put: denominator. > dictionary at: #scale put: scale ] > > > This change has a main drawback, storing fractions and scaled decimal will result in a huge overhead. > > Maybe for ScaledDecimal we can use Pharoâs literal (e.g. 1.12s2). > > For fractions I donât know what literal could be used? > > Or maybe the overhead is fine? > > Cheers, > > Julien > > --- > Julien Delplanque > Doctorant Ã lâUniversitÃ© de Lille > http://juliendelplanque.be/phd.html > Equipe Rmod, Inria > BÃ¢timent B 40, Avenue Halley 59650 Villeneuve d'Ascq > NumÃ©ro de tÃ©lÃ©phone: +333 59 35 86 40 >

Sept. 18, 2018

Re: [Pharo-dev] Platform file encoding for FFI

by Alistair Grant

Hi Guille, Esteban and Henry, Thanks for your replies. On Tue, Sep 18, 2018 at 10:09:02AM +0200, Guillermo Polito wrote: > > > On Mon, Sep 17, 2018 at 6:52 PM Alistair Grant <akgrant0710(a)gmail.com> wrote: > > Hi Esteban, Guille and Everyone, > > I haven't looked at using FFI much, however it is easy to imagine that > different file encoding rules on different platforms will make writing > FFI calls more difficult, > > > Well not really (from my point of view :)) > From the point of view of the FFI call an encoded string is just a bunch of > bytes. FFI does not do any interpretation of them. Right, but getting the appropriately encoded bunch of bytes is the issue. :-) > i.e. some of the different formats are: > > - OSX uses Mac specific decomposed UTF8 encoding > - Windows uses Wide Strings (16 bit Unicode characters) > - Linux allows pretty much anything, but precomposed UTF8 is common > > > > At the image side, we could have an strategy that, depending on the OS, could > encode in one encoding or another, or even not encode at all. > > > Believe it or not, I'm still working on getting the > FileAttributesPlugin working (file name encoding on Windows being the > latest issue - the tests in Pharo need to be extended). > > > I believe you, don't worry ^^. > > > Would it be useful for future FFI work to have primitives available > which convert file names to and from the various platform specific > formats? (Linux is basically a no-op, and Windows could be written > in-image, but OSX requires the platform routines to be called). > > > Maybe... Are the OSX routines exposed as C functions (that we can call through > FFI) or they are objective-C methods/functions (that are more complicated to > map)? The OSX routines are exposed as C functions (and available as Objective-C methods), see convertChars() in platforms/unix/vm/sqUnixCharConv.c. On Tue, Sep 18, 2018 at 11:21:41AM +0200, Esteban Lorenzano wrote: > > self > > ffiCall: #(bool saveContentsToFile(String fileName, String contents)) > > options: #(+stringEncodings( fileName return , platformAPI contents) > > This is cool. > What I do not like is to rely on primitives to do that encoding. > This should be in image??? using FFI if needed (this is all because we > want to rely less and less on plugins :P) I realise of course that this could all be done in FFI, and I agree with all Estaban's arguments in favour of FFI, my main motivation was that the code is already in the VM, and to avoid code duplication with the obvious benefit that if a bug is fixed it will apply everywhere. On Tue, Sep 18, 2018 at 11:23:56AM +0200, Esteban Lorenzano wrote: > > > On 18 Sep 2018, at 11:04, Guillermo Polito <guillermopolito(a)gmail.com> > wrote: > > > > On Tue, Sep 18, 2018 at 10:43 AM Henrik Sperre Johansen < > henrik.s.johansen(a)veloxit.no> wrote: > > It *would* be pretty handy for adding some auto-conversion into the > marshaller based on parameter encoding options though... (other than > filename, could be done in smalltalk using exisiting encoders) > > self > ffiCall: #(bool saveContentsToFile(String fileName, String > contents)) > options: #(+stringEncodings( fileName return , platformAPI > contents) > > > Well, I like this idea. > > > (And yes, I've probably badly mangled the options syntax) > > Is much less verbose than having to manually convert Strings to the > proper > platform Unicode encodings before calling. > Depends a bit on whether the primitive argument is > Byte/Widestrings(latin1/utf32), or if it accepts only utf8 bytes and > one has > to convert first anyways. > > It's not like this isn't a pain point, there are plenty of currently > used > API's that are broken if you try to use non-ascii. > > > Yes, but I think this may be because in general people tend to not know how > encodings work... (even myself I don't feel I know enough :)) > But this makes me think that we should make encoding explicit? > > > Yup, explicit please. Nothing hide behind the carpet :) > > > > Maybe we should force people to specify an encoding if they specify a > callout using a string. > > And then, either they specify it at the level of the callout, or at the > level of the library (like setting a default encoding for all strings). > > > > You can have some global FFI settings (I was thinking on adding some global > options settings for FFI in general, btw) and even ?library based settings?, to > simplify. > > Esteban > > > > Because this raises also the question of what is the default encoding? > And I'd say that in there is no satisfactory default encoding... I'll defer to Sven every time when it comes to character encoding, but my understanding is that the only platform that has consistent encoding rules is OSX, which uses the platform specific decomposed UTF8. Both Windows and Linux use precomposed UTF8, but other character encodings are possible (particularly for older files). So we certainly shouldn't make the encoding hard-coded. UTF8 as the default encoding I think does make sense (this is what FilePlugin currently uses). Cheers, Alistair

Sept. 18, 2018

[Pharo 7.0-dev] Build #1254: 22463 Tag ManifestTextCore and classes in package Text-Core-Manifest

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #1254 was: SUCCESS. The Pull Request #1811 was integrated: "22463 Tag ManifestTextCore and classes in package Text-Core-Manifest" Pull request url: https://github.com/pharo-project/pharo/pull/1811 Issue Url: https://pharo.fogbugz.com/f/cases/22463 Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

Sept. 18, 2018

Re: [Pharo-dev] [rmod] Float should not implement #to:, #to:by:, etc...

by Guillaume Larcheveque

Maybe #to:by: should convert its parameters in Fraction to avoid Floats problems (not sure, just an idea) 2018-09-18 11:25 GMT+02:00 Esteban Lorenzano <estebanlm(a)gmail.com>: > > > On 18 Sep 2018, at 11:13, Guillermo Polito <guillermopolito(a)gmail.com> > wrote: > > > > On Tue, Sep 18, 2018 at 11:06 AM Julien <julien.delplanque(a)inria.fr> > wrote: > >> Hello, >> >> I realised that it is possible to create an interval of floats. >> >> I think this is bad because, since intervals are computed by successively >> adding a number, it might result in precision errors. >> >> (0.0 to: 1.0 by: 0.1) asArray >>> #(0.0 0.1 0.2 0.30000000000000004 0.4 >> 0.5 0.6000000000000001 0.7000000000000001 0.8 0.9 1.0) >> >> The correct (precise) way to do it would be to use ScaledDecimal: >> >> (0.0s1 to: 1.0s1 by: 0.1s1) asArray >>> #(0.0s1 0.1s1 0.2s1 0.3s1 0.4s1 >> 0.5s1 0.6s1 0.7s1 0.8s1 0.9s1 1.0s1) >> >> I opened an issue about it: https://pharo.fogbugz.com/ >> f/cases/22467/Float-should-not-implement-to-to-by-etc >> >> And Iâd like to discuss this with you. >> >> What do you think? >> > > Well, I think it's a matter of balance :) > > #to:by: is defined in Number. So we could, for example, cancel it in Float. > However, people would still be able to do > > 1 to: 1.0 by: 0.1 > > Which would still show problems. > > > Nevertheless, I have seen this a lot of times. > > 0.0 to: 1.0 by: 0.1 > > Is a common use case. > > > And moreover, we could try to do > > 1 to: 7 by: (Margin fromNumber: 1) > > And even worse > > 1 to: Object new by: (Margin fromNumber: 1) > > I think adding type-validations all over the place is not a good solution, > and is kind of opposite to our philosophy... > > So we should > - document the good usages > - document the bad ones > - and live with the fact that we have a relaxed type system that will > fail at runtime :) > > > yup. > But not cancel. > > Esteban > > > Guille > > > -- *Guillaume Larcheveque*

Sept. 18, 2018

Re: [Pharo-dev] [rmod] Float should not implement #to:, #to:by:, etc...

by Esteban Lorenzano

> On 18 Sep 2018, at 11:13, Guillermo Polito <guillermopolito(a)gmail.com> wrote: > > > > On Tue, Sep 18, 2018 at 11:06 AM Julien <julien.delplanque(a)inria.fr <mailto:julien.delplanque@inria.fr>> wrote: > Hello, > > I realised that it is possible to create an interval of floats. > > I think this is bad because, since intervals are computed by successively adding a number, it might result in precision errors. > > (0.0 to: 1.0 by: 0.1) asArray >>> #(0.0 0.1 0.2 0.30000000000000004 0.4 0.5 0.6000000000000001 0.7000000000000001 0.8 0.9 1.0) > > The correct (precise) way to do it would be to use ScaledDecimal: > > (0.0s1 to: 1.0s1 by: 0.1s1) asArray >>> #(0.0s1 0.1s1 0.2s1 0.3s1 0.4s1 0.5s1 0.6s1 0.7s1 0.8s1 0.9s1 1.0s1) > > I opened an issue about it: https://pharo.fogbugz.com/f/cases/22467/Float-should-not-implement-to-to-by… <https://pharo.fogbugz.com/f/cases/22467/Float-should-not-implement-to-to-by…> > > And Iâd like to discuss this with you. > > What do you think? > > Well, I think it's a matter of balance :) > > #to:by: is defined in Number. So we could, for example, cancel it in Float. > However, people would still be able to do > > 1 to: 1.0 by: 0.1 > > Which would still show problems. Nevertheless, I have seen this a lot of times. 0.0 to: 1.0 by: 0.1 Is a common use case. > > And moreover, we could try to do > > 1 to: 7 by: (Margin fromNumber: 1) > > And even worse > > 1 to: Object new by: (Margin fromNumber: 1) > > I think adding type-validations all over the place is not a good solution, and is kind of opposite to our philosophy... > > So we should > - document the good usages > - document the bad ones > - and live with the fact that we have a relaxed type system that will fail at runtime :) yup. But not cancel. Esteban > > Guille

Sept. 18, 2018

[Pharo 7.0-dev] Build #1253: 22456 dynamic left to right columns layout generates an additional unneeded row with null presenters

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #1253 was: SUCCESS. The Pull Request #1803 was integrated: "22456 dynamic left to right columns layout generates an additional unneeded row with null presenters" Pull request url: https://github.com/pharo-project/pharo/pull/1803 Issue Url: https://pharo.fogbugz.com/f/cases/22456 Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

Sept. 18, 2018

Re: [Pharo-dev] Platform file encoding for FFI

by Esteban Lorenzano

> On 18 Sep 2018, at 11:04, Guillermo Polito <guillermopolito(a)gmail.com> wrote: > > > > On Tue, Sep 18, 2018 at 10:43 AM Henrik Sperre Johansen <henrik.s.johansen(a)veloxit.no <mailto:henrik.s.johansen@veloxit.no>> wrote: > Guillermo Polito wrote > > On Mon, Sep 17, 2018 at 6:52 PM Alistair Grant < > > > akgrant0710@ > > > > > > wrote: > > > >> Hi Esteban, Guille and Everyone, > >> > >> I haven't looked at using FFI much, however it is easy to imagine that > >> different file encoding rules on different platforms will make writing > >> FFI calls more difficult, > > > > > > Well not really (from my point of view :)) > > From the point of view of the FFI call an encoded string is just a bunch > > of > > bytes. FFI does not do any interpretation of them. > > It *would* be pretty handy for adding some auto-conversion into the > marshaller based on parameter encoding options though... (other than > filename, could be done in smalltalk using exisiting encoders) > > self > ffiCall: #(bool saveContentsToFile(String fileName, String contents)) > options: #(+stringEncodings( fileName return , platformAPI contents) > > Well, I like this idea. > > (And yes, I've probably badly mangled the options syntax) > > Is much less verbose than having to manually convert Strings to the proper > platform Unicode encodings before calling. > Depends a bit on whether the primitive argument is > Byte/Widestrings(latin1/utf32), or if it accepts only utf8 bytes and one has > to convert first anyways. > > It's not like this isn't a pain point, there are plenty of currently used > API's that are broken if you try to use non-ascii. > > Yes, but I think this may be because in general people tend to not know how encodings work... (even myself I don't feel I know enough :)) > But this makes me think that we should make encoding explicit? Yup, explicit please. Nothing hide behind the carpet :) > > Maybe we should force people to specify an encoding if they specify a callout using a string. > And then, either they specify it at the level of the callout, or at the level of the library (like setting a default encoding for all strings). You can have some global FFI settings (I was thinking on adding some global options settings for FFI in general, btw) and even âlibrary based settingsâ, to simplify. Esteban > > Because this raises also the question of what is the default encoding? > And I'd say that in there is no satisfactory default encoding...

Sept. 18, 2018

Re: [Pharo-dev] Platform file encoding for FFI

by Esteban Lorenzano

> On 18 Sep 2018, at 10:42, Henrik Sperre Johansen <henrik.s.johansen(a)veloxit.no> wrote: > > Guillermo Polito wrote >> On Mon, Sep 17, 2018 at 6:52 PM Alistair Grant < > >> akgrant0710@ > >> > >> wrote: >> >>> Hi Esteban, Guille and Everyone, >>> >>> I haven't looked at using FFI much, however it is easy to imagine that >>> different file encoding rules on different platforms will make writing >>> FFI calls more difficult, >> >> >> Well not really (from my point of view :)) >> From the point of view of the FFI call an encoded string is just a bunch >> of >> bytes. FFI does not do any interpretation of them. > > It *would* be pretty handy for adding some auto-conversion into the > marshaller based on parameter encoding options though... (other than > filename, could be done in smalltalk using exisiting encoders) > > self > ffiCall: #(bool saveContentsToFile(String fileName, String contents)) > options: #(+stringEncodings( fileName return , platformAPI contents) This is cool. What I do not like is to rely on primitives to do that encoding. This should be in imageâ¦ using FFI if needed (this is all because we want to rely less and less on plugins :P) Esteban > > (And yes, I've probably badly mangled the options syntax) > > Is much less verbose than having to manually convert Strings to the proper > platform Unicode encodings before calling. > Depends a bit on whether the primitive argument is > Byte/Widestrings(latin1/utf32), or if it accepts only utf8 bytes and one has > to convert first anyways. > > It's not like this isn't a pain point, there are plenty of currently used > API's that are broken if you try to use non-ascii. > > Cheers, > Henry > > > > > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html >

Sept. 18, 2018

Re: [Pharo-dev] [rmod] Float should not implement #to:, #to:by:, etc...

by Guillermo Polito

On Tue, Sep 18, 2018 at 11:06 AM Julien <julien.delplanque(a)inria.fr> wrote: > Hello, > > I realised that it is possible to create an interval of floats. > > I think this is bad because, since intervals are computed by successively > adding a number, it might result in precision errors. > > (0.0 to: 1.0 by: 0.1) asArray >>> #(0.0 0.1 0.2 0.30000000000000004 0.4 > 0.5 0.6000000000000001 0.7000000000000001 0.8 0.9 1.0) > > The correct (precise) way to do it would be to use ScaledDecimal: > > (0.0s1 to: 1.0s1 by: 0.1s1) asArray >>> #(0.0s1 0.1s1 0.2s1 0.3s1 0.4s1 > 0.5s1 0.6s1 0.7s1 0.8s1 0.9s1 1.0s1) > > I opened an issue about it: > https://pharo.fogbugz.com/f/cases/22467/Float-should-not-implement-to-to-by… > > And Iâd like to discuss this with you. > > What do you think? > Well, I think it's a matter of balance :) #to:by: is defined in Number. So we could, for example, cancel it in Float. However, people would still be able to do 1 to: 1.0 by: 0.1 Which would still show problems. And moreover, we could try to do 1 to: 7 by: (Margin fromNumber: 1) And even worse 1 to: Object new by: (Margin fromNumber: 1) I think adding type-validations all over the place is not a good solution, and is kind of opposite to our philosophy... So we should - document the good usages - document the bad ones - and live with the fact that we have a relaxed type system that will fail at runtime :) Guille

Sept. 18, 2018

Float should not implement #to:, #to:by:, etc...

by Julien

Hello, I realised that it is possible to create an interval of floats. I think this is bad because, since intervals are computed by successively adding a number, it might result in precision errors. (0.0 to: 1.0 by: 0.1) asArray >>> #(0.0 0.1 0.2 0.30000000000000004 0.4 0.5 0.6000000000000001 0.7000000000000001 0.8 0.9 1.0) The correct (precise) way to do it would be to use ScaledDecimal: (0.0s1 to: 1.0s1 by: 0.1s1) asArray >>> #(0.0s1 0.1s1 0.2s1 0.3s1 0.4s1 0.5s1 0.6s1 0.7s1 0.8s1 0.9s1 1.0s1) I opened an issue about it: https://pharo.fogbugz.com/f/cases/22467/Float-should-not-implement-to-to-by… <https://pharo.fogbugz.com/f/cases/22467/Float-should-not-implement-to-to-by…> And Iâd like to discuss this with you. What do you think? Cheers, Julien --- Julien Delplanque Doctorant Ã lâUniversitÃ© de Lille http://juliendelplanque.be/phd.html Equipe Rmod, Inria BÃ¢timent B 40, Avenue Halley 59650 Villeneuve d'Ascq NumÃ©ro de tÃ©lÃ©phone: +333 59 35 86 40

Sept. 18, 2018