- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Platform file encoding for FFI

by Guillermo Polito

On Tue, Sep 18, 2018 at 10:43 AM Henrik Sperre Johansen < henrik.s.johansen(a)veloxit.no> wrote: > Guillermo Polito wrote > > On Mon, Sep 17, 2018 at 6:52 PM Alistair Grant < > > > akgrant0710@ > > > > > > wrote: > > > >> Hi Esteban, Guille and Everyone, > >> > >> I haven't looked at using FFI much, however it is easy to imagine that > >> different file encoding rules on different platforms will make writing > >> FFI calls more difficult, > > > > > > Well not really (from my point of view :)) > > From the point of view of the FFI call an encoded string is just a bunch > > of > > bytes. FFI does not do any interpretation of them. > > It *would* be pretty handy for adding some auto-conversion into the > marshaller based on parameter encoding options though... (other than > filename, could be done in smalltalk using exisiting encoders) > > self > ffiCall: #(bool saveContentsToFile(String fileName, String contents)) > options: #(+stringEncodings( fileName return , platformAPI contents) > Well, I like this idea. (And yes, I've probably badly mangled the options syntax) > > Is much less verbose than having to manually convert Strings to the proper > platform Unicode encodings before calling. > Depends a bit on whether the primitive argument is > Byte/Widestrings(latin1/utf32), or if it accepts only utf8 bytes and one > has > to convert first anyways. > > It's not like this isn't a pain point, there are plenty of currently used > API's that are broken if you try to use non-ascii. > Yes, but I think this may be because in general people tend to not know how encodings work... (even myself I don't feel I know enough :)) But this makes me think that we should make encoding explicit? Maybe we should force people to specify an encoding if they specify a callout using a string. And then, either they specify it at the level of the callout, or at the level of the library (like setting a default encoding for all strings). Because this raises also the question of what is the default encoding? And I'd say that in there is no satisfactory default encoding...

Sept. 18, 2018

Re: [Pharo-dev] Platform file encoding for FFI

by Henrik Sperre Johansen

Guillermo Polito wrote > On Mon, Sep 17, 2018 at 6:52 PM Alistair Grant < > akgrant0710@ > > > wrote: > > >> Would it be useful for future FFI work to have primitives available >> which convert file names to and from the various platform specific >> formats? (Linux is basically a no-op, and Windows could be written >> in-image, but OSX requires the platform routines to be called). >> > > Maybe... Are the OSX routines exposed as C functions (that we can call > through FFI) or they are objective-C methods/functions (that are more > complicated to map)? > > Thanks Alistair! +1. From the image point of view, the non-standard normal form used on OSX is the biggest issue. If it's available through FFI, the platform-specific String encoding options I mentioned previously could be implemented entirely in the image. If there are extra hoops to jump though, like having to provide utf8 to said FFI function, it might still be worth it for the reduced performance overhead. Cheers, Henry -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Sept. 18, 2018

Re: [Pharo-dev] Platform file encoding for FFI

by Henrik Sperre Johansen

Guillermo Polito wrote > On Mon, Sep 17, 2018 at 6:52 PM Alistair Grant < > akgrant0710@ > > > wrote: > >> Hi Esteban, Guille and Everyone, >> >> I haven't looked at using FFI much, however it is easy to imagine that >> different file encoding rules on different platforms will make writing >> FFI calls more difficult, > > > Well not really (from my point of view :)) > From the point of view of the FFI call an encoded string is just a bunch > of > bytes. FFI does not do any interpretation of them. It *would* be pretty handy for adding some auto-conversion into the marshaller based on parameter encoding options though... (other than filename, could be done in smalltalk using exisiting encoders) self ffiCall: #(bool saveContentsToFile(String fileName, String contents)) options: #(+stringEncodings( fileName return , platformAPI contents) (And yes, I've probably badly mangled the options syntax) Is much less verbose than having to manually convert Strings to the proper platform Unicode encodings before calling. Depends a bit on whether the primitive argument is Byte/Widestrings(latin1/utf32), or if it accepts only utf8 bytes and one has to convert first anyways. It's not like this isn't a pain point, there are plenty of currently used API's that are broken if you try to use non-ascii. Cheers, Henry -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Sept. 18, 2018

Fraction and ScaledDecimal should be not be serialized as Float in STON format

by Julien

Hello, I realised that in the current implementation, when a number is serialised by STON it is either as an integer literal or as a float literal. There is a risk to loose precision, especially if what you serialise is a ScaledDecimal or a fraction. I propose a simple change to fix this: 1. Add STONWriter>>#isInJsonMode method which returns true if the STONWriter wants to write JSON 2. Add: Fraction>>#stonOn: stonWriter stonWriter isInJsonMode ifTrue: [ ^ super stonOn: stonWriter ]. stonWriter writeObject: self streamMap: [ :dictionary | dictionary at: #numerator put: numerator. dictionary at: #denominator put: denominator ] 3. Add: ScaledDecimal>>#stonOn: stonWriter stonWriter isInJsonMode ifTrue: [ ^ super stonOn: stonWriter ]. stonWriter writeObject: self streamMap: [ :dictionary | dictionary at: #numerator put: numerator. dictionary at: #denominator put: denominator. dictionary at: #scale put: scale ] This change has a main drawback, storing fractions and scaled decimal will result in a huge overhead. Maybe for ScaledDecimal we can use Pharoâs literal (e.g. 1.12s2). For fractions I donât know what literal could be used? Or maybe the overhead is fine? Cheers, Julien --- Julien Delplanque Doctorant Ã lâUniversitÃ© de Lille http://juliendelplanque.be/phd.html Equipe Rmod, Inria BÃ¢timent B 40, Avenue Halley 59650 Villeneuve d'Ascq NumÃ©ro de tÃ©lÃ©phone: +333 59 35 86 40

Sept. 18, 2018

Re: [Pharo-dev] UndefinedClasses as first class entities for P7

by Guillermo Polito

On Thu, Sep 13, 2018 at 11:07 PM Tim Mackinnon <tim(a)testit.works> wrote: > Kind of related - we did get the ability to save methods with undefined > classes and undeclared variables (vs having to correct this to save). So at > least a nod to that? > Yes, I was following your progress there :) And since you have practical experience with this in excercism, It would be ***really lovely*** if you can check our paper/code and give us some feedback.

Sept. 18, 2018

Re: [Pharo-dev] Confused about FileReference read/write streams

by Guillermo Polito

I've opened an issue https://pharo.fogbugz.com/f/cases/22466/Add-readWriteStream-and-readWriteSt… On Tue, Sep 18, 2018 at 10:14 AM Guillermo Polito <guillermopolito(a)gmail.com> wrote: > Hi Sven, > > On Mon, Sep 17, 2018 at 4:14 PM Sven Van Caekenberghe <sven(a)stfx.eu> > wrote: > >> Hi, >> >> Since ever (or as far as I can remember), FileReference made a >> distinction between read and write streams, see #readStream[Do:] and >> #writeStream[Do:] - these return a ZnCharacterReadStream and >> ZnCharacterWriteStream in Pharo 7. >> > > I think so too. > > >> >> Along the same lines, #binaryReadStream and #binaryWriteStream where >> introduced. > > > Well, when I did the stream changes on FileSystem, there was already > support for binary streams. I just exposed it in the API :) > > >> However, the first returns a ZnBufferedReadStream (OK) while the latter >> returns a ZnBufferedReadWriteStream. Why ? >> > > I cannot tell now the details, but there were some parts of the system > where it was assumed that a write-stream is read-write. > IIRC this was related with the changes/sources file handling. > > >> >> I think it would be generally better if a standard ZnBufferedWriteStream >> where returned. >> >> ZnBufferedReadWriteStream is not a very pretty class to say the least, it >> also strikes me as confusing. >> >> What would break if we changed this ? >> >> If necessary there could be something like #readWriteStream[Do:] I guess. >> > > That could be a cleaner way to do this, yes. > And this would isolate strange uses. > > >> >> Sven >> >> >> > > -- > > > > Guille Polito > > Research Engineer > > Centre de Recherche en Informatique, Signal et Automatique de Lille > > CRIStAL - UMR 9189 > > French National Center for Scientific Research - *http://www.cnrs.fr > <http://www.cnrs.fr>* > > > *Web:* *http://guillep.github.io* <http://guillep.github.io> > > *Phone: *+33 06 52 70 66 13 > -- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13

Sept. 18, 2018

Re: [Pharo-dev] Confused about FileReference read/write streams

by Guillermo Polito

Hi Sven, On Mon, Sep 17, 2018 at 4:14 PM Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > Hi, > > Since ever (or as far as I can remember), FileReference made a distinction > between read and write streams, see #readStream[Do:] and #writeStream[Do:] > - these return a ZnCharacterReadStream and ZnCharacterWriteStream in Pharo > 7. > I think so too. > > Along the same lines, #binaryReadStream and #binaryWriteStream where > introduced. Well, when I did the stream changes on FileSystem, there was already support for binary streams. I just exposed it in the API :) > However, the first returns a ZnBufferedReadStream (OK) while the latter > returns a ZnBufferedReadWriteStream. Why ? > I cannot tell now the details, but there were some parts of the system where it was assumed that a write-stream is read-write. IIRC this was related with the changes/sources file handling. > > I think it would be generally better if a standard ZnBufferedWriteStream > where returned. > > ZnBufferedReadWriteStream is not a very pretty class to say the least, it > also strikes me as confusing. > > What would break if we changed this ? > > If necessary there could be something like #readWriteStream[Do:] I guess. > That could be a cleaner way to do this, yes. And this would isolate strange uses. > > Sven > > > -- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13

Sept. 18, 2018

Re: [Pharo-dev] Platform file encoding for FFI

by Guillermo Polito

On Mon, Sep 17, 2018 at 6:52 PM Alistair Grant <akgrant0710(a)gmail.com> wrote: > Hi Esteban, Guille and Everyone, > > I haven't looked at using FFI much, however it is easy to imagine that > different file encoding rules on different platforms will make writing > FFI calls more difficult, Well not really (from my point of view :)) >From the point of view of the FFI call an encoded string is just a bunch of bytes. FFI does not do any interpretation of them. > i.e. some of the different formats are: > > - OSX uses Mac specific decomposed UTF8 encoding > - Windows uses Wide Strings (16 bit Unicode characters) > - Linux allows pretty much anything, but precomposed UTF8 is common > > At the image side, we could have an strategy that, depending on the OS, could encode in one encoding or another, or even not encode at all. > Believe it or not, I'm still working on getting the > FileAttributesPlugin working (file name encoding on Windows being the > latest issue - the tests in Pharo need to be extended). > I believe you, don't worry ^^. > Would it be useful for future FFI work to have primitives available > which convert file names to and from the various platform specific > formats? (Linux is basically a no-op, and Windows could be written > in-image, but OSX requires the platform routines to be called). > Maybe... Are the OSX routines exposed as C functions (that we can call through FFI) or they are objective-C methods/functions (that are more complicated to map)? Thanks Alistair!

Sept. 18, 2018

Standardized nomenclature for feature requests

by Esteban A. Maringolo

Hi there, Q: Is there a reason why Pharo (as a project) doesn't follow a standardized nomenclature for features/request as other communities does? So for instance something like PFR-005 (Pharo Feature Request) could be "Truly headless images", PFR-006 "Pharo Bootstrapping", etc. I can't really tell whether this would improve to the overall process, but I wanted to know if this was already debated by the board and dismissed, if it is something that wasn't considered at all, of if we have something equivalent and what it is. Regards, Language examples: JSR (Java Specification Requests) https://www.jcp.org/en/jsr/all PEP (Python Enhancement Proposals) https://www.python.org/dev/peps/ RFC (Rust, Request for Comments) https://github.com/rust-lang/rfcs Technologies: BIP (Bitcoin Improvement Proposal) https://github.com/bitcoin/bips EIP (Ethereum Improvement Proposal) https://eips.ethereum.org/ -- Esteban A. Maringolo

Sept. 17, 2018

ZnMimePart Disposition-Content spaces

by Ben Coman

Hi Sven, We encountered a ZnMimePart Content-Disposition header problem with a server expecting a space following the "form-data;" string. The diff here... https://www.diffchecker.com/ETIfCUzx * on the left shows the working request from the default command-line tool * on the right shows the non-working request from Pharo. The specifications are really unclear about the requirement or not for a space, so it may be reasonable to consider the fault lies with the server not being permissive enough, but I notice these examples all have space after semi colons... * https://tools.ietf.org/html/rfc7578 * https://www.ietf.org/rfc/rfc2388.txt so I wonder if this scenario might be widespread (and endemic to Rails apps which this was) and maybe its pragmatic to include the extra spaces to cut off potential issues (it took quite a while to isolate with the live service being HTTPS only.) Steps to reproduce are here... https://github.com/exercism/pharo/issues/96#issuecomment-421224203 In the meantime, once isolated it was simple to work around, but it was almost a show stopper, so I'm interested in your thoughts. cheers -ben

Sept. 17, 2018