NativeBoost and lacking string argument/return handling
I just had a glance at Nix/Win32Environment NB calls, and nearly cried... (context: http://www.youtube.com/watch?v=Um41DPPs5ZA&list=SP843D1D545F9F52B6&index=15&... ) Which got me thinking, maybe an extra opt (possibly non-static, ref database libraries string encodings) if Strings are expected as parameters should be mandatory, for specifying the expected encoding of the library being called? (oh yes, MANDATORY, if you don't know what is required, you shouldn't be passing in Smalltalk strings in the first place) You'd then be able to cut out ALL the manual cruft in the video's getEnvironmentVariableW: wrapper call basically needed for any current call to a method accepting Strings. (assuming you're not ok with mysterious failures once parameter/return is outside ascii range) Might it work with a few choice assembly implementations of encoders (utf8 + utf16) along the lines of NBUTF8StringExample (expanded to handle both Wide- and ByteString arguments/returns), as well as a fallback (or just) using callback which employs in-image converters? (Such a preamble would also be a good place to do embedded 0 character checks for args expected to be c strings: http://www.hakipedia.com/index.php/Poison_Null_Byte) This have some other implications as well of course; -NBExternalAddress readString/writeString should be removed or made encoding aware. In the current form, they are dangerous/useless outside of pure ascii⦠- How to handle char * parameters. Perhaps parameter-specific validity checks? a) NBExternalAddress - Assume correctly manually encoded contents if allowExternalAddressOpt or whatever its called is set? b) ByteArray - Assume correctly manually encoded contents? c) String - Same as above, either encoding opt set, or return error Cheers, Henry
Well, to my thinking , in C 'string' defined as 'a sequence of characters (bytes) terminated by zero byte'. There is nothing about encoding, because it is related to text, but C 'string' can be something else than human-readable text and contain anything and used for different purpose.
From that perspective, if strictly following such definition, i do not see how you can force encoding to be a mandatory part of it. (except from having special encoding, named "NO_ENCODING" ;)
But sure thing, i am not opposed to having a really nice encoding/decoding schemes, which you can easily control. On 5 July 2013 15:45, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
I just had a glance at Nix/Win32Environment NB calls, and nearly cried... (context: http://www.youtube.com/watch?v=Um41DPPs5ZA&list=SP843D1D545F9F52B6&index=15&... )
Which got me thinking, maybe an extra opt (possibly non-static, ref database libraries string encodings) if Strings are expected as parameters should be mandatory, for specifying the expected encoding of the library being called? (oh yes, MANDATORY, if you don't know what is required, you shouldn't be passing in Smalltalk strings in the first place)
You'd then be able to cut out ALL the manual cruft in the video's getEnvironmentVariableW: wrapper call basically needed for any current call to a method accepting Strings. (assuming you're not ok with mysterious failures once parameter/return is outside ascii range)
Might it work with a few choice assembly implementations of encoders (utf8 + utf16) along the lines of NBUTF8StringExample (expanded to handle both Wide- and ByteString arguments/returns), as well as a fallback (or just) using callback which employs in-image converters? (Such a preamble would also be a good place to do embedded 0 character checks for args expected to be c strings: http://www.hakipedia.com/index.php/Poison_Null_Byte)
This have some other implications as well of course; -NBExternalAddress readString/writeString should be removed or made encoding aware. In the current form, they are dangerous/useless outside of pure ascii⦠- How to handle char * parameters. Perhaps parameter-specific validity checks? a) NBExternalAddress - Assume correctly manually encoded contents if allowExternalAddressOpt or whatever its called is set? b) ByteArray - Assume correctly manually encoded contents? c) String - Same as above, either encoding opt set, or return error
The caveat is that options is applicable to whole callout's generated code. What if i have function which takes 2 arguments: - string with utf-8 encoding - string with utf-16 encoding so, you cannot put that in options, because you need to control it on a per-argument basis. The only way i see how to do that is same as in NBUTF8StringExample: you explicitly designate the type of argument e.g. void foo(utf8String x, utf16String y)
Cheers, Henry
-- Best regards, Igor Stasenko.
On Jul 5, 2013, at 4:59 , Igor Stasenko wrote:
Well, to my thinking , in C 'string' defined as 'a sequence of characters (bytes) terminated by zero byte'. Yeah, except, you know, most/all libraries do not really deal with theoretically pure C strings, and we certainly when passing in "a string" from Smalltalk. In practice, we're dealing with passing/converting Smalltalk strings to functions expecting a pointer to bytes in a certain encoding. (either null-terminated or with an additional size arg)
There is nothing about encoding, because it is related to text, but C 'string' can be something else than human-readable text and contain anything and used for different purpose.
From that perspective, if strictly following such definition, i do not see how you can force encoding to be a mandatory part of it. (except from having special encoding, named "NO_ENCODING" ;)
And from that perspective, any user will *never* create safe bindings to/from anything but ASCII ByteStrings by default. FFI, meet 1988. And even those who do, have to jump through an excessive amount of hoops, ref https://dl.dropboxusercontent.com/u/6751081/Environment.zip
But sure thing, i am not opposed to having a really nice encoding/decoding schemes, which you can easily control.
The caveat is that options is applicable to whole callout's generated code. What if i have function which takes 2 arguments: - string with utf-8 encoding - string with utf-16 encoding
so, you cannot put that in options, because you need to control it on a per-argument basis.
The only way i see how to do that is same as in NBUTF8StringExample: you explicitly designate the type of argument e.g.
void foo(utf8String x, utf16String y)
No, not for FFI you don't. Any sane external library will never use different encodings for string parameters to the same method. In fact, I challenge you to find *any* which doesn't. Cheers, Henry
Was leaving work in a hurry there, think I forgot a not and some smileys in the last post ;) Anyways, in combination with an encoding opt, a nice addition (for those functions expecting size of string as separate parameter) would be being able to specify strLength(myString) as the size parameter, and have it computed after conversion, otherwise you still have to do it all in an image-side wrapper to get the correct number... 2 weeks vacation upcoming, I guess I got plans if the weather is bad ;) Cheers, Henry
On 5 July 2013 18:17, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Was leaving work in a hurry there, think I forgot a not and some smileys in the last post ;) Anyways, in combination with an encoding opt, a nice addition (for those functions expecting size of string as separate parameter) would be being able to specify strLength(myString) as the size parameter, and have it computed after conversion, otherwise you still have to do it all in an image-side wrapper to get the correct number...
2 weeks vacation upcoming, I guess I got plans if the weather is bad ;)
i hope so (so you can contribute to NB ;)))
Cheers, Henry
-- Best regards, Igor Stasenko.
On 5 July 2013 17:48, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
On Jul 5, 2013, at 4:59 , Igor Stasenko wrote:
Well, to my thinking , in C 'string' defined as 'a sequence of characters (bytes) terminated by zero byte'. Yeah, except, you know, most/all libraries do not really deal with theoretically pure C strings, and we certainly when passing in "a string" from Smalltalk. In practice, we're dealing with passing/converting Smalltalk strings to functions expecting a pointer to bytes in a certain encoding. (either null-terminated or with an additional size arg)
There is nothing about encoding, because it is related to text, but C 'string' can be something else than human-readable text and contain anything and used for different purpose.
From that perspective, if strictly following such definition, i do not see how you can force encoding to be a mandatory part of it. (except from having special encoding, named "NO_ENCODING" ;)
And from that perspective, any user will *never* create safe bindings to/from anything but ASCII ByteStrings by default. FFI, meet 1988. And even those who do, have to jump through an excessive amount of hoops, ref https://dl.dropboxusercontent.com/u/6751081/Environment.zip
But sure thing, i am not opposed to having a really nice encoding/decoding schemes, which you can easily control.
The caveat is that options is applicable to whole callout's generated code. What if i have function which takes 2 arguments: - string with utf-8 encoding - string with utf-16 encoding
so, you cannot put that in options, because you need to control it on a per-argument basis.
The only way i see how to do that is same as in NBUTF8StringExample: you explicitly designate the type of argument e.g.
void foo(utf8String x, utf16String y)
No, not for FFI you don't. Any sane external library will never use different encodings for string parameters to the same method. In fact, I challenge you to find *any* which doesn't.
i agree. So, what's the plan? Each time user puts 'String' pseudo-type, demand encoding?
Cheers, Henry
-- Best regards, Igor Stasenko.
participants (3)
-
Henrik Johansen -
Henrik Sperre Johansen
-
Igor Stasenko