On Wed, Jul 25, 2018 at 02:20:30PM +0200, Damien Pollet wrote:
On Wed, 25 Jul 2018 at 13:48, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 25 Jul 2018, at 13:39, Damien Pollet <damien.pollet+pharo@gmail.com> wrote: Related issue: command line arguments come from VM system attributes as ByteStrings??? and thus interpreted as iso-8859-1, which is incorrect in most cases nowadays, even though it seems to work as long as you only use ASCII. Decoding them is easy enough, but it requires two copies (asByteString utf8Decoded)
Yes this is a really big issue. Anything coming in as command line arg or environment variable (or clipboard) is in a basically unknown OS determined encoding. I would assume/hope the UTF-8 is the sensible default today, but apparently not. And it is hard to find a cross platform solution.
My point here was that it would make more sense for those to be passed into the image as ByteArrays, revealing the fact that their encoding is unknown. Currently the bytes are correct, but since they've been shoved into ByteStrings by the VM, the characters will be wrong unless your system happens to be using Latin 1.
That sounds right to me. Having said that, there should be no need to change the VM interface to do this. A ByteString is by definition an array of 8 bit wide characters, and conversion between ByteString and ByteArray is trivial. Any necessary changes can be done without touching the VM. Dave
I suppose we can either have a setting for decoding (since it's pretty much arbitrary), or heuristics like checking LC_CTYPE or whatever. Pablo mentioned the Locale class, but it doesn't seem to detect anything correct from the environment.