On Wed, 16 Jan 2019 at 18:37, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Still, one of the conclusions of previous discussions about the encoding of environment variables was/is that there is no single correct solution. OS's are not consistent in how the encoding is done in all (historical) contexts (like sometimes,
1 env var defines the encoding to use for others,
ouch. That one point nearly made my retract my comment next paragraph, but is there much more complexity? or just a case of utf8<==>appSpecificEncoding rather than ascii<==>appSpecificEncoding ? Sorry if I'm rehashing past discussion (do you have a link?), but considering... * 92% of web pages are UTF8 encoded[1] such that pragmatically UTF8 *is* the standard for text * Strings so pervasive in a system ...would there be an overall benefit to adopt UTF8 as the encoding for Strings consistently provided across the cross-platform vm interface? (i.e. fixing platforms that don't comply to the standard due to their historical baggage) And I found it interesting Microsoft are making some moves towards UTF8 [2]... "With insider build 17035 and the April 2018 update (nominal build 17134) for Windows 10, a "Beta: Use Unicode UTF-8 for worldwide language support" checkbox appeared for setting the locale code page to UTF-8.[a] This allows for calling "narrow" functions, including fopen and SetWindowTextA, with UTF-8 strings. " The approach vm-side could be similar to Section 10 How to do text on Windows [3] with the philosophy of "performing the [conversions] as close to API calls as possible, and never holding the [converted] data." [1] https://w3techs.com/technologies/history_overview/character_encoding/ms/y [2] https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows [3] http://utf8everywhere.org/ different applications do different things, and other such nice stuff), and
certainly not across platforms.
So this is really complex.
Do we want to hide this in some obscure VM C code that very few people can see, read, let alone help with ?
The image side is perfectly capable of dealing with platform differences in a clean/clear way, and at least we can then use the full power of our language and our tools.
Big question... Do we currently have primitives of the same name returning different encodings on different platforms? I presume that would be awkward. If the image is handle encoding differences, should separate primitives be used? e.g. utf8GetEnv & utf16getEnv Could I get some feedback on [4] saying... **The Single Most Important Fact About Encodings** If you completely forget everything I just explained, please remember one extremely important fact. It does not make sense to have a string without knowing what encoding it uses. " And so... does our String nowadays require an 'encoding' instance variable such that this is *always* associated? This might remove any need for separate utf8GetEnv & utf16getEnv (if that was even a reasonable idea). cheers -ben [4] https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-softwar...
On 16 Jan 2019, at 10:59, Guillermo Polito <guillermopolito@gmail.com> wrote:
Hi Nicolas,
On Wed, Jan 16, 2019 at 10:25 AM Nicolas Cellier < nicolas.cellier.aka.nice@gmail.com> wrote: IMO, windows VM (and plugins) should do the UCS2 -> UTF8 conversion because the purpose of a VM is to provide an OS independant façade. I made progress recently in this area, but we should finish the job/test/consolidate.
I'm following your changes for windows from the shadows and I think they are awesome :).
If someone bypass the VM and use direct windows API thru FFI, then he takes the responsibility, but uniformity doesn't hurt.
So far we are using FFI for this, as you say we create first Win32WideStrings from utf8 strings and then we use ffi calls to the *W functions. I don't think we can make it for Pharo7.0.0. The cycle to build, do some acceptance tests, and then bless a new VM as stable is far too long for our inminent release :).
But this could be for a 7.1.0, and if you like I can surely give a hand on this.
Guille