What's important is to create abstract layers that insulate the un-needed complexity in lowest layers possible.

The VM excels at insulating of course.

At image side we have to assume the responsibility of not leaking too much by ourself.

As Eliot said, right now the VM (and FFI) just take sequences of uninterpreted bytes (ByteArray) and pass them to API.

The conversion ByteString/WideString <-> specifically-encoded ByteArray is performed at image side.

With FFI, we could eventually make this conversion platform specific instead of always UTF8.

The purpose would be to reduce back and forth conversions in chained API calls for example.

For sanity, then better follow those rules:

- the image does not attempt direct interaction with these opaque data (other than thru OS API)

- nor preserve them across snapshots.

Beware, conversion is not platform specific, but can be library specific (some library on windows will take UTF8).

So we may reify the library and always double dispatch to the library, or we create upper level abstract messages that may chain several low level OS API calls.

We would thus let complexity creep one more level, but only if we have good reason to do so.

We don't want to trade uniformity for small gains.

BTW, note that the xxxW API is already a huge uniformisation progress compared to the code-page specific xxxA API!

Another strategy is to create more complex abstractions (i.e. parameterized) that can deal with a zoo of different underlying conventions.

For example, this would be the EncodedString of VW.

This strategy could be tempting, because it enables dealing with lower level platform-specific-encoded objects and still interact with them in the image transparently.

But I strongly advise to think twice (or more) before introducing such complexity:

- it breaks former invariants (thus potentially lot of code)

- complexity tends to spread in many places

I don't recommend it.

PS: oups, sorry for out of band message, I wanted to send, but it seems that I did not press the button properly...

> On 16 Jan 2019, at 10:59, Guillermo Polito <guillermopolito@gmail.com> wrote:
>
> Hi Nicolas,
>
> On Wed, Jan 16, 2019 at 10:25 AM Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
> IMO, windows VM (and plugins) should do the UCS2 -> UTF8 conversion because the purpose of a VM is to provide an OS independant fa��ade.
> I made progress recently in this area, but we should finish the job/test/consolidate.
>
> I'm following your changes for windows from the shadows and I think they are awesome :).
> 
> If someone bypass the VM and use direct windows API thru FFI, then he takes the responsibility, but uniformity doesn't hurt.
>
>  So far we are using FFI for this, as you say we create first Win32WideStrings from utf8 strings and then we use ffi calls to the *W functions.
> I don't think we can make it for Pharo7.0.0. The cycle to build, do some acceptance tests, and then bless a new VM as stable is far too long for our inminent release :).
>
> But this could be for a 7.1.0, and if you like I can surely give a hand on this.
>
> Guille

--

_,,,^..^,,,_

best, Eliot