File path encoding

older
Pharo Launcher on Windows > Failing

Sven Van Caekenberghe

April 16, 2018

9:51 a.m.

Hi, Some people raise issues with file path encoding with non-ASCII characters. The following works fine for me (macOS, Pharo 7, using new File class): "works" (FileLocator temp / 'foo' , 'txt') writeStreamDo: [ :out | out << 'Les Ã©lÃ¨ves FranÃ§ais' ; crlf ]. (FileLocator temp / 'foo' , 'txt') readStreamDo: [ :in | in upToEnd ]. "works" (FileLocator temp / 'benoÃ®t' , 'txt') writeStreamDo: [ :out | out << 'Les Ã©lÃ¨ves FranÃ§ais' ; crlf ]. (FileLocator temp / 'benoÃ®t' , 'txt') readStreamDo: [ :in | in upToEnd ]. "works" (FileLocator temp / 'benoÃ®t') ensureCreateDirectory. (FileLocator temp / 'benoÃ®t' / 'benoÃ®t' , 'txt') writeStreamDo: [ :out | out << 'Les Ã©lÃ¨ves FranÃ§ais' ; crlf ]. (FileLocator temp / 'benoÃ®t' / 'benoÃ®t' , 'txt') readStreamDo: [ :in | in upToEnd ]. Can Linux/Windows users confirm that it works on their platform ? Sven

Show replies by date

Ben Coman

April 16, 2018

10:52 a.m.

On 16 April 2018 at 17:51, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

Hi,

Some people raise issues with file path encoding with non-ASCII characters.

The following works fine for me (macOS, Pharo 7, using new File class):

"works"

(FileLocator temp / 'foo' , 'txt') writeStreamDo: [ :out | out << 'Les Ã©lÃ¨ves FranÃ§ais' ; crlf ].

(FileLocator temp / 'foo' , 'txt') readStreamDo: [ :in | in upToEnd ].

"works"

(FileLocator temp / 'benoÃ®t' , 'txt') writeStreamDo: [ :out | out << 'Les Ã©lÃ¨ves FranÃ§ais' ; crlf ].

(FileLocator temp / 'benoÃ®t' , 'txt') readStreamDo: [ :in | in upToEnd ].

"works"

(FileLocator temp / 'benoÃ®t') ensureCreateDirectory.

(FileLocator temp / 'benoÃ®t' / 'benoÃ®t' , 'txt') writeStreamDo: [ :out | out << 'Les Ã©lÃ¨ves FranÃ§ais' ; crlf ].

(FileLocator temp / 'benoÃ®t' / 'benoÃ®t' , 'txt') readStreamDo: [ :in | in upToEnd ].

Can Linux/Windows users confirm that it works on their platform ?

Thanks for attacking this. All three work on Windows 10 English locale. I'd hope it wasn't different with French locale, but wait for a report on that. cheers -ben

Ben Coman

11:05 a.m.

On 16 April 2018 at 18:52, Ben Coman <btc@openinworld.com> wrote:

...

On 16 April 2018 at 17:51, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...
Hi,

Some people raise issues with file path encoding with non-ASCII characters.

The following works fine for me (macOS, Pharo 7, using new File class):

"works"

(FileLocator temp / 'foo' , 'txt') writeStreamDo: [ :out | out << 'Les Ã©lÃ¨ves FranÃ§ais' ; crlf ].

(FileLocator temp / 'foo' , 'txt') readStreamDo: [ :in | in upToEnd ].

"works"

(FileLocator temp / 'benoÃ®t' , 'txt') writeStreamDo: [ :out | out << 'Les Ã©lÃ¨ves FranÃ§ais' ; crlf ].

(FileLocator temp / 'benoÃ®t' , 'txt') readStreamDo: [ :in | in upToEnd ].

"works"

(FileLocator temp / 'benoÃ®t') ensureCreateDirectory.

(FileLocator temp / 'benoÃ®t' / 'benoÃ®t' , 'txt') writeStreamDo: [ :out | out << 'Les Ã©lÃ¨ves FranÃ§ais' ; crlf ].

(FileLocator temp / 'benoÃ®t' / 'benoÃ®t' , 'txt') readStreamDo: [ :in | in upToEnd ].

Can Linux/Windows users confirm that it works on their platform ?

Thanks for attacking this. All three work on Windows 10 English locale. I'd hope it wasn't different with French locale, but wait for a report on that.

cheers -ben

I unzipped a fresh download of http://files.pharo.org/platform/Pharo6.1-win.zip * into C:\Temp\Pharo6.1 and immediately copied that directory * into C:\Temp\Pharo6.1.benoÃ®t Double-clicking on Pharo.exe * in the former, opens Pharo an expected. * in the latter, does nothing. cheers -ben

Sven Van Caekenberghe

11:57 a.m.

Ben, On my machine, macOS 10.13.4, $ pwd /tmp/benoÃ®t $ curl get.pharo.org/70+vm | bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3036 100 3036 0 0 38694 0 --:--:-- --:--:-- --:--:-- 38923 Downloading the latest 70 Image: http://files.pharo.org/get-files/70/pharo.zip Pharo.image Downloading the latest pharoVM: http://files.pharo.org/get-files/70/pharo-mac-stable.zip pharo-vm/Pharo.app/Contents/MacOS/Pharo Creating starter scripts pharo and pharo-ui $ ./pharo Pharo.image printVersion [version] 'Pharo7.0' 'Pharo-7.0+alpha.build.769.sha.244850d3ffc1b49ab716c8093f4bcd9af3917be6 (32 Bit)' $ ./pharo Pharo.image eval 'FileLocator image resolve' File @ /private/tmp/benoÃ®t/Pharo.image And the UI version works too. So we're back at 'it is a Windows problem' (but maybe it is only visible on Windows). Can you see what is going on ? Technically, at the File (FilePlugin) level we seem to be able to handle paths with strange characters, even on Windows, but somehow, somewhere a bogus call is made during image startup.

...

On 16 Apr 2018, at 13:05, Ben Coman <btc@openinworld.com> wrote:

On 16 April 2018 at 18:52, Ben Coman <btc@openinworld.com> wrote:

...
On 16 April 2018 at 17:51, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...
Hi,

Some people raise issues with file path encoding with non-ASCII characters.

The following works fine for me (macOS, Pharo 7, using new File class):

"works"

(FileLocator temp / 'foo' , 'txt') writeStreamDo: [ :out | out << 'Les Ã©lÃ¨ves FranÃ§ais' ; crlf ].

(FileLocator temp / 'foo' , 'txt') readStreamDo: [ :in | in upToEnd ].

"works"

(FileLocator temp / 'benoÃ®t' , 'txt') writeStreamDo: [ :out | out << 'Les Ã©lÃ¨ves FranÃ§ais' ; crlf ].

(FileLocator temp / 'benoÃ®t' , 'txt') readStreamDo: [ :in | in upToEnd ].

"works"

(FileLocator temp / 'benoÃ®t') ensureCreateDirectory.

(FileLocator temp / 'benoÃ®t' / 'benoÃ®t' , 'txt') writeStreamDo: [ :out | out << 'Les Ã©lÃ¨ves FranÃ§ais' ; crlf ].

(FileLocator temp / 'benoÃ®t' / 'benoÃ®t' , 'txt') readStreamDo: [ :in | in upToEnd ].

Can Linux/Windows users confirm that it works on their platform ?

Thanks for attacking this. All three work on Windows 10 English locale. I'd hope it wasn't different with French locale, but wait for a report on that.

cheers -ben

I unzipped a fresh download of http://files.pharo.org/platform/Pharo6.1-win.zip * into C:\Temp\Pharo6.1 and immediately copied that directory * into C:\Temp\Pharo6.1.benoÃ®t

Double-clicking on Pharo.exe * in the former, opens Pharo an expected. * in the latter, does nothing.

cheers -ben

Alistair Grant

12:51 p.m.

Hi Sven, On 16 April 2018 at 13:57, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

Technically, at the File (FilePlugin) level we seem to be able to handle paths with strange characters, even on Windows, but somehow, somewhere a bogus call is made during image startup.

I'm not going to have time to look at this in the next few days, sorry. But I have a vague memory that some of the other primitives outside FilePlugin don't add the "\\?\" prefix, so they possibly also don't do wide character conversion. Cheers, Alistair

Sven Van Caekenberghe

1:10 p.m.

...

On 16 Apr 2018, at 14:51, Alistair Grant <akgrant0710@gmail.com> wrote:

Hi Sven,

On 16 April 2018 at 13:57, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...
Technically, at the File (FilePlugin) level we seem to be able to handle paths with strange characters, even on Windows, but somehow, somewhere a bogus call is made during image startup.

I'm not going to have time to look at this in the next few days, sorry.

But I have a vague memory that some of the other primitives outside FilePlugin don't add the "\\?\" prefix, so they possibly also don't do wide character conversion.

Then I hope some other Windows user/developer can have a look. Hell, I believe we got a few French people working on Pharo, n'est pas ?

...

Cheers, Alistair

Sven Van Caekenberghe

5 p.m.

Actually, (from another thread) I think that it is related to how environment variables are accessed. Consider: $ FOO=benoÃ®t ./pharo Pharo.image eval 'OSEnvironment current associations' {'TERM_PROGRAM'->'Apple_Terminal'. 'TERM'->'xterm-256color'. 'SHELL'->'/bin/bash'. 'TMPDIR'->'/var/folders/sy/sndrtj9j1tq06j0lfnshmrl80000gn/T/'. 'FOO'->'benoÃÂ®t'. 'Apple_PubSub_Socket_Render'->'/private/tmp/com.apple.launchd.uWk7pivcLT/Render'. 'TERM_PROGRAM_VERSION'->'404'. 'TERM_SESSION_ID'->'845BECCD-0AB0-4686-B7F9-3A0FF84BDCB7'. 'USER'->'sven'. 'SSH_AUTH_SOCK'->'/private/tmp/com.apple.launchd.y5oCwdUyaG/Listeners'. 'PATH'->'/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/texbin:/opt/X11/bin'. 'PWD'->'/tmp/benoiÃÂt'. 'XPC_FLAGS'->'0x0'. 'XPC_SERVICE_NAME'->'0'. 'HOME'->'/Users/sven'. 'SHLVL'->'2'. 'LOGNAME'->'sven'. 'LC_CTYPE'->'UTF-8'. 'DISPLAY'->'/private/tmp/com.apple.launchd.lsgASYFiWW/org.macosforge.xquartz:0'. 'SECURITYSESSIONID'->'186a9'. 'OLDPWD'->'/tmp/benoiÃÂt'. '_'->'/tmp/benoiÃÂt/pharo-vm/Pharo.app/Contents/MacOS/Pharo'. '__CF_USER_TEXT_ENCODING'->'0x1F5:0x0:0x0'} As you can see, several values are wrongly decoded - and this is on macOS as well.

...

On 16 Apr 2018, at 15:10, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...
On 16 Apr 2018, at 14:51, Alistair Grant <akgrant0710@gmail.com> wrote:

Hi Sven,

On 16 April 2018 at 13:57, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...
Technically, at the File (FilePlugin) level we seem to be able to handle paths with strange characters, even on Windows, but somehow, somewhere a bogus call is made during image startup.

I'm not going to have time to look at this in the next few days, sorry.

But I have a vague memory that some of the other primitives outside FilePlugin don't add the "\\?\" prefix, so they possibly also don't do wide character conversion.

Then I hope some other Windows user/developer can have a look. Hell, I believe we got a few French people working on Pharo, n'est pas ?

...
Cheers, Alistair

Ben Coman

1:15 p.m.

...

...
On 16 Apr 2018, at 13:05, Ben Coman <btc@openinworld.com> wrote:

On 16 April 2018 at 18:52, Ben Coman <btc@openinworld.com> wrote:

...
On 16 April 2018 at 17:51, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...
Hi,

Some people raise issues with file path encoding with non-ASCII characters.

The following works fine for me (macOS, Pharo 7, using new File class):

"works"

(FileLocator temp / 'foo' , 'txt') writeStreamDo: [ :out | out << 'Les Ã©lÃ¨ves FranÃ§ais' ; crlf ].

(FileLocator temp / 'foo' , 'txt') readStreamDo: [ :in | in upToEnd ].

"works"

(FileLocator temp / 'benoÃ®t' , 'txt') writeStreamDo: [ :out | out << 'Les Ã©lÃ¨ves FranÃ§ais' ; crlf ].

(FileLocator temp / 'benoÃ®t' , 'txt') readStreamDo: [ :in | in upToEnd ].

"works"

(FileLocator temp / 'benoÃ®t') ensureCreateDirectory.

(FileLocator temp / 'benoÃ®t' / 'benoÃ®t' , 'txt') writeStreamDo: [ :out | out << 'Les Ã©lÃ¨ves FranÃ§ais' ; crlf ].

(FileLocator temp / 'benoÃ®t' / 'benoÃ®t' , 'txt') readStreamDo: [ :in | in upToEnd ].

Can Linux/Windows users confirm that it works on their platform ?

Thanks for attacking this. All three work on Windows 10 English locale. I'd hope it wasn't different with French locale, but wait for a report on that.

cheers -ben

I unzipped a fresh download of http://files.pharo.org/platform/Pharo6.1-win.zip * into C:\Temp\Pharo6.1 and immediately copied that directory * into C:\Temp\Pharo6.1.benoÃ®t

Double-clicking on Pharo.exe * in the former, opens Pharo an expected. * in the latter, does nothing.

On 16 April 2018 at 19:57, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

Ben,

On my machine, macOS 10.13.4,

$ pwd /tmp/benoÃ®t

$ curl get.pharo.org/70+vm | bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3036 100 3036 0 0 38694 0 --:--:-- --:--:-- --:--:-- 38923 Downloading the latest 70 Image: http://files.pharo.org/get-files/70/pharo.zip Pharo.image Downloading the latest pharoVM: http://files.pharo.org/get-files/70/pharo-mac-stable.zip pharo-vm/Pharo.app/Contents/MacOS/Pharo Creating starter scripts pharo and pharo-ui

$ ./pharo Pharo.image printVersion [version] 'Pharo7.0' 'Pharo-7.0+alpha.build.769.sha.244850d3ffc1b49ab716c8093f4bcd9af3917be6 (32 Bit)'

$ ./pharo Pharo.image eval 'FileLocator image resolve' File @ /private/tmp/benoÃ®t/Pharo.image

And the UI version works too.

So we're back at 'it is a Windows problem' (but maybe it is only visible on Windows).

Can you see what is going on ?

Technically, at the File (FilePlugin) level we seem to be able to handle paths with strange characters, even on Windows, but somehow, somewhere a bogus call is made during image startup.

I tried logging the startup like this... WorkingSession >> runStartup: isImageStarting 'C:\Temp\StartupLog.txt' asFileReference writeStreamDo: [ :log | log nextPutAll: log className; flush. self runList: manager startupList do: [ :each | log nextPutAll: each printString; nextPutAll: String crlf; flush. each startup: isImageStarting ]. self executeDeferredStartupActions: isImageStarting ] which worked in the former and with the latter(benoÃ®t) nothing. But poking around I bumped into an interesting symptom. The following does nothing.... C:\Temp\Pharo6.1.benoÃ®t> pharo but the following successfully opens the Image... C:\Temp\Pharo6.1.benoÃ®t> pharo Pharo6.1.image Not sure where to go from there. Maybe needs a Windows Debug-VM, but I'm out of time for now. cheers -ben

3028

Age (days ago)

3028

Last active (days ago)

List overview

7 comments

3 participants

participants (3)

Alistair Grant
Ben Coman
Sven Van Caekenberghe