Windows 64bit, long filename issue but after a second load, how to categorise and report this?
We noticed a weird problem when testing the pharo exercism project. When loading on Windows10 in a fresh 64bit image with the following evaluate in the playground: Metacello new baseline: 'Exercism'; repository: 'github://exercism/pharo-smalltalk:master/dev/src'; load. The project seemed to load correctly (but we seemed to hit an issue with an http submit giving a 404 error - which weirdly works in 32bit and OSX 64/32). Anyway - the real question here is that when trying to reproduce this in a second fresh 64bit image 2 users now have reported getting the long filename error: LGit_GIT_EEXISTS: Failed to stat file 'C:/Users/sam/Documents/Pharo/images/exercism-test-run-7.0-32...stance/testRaiseUnexpectedMessageWhenTooMuchMessages.st': The filename or extension is too long. And the second user got the same but with a different filename: ruleRBUtlitiesMethodsRuleV1FalsePositve.st I know there is a long standing problem with Windows and file path lengths - but why would it work the first time and then subsequently fail in fresh installs? Is something being cached on windows that isnât on other platforms? (Writing this - Iâm wondering if there is that shared pharo Iceberg setting for Share repos that might be implicated?) Also - given Exercism is in Tonel format and those files arenât in exercism, and in fact Iâm struggling to find where they are defined (my images donât have them) - there is something very weird going on. I wonder if we have another error that is masking all of this? Ultimately my question is - where is the Pharo issue that is tracking this problem? Iâve seen it mentioned lots of times here, but canât see anything in the issues that I can tack the above onto? For reference - our exercism issue is: https://github.com/exercism/pharo-smalltalk/issues/140 Tim
On Thu, 21 Feb 2019 at 20:36, Tim Mackinnon <tim@testit.works> wrote:
We noticed a weird problem when testing the pharo exercism project.
When loading on Windows10 in a fresh 64bit image with the following evaluate in the playground:
Metacello new baseline: 'Exercism'; repository: 'github://exercism/pharo-smalltalk:master/dev/src'; load.
The project seemed to load correctly (but we seemed to hit an issue with an http submit giving a 404 error - which weirdly works in 32bit and OSX 64/32). Anyway - the real question here is that when trying to reproduce this in a second fresh 64bit image 2 users now have reported getting the long filename error:
LGit_GIT_EEXISTS: Failed to stat file 'C:/Users/sam/Documents/Pharo/images/exercism-test-run-7.0-32...stance/testRaiseUnexpectedMessageWhenTooMuchMessages.st': The filename or extension is too long.
And the second user got the same but with a different filename: ruleRBUtlitiesMethodsRuleV1FalsePositve.st
These files are both method names which indicates FileTree format rather than Tonel (and these "failed to stat file" errors on Windows is one of the reasons for the move to Tonel). It may be from one of the dependencies that are pulled in. To identify which, search for that file under `pharo-local/iceberg` directory. I believe the repo may have been cloned successfully, its just the libgit binary can't `stat` long filenames. If that turns out the be untrue, use commandline git to complete the clone and you will likely see it.
I know there is a long standing problem with Windows and file path lengths - but why would it work the first time and then subsequently fail in fresh installs? Is something being cached on windows that isnât on other platforms? (Writing this - Iâm wondering if there is that shared pharo Iceberg setting for Share repos that might be implicated?)
Also - given Exercism is in Tonel format and those files arenât in exercism, and in fact Iâm struggling to find where they are defined
as above.
(my images donât have them) - there is something very weird going on.
I wonder if we have another error that is masking all of this?
Ultimately my question is - where is the Pharo issue that is tracking this problem? Iâve seen it mentioned lots of times here, but canât see anything in the issues that I can tack the above onto?
Filetree on Windows is constrained by long filename issues with libgit. cheers -ben
Hi Ben - while I understand the description below - why would it work the first time (no errors) and then fail the second time in a fresh image? Are we really sure its just down to non-determinism of the load process (possibly - but it seems very suspicious to me). There is something more fishy about windows going on I suspect⦠I will track down the Exercism dependencies to see if any of them are FileTree - having said that, on OSX (where it loads cleanly - I would expect to be able to find one of those methods in my image - but again I canât? They look like valid methods - but why arenât they in my image? Its like the load process is downloading other stuff on its own and then not loading that into the image? Iâm just worried that we keep blaming windows (and yes its dumb having a file length restriction I this day and age) - but maybe we are missing another problem too? Tim
On 21 Feb 2019, at 13:20, Ben Coman <btc@openinworld.com> wrote:
On Thu, 21 Feb 2019 at 20:36, Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>> wrote: We noticed a weird problem when testing the pharo exercism project.
When loading on Windows10 in a fresh 64bit image with the following evaluate in the playground:
Metacello new baseline: 'Exercism'; repository: 'github://exercism/pharo-smalltalk:master/dev/src'; load.
The project seemed to load correctly (but we seemed to hit an issue with an http submit giving a 404 error - which weirdly works in 32bit and OSX 64/32). Anyway - the real question here is that when trying to reproduce this in a second fresh 64bit image 2 users now have reported getting the long filename error:
LGit_GIT_EEXISTS: Failed to stat file 'C:/Users/sam/Documents/Pharo/images/exercism-test-run-7.0-32...stance/testRaiseUnexpectedMessageWhenTooMuchMessages.st': The filename or extension is too long.
And the second user got the same but with a different filename: ruleRBUtlitiesMethodsRuleV1FalsePositve.st
These files are both method names which indicates FileTree format rather than Tonel (and these "failed to stat file" errors on Windows is one of the reasons for the move to Tonel). It may be from one of the dependencies that are pulled in. To identify which, search for that file under `pharo-local/iceberg` directory. I believe the repo may have been cloned successfully, its just the libgit binary can't `stat` long filenames. If that turns out the be untrue, use commandline git to complete the clone and you will likely see it.
I know there is a long standing problem with Windows and file path lengths - but why would it work the first time and then subsequently fail in fresh installs? Is something being cached on windows that isnât on other platforms? (Writing this - Iâm wondering if there is that shared pharo Iceberg setting for Share repos that might be implicated?)
Also - given Exercism is in Tonel format and those files arenât in exercism, and in fact Iâm struggling to find where they are defined
as above.
(my images donât have them) - there is something very weird going on.
I wonder if we have another error that is masking all of this?
Ultimately my question is - where is the Pharo issue that is tracking this problem? Iâve seen it mentioned lots of times here, but canât see anything in the issues that I can tack the above onto?
Filetree on Windows is constrained by long filename issues with libgit.
cheers -ben
Tim Mackinnon wrote
Hi Ben - while I understand the description below - why would it work the first time (no errors) and then fail the second time in a fresh image? Are we really sure its just down to non-determinism of the load process (possibly - but it seems very suspicious to me). There is something more fishy about windows going on I suspectâ¦
The Iceberg cache path is relative to the image folder, so it's at least possible to have success in one image, located in C:\Pharo\testImage\, while another image, reciding in something like C:\Users\sam\Documents\Pharo\images\exercism-test-run-7.0-32\ , fails to load the exact same filetree package... Tim Mackinnon wrote
I will track down the Exercism dependencies to see if any of them are FileTree - having said that, on OSX (where it loads cleanly - I would expect to be able to find one of those methods in my image - but again I canât? They look like valid methods - but why arenât they in my image? Its like the load process is downloading other stuff on its own and then not loading that into the image?
Yeaaah, that sounds fishy. But at least the first method sounds like a test, which may be part of the packages in the git repo that gets cloned without being loaded by Metacello, if the Metacello dependency specifies a *Core group which does not include tests. Tim Mackinnon wrote
Iâm just worried that we keep blaming windows (and yes its dumb having a file length restriction I this day and age) - but maybe we are missing another problem too?
Tim
To be pendantic, (and not of much actual help, sorry :/ ) the problem isn't really with Windows, which provides API's for getting file info that handle long paths just fine, but with libgit, which is probably compiled against some POSIX stat implementation provided by Cygwin/mingw, which does not. Cheers, Henry -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
Some interesting thoughts below - but my understanding is these images are being created by Pharo Launcher , so they should be isolated from one another? I also donât understand why you can load with 32bit (multiple images) but only once with 64bit. Do we load radically different git code with longer paths on 64bit? (Strange isnât it?) Possibly worth caveating this is with 2-3 users so far, and only noticed this pattern with the last one. Also - has anyone created an issue for these path problems - Iâm curious if my results match anything else. Tim Sent from my iPhone
On 21 Feb 2019, at 16:01, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Tim Mackinnon wrote
Hi Ben - while I understand the description below - why would it work the first time (no errors) and then fail the second time in a fresh image? Are we really sure its just down to non-determinism of the load process (possibly - but it seems very suspicious to me). There is something more fishy about windows going on I suspectâ¦
The Iceberg cache path is relative to the image folder, so it's at least possible to have success in one image, located in C:\Pharo\testImage\, while another image, reciding in something like C:\Users\sam\Documents\Pharo\images\exercism-test-run-7.0-32\ , fails to load the exact same filetree package...
Tim Mackinnon wrote
I will track down the Exercism dependencies to see if any of them are FileTree - having said that, on OSX (where it loads cleanly - I would expect to be able to find one of those methods in my image - but again I canât? They look like valid methods - but why arenât they in my image? Its like the load process is downloading other stuff on its own and then not loading that into the image?
Yeaaah, that sounds fishy. But at least the first method sounds like a test, which may be part of the packages in the git repo that gets cloned without being loaded by Metacello, if the Metacello dependency specifies a *Core group which does not include tests.
Tim Mackinnon wrote
Iâm just worried that we keep blaming windows (and yes its dumb having a file length restriction I this day and age) - but maybe we are missing another problem too?
Tim
To be pendantic, (and not of much actual help, sorry :/ ) the problem isn't really with Windows, which provides API's for getting file info that handle long paths just fine, but with libgit, which is probably compiled against some POSIX stat implementation provided by Cygwin/mingw, which does not.
Cheers, Henry
-- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
Just to add some additional notes on this (and I will also file an issue, as I donât think one has been submitted, and I donât think this is fully down to file path length alone) - After some more testing, we have also seen the error on Windows 32bit as well (so while easier to get in 64bit, it does occur in 32bit like others have reported). My real concern however, is that a simple project (which admittedly pulls in a single Filetree prerequisite - PharoWebBrowser) seems to trigger this error - but confusingly references files that donât appear to be in the image. Tracing through PharoWebBrowser - there are some file tree based pre-reques and some that are mcz - and none of them have files like: testRaiseUnexpectedMessageWhenTooMuchMessages.st And this method is not in the image that gets loaded into OSX. So I am wondering where these long files are coming from? So yes, while there is a long filename issue - is there another hidden issue underneath all of this where files we arenât even using are getting reference somehow? Tim
On 22 Feb 2019, at 00:01, Tim Mackinnon <tim@testit.works> wrote:
Some interesting thoughts below - but my understanding is these images are being created by Pharo Launcher , so they should be isolated from one another?
I also donât understand why you can load with 32bit (multiple images) but only once with 64bit. Do we load radically different git code with longer paths on 64bit? (Strange isnât it?)
Possibly worth caveating this is with 2-3 users so far, and only noticed this pattern with the last one.
Also - has anyone created an issue for these path problems - Iâm curious if my results match anything else.
Tim
Sent from my iPhone
On 21 Feb 2019, at 16:01, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Tim Mackinnon wrote
Hi Ben - while I understand the description below - why would it work the first time (no errors) and then fail the second time in a fresh image? Are we really sure its just down to non-determinism of the load process (possibly - but it seems very suspicious to me). There is something more fishy about windows going on I suspectâ¦
The Iceberg cache path is relative to the image folder, so it's at least possible to have success in one image, located in C:\Pharo\testImage\, while another image, reciding in something like C:\Users\sam\Documents\Pharo\images\exercism-test-run-7.0-32\ , fails to load the exact same filetree package...
Tim Mackinnon wrote
I will track down the Exercism dependencies to see if any of them are FileTree - having said that, on OSX (where it loads cleanly - I would expect to be able to find one of those methods in my image - but again I canât? They look like valid methods - but why arenât they in my image? Its like the load process is downloading other stuff on its own and then not loading that into the image?
Yeaaah, that sounds fishy. But at least the first method sounds like a test, which may be part of the packages in the git repo that gets cloned without being loaded by Metacello, if the Metacello dependency specifies a *Core group which does not include tests.
Tim Mackinnon wrote
Iâm just worried that we keep blaming windows (and yes its dumb having a file length restriction I this day and age) - but maybe we are missing another problem too?
Tim
To be pendantic, (and not of much actual help, sorry :/ ) the problem isn't really with Windows, which provides API's for getting file info that handle long paths just fine, but with libgit, which is probably compiled against some POSIX stat implementation provided by Cygwin/mingw, which does not.
Cheers, Henry
-- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
Iâve created - https://github.com/pharo-vcs/iceberg/issues/1204
On 25 Feb 2019, at 17:12, Tim Mackinnon <tim@testit.works> wrote:
Just to add some additional notes on this (and I will also file an issue, as I donât think one has been submitted, and I donât think this is fully down to file path length alone) -
After some more testing, we have also seen the error on Windows 32bit as well (so while easier to get in 64bit, it does occur in 32bit like others have reported).
My real concern however, is that a simple project (which admittedly pulls in a single Filetree prerequisite - PharoWebBrowser) seems to trigger this error - but confusingly references files that donât appear to be in the image. Tracing through PharoWebBrowser - there are some file tree based pre-reques and some that are mcz - and none of them have files like: testRaiseUnexpectedMessageWhenTooMuchMessages.st
And this method is not in the image that gets loaded into OSX. So I am wondering where these long files are coming from?
So yes, while there is a long filename issue - is there another hidden issue underneath all of this where files we arenât even using are getting reference somehow?
Tim
On 22 Feb 2019, at 00:01, Tim Mackinnon <tim@testit.works> wrote:
Some interesting thoughts below - but my understanding is these images are being created by Pharo Launcher , so they should be isolated from one another?
I also donât understand why you can load with 32bit (multiple images) but only once with 64bit. Do we load radically different git code with longer paths on 64bit? (Strange isnât it?)
Possibly worth caveating this is with 2-3 users so far, and only noticed this pattern with the last one.
Also - has anyone created an issue for these path problems - Iâm curious if my results match anything else.
Tim
Sent from my iPhone
On 21 Feb 2019, at 16:01, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Tim Mackinnon wrote
Hi Ben - while I understand the description below - why would it work the first time (no errors) and then fail the second time in a fresh image? Are we really sure its just down to non-determinism of the load process (possibly - but it seems very suspicious to me). There is something more fishy about windows going on I suspectâ¦
The Iceberg cache path is relative to the image folder, so it's at least possible to have success in one image, located in C:\Pharo\testImage\, while another image, reciding in something like C:\Users\sam\Documents\Pharo\images\exercism-test-run-7.0-32\ , fails to load the exact same filetree package...
Tim Mackinnon wrote
I will track down the Exercism dependencies to see if any of them are FileTree - having said that, on OSX (where it loads cleanly - I would expect to be able to find one of those methods in my image - but again I canât? They look like valid methods - but why arenât they in my image? Its like the load process is downloading other stuff on its own and then not loading that into the image?
Yeaaah, that sounds fishy. But at least the first method sounds like a test, which may be part of the packages in the git repo that gets cloned without being loaded by Metacello, if the Metacello dependency specifies a *Core group which does not include tests.
Tim Mackinnon wrote
Iâm just worried that we keep blaming windows (and yes its dumb having a file length restriction I this day and age) - but maybe we are missing another problem too?
Tim
To be pendantic, (and not of much actual help, sorry :/ ) the problem isn't really with Windows, which provides API's for getting file info that handle long paths just fine, but with libgit, which is probably compiled against some POSIX stat implementation provided by Cygwin/mingw, which does not.
Cheers, Henry
-- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
Iâve been able to find out more information about this - and have now got the full filename and error in question: `'C:/Users/sam/Documents/Pharo/images/exercism-test-run-7.0-32bit/pharo-local/iceberg/dionisiydk/StateSpecs/StateSpecs-DSL-PragmaBasedShouldExpressions-Tests.package/SpecOfShouldExpressionDriverTests.class/instance/testRaiseUnexpectedMessageWhenTooMuchMessages.st': The filename or extension is too long.` So the path is 262 characters long. (hence the error) The package this file is in, is a downstream project prerequisite for PharoWebBrowser - which is dragged in because that project doesn't have different groupings for code and tests (something I am working with Sean to try to get resolved - although Iâm scratching my head on how you can test baselines?). We don't see this in the image in OSX because the included baseline doesn't actually pre-req these pragma based tests? However from below, have I now understood that when we reference a project, we end up cloning the whole repository for that repo on disk (so one big read), and then metacello picks out the relevant files from the baseline. So possibly, this might be some orphaned files/packages in the repo - that iceberg no longer knows about, but git does? The longer term fix is to convert PharoWebBrowser to Tonel and see if we can improve the grouping. Tim
On 25 Feb 2019, at 17:19, Tim Mackinnon <tim@testit.works> wrote:
Iâve created - https://github.com/pharo-vcs/iceberg/issues/1204
On 25 Feb 2019, at 17:12, Tim Mackinnon <tim@testit.works> wrote:
Just to add some additional notes on this (and I will also file an issue, as I donât think one has been submitted, and I donât think this is fully down to file path length alone) -
After some more testing, we have also seen the error on Windows 32bit as well (so while easier to get in 64bit, it does occur in 32bit like others have reported).
My real concern however, is that a simple project (which admittedly pulls in a single Filetree prerequisite - PharoWebBrowser) seems to trigger this error - but confusingly references files that donât appear to be in the image. Tracing through PharoWebBrowser - there are some file tree based pre-reques and some that are mcz - and none of them have files like: testRaiseUnexpectedMessageWhenTooMuchMessages.st
And this method is not in the image that gets loaded into OSX. So I am wondering where these long files are coming from?
So yes, while there is a long filename issue - is there another hidden issue underneath all of this where files we arenât even using are getting reference somehow?
Tim
On 22 Feb 2019, at 00:01, Tim Mackinnon <tim@testit.works> wrote:
Some interesting thoughts below - but my understanding is these images are being created by Pharo Launcher , so they should be isolated from one another?
I also donât understand why you can load with 32bit (multiple images) but only once with 64bit. Do we load radically different git code with longer paths on 64bit? (Strange isnât it?)
Possibly worth caveating this is with 2-3 users so far, and only noticed this pattern with the last one.
Also - has anyone created an issue for these path problems - Iâm curious if my results match anything else.
Tim
Sent from my iPhone
On 21 Feb 2019, at 16:01, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Tim Mackinnon wrote
Hi Ben - while I understand the description below - why would it work the first time (no errors) and then fail the second time in a fresh image? Are we really sure its just down to non-determinism of the load process (possibly - but it seems very suspicious to me). There is something more fishy about windows going on I suspectâ¦
The Iceberg cache path is relative to the image folder, so it's at least possible to have success in one image, located in C:\Pharo\testImage\, while another image, reciding in something like C:\Users\sam\Documents\Pharo\images\exercism-test-run-7.0-32\ , fails to load the exact same filetree package...
Tim Mackinnon wrote
I will track down the Exercism dependencies to see if any of them are FileTree - having said that, on OSX (where it loads cleanly - I would expect to be able to find one of those methods in my image - but again I canât? They look like valid methods - but why arenât they in my image? Its like the load process is downloading other stuff on its own and then not loading that into the image?
Yeaaah, that sounds fishy. But at least the first method sounds like a test, which may be part of the packages in the git repo that gets cloned without being loaded by Metacello, if the Metacello dependency specifies a *Core group which does not include tests.
Tim Mackinnon wrote
Iâm just worried that we keep blaming windows (and yes its dumb having a file length restriction I this day and age) - but maybe we are missing another problem too?
Tim
To be pendantic, (and not of much actual help, sorry :/ ) the problem isn't really with Windows, which provides API's for getting file info that handle long paths just fine, but with libgit, which is probably compiled against some POSIX stat implementation provided by Cygwin/mingw, which does not.
Cheers, Henry
-- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
participants (3)
-
Ben Coman -
Henrik Sperre Johansen -
Tim Mackinnon