Issues with recent Pharos reading files
Iâm having trouble reading files with the most recent âstableâ versions of Pharo. Today, I downloaded Pharo (32-bit), loaded my code into the image using git and Iceberg, and half of my tests failed because Pharo claimed that the files that are on my disk are in a directory that does not exist. If I go back to an older image, everything works fine. Iâm seeing a similar thing on Travis-CI: the build under Pharo 6 works fine, and all the tests are green, but the build under Pharo 7 fails:
Loaded -> SmalltalkCI-Coverage-Core-fn.5 --- filetree:///home/travis/smalltalkCI-master/repository [:] --- cache SmalltalkCIPharo class>>deprecationWarning (TestCoverage is Undeclared) SmalltalkCIPharo7 class>>stdout (ZnCrPortableWriteStream is Undeclared) Loaded -> SmalltalkCI-Pharo-Core-fn.62 --- filetree:///home/travis/smalltalkCI-master/repository [:] --- cache ...finished baselineMessageNotUnderstood: receiver of "on:" is nil UndefinedObject(Object)>>doesNotUnderstand: #on: SmalltalkCIPharo7 class>>stdout SmalltalkCIPharo7 class(SmalltalkCI class)>>travisFold:id:block: SmalltalkCIPharo7 class(SmalltalkCI class)>>stage:id:block: SmalltalkCIPharo7(SmalltalkCI)>>reportImageInfo SmalltalkCIPharo7(SmalltalkCI)>>basicLoad [ super basicLoad ] in [ [ super basicLoad ] on: (Smalltalk at: #MCMergeOrLoadWarning) do: [ :ex | ex resume: true ] ] in SmalltalkCIPharo7(SmalltalkCIPharo)>>basicLoad in Block: [ super basicLoad ]
(The logs are here: https://travis-ci.org/apblack/GraceInPharo. Look at build #83. Iâm not sure if they are public â I think that they are!) Whatâs weird is that my co-worker, who I was introducing to Pharo, was able to download a Pharo distribution and install and run the same git commit with no problems. So I decided that maybe my launcher was out of date. I just downloaded and installed a new version of the launcher. When I use it to create a new image, it is unable to launch it, because it canât find the VM. (The VM package was there, but with a âno entryâ sign across it.) I eventually patch up the launcher by hand, and load my code into a new image. Half of my tests fail once again. There are two issues. First, file streams are now ZnCharacterReadStreams instead of MultiByteFileStreams. Ok, thatâs a change that I had been expecting. But ZnCharacterReadStreams donât understand readStream, whereas MultiByteFileStreams did (and returned self). I can code around that. The second issue is one that I canât code around. Iâm getting a "directory does not existâ error from DiskStore>>#basicEntry:path:nodesDo:, even though the sending method, FileSystemStore>>#directoryAt:ifAbsent:nodesDo: has just correctly verified that the directory does exist. Looking at the offending method, I think that the problem is that Primitives lookupEntryIn: encodedPathString index: index. always returns #badDirectoryPath. Since this method contains a call to a primitive, followed by ^ #badDirectoryPath if the primitive fails, Iâm guessing that either I donât have the primitive installed, or I have an old version of the primitive. How do I fix this? More interesting in some ways: if I download a new copy of the launcher, and then use the launcher to install a new copy of the development image, shouldnât all of the necessary files be installed along with it.
From the fact that this isnât working on Travis either, Iâm guessing that there might be a problem with the distribution.
Andrew
On Tue, Mar 20, 2018 at 2:36 PM, Andrew P. Black <apblack@pdx.edu> wrote:
Iâm having trouble reading files with the most recent âstableâ versions of Pharo.
Today, I downloaded Pharo (32-bit), loaded my code into the image using git and Iceberg, and half of my tests failed because Pharo claimed that the files that are on my disk are in a directory that does not exist. If I go back to an older image, everything works fine.
Iâm seeing a similar thing on Travis-CI: the build under Pharo 6 works fine, and all the tests are green, but the build under Pharo 7 fails:
Loaded -> SmalltalkCI-Coverage-Core-fn.5 --- filetree:///home/travis/smalltalkCI-master/repository [:] --- cache SmalltalkCIPharo class>>deprecationWarning (TestCoverage is Undeclared) SmalltalkCIPharo7 class>>stdout (ZnCrPortableWriteStream is Undeclared) Loaded -> SmalltalkCI-Pharo-Core-fn.62 --- filetree:///home/travis/smalltalkCI-master/repository [:] --- cache ...finished baselineMessageNotUnderstood: receiver of "on:" is nil UndefinedObject(Object)>>doesNotUnderstand: #on: SmalltalkCIPharo7 class>>stdout SmalltalkCIPharo7 class(SmalltalkCI class)>>travisFold:id:block: SmalltalkCIPharo7 class(SmalltalkCI class)>>stage:id:block: SmalltalkCIPharo7(SmalltalkCI)>>reportImageInfo SmalltalkCIPharo7(SmalltalkCI)>>basicLoad [ super basicLoad ] in [ [ super basicLoad ] on: (Smalltalk at: #MCMergeOrLoadWarning) do: [ :ex | ex resume: true ] ] in SmalltalkCIPharo7(SmalltalkCIPharo)>>basicLoad in Block: [ super basicLoad ]
(The logs are here: https://travis-ci.org/apblack/GraceInPharo. Look at build #83. Iâm not sure if they are public â I think that they are!)
Whatâs weird is that my co-worker, who I was introducing to Pharo, was able to download a Pharo distribution and install and run the same git commit with no problems.
So I decided that maybe my launcher was out of date. I just downloaded and installed a new version of the launcher. When I use it to create a new image, it is unable to launch it, because it canât find the VM. (The VM package was there, but with a âno entryâ sign across it.)
I eventually patch up the launcher by hand, and load my code into a new image. Half of my tests fail once again. There are two issues.
First, file streams are now ZnCharacterReadStreams instead of MultiByteFileStreams. Ok, thatâs a change that I had been expecting. But ZnCharacterReadStreams donât understand readStream, whereas MultiByteFileStreams did (and returned self). I can code around that.
The second issue is one that I canât code around. Iâm getting a "directory does not existâ error from DiskStore>>#basicEntry:path:nodesDo:, even though the sending method, FileSystemStore>>#directoryAt:ifAbsent:nodesDo: has just correctly verified that the directory does exist. Looking at the offending method, I think that the problem is that
Primitives lookupEntryIn: encodedPathString index: index.
always returns #badDirectoryPath. Since this method contains a call to a primitive, followed by ^ #badDirectoryPath if the primitive fails, Iâm guessing that either I donât have the primitive installed, or I have an old version of the primitive.
How do I fix this? More interesting in some ways: if I download a new copy of the launcher, and then use the launcher to install a new copy of the development image, shouldnât all of the necessary files be installed along with it.
From the fact that this isnât working on Travis either, Iâm guessing that there might be a problem with the distribution.
Andrew
Hi, Fast answer, I did not had the time to check for a long time. I see it the logs: SmalltalkCIPharo7 class>>stdout (ZnCrPortableWriteStream is Undeclared) I think ZnCrPortableWriteStream was renamed into ZnNewLineWriterStream. Maybe it is SmalltalkCI who need to adapt? Probably this method: https://github.com/hpi-swa/smalltalkCI/blob/master/repository/SmalltalkCI-Ph... -- Cyril Ferlicot https://ferlicot.fr
On 20 Mar 2018, at 14:36, Andrew P. Black <apblack@pdx.edu> wrote:
Iâm having trouble reading files with the most recent âstableâ versions of Pharo.
OK, this is no answer to your issue, but Pharo 7 latest is far from stable, it is intentionally the development head. It is very good that you test it and help out, more eyes and all that, but using it is at your own risk, there are no promises about stability.
Just a side note. I had problems with Monticello ftp repo. Ok on pharo 6, not on 7 last week. Iâll try to reproduce. It was in relation to the zip file. Cheers, Cedrick Envoyé de mon iPhone
Le 20 mars 2018 à 15:15, Sven Van Caekenberghe <sven@stfx.eu> a écrit :
On 20 Mar 2018, at 14:36, Andrew P. Black <apblack@pdx.edu> wrote:
Iâm having trouble reading files with the most recent âstableâ versions of Pharo.
OK, this is no answer to your issue, but Pharo 7 latest is far from stable, it is intentionally the development head.
It is very good that you test it and help out, more eyes and all that, but using it is at your own risk, there are no promises about stability.
On 21 Mar 2018, at 03:15 , Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 20 Mar 2018, at 14:36, Andrew P. Black <apblack@pdx.edu> wrote:
Iâm having trouble reading files with the most recent âstableâ versions of Pharo.
OK, this is no answer to your issue, but Pharo 7 latest is far from stable, it is intentionally the development head.
It is very good that you test it and help out, more eyes and all that, but using it is at your own risk, there are no promises about stability.
I understand this. The point of my post was to try and diagnose the issue and help make Pharo 7 more stable. Does my diagnosis of a missing or outdated primitive look reasonable? If so, how do I check? That is, where are is this primitive <primitive: 'primitiveDirectoryLookup' module: 'FilePlugin' error: errorCode > supposed to come from? Iâm guessing that there should be a file with âFilePluginâ somewhere in its name somewhere in one of my VMs. There isnât â although some older Cog vms from back in the day (Iâm talking Pharo 1.4) have something called âlibAsynchFilePlugin.dylibâ If this plugin is now essential to run Pharo, shouldnât downloading the launcher and using the launcher to grab a fresh image and launch it (after checking for the appropriate VM) be all that is required to ensure that it is installed? If not, what other steps should I take? Iâm getting on a l-o-n-g flight in a few hours, and would really like to be able to run Pharo ... Andrew
Hi Andrew, On 20 March 2018 at 21:42, Andrew P. Black <apblack@pdx.edu> wrote:
On 21 Mar 2018, at 03:15 , Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 20 Mar 2018, at 14:36, Andrew P. Black <apblack@pdx.edu> wrote:
Iâm having trouble reading files with the most recent âstableâ versions of Pharo.
OK, this is no answer to your issue, but Pharo 7 latest is far from stable, it is intentionally the development head.
It is very good that you test it and help out, more eyes and all that, but using it is at your own risk, there are no promises about stability.
I understand this. The point of my post was to try and diagnose the issue and help make Pharo 7 more stable.
Does my diagnosis of a missing or outdated primitive look reasonable?
Not really :-)
If so, how do I check?
If you can successfully open World Menu -> Tools -> File Browser then the primitive is working.
That is, where are is this primitive
<primitive: 'primitiveDirectoryLookup' module: 'FilePlugin' error: errorCode >
supposed to come from? Iâm guessing that there should be a file with âFilePluginâ somewhere in its name somewhere in one of my VMs. There isnât â although some older Cog vms from back in the day (Iâm talking Pharo 1.4) have something called âlibAsynchFilePlugin.dylibâ
Plugins can either be built-in or external. FilePlugin is built-in.
If this plugin is now essential to run Pharo, shouldnât downloading the launcher and using the launcher to grab a fresh image and launch it (after checking for the appropriate VM) be all that is required to ensure that it is installed? If not, what other steps should I take?
The error handling in #lookupEntryIn:index: is less than ideal. I'm working on a replacement, so I don't think it is worthwhile spending a lot of time on it. This primitive returns a directory entry by index number. Just taking a couple of wild guesses: One possibility is that the directory was modified while you were iterating over the directory. I'm also not sure how it handles entries that you don't have permission to read. But to help track down your issue, maybe temporarily replace the code with: <primitive: 'primitiveDirectoryLookup' module: 'FilePlugin' error: errorCode > errorCode ifNotNil: [self halt: errorCode asString]. ^ #badDirectoryPath That should allow you to track down the issue a bit more. P.S. If the primitive really is missing, errorCode will be #'not found'.
Iâm getting on a l-o-n-g flight in a few hours, and would really like to be able to run Pharo ...
Andrew
HTH, Alistair
I'll fix smalltalk CI code for Pharo 7 to use the new class. That will solve the problems in travis builds. @Andrew, about the directories issue, can you provide me a full stack trace and even better a way to reproduce it? @Cédrik, kind of the same, if you provide a proper issue (or even a test case?) that would be a real time saver. Issues that are arising now are because there are parts of the system that were badly tested or not tested at all, so ... Tx all, Guille On Tue, Mar 20, 2018 at 10:07 PM, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Andrew,
On 20 March 2018 at 21:42, Andrew P. Black <apblack@pdx.edu> wrote:
On 21 Mar 2018, at 03:15 , Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 20 Mar 2018, at 14:36, Andrew P. Black <apblack@pdx.edu> wrote:
Iâm having trouble reading files with the most recent âstableâ
versions of Pharo.
OK, this is no answer to your issue, but Pharo 7 latest is far from
stable, it is intentionally the development head.
It is very good that you test it and help out, more eyes and all that,
but using it is at your own risk, there are no promises about stability.
I understand this. The point of my post was to try and diagnose the issue and help make Pharo 7 more stable.
Does my diagnosis of a missing or outdated primitive look reasonable?
Not really :-)
If so, how do I check?
If you can successfully open World Menu -> Tools -> File Browser then the primitive is working.
That is, where are is this primitive
<primitive: 'primitiveDirectoryLookup' module: 'FilePlugin' error: errorCode >
supposed to come from? Iâm guessing that there should be a file with âFilePluginâ somewhere in its name somewhere in one of my VMs. There isnât â although some older Cog vms from back in the day (Iâm talking Pharo 1.4) have something called âlibAsynchFilePlugin.dylibâ
Plugins can either be built-in or external. FilePlugin is built-in.
If this plugin is now essential to run Pharo, shouldnât downloading the launcher and using the launcher to grab a fresh image and launch it (after checking for the appropriate VM) be all that is required to ensure that it is installed? If not, what other steps should I take?
The error handling in #lookupEntryIn:index: is less than ideal. I'm working on a replacement, so I don't think it is worthwhile spending a lot of time on it.
This primitive returns a directory entry by index number. Just taking a couple of wild guesses: One possibility is that the directory was modified while you were iterating over the directory. I'm also not sure how it handles entries that you don't have permission to read.
But to help track down your issue, maybe temporarily replace the code with:
<primitive: 'primitiveDirectoryLookup' module: 'FilePlugin' error: errorCode > errorCode ifNotNil: [self halt: errorCode asString]. ^ #badDirectoryPath
That should allow you to track down the issue a bit more.
P.S. If the primitive really is missing, errorCode will be #'not found'.
Iâm getting on a l-o-n-g flight in a few hours, and would really like to be able to run Pharo ...
Andrew
HTH, Alistair
-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
On Wed, Mar 21, 2018 at 9:51 AM, Guillermo Polito <guillermopolito@gmail.com
wrote:
I'll fix smalltalk CI code for Pharo 7 to use the new class. That will solve the problems in travis builds.
Aaand, fix in here, awaiting for integration: https://github.com/hpi-swa/smalltalkCI/pull/361
@Andrew, about the directories issue, can you provide me a full stack trace and even better a way to reproduce it?
@Cédrik, kind of the same, if you provide a proper issue (or even a test case?) that would be a real time saver.
Issues that are arising now are because there are parts of the system that were badly tested or not tested at all, so ...
Tx all, Guille
On Tue, Mar 20, 2018 at 10:07 PM, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Andrew,
On 20 March 2018 at 21:42, Andrew P. Black <apblack@pdx.edu> wrote:
On 21 Mar 2018, at 03:15 , Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 20 Mar 2018, at 14:36, Andrew P. Black <apblack@pdx.edu> wrote:
Iâm having trouble reading files with the most recent âstableâ
versions of Pharo.
OK, this is no answer to your issue, but Pharo 7 latest is far from
stable, it is intentionally the development head.
It is very good that you test it and help out, more eyes and all that,
but using it is at your own risk, there are no promises about stability.
I understand this. The point of my post was to try and diagnose the issue and help make Pharo 7 more stable.
Does my diagnosis of a missing or outdated primitive look reasonable?
Not really :-)
If so, how do I check?
If you can successfully open World Menu -> Tools -> File Browser then the primitive is working.
That is, where are is this primitive
<primitive: 'primitiveDirectoryLookup' module: 'FilePlugin' error: errorCode >
supposed to come from? Iâm guessing that there should be a file with âFilePluginâ somewhere in its name somewhere in one of my VMs. There isnât â although some older Cog vms from back in the day (Iâm talking Pharo 1.4) have something called âlibAsynchFilePlugin.dylibâ
Plugins can either be built-in or external. FilePlugin is built-in.
If this plugin is now essential to run Pharo, shouldnât downloading the launcher and using the launcher to grab a fresh image and launch it (after checking for the appropriate VM) be all that is required to ensure that it is installed? If not, what other steps should I take?
The error handling in #lookupEntryIn:index: is less than ideal. I'm working on a replacement, so I don't think it is worthwhile spending a lot of time on it.
This primitive returns a directory entry by index number. Just taking a couple of wild guesses: One possibility is that the directory was modified while you were iterating over the directory. I'm also not sure how it handles entries that you don't have permission to read.
But to help track down your issue, maybe temporarily replace the code with:
<primitive: 'primitiveDirectoryLookup' module: 'FilePlugin' error: errorCode > errorCode ifNotNil: [self halt: errorCode asString]. ^ #badDirectoryPath
That should allow you to track down the issue a bit more.
P.S. If the primitive really is missing, errorCode will be #'not found'.
Iâm getting on a l-o-n-g flight in a few hours, and would really like to be able to run Pharo ...
Andrew
HTH, Alistair
--
Guille Polito
Research Engineer
Centre de Recherche en Informatique, Signal et Automatique de Lille
CRIStAL - UMR 9189
French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*
*Web:* *http://guillep.github.io* <http://guillep.github.io>
*Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>
-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
On 20 Mar 2018, at 23:01 , Guillermo Polito <guillermopolito@gmail.com> wrote:
@Andrew, about the directories issue, can you provide me a full stack trace and even better a way to reproduce it?
I sent a lengthy message in this thread explaining the circumstances. Is that enough? If not I can try opening a file many times and see if I can reproduce the breaking of the primitive. Iâm pretty sure thatâs what was happening (many hours of debugging on an aeroplane), but did not try to recreate the problem â I was just glad to be able to fix it and move on with my work. And my battery was dead. 9 hours on a B 737-200 with no power :-(. Pharo eats batteries for breakfast, lunch and dinner! Andrew
Hi andrew do you have a reproducible case? or a stackk trace? Stef On Wed, Mar 21, 2018 at 10:50 PM, Andrew P. Black <apblack@pdx.edu> wrote:
On 20 Mar 2018, at 23:01 , Guillermo Polito <guillermopolito@gmail.com> wrote:
@Andrew, about the directories issue, can you provide me a full stack trace and even better a way to reproduce it?
I sent a lengthy message in this thread explaining the circumstances. Is that enough? If not I can try opening a file many times and see if I can reproduce the breaking of the primitive. Iâm pretty sure thatâs what was happening (many hours of debugging on an aeroplane), but did not try to recreate the problem â I was just glad to be able to fix it and move on with my work.
And my battery was dead. 9 hours on a B 737-200 with no power :-(. Pharo eats batteries for breakfast, lunch and dinner!
Andrew
On 21 Mar 2018, at 10:07 , Alistair Grant <akgrant0710@gmail.com> wrote:
But to help track down your issue, maybe temporarily replace the code with:
<primitive: 'primitiveDirectoryLookup' module: 'FilePlugin' error: errorCode > errorCode ifNotNil: [self halt: errorCode asString]. ^ #badDirectoryPath
That should allow you to track down the issue a bit more.
P.S. If the primitive really is missing, errorCode will be #'not found'.
I still got #badDirectoryPath, so I assumed that the primitive was there, and was just returning the wrong answer. The question then became: why? After some hours spent investigating, I figured out what is going on. Here is the story. The new file streams have behaviour that differs in several respects from the old ones. Here are three things that affected my application: 1. sending #readstream to an old file stream answered self; with the new ones, I get a MNU. 2. closing a stream that that already been closed used to be a no-op; now it causes a primitive failure walkback. 3. Repositioning a filestream stream doesnât work. Unfortunately, sending #position: doesnât fail immediately, but the effect in my app (a compiler) is that the file Iâm reading appears to be empty. Now for the interesting bit. Because the file appeared empty, my tests opened the default prelude file. This also appeared to be empty, so it opened the default prelude file again, while it was still open the first time. This continued indefinitely â presumably, until some resource in the plugin was exhausted. Once that happened, the plugin was permanently broken. Probably the unwind machinery should have released the resource, but it didnât, probably because the resource is in the plugin, not in Pharo. You can argue that (1) is the desired behaviour, but it is atypical. Sending #asString to a String, #asNumber to a Number, or #asSet to a Set all return self. I think that sending #readStream to a ReadStream should likewise answer self. It used to. You can argue that (2) ought to fail, and Iâll probably agree with you. However, the failure should be a clean one (âattempt to close a stream that has already been closedâ), not a primitive failure. And then /all/ streams should behave that way, and there should be a #closeIfNecessary operation that does what #close used to do. Incidentally, the predicate that tests whether a stream is closed is named #closed; it should be named #isClosed. (3), as already been discussed here, is just wrong. If a stream has a #position: method, it should work. If we need another wrapper to implement it, so be it. If #position: canât be implemented for a particular kind of stream (e.g., an interactive input stream), then the method either should not be present at all, or should unconditionally raise an exception. I have not investigated exactly how #position: fails, but Iâm pretty sure that this is the culprit. If I replace the `fileStream` by `fileStream contents readStream`, everything works fine again. As for the primitive failing permanently because it runs out of resources â Iâll leave that to smarter people than me to figure out. At very least, it should produce an intelligible ResourceExhausted exception. Andrew
participants (7)
-
Alistair Grant -
Andrew P. Black -
Cyril Ferlicot -
Cédrick Béler -
Guillermo Polito -
Stephane Ducasse -
Sven Van Caekenberghe