Stdio>>#standardIOStreamNamed:forWrite: bug?
Hi Damien, Vincent and Guille, Just a disclaimer: despite having done some work on this, I don't consider myself an expert. If someone else has a better explanation, please jump in. Damien wrote on discord:
is there a notion of standard input/output on windows ?
Yes, console applications can use stdio. PharoConsole.exe when run from a command prompt has stdio. (Note that cygwin terminals are different and don't support stdio (since Pharo uses the native windows functions)).
looks like Alistair added the possibility to open files by number,
Holger Freyther deserves the credit here. My clumsy git merging left his name out as the author (sorry Holger!).
that's probably what was missing at the time Stdio was introduced
I'm not sure that opening files by fd would be useful in this particular case. There are three ways to access files on Windows: file descriptor - posix compatibility FILE* - posix compatibility HANDLE - windows native The VM uses native operations for file i/o on Windows, e.g. Read/WriteConsole() for stdio (including pipe redirection of stdio). Mixing file descriptors and FILE* is possible if you're careful. I don't know how to mix fd / FILE* and HANDLEs. It's probably better to stick with using native windows functions most of the time. We included these functions on Windows since they exist, and there are probably some use cases, but you need to know what you're doing. File class>>stdioHandles calls primitiveFileStdioHandles() which eventually opens the stdio streams (if possible) using the Windows native GetStdHandle() function.
I suspect File needs a nameless variant (I'm not certain how pipes work in unix but I doubt echo foo | cat ever touches the filesystem)
That's my understanding (it doesn't touch the file system).
hmm but the short-term bug is that the TTY logic in Stdio >> #standardIOStreamNamed:forWrite: fails when all 3 standard streams are redirected
OK, my understanding of what you are saying is that if all 3 io streams are redirected the method incorrectly determines that a console isn't present and so falls back to calling #createStdioFileFor:. More below...
I also don't understand the assumption there
Vincent Blondeau added / modified these methods, so he may be able to provide more information. I think the assumption is that if any of the stdio files are connected to a TTY. then it is possible to open all the stdio files. If none are present, then (on Windows) create regular files to act as a substitute. The problem is that it only tests for a TTY, when what we really want is a TTY or a file or a pipe. Assuming I've understood correctly, maybe try modifying (sorry for the loss of formatting, only the last line is changed): File class>>stdioDescriptorIsATTY ^ (0 to: 2) anySatisfy: [ :fdNum | | res | res := self fileDescriptorType: fdNum. res between: 1 and: 3 ] We'd need to rename the method, since it is misleading, but it looks like it better represents the intended use. Cheers, Alistair
Hi, I don't know the windows situation at all, but I'm wondering. ([ File stdioDescriptorIsATTY not ] on: PrimitiveFailed do: [ :ex | "HACK kept for retrocompatibility" Smalltalk os isWin32 ]) ifTrue: [ ^ self createStdioFileFor: moniker ]. the first thing I would do, just trying to keep the same behaviour, is to invert the test... Something like Smalltalk os isWin32 ifTrue: [ [ File stdioDescriptorIsATTY not ] on: PrimitiveFailed do: [ :ex | false ]) ifTrue: [ ^ self createStdioFileFor: moniker ]. ] Like that, this would only penalise windows in the worst case. Then a first question arises in my head: why isWin32 and not isWindows? Is there a reason behind that? Wouldn't that prevent correct behaviour in windows 64 bits? My next impression is that testing for "a TTY or a file or a pipe" seems buggy... We should put a more intention revealing selector like #isInvalidStdioHandle? A more high level test will allow us to do any check we want in the background, and it allows better to understand why we check for TTY or File or Pipe or la mar en coche :) On Thu, Aug 2, 2018 at 8:54 PM Alistair Grant <akgrant0710@gmail.com> wrote:
File class>>stdioHandles calls primitiveFileStdioHandles() which eventually opens the stdio streams (if possible) using the Windows native GetStdHandle() function.
What does #stdioHandles return to the image in the case of a non-console windows VM? I mean, when "it is not possible"?
From the docs, https://docs.microsoft.com/en-us/windows/console/getstdhandle says we may have an INVALID_HANDLE_VALUE or NULL in case it fails. Is the file plugin is masking the error? If so, thats why we need such strange workarounds...
- If it is an invalid handle, why not just testing for "isInvalidHandle" or so? We can have that on the image side and then dispatch to the correct thing in the file plugin maybe? - Maybe the #stdioHandles primitive should just fail in the case we don't have correct handles? that would allow us to do a much more elegant approach on the image side...
I suspect File needs a nameless variant (I'm not certain how pipes
work in unix but I doubt echo foo | cat ever touches the filesystem)
That's my understanding (it doesn't touch the file system).
Well... I see this is just a aesthetic issue and I don't see how this is related to the above more serious issue...
Hi Guille, On Fri, Aug 03, 2018 at 10:34:22AM +0200, Guillermo Polito wrote:
Hi,
I don't know the windows situation at all, but I'm wondering.
([ File stdioDescriptorIsATTY not ] on: PrimitiveFailed do: [ :ex | "HACK kept for retrocompatibility" Smalltalk os isWin32 ]) ifTrue: [ ^ self createStdioFileFor: moniker ].
the first thing I would do, just trying to keep the same behaviour, is to invert the test... Something like
Smalltalk os isWin32 ifTrue: [ [ File stdioDescriptorIsATTY not ] on: PrimitiveFailed do: [ :ex | false ]) ifTrue: [ ^ self createStdioFileFor: moniker ]. ]
Like that, this would only penalise windows in the worst case.
True. Actually I think it would be safe to remove it completely. Unix platforms should always have stdio, and other platforms should act appropriately. But more below...
Then a first question arises in my head: why isWin32 and not isWindows? Is there a reason behind that? Wouldn't that prevent correct behaviour in windows 64 bits?
You're right, it should be #isWindows.
My next impression is that testing for "a TTY or a file or a pipe" seems buggy... We should put a more intention revealing selector like #isInvalidStdioHandle? A more high level test will allow us to do any check we want in the background, and it allows better to understand why we check for TTY or File or Pipe or la mar en coche :)
I made the change so that Damien could test that it gives the behaviour he expects, and did say that the selector will need to be renamed.
What does #stdioHandles return to the image in the case of a non-console windows VM? I mean, when "it is not possible"?
From the docs, https://docs.microsoft.com/en-us/windows/console/getstdhandle says we may have an INVALID_HANDLE_VALUE or NULL in case it fails. Is the file plugin is masking the error? If so, thats why we need such strange workarounds...
We'd still want the ability to test what type of device stdio is connected to. But I agree the primitive should be doing the correct thing.
- If it is an invalid handle, why not just testing for "isInvalidHandle" or so? We can have that on the image side and then dispatch to the correct thing in the file plugin maybe?
- Maybe the #stdioHandles primitive should just fail in the case we don't have correct handles? that would allow us to do a much more elegant approach on the image side...
I don't think the error handling has been changed in a long time, but just looking quickly at the source code, I think you're correct that it isn't returning the correct values. It should be returning nil for those handles that aren't valid. I don't have my Windows VM handy at the moment, is anyone able to quickly confirm? (Start pharo with Pharo.exe (not PharoConsole.exe) and supply the result from: File stdioHandles I've been asking lots of questions lately and not providing much input (I was hoping to have File Attributes integrated by now, but MacOS file encoding and my inability to get the VM to compile on a Mac VM have frustrated me), so I'm more than happy to tidy up the stdio primitives. I'll need to check that it doesn't break anything on Squeak as well (I don't think any of the changes discussed should cause problems). Cheers, Alistair
On Fri, 3 Aug 2018 at 11:24, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Guille,
On Fri, Aug 03, 2018 at 10:34:22AM +0200, Guillermo Polito wrote:
Hi,
I don't know the windows situation at all, but I'm wondering.
([ File stdioDescriptorIsATTY not ] on: PrimitiveFailed do: [ :ex | "HACK kept for retrocompatibility" Smalltalk os isWin32 ]) ifTrue: [ ^ self createStdioFileFor: moniker ].
the first thing I would do, just trying to keep the same behaviour, is to invert the test... Something like
Smalltalk os isWin32 ifTrue: [ [ File stdioDescriptorIsATTY not ] on: PrimitiveFailed do: [ :ex | false ]) ifTrue: [ ^ self createStdioFileFor: moniker ]. ]
Like that, this would only penalise windows in the worst case.
True. Actually I think it would be safe to remove it completely. Unix platforms should always have stdio, and other platforms should act appropriately. But more below...
Then a first question arises in my head: why isWin32 and not isWindows? Is there a reason behind that? Wouldn't that prevent correct behaviour in windows 64 bits?
You're right, it should be #isWindows.
My next impression is that testing for "a TTY or a file or a pipe" seems buggy... We should put a more intention revealing selector like #isInvalidStdioHandle? A more high level test will allow us to do any check we want in the background, and it allows better to understand why we check for TTY or File or Pipe or la mar en coche :)
I made the change so that Damien could test that it gives the behaviour he expects, and did say that the selector will need to be renamed.
What does #stdioHandles return to the image in the case of a non-console windows VM? I mean, when "it is not possible"?
From the docs, https://docs.microsoft.com/en-us/windows/console/getstdhandle says we may have an INVALID_HANDLE_VALUE or NULL in case it fails. Is the file plugin is masking the error? If so, thats why we need such strange workarounds...
We'd still want the ability to test what type of device stdio is connected to. But I agree the primitive should be doing the correct thing.
- If it is an invalid handle, why not just testing for "isInvalidHandle" or so? We can have that on the image side and then dispatch to the correct thing in the file plugin maybe?
- Maybe the #stdioHandles primitive should just fail in the case we don't have correct handles? that would allow us to do a much more elegant approach on the image side...
I don't think the error handling has been changed in a long time, but just looking quickly at the source code, I think you're correct that it isn't returning the correct values. It should be returning nil for those handles that aren't valid.
I don't have my Windows VM handy at the moment, is anyone able to quickly confirm? (Start pharo with Pharo.exe (not PharoConsole.exe) and supply the result from:
File stdioHandles
I've been asking lots of questions lately and not providing much input (I was hoping to have File Attributes integrated by now, but MacOS file encoding and my inability to get the VM to compile on a Mac VM have frustrated me), so I'm more than happy to tidy up the stdio primitives.
I'll need to check that it doesn't break anything on Squeak as well (I don't think any of the changes discussed should cause problems).
Tracking issue: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/274
On Sat, 4 Aug 2018 at 15:43, Alistair Grant <akgrant0710@gmail.com> wrote:
I don't think the error handling has been changed in a long time, but just looking quickly at the source code, I think you're correct that it isn't returning the correct values. It should be returning nil for those handles that aren't valid.
I don't have my Windows VM handy at the moment, is anyone able to quickly confirm? (Start pharo with Pharo.exe (not PharoConsole.exe) and supply the result from:
File stdioHandles
This does indeed fail to return nil even though stdio is not available. Instead a sqFile handle with null HANDLE* is returned. But the image can't test this as the format of sqFile is platform dependent and is private to the VM.
I've been asking lots of questions lately and not providing much input (I was hoping to have File Attributes integrated by now, but MacOS file encoding and my inability to get the VM to compile on a Mac VM have frustrated me), so I'm more than happy to tidy up the stdio primitives.
I'll need to check that it doesn't break anything on Squeak as well (I don't think any of the changes discussed should cause problems).
Tracking issue: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/274
The issue above is for the VM changes. The issue for image changes is: https://pharo.fogbugz.com/f/cases/22296/Stdio-file-creation-fixes Cheers, Alistair
Alistair Grant wrote
Tracking issue: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/274 The issue above is for the VM changes.
The issue for image changes is: https://pharo.fogbugz.com/f/cases/22296/Stdio-file-creation-fixes
Thanks for looking into this, Alistair! Few are brave enough to dive into these dark corners of the system ;-) ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
Damien, Could you check that at least the proposed image-side changes fix your problem for *nixes? On Sat, Aug 4, 2018 at 7:13 PM Sean P. DeNigris <sean@clipperadams.com> wrote:
Alistair Grant wrote
Tracking issue: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/274 The issue above is for the VM changes.
The issue for image changes is: https://pharo.fogbugz.com/f/cases/22296/Stdio-file-creation-fixes
Thanks for looking into this, Alistair! Few are brave enough to dive into these dark corners of the system ;-)
----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
Yes, they do, at least on my macOS. I have some doubt about #stdioIsAvailable though⦠before there was a possible value of 4 (cygwin terminal) for the file descriptor type, which is now out of the tested range, is that intentional On Mon, 6 Aug 2018 at 12:00, Guillermo Polito <guillermopolito@gmail.com> wrote:
Damien,
Could you check that at least the proposed image-side changes fix your problem for *nixes?
On Sat, Aug 4, 2018 at 7:13 PM Sean P. DeNigris <sean@clipperadams.com> wrote:
Alistair Grant wrote
Tracking issue: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/274 The issue above is for the VM changes.
The issue for image changes is: https://pharo.fogbugz.com/f/cases/22296/Stdio-file-creation-fixes
Thanks for looking into this, Alistair! Few are brave enough to dive into these dark corners of the system ;-)
----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
--
Guille Polito
Research Engineer
Centre de Recherche en Informatique, Signal et Automatique de Lille
CRIStAL - UMR 9189
French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*
*Web:* *http://guillep.github.io* <http://guillep.github.io>
*Phone: *+33 06 52 70 66 13
-- Damien Pollet type less, do more [ | ] http://people.untyped.org/damien.pollet
On Mon, 6 Aug 2018 at 12:18, Damien Pollet <damien.pollet@gmail.com> wrote:
Yes, they do, at least on my macOS.
I have some doubt about #stdioIsAvailable though⦠before there was a possible value of 4 (cygwin terminal) for the file descriptor type, which is now out of the tested range, is that intentional
Yep. Pharo can't write to a cygwin terminal since it uses the Windows Read/WriteConsole() functions. cygwin terminals are actually written to through pipes, and we can get the pipe information, and thus determine it is a cygwin terminal. But that's it for now. Cheers, Alistair
On Mon, 6 Aug 2018 at 12:00, Guillermo Polito <guillermopolito@gmail.com> wrote:
Damien,
Could you check that at least the proposed image-side changes fix your problem for *nixes?
On Sat, Aug 4, 2018 at 7:13 PM Sean P. DeNigris <sean@clipperadams.com> wrote:
Alistair Grant wrote
Tracking issue: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/274 The issue above is for the VM changes.
The issue for image changes is: https://pharo.fogbugz.com/f/cases/22296/Stdio-file-creation-fixes
Thanks for looking into this, Alistair! Few are brave enough to dive into these dark corners of the system ;-)
----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
--
Guille Polito
Research Engineer
Centre de Recherche en Informatique, Signal et Automatique de Lille
CRIStAL - UMR 9189
French National Center for Scientific Research - http://www.cnrs.fr
Phone: +33 06 52 70 66 13
-- Damien Pollet type less, do more [ | ] http://people.untyped.org/damien.pollet
participants (4)
-
Alistair Grant -
Damien Pollet -
Guillermo Polito -
Sean P. DeNigris