April 2018 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Alistair Grant

Hi Denis, On 11 April 2018 at 17:16, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > 2018-04-11 17:02 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: >> >> >> >> > On 11 Apr 2018, at 16:36, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> > >> > I can make your example, using the Zn variants, work with the following >> > change: >> > >> > StdioStream>>#atEnd >> > ^ peekBuffer isNil or: [ (peekBuffer := self next) isNil ] >> >> Argh, make that >> >> atEnd >> ^ peekBuffer isNil and: [ (peekBuffer := self next) isNil ] > > > But discussion exactly about "self next isNil": how to avoid it. Apologies in advance for being pedantic, but... Do you really mean this (that the discussion is about how to avoid testing "self next isNil")? My argument has been that without making #atEnd blocking it is not possible to avoid the test (and we don't want to make #atEnd blocking). All the existing stream code that deals with character / byte streams does this test (see my "unwritten rules" from a previous reply). We don't want to make #atEnd blocking, so we need to keep the test (and my personal opinion is that changing the Zinc streams to adopt this approach does not add any significant architectural complexity). Assuming we reach agreement on the above, we do need to make the "unwritten rules" written. If we reach a different agreement we should document that. Cheers, Alistair

April 11, 2018

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Sven Van Caekenberghe

I think we have to reset this whole discussion. FileStream stdin and Stdio stdin are completely different ! We'll have to check that first, before talking about the issues raised in this thread. And BTW these terminal streams are a real pain to test ;-) > On 11 Apr 2018, at 17:20, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > >> On 11 Apr 2018, at 17:16, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: >> >> >> 2018-04-11 17:02 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: >> >> >>> On 11 Apr 2018, at 16:36, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>> >>> I can make your example, using the Zn variants, work with the following change: >>> >>> StdioStream>>#atEnd >>> ^ peekBuffer isNil or: [ (peekBuffer := self next) isNil ] >> >> Argh, make that >> >> atEnd >> ^ peekBuffer isNil and: [ (peekBuffer := self next) isNil ] >> >> But discussion exactly about "self next isNil": how to avoid it. > > I know, but like this it could/might become an implementation detail. > > The more things that I try, the more that I feel that stdin is so special that it does not fit in the rest of the stream zoo. > >> but I am still testing, this is probably not the final answer/solution. >> >>> Which is a literal implementation of your statement that you can only know that you are atEnd by reading (and thus waiting/blocking) and checking for nil, which seems logical to me, given the fact that you *are* waiting for user input. >>> >>> BTW, at least on macOS you have to enter ctrl-D (^D) on a separate line, I am not sure how relevant that is, but that is probably another argument that stdin is special (being line-buffered by the OS, EOF needing to be on a separate line). >>> >>> And FWIW, I have been writing networking code in Pharo for years, and I have never had issues with unclear semantics of these primitives (#atEnd, #next, #peek) on network streams, either the classic SocketStream or the Zdc* streams (TLS or not). That is why I think we have to be careful. >>> >>> That being said, it is important to continue this discussion, I find it very interesting. I am trying to write some test code using stdin myself, to better understand the topic. >>> >>>> On 11 Apr 2018, at 16:06, Alistair Grant <akgrant0710(a)gmail.com> wrote: >>>> >>>> On 11 April 2018 at 15:11, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>>>> >>>>> >>>>>> On 11 Apr 2018, at 11:12, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>>>>> >>>>>> How does one modify #atEnd to block ? I suppose you are talking about StdioStream>>#atEnd ? >>>>>> >>>>>> ^ self peek isNil >>>>>> >>>>>> ? >>>>> >>>>> Still the same question, how do you implement a blocking #atEnd for stdin ? >>>>> >>>>> I have seen your stdio.cs which is indeed needed as the current StdioStream>>#atEnd is bogus for sure. >>>>> >>>>> But that is still a non-blocking one, right ? >>>>> >>>>> Since there is a peekBuffer in StdioStream, why can't that be used ? >>>> >>>> I think you've created a chicken-and-egg problem with this question, >>>> but ignoring that for now: >>>> >>>> >>>> StdioStream>>peek >>>> "Answer what would be returned if the message next were sent to the >>>> receiver. If the receiver is at the end, answer nil. " >>>> >>>> self atEnd ifTrue: [^ nil ]. >>>> >>>> peekBuffer ifNotNil: [ ^ peekBuffer ]. >>>> >>>> ^ peekBuffer := self next. >>>> >>>> >>>> >>>> So when we first start the program, i.e. the user hasn't entered any >>>> input yet, and #peek is called: >>>> >>>> 1. #atEnd returns false because Ctrl-D (or similar) hasn't been >>>> entered (assuming it is non-blocking). >>>> 2. peekBuffer is nil because we haven't previously called #peek. >>>> 3. The system now blocks on "self next". >>>> >>>> >>>> Just a reminder: for terminal input the end-of-file isn't reached >>>> until the user explicitly enters the end of file key (Ctrl-D). >>>> >>>> So, if there is no buffered input (either none has been entered yet, >>>> or all input has been consumed) >>>> >>>> #atEnd (after the patch) calls #primAtEnd:. >>>> >>>> At the moment, #primAtEnd: ends up calling the libc function feof(), >>>> which is non-blocking and answers the end-of-file flag for the FILE*. >>>> Since the user hasn't entered Ctrl-D, that's false. >>>> >>>> If we want to control iteration over the stream and ensure that we >>>> don't need to do a "stream next == nil" check, then #primAtEnd: is >>>> going to have to peek for the next character, and that means waiting >>>> for the user to enter that character. >>>> >>>> In c that is typically done using: >>>> >>>> atEnd = ungetc(fgetc(fp), fp); >>>> >>>> and fgetc() will block until the user enters something. >>>> >>>>> I have run your example testAtEnd.st now, and it works/fails as advertised. >>>> >>>> :-) >>>> >>>> >>>> Cheers, >>>> Alistair

April 11, 2018

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Sven Van Caekenberghe

> On 11 Apr 2018, at 17:16, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > > 2018-04-11 17:02 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > > > > On 11 Apr 2018, at 16:36, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > > I can make your example, using the Zn variants, work with the following change: > > > > StdioStream>>#atEnd > > ^ peekBuffer isNil or: [ (peekBuffer := self next) isNil ] > > Argh, make that > > atEnd > ^ peekBuffer isNil and: [ (peekBuffer := self next) isNil ] > > But discussion exactly about "self next isNil": how to avoid it. I know, but like this it could/might become an implementation detail. The more things that I try, the more that I feel that stdin is so special that it does not fit in the rest of the stream zoo. > but I am still testing, this is probably not the final answer/solution. > > > Which is a literal implementation of your statement that you can only know that you are atEnd by reading (and thus waiting/blocking) and checking for nil, which seems logical to me, given the fact that you *are* waiting for user input. > > > > BTW, at least on macOS you have to enter ctrl-D (^D) on a separate line, I am not sure how relevant that is, but that is probably another argument that stdin is special (being line-buffered by the OS, EOF needing to be on a separate line). > > > > And FWIW, I have been writing networking code in Pharo for years, and I have never had issues with unclear semantics of these primitives (#atEnd, #next, #peek) on network streams, either the classic SocketStream or the Zdc* streams (TLS or not). That is why I think we have to be careful. > > > > That being said, it is important to continue this discussion, I find it very interesting. I am trying to write some test code using stdin myself, to better understand the topic. > > > >> On 11 Apr 2018, at 16:06, Alistair Grant <akgrant0710(a)gmail.com> wrote: > >> > >> On 11 April 2018 at 15:11, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > >>> > >>> > >>>> On 11 Apr 2018, at 11:12, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > >>>> > >>>> How does one modify #atEnd to block ? I suppose you are talking about StdioStream>>#atEnd ? > >>>> > >>>> ^ self peek isNil > >>>> > >>>> ? > >>> > >>> Still the same question, how do you implement a blocking #atEnd for stdin ? > >>> > >>> I have seen your stdio.cs which is indeed needed as the current StdioStream>>#atEnd is bogus for sure. > >>> > >>> But that is still a non-blocking one, right ? > >>> > >>> Since there is a peekBuffer in StdioStream, why can't that be used ? > >> > >> I think you've created a chicken-and-egg problem with this question, > >> but ignoring that for now: > >> > >> > >> StdioStream>>peek > >> "Answer what would be returned if the message next were sent to the > >> receiver. If the receiver is at the end, answer nil. " > >> > >> self atEnd ifTrue: [^ nil ]. > >> > >> peekBuffer ifNotNil: [ ^ peekBuffer ]. > >> > >> ^ peekBuffer := self next. > >> > >> > >> > >> So when we first start the program, i.e. the user hasn't entered any > >> input yet, and #peek is called: > >> > >> 1. #atEnd returns false because Ctrl-D (or similar) hasn't been > >> entered (assuming it is non-blocking). > >> 2. peekBuffer is nil because we haven't previously called #peek. > >> 3. The system now blocks on "self next". > >> > >> > >> Just a reminder: for terminal input the end-of-file isn't reached > >> until the user explicitly enters the end of file key (Ctrl-D). > >> > >> So, if there is no buffered input (either none has been entered yet, > >> or all input has been consumed) > >> > >> #atEnd (after the patch) calls #primAtEnd:. > >> > >> At the moment, #primAtEnd: ends up calling the libc function feof(), > >> which is non-blocking and answers the end-of-file flag for the FILE*. > >> Since the user hasn't entered Ctrl-D, that's false. > >> > >> If we want to control iteration over the stream and ensure that we > >> don't need to do a "stream next == nil" check, then #primAtEnd: is > >> going to have to peek for the next character, and that means waiting > >> for the user to enter that character. > >> > >> In c that is typically done using: > >> > >> atEnd = ungetc(fgetc(fp), fp); > >> > >> and fgetc() will block until the user enters something. > >> > >>> I have run your example testAtEnd.st now, and it works/fails as advertised. > >> > >> :-) > >> > >> > >> Cheers, > >> Alistair > >

April 11, 2018

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Denis Kudriashov

2018-04-11 17:02 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > > > > On 11 Apr 2018, at 16:36, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > > I can make your example, using the Zn variants, work with the following > change: > > > > StdioStream>>#atEnd > > ^ peekBuffer isNil or: [ (peekBuffer := self next) isNil ] > > Argh, make that > > atEnd > ^ peekBuffer isNil and: [ (peekBuffer := self next) isNil ] > But discussion exactly about "self next isNil": how to avoid it. > > but I am still testing, this is probably not the final answer/solution. > > > Which is a literal implementation of your statement that you can only > know that you are atEnd by reading (and thus waiting/blocking) and checking > for nil, which seems logical to me, given the fact that you *are* waiting > for user input. > > > > BTW, at least on macOS you have to enter ctrl-D (^D) on a separate line, > I am not sure how relevant that is, but that is probably another argument > that stdin is special (being line-buffered by the OS, EOF needing to be on > a separate line). > > > > And FWIW, I have been writing networking code in Pharo for years, and I > have never had issues with unclear semantics of these primitives (#atEnd, > #next, #peek) on network streams, either the classic SocketStream or the > Zdc* streams (TLS or not). That is why I think we have to be careful. > > > > That being said, it is important to continue this discussion, I find it > very interesting. I am trying to write some test code using stdin myself, > to better understand the topic. > > > >> On 11 Apr 2018, at 16:06, Alistair Grant <akgrant0710(a)gmail.com> wrote: > >> > >> On 11 April 2018 at 15:11, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > >>> > >>> > >>>> On 11 Apr 2018, at 11:12, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > >>>> > >>>> How does one modify #atEnd to block ? I suppose you are talking about > StdioStream>>#atEnd ? > >>>> > >>>> ^ self peek isNil > >>>> > >>>> ? > >>> > >>> Still the same question, how do you implement a blocking #atEnd for > stdin ? > >>> > >>> I have seen your stdio.cs which is indeed needed as the current > StdioStream>>#atEnd is bogus for sure. > >>> > >>> But that is still a non-blocking one, right ? > >>> > >>> Since there is a peekBuffer in StdioStream, why can't that be used ? > >> > >> I think you've created a chicken-and-egg problem with this question, > >> but ignoring that for now: > >> > >> > >> StdioStream>>peek > >> "Answer what would be returned if the message next were sent to the > >> receiver. If the receiver is at the end, answer nil. " > >> > >> self atEnd ifTrue: [^ nil ]. > >> > >> peekBuffer ifNotNil: [ ^ peekBuffer ]. > >> > >> ^ peekBuffer := self next. > >> > >> > >> > >> So when we first start the program, i.e. the user hasn't entered any > >> input yet, and #peek is called: > >> > >> 1. #atEnd returns false because Ctrl-D (or similar) hasn't been > >> entered (assuming it is non-blocking). > >> 2. peekBuffer is nil because we haven't previously called #peek. > >> 3. The system now blocks on "self next". > >> > >> > >> Just a reminder: for terminal input the end-of-file isn't reached > >> until the user explicitly enters the end of file key (Ctrl-D). > >> > >> So, if there is no buffered input (either none has been entered yet, > >> or all input has been consumed) > >> > >> #atEnd (after the patch) calls #primAtEnd:. > >> > >> At the moment, #primAtEnd: ends up calling the libc function feof(), > >> which is non-blocking and answers the end-of-file flag for the FILE*. > >> Since the user hasn't entered Ctrl-D, that's false. > >> > >> If we want to control iteration over the stream and ensure that we > >> don't need to do a "stream next == nil" check, then #primAtEnd: is > >> going to have to peek for the next character, and that means waiting > >> for the user to enter that character. > >> > >> In c that is typically done using: > >> > >> atEnd = ungetc(fgetc(fp), fp); > >> > >> and fgetc() will block until the user enters something. > >> > >>> I have run your example testAtEnd.st now, and it works/fails as > advertised. > >> > >> :-) > >> > >> > >> Cheers, > >> Alistair > > > > >

April 11, 2018

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Sven Van Caekenberghe

> On 11 Apr 2018, at 16:36, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > I can make your example, using the Zn variants, work with the following change: > > StdioStream>>#atEnd > ^ peekBuffer isNil or: [ (peekBuffer := self next) isNil ] Argh, make that atEnd ^ peekBuffer isNil and: [ (peekBuffer := self next) isNil ] but I am still testing, this is probably not the final answer/solution. > Which is a literal implementation of your statement that you can only know that you are atEnd by reading (and thus waiting/blocking) and checking for nil, which seems logical to me, given the fact that you *are* waiting for user input. > > BTW, at least on macOS you have to enter ctrl-D (^D) on a separate line, I am not sure how relevant that is, but that is probably another argument that stdin is special (being line-buffered by the OS, EOF needing to be on a separate line). > > And FWIW, I have been writing networking code in Pharo for years, and I have never had issues with unclear semantics of these primitives (#atEnd, #next, #peek) on network streams, either the classic SocketStream or the Zdc* streams (TLS or not). That is why I think we have to be careful. > > That being said, it is important to continue this discussion, I find it very interesting. I am trying to write some test code using stdin myself, to better understand the topic. > >> On 11 Apr 2018, at 16:06, Alistair Grant <akgrant0710(a)gmail.com> wrote: >> >> On 11 April 2018 at 15:11, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>> >>> >>>> On 11 Apr 2018, at 11:12, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>>> >>>> How does one modify #atEnd to block ? I suppose you are talking about StdioStream>>#atEnd ? >>>> >>>> ^ self peek isNil >>>> >>>> ? >>> >>> Still the same question, how do you implement a blocking #atEnd for stdin ? >>> >>> I have seen your stdio.cs which is indeed needed as the current StdioStream>>#atEnd is bogus for sure. >>> >>> But that is still a non-blocking one, right ? >>> >>> Since there is a peekBuffer in StdioStream, why can't that be used ? >> >> I think you've created a chicken-and-egg problem with this question, >> but ignoring that for now: >> >> >> StdioStream>>peek >> "Answer what would be returned if the message next were sent to the >> receiver. If the receiver is at the end, answer nil. " >> >> self atEnd ifTrue: [^ nil ]. >> >> peekBuffer ifNotNil: [ ^ peekBuffer ]. >> >> ^ peekBuffer := self next. >> >> >> >> So when we first start the program, i.e. the user hasn't entered any >> input yet, and #peek is called: >> >> 1. #atEnd returns false because Ctrl-D (or similar) hasn't been >> entered (assuming it is non-blocking). >> 2. peekBuffer is nil because we haven't previously called #peek. >> 3. The system now blocks on "self next". >> >> >> Just a reminder: for terminal input the end-of-file isn't reached >> until the user explicitly enters the end of file key (Ctrl-D). >> >> So, if there is no buffered input (either none has been entered yet, >> or all input has been consumed) >> >> #atEnd (after the patch) calls #primAtEnd:. >> >> At the moment, #primAtEnd: ends up calling the libc function feof(), >> which is non-blocking and answers the end-of-file flag for the FILE*. >> Since the user hasn't entered Ctrl-D, that's false. >> >> If we want to control iteration over the stream and ensure that we >> don't need to do a "stream next == nil" check, then #primAtEnd: is >> going to have to peek for the next character, and that means waiting >> for the user to enter that character. >> >> In c that is typically done using: >> >> atEnd = ungetc(fgetc(fp), fp); >> >> and fgetc() will block until the user enters something. >> >>> I have run your example testAtEnd.st now, and it works/fails as advertised. >> >> :-) >> >> >> Cheers, >> Alistair >

April 11, 2018

Re: [Pharo-dev] help wanted: normalising LF on tonel for Pharo project

by Ben Coman

On 11 April 2018 at 05:05, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: > Hi, > > Iâve been wondering how to better fix the problem of having windows and > linux/macOS people contributing and the fact that files are written in > their native system format (crlf windows, lf for the rest of the world). > > > I digged a bit and I found a couple a link that helped me (after trying to > understand the doc): https://stackoverflow.com/questions/170961/whats- > the-best-crlf-carriage-return-line-feed-handling-strategy-with-git > > and it seems adding a .gitattributes file with this content: > > # Auto detect text files and perform LF normalization > * text=auto > I see a few posts around that recommend reading http://adaptivepatchwork.com/2012/03/01/mind-the-end-of-your-line/ which about the above line says... "This is certainly better than requiring everyone to be on the same global setting for core.autocrlf, but it means that you really trust Git to do binary detection properly. In my opinion it is better to explicitly specify your text files that you want normalized." and https://tinyurl.com/ya9xsprx says "We had a repo with * text=auto, and Git guessed wrong for an image file that it was a text file, causing it to corrupt it as it replaced CR + LF bytes with LF bytes in the object database." I'm unsure. Without it the system is subject to different users' different global settings and I'd guess that may be a more frequent problem than Git guessing wrong. The latter can be fixed by a user adding an extra .gitattributes entry explicitly specifying the file was binary, whereas the former seems to introduce a confounding factor. So probably a good line to have. > *.st text merge=union eol=lf > > could fix the problem? > can someone confirm this? > "eol=lf" looks appropriate... https://www.scivision.co/git-line-endings-windows-cygwin-wsl/ Most editors on Windows transparently handle LF line endings. https://en.wikipedia.org/wiki/Comparison_of_text_editors#Newline_support " merge=union" I am not familiar with, but I read at... https://git-scm.com/docs/gitattributes "union = Run 3-way file level merge for text files, but take lines from both versions, instead of leaving conflict markers. This tends to leave the added lines in the resulting file in random** order and the user should verify the result. Do not use this if you do not understand the implications." What are the implications of lines being merged in a random order? btw, has doing a callback from libgit to a custom merge driver in Pharo been considered? https://libgit2.github.com/libgit2/#HEAD/group/callback/git_merge_driver_ap… btw2, I found (https://githubengineering.com/move-fast/) interesting... saying... "Despite being a C library, libgit2 contains many powerful abstractions to accomplish complex tasks that Git simply cannot do. One of these features are indexes that exist solely in memory and allow work-tree related operations to be performed without an actual working directory. [...] With the in-memory index, libgit2 is capable of merging two trees in a repository without having to check out any of their files on disk." On 11 April 2018 at 05:55, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: > or a .iceberg file? > > Esteban > > ps: yep, we need itâ¦ we will have it, why not start now? > Do you mean Iceberg would clone a repo, and from its included .iceberg file a matching .gitattributes file would be created? That seems like double handling. Why not have the user edit the .gitattributes file directly from Iceberg? Iceberg might provide some appropriate templates. cheers -ben

April 11, 2018

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Sven Van Caekenberghe

I can make your example, using the Zn variants, work with the following change: StdioStream>>#atEnd ^ peekBuffer isNil or: [ (peekBuffer := self next) isNil ] Which is a literal implementation of your statement that you can only know that you are atEnd by reading (and thus waiting/blocking) and checking for nil, which seems logical to me, given the fact that you *are* waiting for user input. BTW, at least on macOS you have to enter ctrl-D (^D) on a separate line, I am not sure how relevant that is, but that is probably another argument that stdin is special (being line-buffered by the OS, EOF needing to be on a separate line). And FWIW, I have been writing networking code in Pharo for years, and I have never had issues with unclear semantics of these primitives (#atEnd, #next, #peek) on network streams, either the classic SocketStream or the Zdc* streams (TLS or not). That is why I think we have to be careful. That being said, it is important to continue this discussion, I find it very interesting. I am trying to write some test code using stdin myself, to better understand the topic. > On 11 Apr 2018, at 16:06, Alistair Grant <akgrant0710(a)gmail.com> wrote: > > On 11 April 2018 at 15:11, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> >> >>> On 11 Apr 2018, at 11:12, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>> >>> How does one modify #atEnd to block ? I suppose you are talking about StdioStream>>#atEnd ? >>> >>> ^ self peek isNil >>> >>> ? >> >> Still the same question, how do you implement a blocking #atEnd for stdin ? >> >> I have seen your stdio.cs which is indeed needed as the current StdioStream>>#atEnd is bogus for sure. >> >> But that is still a non-blocking one, right ? >> >> Since there is a peekBuffer in StdioStream, why can't that be used ? > > I think you've created a chicken-and-egg problem with this question, > but ignoring that for now: > > > StdioStream>>peek > "Answer what would be returned if the message next were sent to the > receiver. If the receiver is at the end, answer nil. " > > self atEnd ifTrue: [^ nil ]. > > peekBuffer ifNotNil: [ ^ peekBuffer ]. > > ^ peekBuffer := self next. > > > > So when we first start the program, i.e. the user hasn't entered any > input yet, and #peek is called: > > 1. #atEnd returns false because Ctrl-D (or similar) hasn't been > entered (assuming it is non-blocking). > 2. peekBuffer is nil because we haven't previously called #peek. > 3. The system now blocks on "self next". > > > Just a reminder: for terminal input the end-of-file isn't reached > until the user explicitly enters the end of file key (Ctrl-D). > > So, if there is no buffered input (either none has been entered yet, > or all input has been consumed) > > #atEnd (after the patch) calls #primAtEnd:. > > At the moment, #primAtEnd: ends up calling the libc function feof(), > which is non-blocking and answers the end-of-file flag for the FILE*. > Since the user hasn't entered Ctrl-D, that's false. > > If we want to control iteration over the stream and ensure that we > don't need to do a "stream next == nil" check, then #primAtEnd: is > going to have to peek for the next character, and that means waiting > for the user to enter that character. > > In c that is typically done using: > > atEnd = ungetc(fgetc(fp), fp); > > and fgetc() will block until the user enters something. > >> I have run your example testAtEnd.st now, and it works/fails as advertised. > > :-) > > > Cheers, > Alistair

April 11, 2018

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Alistair Grant

On 11 April 2018 at 15:11, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > >> On 11 Apr 2018, at 11:12, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> >> How does one modify #atEnd to block ? I suppose you are talking about StdioStream>>#atEnd ? >> >> ^ self peek isNil >> >> ? > > Still the same question, how do you implement a blocking #atEnd for stdin ? > > I have seen your stdio.cs which is indeed needed as the current StdioStream>>#atEnd is bogus for sure. > > But that is still a non-blocking one, right ? > > Since there is a peekBuffer in StdioStream, why can't that be used ? I think you've created a chicken-and-egg problem with this question, but ignoring that for now: StdioStream>>peek "Answer what would be returned if the message next were sent to the receiver. If the receiver is at the end, answer nil. " self atEnd ifTrue: [^ nil ]. peekBuffer ifNotNil: [ ^ peekBuffer ]. ^ peekBuffer := self next. So when we first start the program, i.e. the user hasn't entered any input yet, and #peek is called: 1. #atEnd returns false because Ctrl-D (or similar) hasn't been entered (assuming it is non-blocking). 2. peekBuffer is nil because we haven't previously called #peek. 3. The system now blocks on "self next". Just a reminder: for terminal input the end-of-file isn't reached until the user explicitly enters the end of file key (Ctrl-D). So, if there is no buffered input (either none has been entered yet, or all input has been consumed) #atEnd (after the patch) calls #primAtEnd:. At the moment, #primAtEnd: ends up calling the libc function feof(), which is non-blocking and answers the end-of-file flag for the FILE*. Since the user hasn't entered Ctrl-D, that's false. If we want to control iteration over the stream and ensure that we don't need to do a "stream next == nil" check, then #primAtEnd: is going to have to peek for the next character, and that means waiting for the user to enter that character. In c that is typically done using: atEnd = ungetc(fgetc(fp), fp); and fgetc() will block until the user enters something. > I have run your example testAtEnd.st now, and it works/fails as advertised. :-) Cheers, Alistair

April 11, 2018

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by David T. Lewis

OS pipes are a similar case. On Pharo, you can run CommandShellTestCase to provide some test coverage for this. Dave On Wed, Apr 11, 2018 at 03:13:35PM +0200, Denis Kudriashov wrote: > Thanks for explanation. > > I think it would be same scenario for socket stream where #atEnd is not > blocking. So I agree that it is expected behaviour. > > The example is general enough to expect it to be working for any given pair > of in and out streams. So our streams should support this. > > > 2018-04-11 14:56 GMT+02:00 Alistair Grant <akgrant0710(a)gmail.com>: > > > Hi Sven & Dennis, > > > > On 11 April 2018 at 12:04, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > > Hi Alistair. > > > > > > I don't think anybody is annoyed by you. You are doing really good job. > > And > > > nice thing that you are super patient to continue :) > > > > > > > On 11 April 2018 at 12:13, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > > > > Yes, Alistair, you are a top notch open source contributor ! > > > > > > For me, this discussion is about the difference between looking from low > > level details/issues/changes up, vs, from a higher level down. > > > > Thanks for your kind words. > > > > > > > > > What I try to understand is why blocking atEnd is bad? > > > Here is code from VMMaker: > > > > > > [stdin atEnd] whileFalse: > > > [| nextChunk | > > > stdout nextPutAll: 'squeak> '; flush. > > > nextChunk := stdin nextChunkNoTag. > > > [nextChunk notEmpty and: [nextChunk first isSeparator]] whileTrue: > > > [nextChunk := nextChunk allButFirst]. > > > Transcript cr; nextPutAll: nextChunk; cr; flush. > > > [stdout print: (Compiler evaluate: nextChunk); cr; flush] > > > on: Error > > > do: [:ex| self logError: ex description inContext: ex signalerContext to: > > > stderr]]. > > > quitOnEof ifTrue: > > > [SourceFiles at: 2 put: nil. > > > Smalltalk snapshot: false andQuit: true] > > > > > > > > > I am not see why it breaks with blocking #atEnd. Can you explain? > > > > > > First consider the case where #atEnd doesn't block and we just want to > > evaluate 4+3: > > > > 1. #atEnd will return false > > 2. the loop will print the prompt > > 3. wait for input (stdin nextChunkNoTag) > > 4. print the result > > 5. goto 1. > > > > So the screen will look like: > > > > squeak> 4+3! > > 7 > > squeak> [cursor here] > > > > Which is what we expect (prompt, input, result, prompt). > > > > If #atEnd is blocking the VM will hang at step 1 until the user enters > > something in the terminal. In Ubuntu at least terminal input appears to > > be line buffered, so for the example above the terminal will look like: > > > > 4+3! > > squeak> 7 > > [cursor here] > > > > We don't get the prompt when the program is started, the result is > > printed after the prompt, and then there's just a cursor sitting at the > > start of the next line. > > > > Obviously the program could be re-written to have the correct output > > with #atEnd blocking. But I'm arguing that this program is > > representative of many others, and we don't want to break backward > > compatibility in this case. > > > > Cheers, > > Alistair > > > > > > > > > > > > > > > > > > > 2018-04-11 11:41 GMT+02:00 Alistair Grant <akgrant0710(a)gmail.com>: > > >> > > >> Hi Sven, > > >> > > >> Oh dear. I feel as though I'm not getting my concerns across at all > > >> well, and I'm pushing hard enough that all I'm going to do is make > > >> people annoyed. So let me try to restate the issue one last time > > >> before answering your questions directly. > > >> > > >> Pharo & Squeak have unwritten rules about stream usage that I suspect > > >> have just emerged over time without being designed. > > >> > > >> If you want to be able to iterate over any stream, and in particular > > >> stdin from a terminal (which, as far as I know, is the outlier that > > >> causes all the problems) you have to follow these rules: > > >> > > >> 1. If the stream is character / byte oriented you have to check for > > >> EOF using "stream next == nil". #atEnd can be used, but you'll still > > >> have to do the nil check. > > >> > > >> 2. All other streams have to check for EOF (end of stream) using > > >> #atEnd. "stream next == nil" can be used, but you'll still need to > > >> test #atEnd to determine whether nil is a value returned by the > > >> stream. > > >> > > >> If you write code that you want to be able to consume characters, > > >> bytes or any other object, you'll have to test both "stream next == > > >> nil" and #atEnd. > > >> > > >> The rules are the result of the original blue book design being that > > >> #atEnd should be used, and then character input from a terminal being > > >> added later, but always returning an EOF character (nil) before #atEnd > > >> answers correctly. > > >> > > >> At the moment, ZnCharacterEncoder uses #atEnd on character / byte > > >> streams, so fails for stdin on a terminal. > > >> > > >> Back to your questions: > > >> > > >> On 11 April 2018 at 11:12, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > >> > > > >> > > > >> >> On 11 Apr 2018, at 10:29, Alistair Grant <akgrant0710(a)gmail.com> > > wrote: > > >> >> > > >> >> Hi Denis, > > >> >> > > >> >> On 11 April 2018 at 10:02, Denis Kudriashov <dionisiydk(a)gmail.com> > > >> >> wrote: > > >> >>> > > >> >>> 2018-04-11 8:32 GMT+02:00 Alistair Grant <akgrant0710(a)gmail.com>: > > >> >>>> > > >> >>>>>>> Where is it being said that #next and/or #atEnd should be > > blocking > > >> >>>>>>> or > > >> >>>>>>> non-blocking ? > > >> >>>>>> > > >> >>>>>> There is existing code that assumes that #atEnd is non-blocking > > and > > >> >>>>>> that #next is allowed block. I believe that we should keep those > > >> >>>>>> conditions. > > >> >>>>> > > >> >>>>> I fail to see where that is written down, either way. Can you > > point > > >> >>>>> me > > >> >>>>> to comments stating that, I would really like to know ? > > >> >>>> > > >> >>>> I'm not aware of it being written down, just that ever existing > > >> >>>> implementation I'm aware of behaves this way. > > >> >>>> > > >> >>>> On the other hand, making #atEnd blocking breaks Eliot's REPL > > sample > > >> >>>> (in Squeak). > > >> >>> > > >> >>> > > >> >>> Could you write here this example, please? > > >> >> > > >> >> The code is loaded in squeak using: > > >> >> > > >> >> > > >> >> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/ > > buildspurtrunkreaderimage.sh > > >> >> > > >> >> for 32 bit images. It loads: > > >> >> > > >> >> > > >> >> https://github.com/OpenSmalltalk/opensmalltalk- > > vm/blob/Cog/image/LoadReader.st > > >> >> > > >> >> which loads package CogTools-Listener in > > >> >> http://source.squeak.org/VMMaker > > >> >> > > >> >> An image that automatically runs the code and nothing else is created > > >> >> in: > > >> >> > > >> >> > > >> >> https://github.com/OpenSmalltalk/opensmalltalk- > > vm/blob/Cog/image/StartReader.st > > >> >> > > >> >> > > >> >> If you want to run it interactively you can load CogTools-Listener > > and > > >> >> do something like: > > >> >> > > >> >> StdioListener new > > >> >> quitOnEof: false; > > >> >> run > > >> > > > >> > What does #quitOnEof: do ? Can the StdioListener code be > > browsed/viewed > > >> > online somewhere ? > > >> > > >> I just referenced this as an example of making #atEnd (really > > >> FilePlugin>>primitiveFileAtEnd) blocking causing problems. I wasn't > > >> expecting people to go and look at the code or use it as a test. > > >> > > >> If you really want to look at it (from Pharo): > > >> > > >> 1. Add http://source.squeak.org/VMMaker as a repository. > > >> 2. Browse the CogTools-Listener package > > >> > > >> > > >> >> If you modify #atEnd to block it will result in the "squeak>" input > > >> >> prompt being printed in the terminal after the input has been > > entered. > > >> > > > >> > How does one modify #atEnd to block ? I suppose you are talking about > > >> > StdioStream>>#atEnd ? > > >> > > >> I meant the primitive, i.e. FilePlugin>>primitiveFileAtEnd / > > >> FilePluginPrims>>atEnd:. > > >> > > >> > > >> > ^ self peek isNil > > >> > > > >> > ? > > >> > > > >> > PS: I liked your runnable example better, I will try it later on. Thx! > > >> > > >> Right. My code is meant to be minimal and trigger the problem I'm > > >> actually focused on - that ZnCharacterEncoder doesn't work with stdin > > >> from a terminal. > > >> > > >> Sven has expressed a hesitation to change the internal operation of > > >> the Zinc streams from using #atEnd to "stream peek == nil" and this > > >> whole discussion is really about us trying to resolve our different > > >> perspective of the best path forward. I respect Sven and his work so > > >> I'm trying to justify the change (but I'm not expressing it at all > > >> well, obviously). > > >> > > >> Cheers, > > >> Alistair > > >> > > >> > > >> > > >> >> The code can be loaded in to Pharo and basically works, but the > > output > > >> >> tends to be hidden behind the next input prompt because it uses #cr > > >> >> instead of #lf. You can easily modify StdioListener>>initialize to > > >> >> set the line end convention in stdout. > > >> >> > > >> >> NOTE: It is not intended to be a release quality implementation of a > > >> >> evaluation loop. The whole purpose as I understand it is for it to > > be > > >> >> as simple as possible to assist in tracking down issues using the VM > > >> >> simulator. It runs minimal code to get to the point of waiting for > > >> >> user input and then allows an expression that causes problems to be > > >> >> entered and traced using the simulator. > > >> >> > > >> >> Cheers, > > >> >> Alistair > > >> >> > > >> >> > > >> >> > > >> >>>>>>> How is this related to how EOF is signalled ? > > >> >>>>>> > > >> >>>>>> Because, combined with terminal EOF not being known until the > > user > > >> >>>>>> explicitly flags it (with Ctrl-D) it means that #atEnd can't be > > >> >>>>>> used > > >> >>>>>> for iterating over input from stdin connected to a terminal. > > >> >>>>> > > >> >>>>> This seems to me like an exception that only holds for one > > >> >>>>> particular > > >> >>>>> stream in one particular scenario (interactive stdin). I might be > > >> >>>>> wrong. > > >> >>>>> > > >> >>>>>>> It seems to me that there are quite a few classes of streams > > that > > >> >>>>>>> are > > >> >>>>>>> 'special' in the sense that #next could be blocking and/or > > #atEnd > > >> >>>>>>> could be > > >> >>>>>>> unclear - socket/network streams, serial streams, maybe stdio > > >> >>>>>>> (interactive > > >> >>>>>>> or not). Without a message like #isDataAvailable you cannot > > handle > > >> >>>>>>> those > > >> >>>>>>> without blocking. > > >> >>>>>> > > >> >>>>>> Right. I think this is a distraction (I was trying to explain > > some > > >> >>>>>> details, but it's causing more confusion instead of helping). > > >> >>>>>> > > >> >>>>>> The important point is that #atEnd doesn't work for iterating > > over > > >> >>>>>> streams with terminal input > > >> >>>>> > > >> >>>>> Maybe you should also point to the actual code that fails. I mean > > >> >>>>> you > > >> >>>>> showed a partial stack trace, but not how you got there, > > precisely. > > >> >>>>> How does > > >> >>>>> the application reading from an interactive stdin do to get into > > >> >>>>> trouble ? > > >> >>>> > > >> >>>> Included below. > > >> >>>> > > >> >>>> > > >> >>>>>>> Reading from stdin seems like a very rare case for a Smalltalk > > >> >>>>>>> system > > >> >>>>>>> (not that it should not be possible). > > >> >>>>>> > > >> >>>>>> There's been quite a bit of discussion and several projects > > >> >>>>>> recently > > >> >>>>>> related to using pharo for scripting, so it may become more > > common. > > >> >>>>>> E.g. > > >> >>>>>> > > >> >>>>>> > > >> >>>>>> > > >> >>>>>> https://www.quora.com/Can-Smalltalk-be-a-batch-file- > > scripting-language/answer/Philippe-Back-1?share=c19bfc95 > > >> >>>>>> https://github.com/rajula96reddy/pharo-cli > > >> >>>>> > > >> >>>>> Still, it is not common at all. > > >> >>>>> > > >> >>>>>>> I have a feeling that too much functionality is being pushed > > into > > >> >>>>>>> too > > >> >>>>>>> small an API. > > >> >>>>>> > > >> >>>>>> This is just about how should Zinc streams be iterating over the > > >> >>>>>> underlying streams. You didn't like checking the result of #next > > >> >>>>>> for > > >> >>>>>> nil since it isn't general, correctly pointing out that nil is a > > >> >>>>>> valid > > >> >>>>>> value for non-byte oriented streams. But #atEnd doesn't work for > > >> >>>>>> stdin from a terminal. > > >> >>>>>> > > >> >>>>>> > > >> >>>>>> At this point I think there are three options: > > >> >>>>>> > > >> >>>>>> 1. Modify Zinc to check the return value of #next instead of > > using > > >> >>>>>> #atEnd. > > >> >>>>>> > > >> >>>>>> This is what all existing character / byte oriented streams in > > >> >>>>>> Squeak > > >> >>>>>> and Pharo do. At that point the Zinc streams can be used on all > > >> >>>>>> file > > >> >>>>>> / stdio input and output. > > >> >>>>> > > >> >>>>> I agree that such code exists in many places, but there is lots of > > >> >>>>> stream reading that does not check for nils. > > >> >>>> > > >> >>>> Right. Streams can be categorised in many ways, but for this > > >> >>>> discussion I think streams are broken in to two types: > > >> >>>> > > >> >>>> 1) Byte / Character oriented > > >> >>>> 2) All others > > >> >>>> > > >> >>>> For historical reasons, byte / character oriented streams need to > > >> >>>> check for EOF by using "stream next == nil" and all other streams > > >> >>>> should use #atEnd. > > >> >>>> > > >> >>>> This avoids the "nil being part of the domain" issue that was > > >> >>>> discussed earlier in the thread. > > >> >>>> > > >> >>>> > > >> >>>>>> 2. Modify all streams to signal EOF in some other way, i.e. a > > >> >>>>>> sentinel > > >> >>>>>> or notification / exception. > > >> >>>>>> > > >> >>>>>> This is what we were discussing below. But it is a decent chunk > > of > > >> >>>>>> work with significant impact on the existing code base. > > >> >>>>> > > >> >>>>> Agreed. This would be a future extension. > > >> >>>>> > > >> >>>>>> 3. Require anyone who wants to read from stdin to code around > > >> >>>>>> Zinc's > > >> >>>>>> inability to handle terminal input. > > >> >>>>>> > > >> >>>>>> I'd prefer to avoid this option if possible. > > >> >>>>> > > >> >>>>> See higher for a more concrete usage example request. > > >> >>>> > > >> >>>> > > >> >>>> testAtEnd.st > > >> >>>> -- > > >> >>>> | ch stream string stdin | > > >> >>>> > > >> >>>> 'stdio.cs' asFileReference fileIn. > > >> >>>> "stdin := FileStream stdin." > > >> >>>> stdin := ZnCharacterReadStream on: > > >> >>>> (ZnBufferedReadStream on: > > >> >>>> Stdio stdin). > > >> >>>> stream := (String new: 100) writeStream. > > >> >>>> ch := stdin next. > > >> >>>> [ ch == nil ] whileFalse: [ > > >> >>>> stream nextPut: ch. > > >> >>>> ch := stdin next. ]. > > >> >>>> string := stream contents. > > >> >>>> FileStream stdout > > >> >>>> nextPutAll: string; lf; > > >> >>>> nextPutAll: 'Characters read: '; > > >> >>>> nextPutAll: string size asString; > > >> >>>> lf. > > >> >>>> Smalltalk snapshot: false andQuit: true. > > >> >>>> -- > > >> >>>> > > >> >>>> Execute with: > > >> >>>> > > >> >>>> ./pharo --headless Pharo7.0-64bit-e76f1a2.image testAtEnd.st > > >> >>>> > > >> >>>> and type Ctrl-D gives: > > >> >>>> > > >> >>>> > > >> >>>> 'Errors in script loaded from testAtEnd.st' > > >> >>>> MessageNotUnderstood: receiver of "<" is nil > > >> >>>> UndefinedObject(Object)>>doesNotUnderstand: #< > > >> >>>> ZnUTF8Encoder>>nextCodePointFromStream: > > >> >>>> ZnUTF8Encoder(ZnCharacterEncoder)>>nextFromStream: > > >> >>>> ZnCharacterReadStream>>nextElement > > >> >>>> ZnCharacterReadStream(ZnEncodedReadStream)>>next > > >> >>>> UndefinedObject>>DoIt > > >> >>>> OpalCompiler>>evaluate > > >> >>>> > > >> >>>> > > >> >>>> Using #atEnd to control the loop instead of "stdin next == nil" > > >> >>>> produces the same result. > > >> >>>> > > >> >>>> Replacing stdin with FileStream stdin makes the script work. > > >> >>>> > > >> >>>> stdio.cs fixes a bug in StdioStream which really isn't part of this > > >> >>>> discussion (PR to be submitted). > > >> >>>> > > >> >>>> Cheers, > > >> >>>> Alistair > > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>>>> Does that clarify the situation? > > >> >>>>> > > >> >>>>> Yes, it helps. Thanks. But questions remain. > > >> >>>>> > > >> >>>>>> Thanks, > > >> >>>>>> Alistair > > > >

April 11, 2018

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Nicolas Cellier

Hi Alistair, I must take my part too: I suggested that we could use a pair of getc/ungetc to know if we are atEnd(OfData), but this obviously works well with AsyncFileIO only, else it blocks. For files, this generally isn't a problem (but maybe for network mounted partitions), the latency is bearable, but for sockets and pipes, that is not the right thing. 2018-04-11 14:56 GMT+02:00 Alistair Grant <akgrant0710(a)gmail.com>: > Hi Sven & Dennis, > > On 11 April 2018 at 12:04, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > Hi Alistair. > > > > I don't think anybody is annoyed by you. You are doing really good job. > And > > nice thing that you are super patient to continue :) > > > > On 11 April 2018 at 12:13, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > > Yes, Alistair, you are a top notch open source contributor ! > > > > For me, this discussion is about the difference between looking from low > level details/issues/changes up, vs, from a higher level down. > > Thanks for your kind words. > > > > > What I try to understand is why blocking atEnd is bad? > > Here is code from VMMaker: > > > > [stdin atEnd] whileFalse: > > [| nextChunk | > > stdout nextPutAll: 'squeak> '; flush. > > nextChunk := stdin nextChunkNoTag. > > [nextChunk notEmpty and: [nextChunk first isSeparator]] whileTrue: > > [nextChunk := nextChunk allButFirst]. > > Transcript cr; nextPutAll: nextChunk; cr; flush. > > [stdout print: (Compiler evaluate: nextChunk); cr; flush] > > on: Error > > do: [:ex| self logError: ex description inContext: ex signalerContext to: > > stderr]]. > > quitOnEof ifTrue: > > [SourceFiles at: 2 put: nil. > > Smalltalk snapshot: false andQuit: true] > > > > > > I am not see why it breaks with blocking #atEnd. Can you explain? > > > First consider the case where #atEnd doesn't block and we just want to > evaluate 4+3: > > 1. #atEnd will return false > 2. the loop will print the prompt > 3. wait for input (stdin nextChunkNoTag) > 4. print the result > 5. goto 1. > > So the screen will look like: > > squeak> 4+3! > 7 > squeak> [cursor here] > > Which is what we expect (prompt, input, result, prompt). > > If #atEnd is blocking the VM will hang at step 1 until the user enters > something in the terminal. In Ubuntu at least terminal input appears to > be line buffered, so for the example above the terminal will look like: > > 4+3! > squeak> 7 > [cursor here] > > We don't get the prompt when the program is started, the result is > printed after the prompt, and then there's just a cursor sitting at the > start of the next line. > > Obviously the program could be re-written to have the correct output > with #atEnd blocking. But I'm arguing that this program is > representative of many others, and we don't want to break backward > compatibility in this case. > > Cheers, > Alistair > > > > > > > > > > 2018-04-11 11:41 GMT+02:00 Alistair Grant <akgrant0710(a)gmail.com>: > >> > >> Hi Sven, > >> > >> Oh dear. I feel as though I'm not getting my concerns across at all > >> well, and I'm pushing hard enough that all I'm going to do is make > >> people annoyed. So let me try to restate the issue one last time > >> before answering your questions directly. > >> > >> Pharo & Squeak have unwritten rules about stream usage that I suspect > >> have just emerged over time without being designed. > >> > >> If you want to be able to iterate over any stream, and in particular > >> stdin from a terminal (which, as far as I know, is the outlier that > >> causes all the problems) you have to follow these rules: > >> > >> 1. If the stream is character / byte oriented you have to check for > >> EOF using "stream next == nil". #atEnd can be used, but you'll still > >> have to do the nil check. > >> > >> 2. All other streams have to check for EOF (end of stream) using > >> #atEnd. "stream next == nil" can be used, but you'll still need to > >> test #atEnd to determine whether nil is a value returned by the > >> stream. > >> > >> If you write code that you want to be able to consume characters, > >> bytes or any other object, you'll have to test both "stream next == > >> nil" and #atEnd. > >> > >> The rules are the result of the original blue book design being that > >> #atEnd should be used, and then character input from a terminal being > >> added later, but always returning an EOF character (nil) before #atEnd > >> answers correctly. > >> > >> At the moment, ZnCharacterEncoder uses #atEnd on character / byte > >> streams, so fails for stdin on a terminal. > >> > >> Back to your questions: > >> > >> On 11 April 2018 at 11:12, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > >> > > >> > > >> >> On 11 Apr 2018, at 10:29, Alistair Grant <akgrant0710(a)gmail.com> > wrote: > >> >> > >> >> Hi Denis, > >> >> > >> >> On 11 April 2018 at 10:02, Denis Kudriashov <dionisiydk(a)gmail.com> > >> >> wrote: > >> >>> > >> >>> 2018-04-11 8:32 GMT+02:00 Alistair Grant <akgrant0710(a)gmail.com>: > >> >>>> > >> >>>>>>> Where is it being said that #next and/or #atEnd should be > blocking > >> >>>>>>> or > >> >>>>>>> non-blocking ? > >> >>>>>> > >> >>>>>> There is existing code that assumes that #atEnd is non-blocking > and > >> >>>>>> that #next is allowed block. I believe that we should keep those > >> >>>>>> conditions. > >> >>>>> > >> >>>>> I fail to see where that is written down, either way. Can you > point > >> >>>>> me > >> >>>>> to comments stating that, I would really like to know ? > >> >>>> > >> >>>> I'm not aware of it being written down, just that ever existing > >> >>>> implementation I'm aware of behaves this way. > >> >>>> > >> >>>> On the other hand, making #atEnd blocking breaks Eliot's REPL > sample > >> >>>> (in Squeak). > >> >>> > >> >>> > >> >>> Could you write here this example, please? > >> >> > >> >> The code is loaded in squeak using: > >> >> > >> >> > >> >> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/ > buildspurtrunkreaderimage.sh > >> >> > >> >> for 32 bit images. It loads: > >> >> > >> >> > >> >> https://github.com/OpenSmalltalk/opensmalltalk- > vm/blob/Cog/image/LoadReader.st > >> >> > >> >> which loads package CogTools-Listener in > >> >> http://source.squeak.org/VMMaker > >> >> > >> >> An image that automatically runs the code and nothing else is created > >> >> in: > >> >> > >> >> > >> >> https://github.com/OpenSmalltalk/opensmalltalk- > vm/blob/Cog/image/StartReader.st > >> >> > >> >> > >> >> If you want to run it interactively you can load CogTools-Listener > and > >> >> do something like: > >> >> > >> >> StdioListener new > >> >> quitOnEof: false; > >> >> run > >> > > >> > What does #quitOnEof: do ? Can the StdioListener code be > browsed/viewed > >> > online somewhere ? > >> > >> I just referenced this as an example of making #atEnd (really > >> FilePlugin>>primitiveFileAtEnd) blocking causing problems. I wasn't > >> expecting people to go and look at the code or use it as a test. > >> > >> If you really want to look at it (from Pharo): > >> > >> 1. Add http://source.squeak.org/VMMaker as a repository. > >> 2. Browse the CogTools-Listener package > >> > >> > >> >> If you modify #atEnd to block it will result in the "squeak>" input > >> >> prompt being printed in the terminal after the input has been > entered. > >> > > >> > How does one modify #atEnd to block ? I suppose you are talking about > >> > StdioStream>>#atEnd ? > >> > >> I meant the primitive, i.e. FilePlugin>>primitiveFileAtEnd / > >> FilePluginPrims>>atEnd:. > >> > >> > >> > ^ self peek isNil > >> > > >> > ? > >> > > >> > PS: I liked your runnable example better, I will try it later on. Thx! > >> > >> Right. My code is meant to be minimal and trigger the problem I'm > >> actually focused on - that ZnCharacterEncoder doesn't work with stdin > >> from a terminal. > >> > >> Sven has expressed a hesitation to change the internal operation of > >> the Zinc streams from using #atEnd to "stream peek == nil" and this > >> whole discussion is really about us trying to resolve our different > >> perspective of the best path forward. I respect Sven and his work so > >> I'm trying to justify the change (but I'm not expressing it at all > >> well, obviously). > >> > >> Cheers, > >> Alistair > >> > >> > >> > >> >> The code can be loaded in to Pharo and basically works, but the > output > >> >> tends to be hidden behind the next input prompt because it uses #cr > >> >> instead of #lf. You can easily modify StdioListener>>initialize to > >> >> set the line end convention in stdout. > >> >> > >> >> NOTE: It is not intended to be a release quality implementation of a > >> >> evaluation loop. The whole purpose as I understand it is for it to > be > >> >> as simple as possible to assist in tracking down issues using the VM > >> >> simulator. It runs minimal code to get to the point of waiting for > >> >> user input and then allows an expression that causes problems to be > >> >> entered and traced using the simulator. > >> >> > >> >> Cheers, > >> >> Alistair > >> >> > >> >> > >> >> > >> >>>>>>> How is this related to how EOF is signalled ? > >> >>>>>> > >> >>>>>> Because, combined with terminal EOF not being known until the > user > >> >>>>>> explicitly flags it (with Ctrl-D) it means that #atEnd can't be > >> >>>>>> used > >> >>>>>> for iterating over input from stdin connected to a terminal. > >> >>>>> > >> >>>>> This seems to me like an exception that only holds for one > >> >>>>> particular > >> >>>>> stream in one particular scenario (interactive stdin). I might be > >> >>>>> wrong. > >> >>>>> > >> >>>>>>> It seems to me that there are quite a few classes of streams > that > >> >>>>>>> are > >> >>>>>>> 'special' in the sense that #next could be blocking and/or > #atEnd > >> >>>>>>> could be > >> >>>>>>> unclear - socket/network streams, serial streams, maybe stdio > >> >>>>>>> (interactive > >> >>>>>>> or not). Without a message like #isDataAvailable you cannot > handle > >> >>>>>>> those > >> >>>>>>> without blocking. > >> >>>>>> > >> >>>>>> Right. I think this is a distraction (I was trying to explain > some > >> >>>>>> details, but it's causing more confusion instead of helping). > >> >>>>>> > >> >>>>>> The important point is that #atEnd doesn't work for iterating > over > >> >>>>>> streams with terminal input > >> >>>>> > >> >>>>> Maybe you should also point to the actual code that fails. I mean > >> >>>>> you > >> >>>>> showed a partial stack trace, but not how you got there, > precisely. > >> >>>>> How does > >> >>>>> the application reading from an interactive stdin do to get into > >> >>>>> trouble ? > >> >>>> > >> >>>> Included below. > >> >>>> > >> >>>> > >> >>>>>>> Reading from stdin seems like a very rare case for a Smalltalk > >> >>>>>>> system > >> >>>>>>> (not that it should not be possible). > >> >>>>>> > >> >>>>>> There's been quite a bit of discussion and several projects > >> >>>>>> recently > >> >>>>>> related to using pharo for scripting, so it may become more > common. > >> >>>>>> E.g. > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> https://www.quora.com/Can-Smalltalk-be-a-batch-file- > scripting-language/answer/Philippe-Back-1?share=c19bfc95 > >> >>>>>> https://github.com/rajula96reddy/pharo-cli > >> >>>>> > >> >>>>> Still, it is not common at all. > >> >>>>> > >> >>>>>>> I have a feeling that too much functionality is being pushed > into > >> >>>>>>> too > >> >>>>>>> small an API. > >> >>>>>> > >> >>>>>> This is just about how should Zinc streams be iterating over the > >> >>>>>> underlying streams. You didn't like checking the result of #next > >> >>>>>> for > >> >>>>>> nil since it isn't general, correctly pointing out that nil is a > >> >>>>>> valid > >> >>>>>> value for non-byte oriented streams. But #atEnd doesn't work for > >> >>>>>> stdin from a terminal. > >> >>>>>> > >> >>>>>> > >> >>>>>> At this point I think there are three options: > >> >>>>>> > >> >>>>>> 1. Modify Zinc to check the return value of #next instead of > using > >> >>>>>> #atEnd. > >> >>>>>> > >> >>>>>> This is what all existing character / byte oriented streams in > >> >>>>>> Squeak > >> >>>>>> and Pharo do. At that point the Zinc streams can be used on all > >> >>>>>> file > >> >>>>>> / stdio input and output. > >> >>>>> > >> >>>>> I agree that such code exists in many places, but there is lots of > >> >>>>> stream reading that does not check for nils. > >> >>>> > >> >>>> Right. Streams can be categorised in many ways, but for this > >> >>>> discussion I think streams are broken in to two types: > >> >>>> > >> >>>> 1) Byte / Character oriented > >> >>>> 2) All others > >> >>>> > >> >>>> For historical reasons, byte / character oriented streams need to > >> >>>> check for EOF by using "stream next == nil" and all other streams > >> >>>> should use #atEnd. > >> >>>> > >> >>>> This avoids the "nil being part of the domain" issue that was > >> >>>> discussed earlier in the thread. > >> >>>> > >> >>>> > >> >>>>>> 2. Modify all streams to signal EOF in some other way, i.e. a > >> >>>>>> sentinel > >> >>>>>> or notification / exception. > >> >>>>>> > >> >>>>>> This is what we were discussing below. But it is a decent chunk > of > >> >>>>>> work with significant impact on the existing code base. > >> >>>>> > >> >>>>> Agreed. This would be a future extension. > >> >>>>> > >> >>>>>> 3. Require anyone who wants to read from stdin to code around > >> >>>>>> Zinc's > >> >>>>>> inability to handle terminal input. > >> >>>>>> > >> >>>>>> I'd prefer to avoid this option if possible. > >> >>>>> > >> >>>>> See higher for a more concrete usage example request. > >> >>>> > >> >>>> > >> >>>> testAtEnd.st > >> >>>> -- > >> >>>> | ch stream string stdin | > >> >>>> > >> >>>> 'stdio.cs' asFileReference fileIn. > >> >>>> "stdin := FileStream stdin." > >> >>>> stdin := ZnCharacterReadStream on: > >> >>>> (ZnBufferedReadStream on: > >> >>>> Stdio stdin). > >> >>>> stream := (String new: 100) writeStream. > >> >>>> ch := stdin next. > >> >>>> [ ch == nil ] whileFalse: [ > >> >>>> stream nextPut: ch. > >> >>>> ch := stdin next. ]. > >> >>>> string := stream contents. > >> >>>> FileStream stdout > >> >>>> nextPutAll: string; lf; > >> >>>> nextPutAll: 'Characters read: '; > >> >>>> nextPutAll: string size asString; > >> >>>> lf. > >> >>>> Smalltalk snapshot: false andQuit: true. > >> >>>> -- > >> >>>> > >> >>>> Execute with: > >> >>>> > >> >>>> ./pharo --headless Pharo7.0-64bit-e76f1a2.image testAtEnd.st > >> >>>> > >> >>>> and type Ctrl-D gives: > >> >>>> > >> >>>> > >> >>>> 'Errors in script loaded from testAtEnd.st' > >> >>>> MessageNotUnderstood: receiver of "<" is nil > >> >>>> UndefinedObject(Object)>>doesNotUnderstand: #< > >> >>>> ZnUTF8Encoder>>nextCodePointFromStream: > >> >>>> ZnUTF8Encoder(ZnCharacterEncoder)>>nextFromStream: > >> >>>> ZnCharacterReadStream>>nextElement > >> >>>> ZnCharacterReadStream(ZnEncodedReadStream)>>next > >> >>>> UndefinedObject>>DoIt > >> >>>> OpalCompiler>>evaluate > >> >>>> > >> >>>> > >> >>>> Using #atEnd to control the loop instead of "stdin next == nil" > >> >>>> produces the same result. > >> >>>> > >> >>>> Replacing stdin with FileStream stdin makes the script work. > >> >>>> > >> >>>> stdio.cs fixes a bug in StdioStream which really isn't part of this > >> >>>> discussion (PR to be submitted). > >> >>>> > >> >>>> Cheers, > >> >>>> Alistair > >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>>>> Does that clarify the situation? > >> >>>>> > >> >>>>> Yes, it helps. Thanks. But questions remain. > >> >>>>> > >> >>>>>> Thanks, > >> >>>>>> Alistair > >

April 11, 2018