- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Eliot Miranda

On Wed, Apr 4, 2018 at 8:37 AM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > > On 4 Apr 2018, at 17:32, K K Subbu <kksubbu.ml(a)gmail.com> wrote: > > > > On Wednesday 04 April 2018 04:06 PM, Nicolas Cellier wrote: > >>> IIRC, someone said it is implemented as 'remaining size being zero' > >>> and some virtual unix files like /dev/random are zero sized. > >> Currently, for files other than sdio (stdout, stderr, stdin) it is > >> effectively defined as: > >> atEnd := stream position >= stream size > > I see a confusion between Stream and its underlying collection. Stream > is an iterator and just does next, nextPut, peek, reset etc. But methods > like size or atEnd depend on its collection and there is no guarantee that > this collection has a known and finite size. > > > > Essentially, a collection's size may be known and finite, unknown but > finite size or infinite. This has nothing do with file descriptor being > std{in,out,err}. If std* is a regular file or special file like /dev/mem, > /dev/null, it's size is known at open time. With console streams or sysfs > files, size is unknown until EOT (^D) or NUL is received. Lastly, special > files like /dev/zero, /dev/random or /proc/cpuinfo don't have a finite size > but report it as zero (!). > > > > [ stream atEnd ] whileFalse: [ stream next. .. ] > > > > will only terminate if its collection size is finite. It won't terminate > for infinite collections. > > > > Regards .. Subbu > > Good summary, I agree. > > Still, what are the semantics of #next - does the caller always have to > check for nil ? Do we think this is ugly (as the return value is outside > the domain) ? The problem is not that the value is outside the domain. The problem is that it may be within the domain. Answering an element within the domain when there are no more elements is insane. Not even C does that. > Do we then still need #atEnd ? > For backward compatibility, yes, even if deprecated. There is so much existing code that uses it, it'll be useful to have. > Sven > _,,,^..^,,,_ best, Eliot

April 4, 2018

Idea and proof of concept: Accept for Test

by Marcus Denker

Hi, Some code is very âactiveâ, executed all the time, e.g. collections, graphicsâ¦ or imagine if you work on the compiler or debugger. It would really be nice if we could test a change before accepting it. Something like âAccept for Testâ where magically the original method does not change, yet, when running tests, the version of the code we accepted for testing is executed. So how do we implement that? 1) menu in the browser. I did a quick hack, a Suggestions AST menu shows for all nodes (to be replaced with the âcorrectâ way later). SugsSuggestion subclass: #SugsAcceptForTest instanceVariableNames: '' classVariableNames: '' package: 'SmartSuggestions-Suggestionâ label ^'Accept for testâ So, now we can do everything in the #execute method. What do we need? 1) How we know that we are in a test? â> CurrentExecutionEnvironment value isTest 2) can we compile the buffer even if there are syntax errors (kind of what I expect to happen in non-accepted code)? â> yes, remember, we use the same Parser for syntax highlighting and we extended code-generation for SyntaxError Nodes in the past (it raises a syntax error at runtime, isnât that kind off odd in a fun way?). newMethod := context selectedClass compiler source: context code; options: #(+ optionParseErrors); compile. 3) Now how to we âhook intoâ the execution of the original unchanged method? We use a MetaLink. We install it as a before link on the RBMethodNode, the meta-object is a block. This block gets as arguments the thisContext and the arguments of the original method and then just does what you want it do do: if we are in test, return the result of the execution of the method we just compiled. (Closures are fun, we can just reference the outer temp newMethod, no need to store it anywhere else explicitly): [ :aContext :args | CurrentExecutionEnvironment value isTest ifTrue: [ aContext return: (newMethod valueWithReceiver: aContext receiver arguments: args) ] ]; Thus, all the code is this: execute | newMethod metaLink | âwe compile a new method from the not-accepted text editor. Allow syntax errors" newMethod := context selectedClass compiler source: context code; options: #(+ optionParseErrors); compile. "the link executes the method we just created and returns" metaLink := MetaLink new metaObject: [ :aContext :args | CurrentExecutionEnvironment value isTest ifTrue: [ aContext return: (newMethod valueWithReceiver: aContext receiver arguments: args) ] ]; selector: #value:value:; arguments: #(context arguments). context selectedMethod ast link: metaLink Cleanup we do not care as everything will be GCed as soon as we accept the method for real. And it works! I did not test it too much, but it seems to do what it is supposed to do. (but more tests and testing are for sure needed) https://github.com/pharo-project/pharo/pull/1182 Next step: move it to a better place in the menu. Marcus

April 4, 2018

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Sven Van Caekenberghe

> On 4 Apr 2018, at 17:32, K K Subbu <kksubbu.ml(a)gmail.com> wrote: > > On Wednesday 04 April 2018 04:06 PM, Nicolas Cellier wrote: >>> IIRC, someone said it is implemented as 'remaining size being zero' >>> and some virtual unix files like /dev/random are zero sized. >> Currently, for files other than sdio (stdout, stderr, stdin) it is >> effectively defined as: >> atEnd := stream position >= stream size > I see a confusion between Stream and its underlying collection. Stream is an iterator and just does next, nextPut, peek, reset etc. But methods like size or atEnd depend on its collection and there is no guarantee that this collection has a known and finite size. > > Essentially, a collection's size may be known and finite, unknown but finite size or infinite. This has nothing do with file descriptor being std{in,out,err}. If std* is a regular file or special file like /dev/mem, /dev/null, it's size is known at open time. With console streams or sysfs files, size is unknown until EOT (^D) or NUL is received. Lastly, special files like /dev/zero, /dev/random or /proc/cpuinfo don't have a finite size but report it as zero (!). > > [ stream atEnd ] whileFalse: [ stream next. .. ] > > will only terminate if its collection size is finite. It won't terminate for infinite collections. > > Regards .. Subbu Good summary, I agree. Still, what are the semantics of #next - does the caller always have to check for nil ? Do we think this is ugly (as the return value is outside the domain) ? Do we then still need #atEnd ? Sven

April 4, 2018

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by K K Subbu

On Wednesday 04 April 2018 04:06 PM, Nicolas Cellier wrote: >> IIRC, someone said it is implemented as 'remaining size being zero' >> and some virtual unix files like /dev/random are zero sized. > > Currently, for files other than sdio (stdout, stderr, stdin) it is > effectively defined as: > > atEnd := stream position >= stream size I see a confusion between Stream and its underlying collection. Stream is an iterator and just does next, nextPut, peek, reset etc. But methods like size or atEnd depend on its collection and there is no guarantee that this collection has a known and finite size. Essentially, a collection's size may be known and finite, unknown but finite size or infinite. This has nothing do with file descriptor being std{in,out,err}. If std* is a regular file or special file like /dev/mem, /dev/null, it's size is known at open time. With console streams or sysfs files, size is unknown until EOT (^D) or NUL is received. Lastly, special files like /dev/zero, /dev/random or /proc/cpuinfo don't have a finite size but report it as zero (!). [ stream atEnd ] whileFalse: [ stream next. .. ] will only terminate if its collection size is finite. It won't terminate for infinite collections. Regards .. Subbu

April 4, 2018

Re: [Pharo-dev] [squeak-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Eliot Miranda

Hi Nicolas, On Wed, Apr 4, 2018 at 7:24 AM, Nicolas Cellier < nicolas.cellier.aka.nice(a)gmail.com> wrote: > > > 2018-04-04 16:12 GMT+02:00 Eliot Miranda <eliot.miranda(a)gmail.com>: > >> Hi Denis, Hi Sven, >> >> On Apr 4, 2018, at 3:16 AM, Denis Kudriashov <dionisiydk(a)gmail.com> >> wrote: >> >> Hi. >> >> #next returning nil is definitely looks bad because we will always have >> case when nil value will not means the end: >> >> #(1 nil 2) readSteam next; next; atEnd >> >> But I understand that in many cases analysing #next for nil is much more >> suitable than #atEnd. I am sure that it can be always avoided but it can be >> not easy to do. >> Alternatively we can reify real EndOfStream object which will be used as >> result of reading operation when stream is at end. >> It can provide nice API like #ifEndOfStream:. which will reflect >> intention in much cleaner way than ifNil: checks. >> >> >> There is a much better scheme which is to allow streams to have an >> end-of-file element in an inst var, "endOfStream". It is initialised with >> the stream itself (nil cannot be used as we want it as a value). If the >> end of the stream is reached and endOfStream == self an EndIfStream >> exception is raised. Otherwise the value of endOfStream is answered. We >> can't answer nil because in some applications (streams of objects) nil is a >> valid element to read from the steam and we need to manufacture a unique >> sentinel. >> >> We made this change in VisualWorks and it works well. One can avoid >> using atEnd and lion for the sentinel instead, which is much cleaner and >> faster and less problematic than atEnd. >> >> >> Hi Eliot, > Using self is a nice hack for saving an inst. var. :) > > If we want to afford an inst.var. then we can also reify an > endOfStreamAction inst.var. like I did in ExtendedStream > ^endOfStreamAction value > We are then free to use Notification/Error, default value (reified > EndOfStream object, nil, ...), block with block return... > The problem I had is when I started using stream wrappers like in Xtreams. > Each wrapper has it's own action, and it's not obvious how we should > compose these. > I suppose one could send value to the endOfStream element. Then if one wanted to answer a block, rather than the evaluation of a block, one would wrap it in a block or an association. e.g. pastEnd ^endOfStream == self ifTrue: [self class endOfStreamSignal signal] ifFalse: [endOfStream value] and then myStream setEndOfStreamElement: nil -> ['I want it to answer a block on end of stream goddammit!!!'] as opposed to myStream setEndOfStreamElement: ['I want it to answer this string on end of stream goddammit!!!'] P.S. Note that this hack is used in CompiledMethod>>messages in the form of InstructionStream>>selectorToSendOrSelf because notionally any object can function as a message selector, not just symbols. Of course nil could be used here because nil is the value used for an empty key in a method dictionary. But that's implementation. A better representation for method dictionaries is as a flat sequence of pairs, ordered by selector identity hash; but that's a separate discussion :-) > Nicolas > > Sorry to double post, I should have respond-to-all as default action... > > >> >> >> 2018-04-04 12:00 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: >> >>> >>> >>> > On 4 Apr 2018, at 11:38, Nicolas Cellier < >>> nicolas.cellier.aka.nice(a)gmail.com> wrote: >>> > >>> > Hi Sven, >>> > See also discussion at https://github.com/OpenSmallta >>> lk/opensmalltalk-vm/pull/232 >>> >>> Thanks Nicolas, I had already seen parts of it. >>> >>> Now, I still want image level changes to be based on clear semantic >>> definitions of the abstract stream API, we're not just talking about file >>> or unix streams, the general Smalltalk concept. >>> >>> For reading, they are IMHO, >>> >>> #next >>> #readInto:startingAt:count: >>> #peek >>> #atEnd >>> #upToEnd (can be derived but still the semantics are important in >>> relation to #atEnd) >>> >>> For writing, we have >>> >>> #nextPut: >>> #next:putAll:startingAt: >>> #flush >>> >>> For both, we have >>> >>> #atEnd >>> #close >>> #closed (new) >>> >>> So, I know, #next returning nil exists, but is it universally/officially >>> defined as such ? Where is that documented ? >>> >>> Positioning, sizing are not universal, IMHO, but should be clearly >>> defined as well. >>> >>> > 2018-04-04 11:32 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: >>> > Somehow, somewhere there was a change to the implementation of the >>> primitive called by some streams' #atEnd. >>> > >>> > IIRC, someone said it is implemented as 'remaining size being zero' >>> and some virtual unix files like /dev/random are zero sized. >>> > >>> > Now, all kinds of changes are being done image size to work around >>> this. >>> > >>> > I am a strong believer in simple, real (i.e. infinite) streams, but I >>> am not sure we are doing the right thing here. >>> > >>> > Point is, I am not sure #next returning nil is official and universal. >>> > >>> > Consider the comments: >>> > >>> > Stream>>#next >>> > "Answer the next object accessible by the receiver." >>> > >>> > ReadStream>>#next >>> > "Primitive. Answer the next object in the Stream represented by the >>> > receiver. Fail if the collection of this stream is not an Array or a >>> String. >>> > Fail if the stream is positioned at its end, or if the position is >>> out of >>> > bounds in the collection. Optional. See Object documentation >>> > whatIsAPrimitive." >>> > >>> > Note how there is no talk about returning nil ! >>> > >>> > I think we should discuss about this first. >>> > >>> > Was the low level change really correct and the right thing to do ? >>> > >>> > Note also that a Guille introduced something new, #closed which is >>> related to the difference between having no more elements (maybe right now, >>> like an open network stream) and never ever being able to produce more data. >>> > >>> > Sven >>> > >>> > >>> > >>> > >>> >>> >>> >> >> >> >> > > > > -- _,,,^..^,,,_ best, Eliot

April 4, 2018

Re: [Pharo-dev] [squeak-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Nicolas Cellier

2018-04-04 16:12 GMT+02:00 Eliot Miranda <eliot.miranda(a)gmail.com>: > Hi Denis, Hi Sven, > > On Apr 4, 2018, at 3:16 AM, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > Hi. > > #next returning nil is definitely looks bad because we will always have > case when nil value will not means the end: > > #(1 nil 2) readSteam next; next; atEnd > > But I understand that in many cases analysing #next for nil is much more > suitable than #atEnd. I am sure that it can be always avoided but it can be > not easy to do. > Alternatively we can reify real EndOfStream object which will be used as > result of reading operation when stream is at end. > It can provide nice API like #ifEndOfStream:. which will reflect intention > in much cleaner way than ifNil: checks. > > > There is a much better scheme which is to allow streams to have an > end-of-file element in an inst var, "endOfStream". It is initialised with > the stream itself (nil cannot be used as we want it as a value). If the > end of the stream is reached and endOfStream == self an EndIfStream > exception is raised. Otherwise the value of endOfStream is answered. We > can't answer nil because in some applications (streams of objects) nil is a > valid element to read from the steam and we need to manufacture a unique > sentinel. > > We made this change in VisualWorks and it works well. One can avoid using > atEnd and lion for the sentinel instead, which is much cleaner and faster > and less problematic than atEnd. > > > Hi Eliot, Using self is a nice hack for saving an inst. var. :) If we want to afford an inst.var. then we can also reify an endOfStreamAction inst.var. like I did in ExtendedStream ^endOfStreamAction value We are then free to use Notification/Error, default value (reified EndOfStream object, nil, ...), block with block return... The problem I had is when I started using stream wrappers like in Xtreams. Each wrapper has it's own action, and it's not obvious how we should compose these. Nicolas Sorry to double post, I should have respond-to-all as default action... > > > 2018-04-04 12:00 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > >> >> >> > On 4 Apr 2018, at 11:38, Nicolas Cellier <nicolas.cellier.aka.nice@gmai >> l.com> wrote: >> > >> > Hi Sven, >> > See also discussion at https://github.com/OpenSmallta >> lk/opensmalltalk-vm/pull/232 >> >> Thanks Nicolas, I had already seen parts of it. >> >> Now, I still want image level changes to be based on clear semantic >> definitions of the abstract stream API, we're not just talking about file >> or unix streams, the general Smalltalk concept. >> >> For reading, they are IMHO, >> >> #next >> #readInto:startingAt:count: >> #peek >> #atEnd >> #upToEnd (can be derived but still the semantics are important in >> relation to #atEnd) >> >> For writing, we have >> >> #nextPut: >> #next:putAll:startingAt: >> #flush >> >> For both, we have >> >> #atEnd >> #close >> #closed (new) >> >> So, I know, #next returning nil exists, but is it universally/officially >> defined as such ? Where is that documented ? >> >> Positioning, sizing are not universal, IMHO, but should be clearly >> defined as well. >> >> > 2018-04-04 11:32 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: >> > Somehow, somewhere there was a change to the implementation of the >> primitive called by some streams' #atEnd. >> > >> > IIRC, someone said it is implemented as 'remaining size being zero' and >> some virtual unix files like /dev/random are zero sized. >> > >> > Now, all kinds of changes are being done image size to work around this. >> > >> > I am a strong believer in simple, real (i.e. infinite) streams, but I >> am not sure we are doing the right thing here. >> > >> > Point is, I am not sure #next returning nil is official and universal. >> > >> > Consider the comments: >> > >> > Stream>>#next >> > "Answer the next object accessible by the receiver." >> > >> > ReadStream>>#next >> > "Primitive. Answer the next object in the Stream represented by the >> > receiver. Fail if the collection of this stream is not an Array or a >> String. >> > Fail if the stream is positioned at its end, or if the position is >> out of >> > bounds in the collection. Optional. See Object documentation >> > whatIsAPrimitive." >> > >> > Note how there is no talk about returning nil ! >> > >> > I think we should discuss about this first. >> > >> > Was the low level change really correct and the right thing to do ? >> > >> > Note also that a Guille introduced something new, #closed which is >> related to the difference between having no more elements (maybe right now, >> like an open network stream) and never ever being able to produce more data. >> > >> > Sven >> > >> > >> > >> > >> >> >> > > > >

April 4, 2018

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Eliot Miranda

Hi Denis, Hi Sven, > On Apr 4, 2018, at 3:16 AM, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > Hi. > > #next returning nil is definitely looks bad because we will always have case when nil value will not means the end: > > #(1 nil 2) readSteam next; next; atEnd > > But I understand that in many cases analysing #next for nil is much more suitable than #atEnd. I am sure that it can be always avoided but it can be not easy to do. > Alternatively we can reify real EndOfStream object which will be used as result of reading operation when stream is at end. > It can provide nice API like #ifEndOfStream:. which will reflect intention in much cleaner way than ifNil: checks. There is a much better scheme which is to allow streams to have an end-of-file element in an inst var, "endOfStream". It is initialised with the stream itself (nil cannot be used as we want it as a value). If the end of the stream is reached and endOfStream == self an EndIfStream exception is raised. Otherwise the value of endOfStream is answered. We can't answer nil because in some applications (streams of objects) nil is a valid element to read from the steam and we need to manufacture a unique sentinel. We made this change in VisualWorks and it works well. One can avoid using atEnd and lion for the sentinel instead, which is much cleaner and faster and less problematic than atEnd. > > > 2018-04-04 12:00 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: >> >> >> > On 4 Apr 2018, at 11:38, Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com> wrote: >> > >> > Hi Sven, >> > See also discussion at https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/232 >> >> Thanks Nicolas, I had already seen parts of it. >> >> Now, I still want image level changes to be based on clear semantic definitions of the abstract stream API, we're not just talking about file or unix streams, the general Smalltalk concept. >> >> For reading, they are IMHO, >> >> #next >> #readInto:startingAt:count: >> #peek >> #atEnd >> #upToEnd (can be derived but still the semantics are important in relation to #atEnd) >> >> For writing, we have >> >> #nextPut: >> #next:putAll:startingAt: >> #flush >> >> For both, we have >> >> #atEnd >> #close >> #closed (new) >> >> So, I know, #next returning nil exists, but is it universally/officially defined as such ? Where is that documented ? >> >> Positioning, sizing are not universal, IMHO, but should be clearly defined as well. >> >> > 2018-04-04 11:32 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: >> > Somehow, somewhere there was a change to the implementation of the primitive called by some streams' #atEnd. >> > >> > IIRC, someone said it is implemented as 'remaining size being zero' and some virtual unix files like /dev/random are zero sized. >> > >> > Now, all kinds of changes are being done image size to work around this. >> > >> > I am a strong believer in simple, real (i.e. infinite) streams, but I am not sure we are doing the right thing here. >> > >> > Point is, I am not sure #next returning nil is official and universal. >> > >> > Consider the comments: >> > >> > Stream>>#next >> > "Answer the next object accessible by the receiver." >> > >> > ReadStream>>#next >> > "Primitive. Answer the next object in the Stream represented by the >> > receiver. Fail if the collection of this stream is not an Array or a String. >> > Fail if the stream is positioned at its end, or if the position is out of >> > bounds in the collection. Optional. See Object documentation >> > whatIsAPrimitive." >> > >> > Note how there is no talk about returning nil ! >> > >> > I think we should discuss about this first. >> > >> > Was the low level change really correct and the right thing to do ? >> > >> > Note also that a Guille introduced something new, #closed which is related to the difference between having no more elements (maybe right now, like an open network stream) and never ever being able to produce more data. >> > >> > Sven >> > >> > >> > >> > >> >> >

April 4, 2018

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Sven Van Caekenberghe

Alistair, First off, thanks for the discussions and your contributions, I really appreciate them. But I want to have a discussion at the high level of the definition and semantics of the stream API in Pharo. > On 4 Apr 2018, at 13:20, Alistair Grant <akgrant0710(a)gmail.com> wrote: > > On 4 April 2018 at 12:56, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> Playing a bit devil's advocate, the idea is that, in general, >> >> [ stream atEnd] whileFalse: [ stream next. "..." ]. >> >> is no longer allowed ? > > It hasn't been allowed "forever" [1]. It's just been misused for > almost as long. > > [1] Time began when stdio stream support was introduced. :-) I am still not convinced. Another way to put it would be that the old #atEnd or #upToEnd do not make sense for these streams and some new loop is needed, based on a new test (it exists for socket streams already). [ stream isDataAvailable ] whileTrue: [ stream next ] >> And you want to replace it with >> >> [ stream next ifNil: [ false ] ifNotNil: [ :x | "..." true ] whileTrue. >> >> That is a pretty big change, no ? > > That's the way quite a bit of code already operates. > > As Denis pointed out, it's obviously problematic in the general sense, > since nil can be embedded in non-byte oriented streams. I suspect > that in practice not many people write code that reads streams from > both byte oriented and non-byte oriented streams. Maybe yes, maybe no. As Denis' example shows there is a clear definition problem. And I do use streams of byte arrays or strings all the time, this is really important. I want my parsers to work on all kinds of streams. >> I think/feel like a proper EOF exception would be better, more correct. >> >> [ [ stream next. "..." true ] on: EOF do: [ false ] ] whileTrue. > > I agree, but the email thread Nicolas pointed to raises some > performance questions about this approach. It should be > straightforward to do a basic performance comparison which I'll get > around to if other objections aren't raised. Reading in bigger blocks, using #readInto:startingAt:count: (which is basically Unix's (2) Read sys call), would solve performance problems, I think. >> Will we throw away #atEnd then ? Do we need it if we cannot use it ? > > Unix file i/o returns EOF if the end of file has been reach OR if an > error occurs. You should still check #atEnd after reading past the > end of the file to make sure no error occurred. Another part of the > primitive change I'm proposing is to return additional information > about what went wrong in the event of an error. I am sorry, but this kind of semantics (the OR) is way too complex at the general image level, it is too specific and based on certain underlying implementation details. Sven > We could modify the read primitive so that it fails if an error has > occurred, and then #atEnd wouldn't be required. > > Cheers, > Alistair > > > >>> On 4 Apr 2018, at 12:41, Alistair Grant <akgrant0710(a)gmail.com> wrote: >>> >>> Hi Nicolas, >>> >>> On 4 April 2018 at 12:36, Nicolas Cellier >>> <nicolas.cellier.aka.nice(a)gmail.com> wrote: >>>> >>>> >>>> 2018-04-04 12:18 GMT+02:00 Alistair Grant <akgrant0710(a)gmail.com>: >>>>> >>>>> Hi Sven, >>>>> >>>>> On Wed, Apr 04, 2018 at 11:32:02AM +0200, Sven Van Caekenberghe wrote: >>>>>> Somehow, somewhere there was a change to the implementation of the >>>>>> primitive called by some streams' #atEnd. >>>>> >>>>> That's a proposed change by me, but it hasn't been integrated yet. So >>>>> the discussion below should apply to the current stable vm (from August >>>>> last year). >>>>> >>>>> >>>>>> IIRC, someone said it is implemented as 'remaining size being zero' >>>>>> and some virtual unix files like /dev/random are zero sized. >>>>> >>>>> Currently, for files other than sdio (stdout, stderr, stdin) it is >>>>> effectively defined as: >>>>> >>>>> atEnd := stream position >= stream size >>>>> >>>>> >>>>> And, as you say, plenty of virtual unix files report size 0. >>>>> >>>>> >>>>> >>>>>> Now, all kinds of changes are being done image size to work around this. >>>>> >>>>> I would phrase this slightly differently :-) >>>>> >>>>> Some code does the right thing, while other code doesn't. E.g.: >>>>> >>>>> MultiByteFileStream>>upToEnd is good, while >>>>> FileStream>>contents is incorrect >>>>> >>>>> >>>>>> I am a strong believer in simple, real (i.e. infinite) streams, but I >>>>>> am not sure we are doing the right thing here. >>>>>> >>>>>> Point is, I am not sure #next returning nil is official and universal. >>>>>> >>>>>> Consider the comments: >>>>>> >>>>>> Stream>>#next >>>>>> "Answer the next object accessible by the receiver." >>>>>> >>>>>> ReadStream>>#next >>>>>> "Primitive. Answer the next object in the Stream represented by the >>>>>> receiver. Fail if the collection of this stream is not an Array or a >>>>>> String. >>>>>> Fail if the stream is positioned at its end, or if the position is out >>>>>> of >>>>>> bounds in the collection. Optional. See Object documentation >>>>>> whatIsAPrimitive." >>>>>> >>>>>> Note how there is no talk about returning nil ! >>>>>> >>>>>> I think we should discuss about this first. >>>>>> >>>>>> Was the low level change really correct and the right thing to do ? >>>>> >>>>> The primitive change proposed doesn't affect this discussion. It will >>>>> mean that #atEnd returns false (correctly) sometimes, while currently it >>>>> returns true (incorrectly). The end result is still incorrect, e.g. >>>>> #contents returns an empty string for /proc/cpuinfo. >>>>> >>>>> You're correct about no mention of nil, but we have: >>>>> >>>>> FileStream>>next >>>>> >>>>> (position >= readLimit and: [self atEnd]) >>>>> ifTrue: [^nil] >>>>> ifFalse: [^collection at: (position := position + 1)] >>>>> >>>>> >>>>> which has been around for a long time (I suspect, before Pharo existed). >>>>> >>>>> Having said that, I think that raising an exception is a better >>>>> solution, but it is a much, much bigger change than the one I proposed >>>>> in https://github.com/pharo-project/pharo/pull/1180. >>>>> >>>>> >>>>> Cheers, >>>>> Alistair >>>>> >>>> >>>> Hi, >>>> yes, if you are after universal behavior englobing Unix streams, the >>>> Exception might be the best way. >>>> Because on special stream you can't allways say in advance, you have to try. >>>> That's the solution adopted by authors of Xtreams. >>>> But there is a runtime penalty associated to it. >>>> >>>> The penalty once was so high that my proposal to generalize EndOfStream >>>> usage was rejected a few years ago by AndreaRaab. >>>> http://forum.world.st/EndOfStream-unused-td68806.html >>> >>> Thanks for this, I'll definitely take a look. >>> >>> Do you have a sense of how Denis' suggestion of using an EndOfStream >>> object would compare? >>> >>> It would keep the same coding style, but avoid the problems with nil. >>> >>> Thanks, >>> Alistair >>> >>> >>> >>>> I have regularly benched Xtreams, but stopped a few years ago. >>>> Maybe i can excavate and pass on newer VM. >>>> >>>> In the mean time, i had experimented a programmable end of stream behavior >>>> (via a block, or any other valuable) >>>> http://www.squeaksource.com/XTream.htm >>>> so as to reconcile performance and universality, but it was a source of >>>> complexification at implementation side. >>>> >>>> Nicolas >>>> >>>>> >>>>> >>>>>> Note also that a Guille introduced something new, #closed which is >>>>>> related to the difference between having no more elements (maybe right now, >>>>>> like an open network stream) and never ever being able to produce more data. >>>>>> >>>>>> Sven >> >> >

April 4, 2018

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Alistair Grant

On 4 April 2018 at 12:56, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > Playing a bit devil's advocate, the idea is that, in general, > > [ stream atEnd] whileFalse: [ stream next. "..." ]. > > is no longer allowed ? It hasn't been allowed "forever" [1]. It's just been misused for almost as long. [1] Time began when stdio stream support was introduced. :-) > And you want to replace it with > > [ stream next ifNil: [ false ] ifNotNil: [ :x | "..." true ] whileTrue. > > That is a pretty big change, no ? That's the way quite a bit of code already operates. As Denis pointed out, it's obviously problematic in the general sense, since nil can be embedded in non-byte oriented streams. I suspect that in practice not many people write code that reads streams from both byte oriented and non-byte oriented streams. > I think/feel like a proper EOF exception would be better, more correct. > > [ [ stream next. "..." true ] on: EOF do: [ false ] ] whileTrue. I agree, but the email thread Nicolas pointed to raises some performance questions about this approach. It should be straightforward to do a basic performance comparison which I'll get around to if other objections aren't raised. > Will we throw away #atEnd then ? Do we need it if we cannot use it ? Unix file i/o returns EOF if the end of file has been reach OR if an error occurs. You should still check #atEnd after reading past the end of the file to make sure no error occurred. Another part of the primitive change I'm proposing is to return additional information about what went wrong in the event of an error. We could modify the read primitive so that it fails if an error has occurred, and then #atEnd wouldn't be required. Cheers, Alistair >> On 4 Apr 2018, at 12:41, Alistair Grant <akgrant0710(a)gmail.com> wrote: >> >> Hi Nicolas, >> >> On 4 April 2018 at 12:36, Nicolas Cellier >> <nicolas.cellier.aka.nice(a)gmail.com> wrote: >>> >>> >>> 2018-04-04 12:18 GMT+02:00 Alistair Grant <akgrant0710(a)gmail.com>: >>>> >>>> Hi Sven, >>>> >>>> On Wed, Apr 04, 2018 at 11:32:02AM +0200, Sven Van Caekenberghe wrote: >>>>> Somehow, somewhere there was a change to the implementation of the >>>>> primitive called by some streams' #atEnd. >>>> >>>> That's a proposed change by me, but it hasn't been integrated yet. So >>>> the discussion below should apply to the current stable vm (from August >>>> last year). >>>> >>>> >>>>> IIRC, someone said it is implemented as 'remaining size being zero' >>>>> and some virtual unix files like /dev/random are zero sized. >>>> >>>> Currently, for files other than sdio (stdout, stderr, stdin) it is >>>> effectively defined as: >>>> >>>> atEnd := stream position >= stream size >>>> >>>> >>>> And, as you say, plenty of virtual unix files report size 0. >>>> >>>> >>>> >>>>> Now, all kinds of changes are being done image size to work around this. >>>> >>>> I would phrase this slightly differently :-) >>>> >>>> Some code does the right thing, while other code doesn't. E.g.: >>>> >>>> MultiByteFileStream>>upToEnd is good, while >>>> FileStream>>contents is incorrect >>>> >>>> >>>>> I am a strong believer in simple, real (i.e. infinite) streams, but I >>>>> am not sure we are doing the right thing here. >>>>> >>>>> Point is, I am not sure #next returning nil is official and universal. >>>>> >>>>> Consider the comments: >>>>> >>>>> Stream>>#next >>>>> "Answer the next object accessible by the receiver." >>>>> >>>>> ReadStream>>#next >>>>> "Primitive. Answer the next object in the Stream represented by the >>>>> receiver. Fail if the collection of this stream is not an Array or a >>>>> String. >>>>> Fail if the stream is positioned at its end, or if the position is out >>>>> of >>>>> bounds in the collection. Optional. See Object documentation >>>>> whatIsAPrimitive." >>>>> >>>>> Note how there is no talk about returning nil ! >>>>> >>>>> I think we should discuss about this first. >>>>> >>>>> Was the low level change really correct and the right thing to do ? >>>> >>>> The primitive change proposed doesn't affect this discussion. It will >>>> mean that #atEnd returns false (correctly) sometimes, while currently it >>>> returns true (incorrectly). The end result is still incorrect, e.g. >>>> #contents returns an empty string for /proc/cpuinfo. >>>> >>>> You're correct about no mention of nil, but we have: >>>> >>>> FileStream>>next >>>> >>>> (position >= readLimit and: [self atEnd]) >>>> ifTrue: [^nil] >>>> ifFalse: [^collection at: (position := position + 1)] >>>> >>>> >>>> which has been around for a long time (I suspect, before Pharo existed). >>>> >>>> Having said that, I think that raising an exception is a better >>>> solution, but it is a much, much bigger change than the one I proposed >>>> in https://github.com/pharo-project/pharo/pull/1180. >>>> >>>> >>>> Cheers, >>>> Alistair >>>> >>> >>> Hi, >>> yes, if you are after universal behavior englobing Unix streams, the >>> Exception might be the best way. >>> Because on special stream you can't allways say in advance, you have to try. >>> That's the solution adopted by authors of Xtreams. >>> But there is a runtime penalty associated to it. >>> >>> The penalty once was so high that my proposal to generalize EndOfStream >>> usage was rejected a few years ago by AndreaRaab. >>> http://forum.world.st/EndOfStream-unused-td68806.html >> >> Thanks for this, I'll definitely take a look. >> >> Do you have a sense of how Denis' suggestion of using an EndOfStream >> object would compare? >> >> It would keep the same coding style, but avoid the problems with nil. >> >> Thanks, >> Alistair >> >> >> >>> I have regularly benched Xtreams, but stopped a few years ago. >>> Maybe i can excavate and pass on newer VM. >>> >>> In the mean time, i had experimented a programmable end of stream behavior >>> (via a block, or any other valuable) >>> http://www.squeaksource.com/XTream.htm >>> so as to reconcile performance and universality, but it was a source of >>> complexification at implementation side. >>> >>> Nicolas >>> >>>> >>>> >>>>> Note also that a Guille introduced something new, #closed which is >>>>> related to the difference between having no more elements (maybe right now, >>>>> like an open network stream) and never ever being able to produce more data. >>>>> >>>>> Sven > >

April 4, 2018

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

by Sven Van Caekenberghe

Playing a bit devil's advocate, the idea is that, in general, [ stream atEnd] whileFalse: [ stream next. "..." ]. is no longer allowed ? And you want to replace it with [ stream next ifNil: [ false ] ifNotNil: [ :x | "..." true ] whileTrue. That is a pretty big change, no ? I think/feel like a proper EOF exception would be better, more correct. [ [ stream next. "..." true ] on: EOF do: [ false ] ] whileTrue. Will we throw away #atEnd then ? Do we need it if we cannot use it ? > On 4 Apr 2018, at 12:41, Alistair Grant <akgrant0710(a)gmail.com> wrote: > > Hi Nicolas, > > On 4 April 2018 at 12:36, Nicolas Cellier > <nicolas.cellier.aka.nice(a)gmail.com> wrote: >> >> >> 2018-04-04 12:18 GMT+02:00 Alistair Grant <akgrant0710(a)gmail.com>: >>> >>> Hi Sven, >>> >>> On Wed, Apr 04, 2018 at 11:32:02AM +0200, Sven Van Caekenberghe wrote: >>>> Somehow, somewhere there was a change to the implementation of the >>>> primitive called by some streams' #atEnd. >>> >>> That's a proposed change by me, but it hasn't been integrated yet. So >>> the discussion below should apply to the current stable vm (from August >>> last year). >>> >>> >>>> IIRC, someone said it is implemented as 'remaining size being zero' >>>> and some virtual unix files like /dev/random are zero sized. >>> >>> Currently, for files other than sdio (stdout, stderr, stdin) it is >>> effectively defined as: >>> >>> atEnd := stream position >= stream size >>> >>> >>> And, as you say, plenty of virtual unix files report size 0. >>> >>> >>> >>>> Now, all kinds of changes are being done image size to work around this. >>> >>> I would phrase this slightly differently :-) >>> >>> Some code does the right thing, while other code doesn't. E.g.: >>> >>> MultiByteFileStream>>upToEnd is good, while >>> FileStream>>contents is incorrect >>> >>> >>>> I am a strong believer in simple, real (i.e. infinite) streams, but I >>>> am not sure we are doing the right thing here. >>>> >>>> Point is, I am not sure #next returning nil is official and universal. >>>> >>>> Consider the comments: >>>> >>>> Stream>>#next >>>> "Answer the next object accessible by the receiver." >>>> >>>> ReadStream>>#next >>>> "Primitive. Answer the next object in the Stream represented by the >>>> receiver. Fail if the collection of this stream is not an Array or a >>>> String. >>>> Fail if the stream is positioned at its end, or if the position is out >>>> of >>>> bounds in the collection. Optional. See Object documentation >>>> whatIsAPrimitive." >>>> >>>> Note how there is no talk about returning nil ! >>>> >>>> I think we should discuss about this first. >>>> >>>> Was the low level change really correct and the right thing to do ? >>> >>> The primitive change proposed doesn't affect this discussion. It will >>> mean that #atEnd returns false (correctly) sometimes, while currently it >>> returns true (incorrectly). The end result is still incorrect, e.g. >>> #contents returns an empty string for /proc/cpuinfo. >>> >>> You're correct about no mention of nil, but we have: >>> >>> FileStream>>next >>> >>> (position >= readLimit and: [self atEnd]) >>> ifTrue: [^nil] >>> ifFalse: [^collection at: (position := position + 1)] >>> >>> >>> which has been around for a long time (I suspect, before Pharo existed). >>> >>> Having said that, I think that raising an exception is a better >>> solution, but it is a much, much bigger change than the one I proposed >>> in https://github.com/pharo-project/pharo/pull/1180. >>> >>> >>> Cheers, >>> Alistair >>> >> >> Hi, >> yes, if you are after universal behavior englobing Unix streams, the >> Exception might be the best way. >> Because on special stream you can't allways say in advance, you have to try. >> That's the solution adopted by authors of Xtreams. >> But there is a runtime penalty associated to it. >> >> The penalty once was so high that my proposal to generalize EndOfStream >> usage was rejected a few years ago by AndreaRaab. >> http://forum.world.st/EndOfStream-unused-td68806.html > > Thanks for this, I'll definitely take a look. > > Do you have a sense of how Denis' suggestion of using an EndOfStream > object would compare? > > It would keep the same coding style, but avoid the problems with nil. > > Thanks, > Alistair > > > >> I have regularly benched Xtreams, but stopped a few years ago. >> Maybe i can excavate and pass on newer VM. >> >> In the mean time, i had experimented a programmable end of stream behavior >> (via a block, or any other valuable) >> http://www.squeaksource.com/XTream.htm >> so as to reconcile performance and universality, but it was a source of >> complexification at implementation side. >> >> Nicolas >> >>> >>> >>>> Note also that a Guille introduced something new, #closed which is >>>> related to the difference between having no more elements (maybe right now, >>>> like an open network stream) and never ever being able to produce more data. >>>> >>>> Sven

April 4, 2018