Re: [Pharo-project] XTream was Re: Re: Ocean (was Re: Less plugins, more Smalltalk code!)
Thanks nicolas Indeed we would love to have a clean covered with test composable stream hierarchy the idea behind nile was to make it compatible, integrate it and then clean but may be we should have done otherwise. It would be good to have a path to get streams cleaned, but my time is counted. Stef On Oct 7, 2010, at 9:41 PM, Nicolas Cellier wrote:
Yes, every time I must warn and apologize because I'm guilty of hijacking the name. The goals are more or less the same, because I also hijacked some ideas (though not all invented at VW): - separate read/write stream - use stream wrappers - implement main sequenceable collection API (more to be found at http://www.squeaksource.com/XTream.html wiki page)
There are also Squeak specific goals: - clean the mess by first principles (just an excerpt http://lists.squeakfoundation.org/pipermail/squeak-dev/2009-December/141786.... ) - accelerate MultiByteFileStream
At time Squeak Xtream started, there was no license for VW Xtream, so it's a full rewrite from scratch. Thus implementation differs. Mainly I did not reify the Buffer (the master piece of original VW Xtream). Due to this, I've got a BufferedReadWriteStream, which is contradictory to goal 1). It might change in future. I also do not want to depend too much on Exception handling, which current VW Buffer implementation does.
Also, I did not replace the whole protocol, but rather reused existing one. That's because further goal is to provide a replacement for Squeak Streams (could apply to Pharo too, but there's a possible overlap with Nile* See note below). However, I currently made no plan to reach this goal. I just have some toy images that I break regularly due to lack of compatibility... Either I implement all the messy protocol that flourished in Squeak, or I try to stay a bit cleaner but with uncompatibilities... Maybe a separate compatibility package with a deprecation policy might help...
Be aware that recent Squeak and Pharo acceleration thanks to buffering was first experimented in Squeak Xtream, and then ported by talents of Levente, that's already a little reward. Some big acceleration could be gained other UTF8 streams when a majority of characters are ASCII (the case of source code management and most latin characters texts), both in cog and traditional vm. My current tests (cog) are:
{ [| tmp | tmp := (MultiByteFileStream readOnlyFileNamed: (SourceFiles at: 2) name) ascii; wantsLineEndConversion: false; converter: UTF8TextConverter new. 1 to: 20000 do: [:i | tmp upTo: Character cr]. tmp close] timeToRun. [| tmp | tmp := ((StandardFileStream readOnlyFileNamed: (SourceFiles at: 2) name) readXtream binary buffered ~> UTF8Decoder) buffered. 1 to: 20000 do: [:i | tmp upTo: Character cr]. tmp close] timeToRun. } #(163 21)
So, Xtream is around 8x faster at reading first 20000 lines of change log... No surprise it's more than 99% pure ASCII.
Last, concerning the errors, it's more a matter of fixing the tests themselves... They use #squeakToUtf8 as a reference, and some tests are for other unloaded XTream packages. I started to change the test last night but did not publish all...
See also: http://lists.squeakfoundation.org/pipermail/squeak-dev/2010-September/153578... http://lists.squeakfoundation.org/pipermail/squeak-dev/2009-December/141539.... http://lists.squeakfoundation.org/pipermail/squeak-dev/2009-December/141647.... and more mails in november/december 2009
more dated material http://bugs.squeak.org/view.php?id=6755 and various attempts like LazyStream/LazyCollections...
Nicolas
* Note about Nile: Nile has a different approach: do not eliminate ReadWriteStreams, but compose with traits. Nile also has high test coverage level, which is good. Nile was designed as a Stream replacement for Squeak Stream... As such, it carries some cruft (same dilemna as Squeak Xtream). Pharo goal is to perform major clean-up. As a mean to reach this goal, it shall not encumber with backward compatibility. This means that once Nile adopted (with historical compatibility cruft), it should be cleaned up again...
2010/10/7 Stéphane Ducasse <stephane.ducasse@inria.fr>:
pay attention that there are two xtreams package frameworks from what I understand
one presented at esug and one developed by nicolas
Stef
On Oct 7, 2010, at 4:58 PM, Sven Van Caekenberghe wrote:
On 06 Oct 2010, at 08:48, Nicolas Cellier wrote:
See also hijacked http://www.squeaksource.com/XTream.html borrowing some of the ideas, but somehow less extreme. Most packages should now load in Pharo.
Hi Nicolas,
I have been looking at the ESUG 2010 slides & google code project pages of xstreams and I must say that I like this very much.
Your code on SS does indeed load in Pharo 1.1. 8 tests fail out of 37 with errors that I think are probably fixable. I am browsing it now.
Now my question is, first, do you have some write up somewhere explaining what you did and why, and second, what is the current state of your project and what are your future plans/goals ?
Thx,
Sven
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Stef, Do you have any interest in my alternate protocol idea? In short, #next is misleading even on VW. When I learned that, I pretty much gave up on a robust implementation and did the logical thing: made the alternate selectors and started writing all of my new code in terms of them. It is a lot quicker to type a few extra characters than to spend a lot of time chasing down side effects rather than original errors. By simply making new code work better, the backward compatibility problem largely evaporates. It's a shame that perfectly good selectors #next and #next: are usurped, but that's how it is unless we want to break a lot of old code. One interesting observation (not mine) is that many #next and #next: senders in the image are vulnerable to defects if they get other than expected results. Robust semantics would break some things, but call attention to lurking bugs elsewhere. I will happily make the alternate selector code available if you want it; in fact, an old version is in the in box. In the past, you expressed concerns about getting exhaustion errors when you don't want them. My answer is that #nextAvailable: exists for that purpose - it "authorizes" truncation. You can also add a #nextOrNil if you must (I might even have it for benefit of some of my older code). In Pharo, #nextOrNil is trivial: just delegate to #next. Bill ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Stéphane Ducasse [stephane.ducasse@inria.fr] Sent: Thursday, October 07, 2010 4:25 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] XTream was Re: Re: Ocean (was Re: Less plugins, more Smalltalk code!) Thanks nicolas Indeed we would love to have a clean covered with test composable stream hierarchy the idea behind nile was to make it compatible, integrate it and then clean but may be we should have done otherwise. It would be good to have a path to get streams cleaned, but my time is counted. Stef On Oct 7, 2010, at 9:41 PM, Nicolas Cellier wrote:
Yes, every time I must warn and apologize because I'm guilty of hijacking the name. The goals are more or less the same, because I also hijacked some ideas (though not all invented at VW): - separate read/write stream - use stream wrappers - implement main sequenceable collection API (more to be found at http://www.squeaksource.com/XTream.html wiki page)
There are also Squeak specific goals: - clean the mess by first principles (just an excerpt http://lists.squeakfoundation.org/pipermail/squeak-dev/2009-December/141786.... ) - accelerate MultiByteFileStream
At time Squeak Xtream started, there was no license for VW Xtream, so it's a full rewrite from scratch. Thus implementation differs. Mainly I did not reify the Buffer (the master piece of original VW Xtream). Due to this, I've got a BufferedReadWriteStream, which is contradictory to goal 1). It might change in future. I also do not want to depend too much on Exception handling, which current VW Buffer implementation does.
Also, I did not replace the whole protocol, but rather reused existing one. That's because further goal is to provide a replacement for Squeak Streams (could apply to Pharo too, but there's a possible overlap with Nile* See note below). However, I currently made no plan to reach this goal. I just have some toy images that I break regularly due to lack of compatibility... Either I implement all the messy protocol that flourished in Squeak, or I try to stay a bit cleaner but with uncompatibilities... Maybe a separate compatibility package with a deprecation policy might help...
Be aware that recent Squeak and Pharo acceleration thanks to buffering was first experimented in Squeak Xtream, and then ported by talents of Levente, that's already a little reward. Some big acceleration could be gained other UTF8 streams when a majority of characters are ASCII (the case of source code management and most latin characters texts), both in cog and traditional vm. My current tests (cog) are:
{ [| tmp | tmp := (MultiByteFileStream readOnlyFileNamed: (SourceFiles at: 2) name) ascii; wantsLineEndConversion: false; converter: UTF8TextConverter new. 1 to: 20000 do: [:i | tmp upTo: Character cr]. tmp close] timeToRun. [| tmp | tmp := ((StandardFileStream readOnlyFileNamed: (SourceFiles at: 2) name) readXtream binary buffered ~> UTF8Decoder) buffered. 1 to: 20000 do: [:i | tmp upTo: Character cr]. tmp close] timeToRun. } #(163 21)
So, Xtream is around 8x faster at reading first 20000 lines of change log... No surprise it's more than 99% pure ASCII.
Last, concerning the errors, it's more a matter of fixing the tests themselves... They use #squeakToUtf8 as a reference, and some tests are for other unloaded XTream packages. I started to change the test last night but did not publish all...
See also: http://lists.squeakfoundation.org/pipermail/squeak-dev/2010-September/153578... http://lists.squeakfoundation.org/pipermail/squeak-dev/2009-December/141539.... http://lists.squeakfoundation.org/pipermail/squeak-dev/2009-December/141647.... and more mails in november/december 2009
more dated material http://bugs.squeak.org/view.php?id=6755 and various attempts like LazyStream/LazyCollections...
Nicolas
* Note about Nile: Nile has a different approach: do not eliminate ReadWriteStreams, but compose with traits. Nile also has high test coverage level, which is good. Nile was designed as a Stream replacement for Squeak Stream... As such, it carries some cruft (same dilemna as Squeak Xtream). Pharo goal is to perform major clean-up. As a mean to reach this goal, it shall not encumber with backward compatibility. This means that once Nile adopted (with historical compatibility cruft), it should be cleaned up again...
2010/10/7 Stéphane Ducasse <stephane.ducasse@inria.fr>:
pay attention that there are two xtreams package frameworks from what I understand
one presented at esug and one developed by nicolas
Stef
On Oct 7, 2010, at 4:58 PM, Sven Van Caekenberghe wrote:
On 06 Oct 2010, at 08:48, Nicolas Cellier wrote:
See also hijacked http://www.squeaksource.com/XTream.html borrowing some of the ideas, but somehow less extreme. Most packages should now load in Pharo.
Hi Nicolas,
I have been looking at the ESUG 2010 slides & google code project pages of xstreams and I must say that I like this very much.
Your code on SS does indeed load in Pharo 1.1. 8 tests fail out of 37 with errors that I think are probably fixable. I am browsing it now.
Now my question is, first, do you have some write up somewhere explaining what you did and why, and second, what is the current state of your project and what are your future plans/goals ?
Thx,
Sven
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
2010/10/7 Schwab,Wilhelm K <bschwab@anest.ufl.edu>:
Stef,
Do you have any interest in my alternate protocol idea? Â In short, #next is misleading even on VW. Â When I learned that, I pretty much gave up on a robust implementation and did the logical thing: made the alternate selectors and started writing all of my new code in terms of them. Â It is a lot quicker to type a few extra characters than to spend a lot of time chasing down side effects rather than original errors. Â By simply making new code work better, the backward compatibility problem largely evaporates. Â It's a shame that perfectly good selectors #next and #next: are usurped, but that's how it is unless we want to break a lot of old code.
One interesting observation (not mine) is that many #next and #next: senders in the image are vulnerable to defects if they get other than expected results. Â Robust semantics would break some things, but call attention to lurking bugs elsewhere.
I will happily make the alternate selector code available if you want it; in fact, an old version is in the in box. Â In the past, you expressed concerns about getting exhaustion errors when you don't want them. Â My answer is that #nextAvailable: exists for that purpose - it "authorizes" truncation. Â You can also add a #nextOrNil if you must (I might even have it for benefit of some of my older code). Â In Pharo, #nextOrNil is trivial: just delegate to #next.
Bill
Alternatively, the Squeak Xtream solution is to set appropriate endOfStreamAction: at creation time for example. For example, you could have a different factory SequenceableCollection>>#pedanticReadXtream ^self readXtream endOfStreamAction: [self error: 'no more data in this stream']; yourself Nicolas
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Stéphane Ducasse [stephane.ducasse@inria.fr] Sent: Thursday, October 07, 2010 4:25 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] XTream was Re: Re: Ocean (was Re: Less plugins,   more Smalltalk code!)
Thanks nicolas Indeed we would love to have a clean covered with test composable stream hierarchy the idea behind nile was to make it compatible, integrate it and then clean but may be we should have done otherwise.
It would be good to have a path to get streams cleaned, but my time is counted.
Stef On Oct 7, 2010, at 9:41 PM, Nicolas Cellier wrote:
Yes, every time I must warn and apologize because I'm guilty of hijacking the name. The goals are more or less the same, because I also hijacked some ideas (though not all invented at VW): - separate read/write stream - use stream wrappers - implement main sequenceable collection API (more to be found at http://www.squeaksource.com/XTream.html wiki page)
There are also Squeak specific goals: - clean the mess by first principles (just an excerpt http://lists.squeakfoundation.org/pipermail/squeak-dev/2009-December/141786.... ) - accelerate MultiByteFileStream
At time Squeak Xtream started, there was no license for VW Xtream, so it's a full rewrite from scratch. Thus implementation differs. Mainly I did not reify the Buffer (the master piece of original VW Xtream). Due to this, I've got a BufferedReadWriteStream, which is contradictory to goal 1). It might change in future. I also do not want to depend too much on Exception handling, which current VW Buffer implementation does.
Also, I did not replace the whole protocol, but rather reused existing one. That's because further goal is to provide a replacement for Squeak Streams (could apply to Pharo too, but there's a possible overlap with Nile* See note below). However, I currently made no plan to reach this goal. I just have some toy images that I break regularly due to lack of compatibility... Either I implement all the messy protocol that flourished in Squeak, or I try to stay a bit cleaner but with uncompatibilities... Maybe a separate compatibility package with a deprecation policy might help...
Be aware that recent Squeak and Pharo acceleration thanks to buffering was first experimented in Squeak Xtream, and then ported by talents of Levente, that's already a little reward. Some big acceleration could be gained other UTF8 streams when a majority of characters are ASCII (the case of source code management and most latin characters texts), both in cog and traditional vm. My current tests (cog) are:
{ [| tmp | tmp := (MultiByteFileStream readOnlyFileNamed: (SourceFiles at: 2) name) ascii; wantsLineEndConversion: false; converter: UTF8TextConverter new. Â Â Â 1 to: 20000 do: [:i | tmp upTo: Character cr]. tmp close] timeToRun. [| tmp | tmp := ((StandardFileStream readOnlyFileNamed: (SourceFiles at: 2) name) readXtream binary buffered ~> UTF8Decoder) buffered. Â Â Â 1 to: 20000 do: [:i | tmp upTo: Character cr]. tmp close] timeToRun. } #(163 21)
So, Xtream is around 8x faster at reading first 20000 lines of change log... No surprise it's more than 99% pure ASCII.
Last, concerning the errors, it's more a matter of fixing the tests themselves... They use #squeakToUtf8 as a reference, and some tests are for other unloaded XTream packages. I started to change the test last night but did not publish all...
See also: http://lists.squeakfoundation.org/pipermail/squeak-dev/2010-September/153578... http://lists.squeakfoundation.org/pipermail/squeak-dev/2009-December/141539.... http://lists.squeakfoundation.org/pipermail/squeak-dev/2009-December/141647.... and more mails in november/december 2009
more dated material http://bugs.squeak.org/view.php?id=6755 and various attempts like LazyStream/LazyCollections...
Nicolas
* Note about Nile: Nile has a different approach: do not eliminate ReadWriteStreams, but compose with traits. Nile also has high test coverage level, which is good. Nile was designed as a Stream replacement for Squeak Stream... As such, it carries some cruft (same dilemna as Squeak Xtream). Pharo goal is to perform major clean-up. As a mean to reach this goal, it shall not encumber with backward compatibility. This means that once Nile adopted (with historical compatibility cruft), it should be cleaned up again...
2010/10/7 Stéphane Ducasse <stephane.ducasse@inria.fr>:
pay attention that there are two xtreams package frameworks from what I understand
    one presented at esug and one developed by nicolas
Stef
On Oct 7, 2010, at 4:58 PM, Sven Van Caekenberghe wrote:
On 06 Oct 2010, at 08:48, Nicolas Cellier wrote:
See also hijacked  http://www.squeaksource.com/XTream.html borrowing some of the ideas, but somehow less extreme. Most packages should now load in Pharo.
Hi Nicolas,
I have been looking at the ESUG 2010 slides & google code project pages of xstreams and I must say that I like this very much.
Your code on SS does indeed load in Pharo 1.1. 8 tests fail out of 37 with errors that I think are probably fixable. I am browsing it now.
Now my question is, first, do you have some write up somewhere explaining what you did and why, and second, what is the current state of your project and what are your future plans/goals ?
Thx,
Sven
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
participants (3)
-
Nicolas Cellier -
Schwab,Wilhelm K -
Stéphane Ducasse