[Pharo-project] Xtreams : first embryonary port on Squeak
So, I started a quick port of VW Xtreams to Squeak this evening. Only the Xtreams-Core and the trivial Xtreams-Terminals (no file/pipe/socket/pointer). You'll find code at http://www.squeaksource.com/XTream. My previous experimental Xtream project has been renamed SqueaXTream to reduce confusion, but still sits in the same project. Please find my report below. cheers Nicolas --------------------- REPORT --------------------- THE CONTENTS: This first port is made of 3 packages - Xtreams-Core - Xtreams-Terminals - Xtreams-VWCompatible Plus 2 tests - Xtreams-CoreTests - Xtreams-TerminalsTests Please notice I removed one hyphen from test package names for MC compatibility. THE PROCEDURE: I didn't use FileOut30 not any more advanced tool, but just a simple copy/paste strategy. This gave me a chance to do a fast review of code. For porting the future evolutions of VW Xtreams, more advanced tools will be necessary, the manual approach does not scale. THE MODIFICATIONS: The changes I made are quite restricted, which indicates that Xtreams is not that hard to port. Main changes are due to lack of namespace in Squeak: - I added a XT prefix to every Xtreams classes (no Namespace in Squeak) - I added XTPools, a SharedPool for holding the single Xtreams namespace variable DefaultBufferSize - I transformed class SharedVariables into class variables and added proper class side initialization for these in XTWriteStream For the rest, I tried to not change Xtreams contents. but rather did implement VW compatible messages when possible in the Xtreams-VWCompatible package when they did not exist in Squeak. These message are a full rewrite and are also released under MIT license. The few exceptions are: - I modified originator -> self originator in XTIncomplete. - I changed the senders of #newInFixedSpace: to rather send an #error: 'not implemented in Squeak' in XTRecyclingBuffer - I replaced #waitIfCurtailedSignal/#signal with a #critical: section in XTRecyclingBuffer - I replaced Core.Timer after:do: with self after:do: in Xtreams-CoreTests A DISCUSSION OF TECHNICAL DETAILS: I first attempted to replace #growToAtLeast: with #grownBy: because Squeak become: is notoriously inefficient. But obviously, this did not work, the tests rely on destination preserving its identity. This clearly is going to degrade micro benchmarks when the destination collection capacity is not well adjusted in advance. I think we should discuss this particular point. Is identity preservation absolutely required, or just convenient ? THE STATUS OF TESTS: Tests do not all pass. There seems to be a bug in Squeak #replace:from:to:with: when the replacement is the collection itself, moved to the right (this is with a COG VM). THE FUTURE: I will probably continue porting with reduced activity, so I invite any interested person to help. Right now, the project is world writeable, code commited here falling under MIT license.
On Sun, 10 Oct 2010, Nicolas Cellier wrote:
So, I started a quick port of VW Xtreams to Squeak this evening. Only the Xtreams-Core and the trivial Xtreams-Terminals (no file/pipe/socket/pointer). You'll find code at http://www.squeaksource.com/XTream. My previous experimental Xtream project has been renamed SqueaXTream to reduce confusion, but still sits in the same project. Please find my report below.
Great. :)
cheers
Nicolas
--------------------- REPORT ---------------------
THE CONTENTS:
This first port is made of 3 packages - Xtreams-Core - Xtreams-Terminals - Xtreams-VWCompatible Plus 2 tests - Xtreams-CoreTests - Xtreams-TerminalsTests Please notice I removed one hyphen from test package names for MC compatibility.
THE PROCEDURE:
I didn't use FileOut30 not any more advanced tool, but just a simple copy/paste strategy. This gave me a chance to do a fast review of code. For porting the future evolutions of VW Xtreams, more advanced tools will be necessary, the manual approach does not scale.
If you can create a diff from the VW changes, then porting "by hand" shouldn't be hard.
THE MODIFICATIONS:
The changes I made are quite restricted, which indicates that Xtreams is not that hard to port. Main changes are due to lack of namespace in Squeak: - I added a XT prefix to every Xtreams classes (no Namespace in Squeak) - I added XTPools, a SharedPool for holding the single Xtreams namespace variable DefaultBufferSize - I transformed class SharedVariables into class variables and added proper class side initialization for these in XTWriteStream
For the rest, I tried to not change Xtreams contents. but rather did implement VW compatible messages when possible in the Xtreams-VWCompatible package when they did not exist in Squeak. These message are a full rewrite and are also released under MIT license. The few exceptions are: - I modified originator -> self originator in XTIncomplete. - I changed the senders of #newInFixedSpace: to rather send an #error: 'not implemented in Squeak' in XTRecyclingBuffer - I replaced #waitIfCurtailedSignal/#signal with a #critical: section in XTRecyclingBuffer - I replaced Core.Timer after:do: with self after:do: in Xtreams-CoreTests
A DISCUSSION OF TECHNICAL DETAILS:
I first attempted to replace #growToAtLeast: with #grownBy: because Squeak become: is notoriously inefficient. But obviously, this did not work, the tests rely on destination preserving its identity. This clearly is going to degrade micro benchmarks when the destination collection capacity is not well adjusted in advance. I think we should discuss this particular point. Is identity preservation absolutely required, or just convenient ?
I've never seen a real world example where this identity preservation was useful, but it causes trouble when literals aren't immutable. IMHO the #become: send is just convenient with VW's stream implementation and it has no advantage over Squeak's approach.
THE STATUS OF TESTS:
Tests do not all pass. There seems to be a bug in Squeak #replace:from:to:with: when the replacement is the collection itself, moved to the right (this is with a COG VM).
#replace:from:to:with: is not supposed to work for moves. IIRC the primitive version uses memcpy (or strncpy), so moving to the left works just because the undelying C function supports it. I used to (ab)use this feature though. Levente
THE FUTURE:
I will probably continue porting with reduced activity, so I invite any interested person to help. Right now, the project is world writeable, code commited here falling under MIT license.
2010/10/10 Levente Uzonyi <leves@elte.hu>:
On Sun, 10 Oct 2010, Nicolas Cellier wrote:
THE STATUS OF TESTS:
Tests do not all pass. There seems to be a bug in Squeak #replace:from:to:with: when the replacement is the collection itself, moved to the right (this is with a COG VM).
#replace:from:to:with: is not supposed to work for moves. IIRC the primitive version uses memcpy (or strncpy), so moving to the left works just because the undelying C function supports it. I used to (ab)use this feature though.
Levente
OK, since we use memcpy rather than memmove in Squeak VM, I implemented the trivial work around - move through a motionBuffer temporary copy. I now have a single test failing, XTCollectionWriteStream>>#testReadWriteLargeAmount and this must be due to my #after:do: simplistic implementation. Messing with #fork in SUnit TestCase is tricky, and I need help on this point. Nicolas
Nicolas, #fork in a test? Do you mean like the code below? The thread running the test waits on something signaled toward the end of the forked thread. If you want a watchdog timer, you could fork another thread that waits on a delay, sets an error flag (best to use a shared queue or a mutex to avoid weirdness) and then signals the test semaphore. You might also want to use #ifCurtailed: to set an error flag and wrap that with #ensure: to signal the semaphore. You don't necessarily need to be perfect about signaling the semaphore because failure to do so is not much different from your test going into an infinite loop (actually more benign because the stack isn't growing); you'll realize it, break, and figure out what didn't happen or at least "that it didn't" and try again forewarned. Does that help? Bill testLocalServerAndClient "11-09 - try a server and a client in the same image." | server port client serverSocket serverSocketStream text thread done read | port := 4900. done := Semaphore new. "11-09 - gag the prompter below" serverSocket := nil. server := ConnectionQueue portNumber:port queueLength:5. [ client := SocketStream openConnectionToHostNamed:'localhost' port:port. [ serverSocket isNil ] whileTrue:[ serverSocket := server getConnectionOrNil. ]. serverSocketStream := SocketStream on:serverSocket. text := 'Long live Tux!'. serverSocketStream nextPutAll:text; flush. "11-09 - this hung up w/o the thread. 11-09 - as you were; the problem was a lack of #flush on the send side, so there was nothing for the client side to read." thread := [ read := client nextMany:text size. done signal. ] forkAt:Processor userBackgroundPriority. thread name:'READ'. done wait. self should:[ read = text ]. Transcript nextPutAll:'Just to make the point, we read: '; nextPutAll:read; cr; flush. ] ensure:[ server destroy. serverSocketStream close. client close. ]. "11-09 - no prompts" serverSocket := serverSocket. client := client. serverSocketStream := serverSocketStream. thread := thread. " NetworkSmokeTestCase prod:#testLocalServerAndClient. " ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Nicolas Cellier [nicolas.cellier.aka.nice@gmail.com] Sent: Sunday, October 10, 2010 7:05 AM To: The general-purpose Squeak developers list Cc: Pharo Development Subject: Re: [Pharo-project] [squeak-dev] Xtreams : first embryonary port on Squeak 2010/10/10 Levente Uzonyi <leves@elte.hu>:
On Sun, 10 Oct 2010, Nicolas Cellier wrote:
THE STATUS OF TESTS:
Tests do not all pass. There seems to be a bug in Squeak #replace:from:to:with: when the replacement is the collection itself, moved to the right (this is with a COG VM).
#replace:from:to:with: is not supposed to work for moves. IIRC the primitive version uses memcpy (or strncpy), so moving to the left works just because the undelying C function supports it. I used to (ab)use this feature though.
Levente
OK, since we use memcpy rather than memmove in Squeak VM, I implemented the trivial work around - move through a motionBuffer temporary copy. I now have a single test failing, XTCollectionWriteStream>>#testReadWriteLargeAmount and this must be due to my #after:do: simplistic implementation. Messing with #fork in SUnit TestCase is tricky, and I need help on this point. Nicolas _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
FWIW, SIF might help you: http://www.pocketsmalltalk.com/sif/ I have put a lot of code through it in the past couple of years. ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Nicolas Cellier [nicolas.cellier.aka.nice@gmail.com] Sent: Saturday, October 09, 2010 6:33 PM To: The general-purpose Squeak developers list; Pharo Development; vwnc@cs.uiuc.edu Subject: [Pharo-project] Xtreams : first embryonary port on Squeak So, I started a quick port of VW Xtreams to Squeak this evening. Only the Xtreams-Core and the trivial Xtreams-Terminals (no file/pipe/socket/pointer). You'll find code at http://www.squeaksource.com/XTream. My previous experimental Xtream project has been renamed SqueaXTream to reduce confusion, but still sits in the same project. Please find my report below. cheers Nicolas --------------------- REPORT --------------------- THE CONTENTS: This first port is made of 3 packages - Xtreams-Core - Xtreams-Terminals - Xtreams-VWCompatible Plus 2 tests - Xtreams-CoreTests - Xtreams-TerminalsTests Please notice I removed one hyphen from test package names for MC compatibility. THE PROCEDURE: I didn't use FileOut30 not any more advanced tool, but just a simple copy/paste strategy. This gave me a chance to do a fast review of code. For porting the future evolutions of VW Xtreams, more advanced tools will be necessary, the manual approach does not scale. THE MODIFICATIONS: The changes I made are quite restricted, which indicates that Xtreams is not that hard to port. Main changes are due to lack of namespace in Squeak: - I added a XT prefix to every Xtreams classes (no Namespace in Squeak) - I added XTPools, a SharedPool for holding the single Xtreams namespace variable DefaultBufferSize - I transformed class SharedVariables into class variables and added proper class side initialization for these in XTWriteStream For the rest, I tried to not change Xtreams contents. but rather did implement VW compatible messages when possible in the Xtreams-VWCompatible package when they did not exist in Squeak. These message are a full rewrite and are also released under MIT license. The few exceptions are: - I modified originator -> self originator in XTIncomplete. - I changed the senders of #newInFixedSpace: to rather send an #error: 'not implemented in Squeak' in XTRecyclingBuffer - I replaced #waitIfCurtailedSignal/#signal with a #critical: section in XTRecyclingBuffer - I replaced Core.Timer after:do: with self after:do: in Xtreams-CoreTests A DISCUSSION OF TECHNICAL DETAILS: I first attempted to replace #growToAtLeast: with #grownBy: because Squeak become: is notoriously inefficient. But obviously, this did not work, the tests rely on destination preserving its identity. This clearly is going to degrade micro benchmarks when the destination collection capacity is not well adjusted in advance. I think we should discuss this particular point. Is identity preservation absolutely required, or just convenient ? THE STATUS OF TESTS: Tests do not all pass. There seems to be a bug in Squeak #replace:from:to:with: when the replacement is the collection itself, moved to the right (this is with a COG VM). THE FUTURE: I will probably continue porting with reduced activity, so I invite any interested person to help. Right now, the project is world writeable, code commited here falling under MIT license. _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
This still seems confusing. Could we just have a new SqueakSource project called "Xtreams" (small T + an S)? It's not like projects are expensive to create... Julian On Sat, Oct 9, 2010 at 11:33 PM, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
So, I started a quick port of VW Xtreams to Squeak this evening. Only the Xtreams-Core and the trivial Xtreams-Terminals (no file/pipe/socket/pointer). You'll find code at http://www.squeaksource.com/XTream. My previous experimental Xtream project has been renamed SqueaXTream to reduce confusion, but still sits in the same project. Please find my report below.
participants (4)
-
Julian Fitzell -
Levente Uzonyi -
Nicolas Cellier -
Schwab,Wilhelm K