Maybe i'm off track, but here my thoughts: - one big misconception (and most problems which arising from it), that people tend to use a bidirectional communication models instead of unidirectional. In bidirectional you tend to think as if you using same channel for sending and receiving data. The truth is, that it is two unidirectional channels, wired in the way so they look like single bidirectional channel. And what it means, that coupling sending and receiving activities in a single thread is wrong: - you should not wait for data only if you sent something - you should not expect that activity from other end is possible only after you _send_ something. If you treat connection as two unidirectional, loosely coupled channels, instead of single bidirectional one, you will find that in such setup you don't have many problems. You should definitely go for (2), e.g. you should decouple a transport/communication layer from your program/logic flow: if some process wants to send data and block until some response to that message arrived, you should do so. You can simply put that process on semaphore, and adding it to the list of 'awaiting answer' processes in your scheduler which runs in separate process to handle incoming data. Then you can have different strategies, what to do if a) data seems taking too long to arrive (signal timeout) b) connection lost c) etc.. ohh.. with smalltalk it is so easy to operate with those things (the fact the VM is green-threaded doesn't changes much). On 28 October 2013 23:52, Norbert Hartl <norbert@hartl.name> wrote:

Am 28.10.2013 um 23:50 schrieb Norbert Hartl <norbert@hartl.name>:

Are there any approaches to simulate coroutines in a single thread environment or approaches to deal with multiple processes within one process?

That should have been

Are there any approaches to simulate coroutines in a single thread environment or approaches to deal with multiple processes within one test?

Norbert

-- Best regards, Igor Stasenko.

Am 29.10.2013 um 02:08 schrieb David T. Lewis <lewis@mail.msen.com>:

I have some experience with this, not related to Smalltalk but maybe still relevant. I have worked with systems that were designed around networked message passing, implemented on platforms like MS-DOS with time slicing kernels, Series I, OS/2, etc. These systems were designed to operate with asynchronous message passing protocols. The asynchronous designs were intended to support primitive OS environments, including MS-DOS with software interrupts for multitasking.

These are production-critical manufacturing applications, and in the course of modernizing them I have reimplemented the network protocols in Java and moved them to new computing platforms. Along the way I have found that the asynchronous designs could always be replaced by bidirectional send-receive transactions running in the context of a thread or process. Even the systems that appeared to be most complex have turned out to work perfectly well when modeled as synchronous send-receive transactions on a bidirectional communication channel. The software is much simpler to understand and debug when structured in this manner.

My conclusion is that designing around asynchronous, loosely coupled channels sounds like a good idea, but in practice it leads to unnecessary complexity, and to systems that are difficult to debug and support.

I agree. That was the reason for me to look out for a simpler approach. At least for testing. Can you elaborate on the „bidirectional send-receive transaction“. I’m not sure I understand what happens there. I explained my case a bit more in my response to Igor. thanks, Norbert

$0.02

Dave

On Tue, Oct 29, 2013 at 01:19:17AM +0100, Igor Stasenko wrote:

...

Maybe i'm off track, but here my thoughts:

- one big misconception (and most problems which arising from it), that people tend to use a bidirectional communication models instead of unidirectional. In bidirectional you tend to think as if you using same channel for sending and receiving data. The truth is, that it is two unidirectional channels, wired in the way so they look like single bidirectional channel.

And what it means, that coupling sending and receiving activities in a single thread is wrong: - you should not wait for data only if you sent something - you should not expect that activity from other end is possible only after you _send_ something.

If you treat connection as two unidirectional, loosely coupled channels, instead of single bidirectional one, you will find that in such setup you don't have many problems.

You should definitely go for (2), e.g. you should decouple a transport/communication layer from your program/logic flow: if some process wants to send data and block until some response to that message arrived, you should do so. You can simply put that process on semaphore, and adding it to the list of 'awaiting answer' processes in your scheduler which runs in separate process to handle incoming data. Then you can have different strategies, what to do if a) data seems taking too long to arrive (signal timeout) b) connection lost c) etc..

ohh.. with smalltalk it is so easy to operate with those things (the fact the VM is green-threaded doesn't changes much).

On 28 October 2013 23:52, Norbert Hartl <norbert@hartl.name> wrote:

...

Am 28.10.2013 um 23:50 schrieb Norbert Hartl <norbert@hartl.name>:

Are there any approaches to simulate coroutines in a single thread environment or approaches to deal with multiple processes within one process?

That should have been

Are there any approaches to simulate coroutines in a single thread environment or approaches to deal with multiple processes within one test?

Norbert

-- Best regards, Igor Stasenko.

On 29 October 2013 02:08, David T. Lewis <lewis@mail.msen.com> wrote:

I have some experience with this, not related to Smalltalk but maybe still relevant. I have worked with systems that were designed around networked message passing, implemented on platforms like MS-DOS with time slicing kernels, Series I, OS/2, etc. These systems were designed to operate with asynchronous message passing protocols. The asynchronous designs were intended to support primitive OS environments, including MS-DOS with software interrupts for multitasking.

These are production-critical manufacturing applications, and in the course of modernizing them I have reimplemented the network protocols in Java and moved them to new computing platforms. Along the way I have found that the asynchronous designs could always be replaced by bidirectional send-receive transactions running in the context of a thread or process. Even the systems that appeared to be most complex have turned out to work perfectly well when modeled as synchronous send-receive transactions on a bidirectional communication channel. The software is much simpler to understand and debug when structured in this manner.

My conclusion is that designing around asynchronous, loosely coupled channels sounds like a good idea, but in practice it leads to unnecessary complexity, and to systems that are difficult to debug and support.

$0.02

Dave

Example from my life: real-time voice over IP project.

From very starting, it was designed as a bidirectional connection between two peers. Even though, that outgoing/incoming traffic was handled separately, the model was still there and treated as 'single conversation channel'.

Later we wanted to introduce many-peer conversation and found that the above bi-directional model simply does not fits.. everything falls apart. Simple example: you have 3 peers A, B , C. So, you can represent is in this way: A<->B , A<->C , B<->C, or this: A -> (B,C) B -> (A, C) C -> (A,B) The bidirectional model implies that you have symmetry in number of outgoing and incoming channels (hence <->).. and if you send data over one of them, only single node will receive it. Logically, that leads to sending same data to two different channels. So, at very beginning you nailing down how the traffic is going to flow. With unidirectional model, as illustrated when you sending data over one channel, it will be delivered to all participants, and you don't have to do it yourself - the transport layer handles it for you. The thing is, that the way how data gets to receivers is heavily depends on network topology: imagine that two peers, B, C belong to same subnet, and latency between them is miniscule comparing to latency between A and one of them. The logical choice then should be to route the signal from A to either B or C and then broadcast from middle point to the listener which is left. Not possible with bidirectional model, because A insists on having direct socket connection with both B and C. Same goes for proxies: imagine that node A is connected via proxy.. and not directly to nodes B and C.. that means that to deliver its voice traffic to both of them it should just send a single packet to proxy, and then proxy will broadcast it to listeners. Not going to happen with bidirectional model, A will still send two very same packets to same proxy.. but with different destination address. Or even, it could be, that receiving traffic from A to C goes faster via B, while sending traffic from C to A is faster to do directly.. So try to fit that into bidirectional model of communication and see where it goes. Yes, i agree it is easier to put everything into sequential bidirectional model which makes everything so simple. If it fits. But if its not, you should not even try to fit it.. My point is, that IP is inherently asynchronous and unidirectional. -- Best regards, Igor Stasenko.

Igor, it seems I did a terrible job explaining it. Comments inline! Am 29.10.2013 um 01:19 schrieb Igor Stasenko <siguctua@gmail.com>:

Maybe i'm off track, but here my thoughts:

- one big misconception (and most problems which arising from it), that people tend to use a bidirectional communication models instead of unidirectional. In bidirectional you tend to think as if you using same channel for sending and receiving data. The truth is, that it is two unidirectional channels, wired in the way so they look like single bidirectional channel.

And what it means, that coupling sending and receiving activities in a single thread is wrong: - you should not wait for data only if you sent something - you should not expect that activity from other end is possible only after you _send_ something.

If you treat connection as two unidirectional, loosely coupled channels, instead of single bidirectional one, you will find that in such setup you don't have many problems.

I don’t know what gave you the impression it could be that way but it isn’t. There is a component called NetworkService that I’m using to do sends and where I’m registering for incoming messages. I was not talking about the architecture of the application but about ways how to test those setups. For this I have a TestNetworkService that has an instVar otherService. On connection time the NetworkServices set themself via a setter in the other service and a send is done like send: someData otherService doReceive: someData This is the place where I change the execution model for the tests. What would be normally something asynchronous is now synchronous with the mentioned side effect that a send with response will look like A send1 B receive1 B send2 A receive2 A return from receive2 B return from send2 B return from receive1 A return from send1 I really like the tests to be single threaded. It makes debugging a lot easier. So I was evaluating a few possibilities. As long as all of the calls in the above chain are tail calls it should be the same of the calls are nested or not. But that would be a restriction and I’m not sure yet if it is a good one or not. The next approach I do at the moment is that e.g. a send call goes roughly like this p := [ a block for preparing send data ] s := [ a block that sends the data ] s value: p value What I do is to rearrange flow. At the moment where s would be evaluated it is instead made the sender of p and the sender of s will be the former sender of p. So everything that happens in p will return before s is executed. For my execution model this mimicks parallel behaviour. But now the receive/sends are nested and my simple model doesn’t work. I could improve it but it seems like I need to tweak a tweak of a workaround that probably wasn’t the right choice in the first place. What my experience showed me is that often it is a good way to remove a hack instead of hacking a hack. So I’m thinking about if this is the point to let go :)

You should definitely go for (2), e.g. you should decouple a transport/communication layer from your program/logic flow: if some process wants to send data and block until some response to that message arrived, you should do so. You can simply put that process on semaphore, and adding it to the list of 'awaiting answer' processes in your scheduler which runs in separate process to handle incoming data. Then you can have different strategies, what to do if a) data seems taking too long to arrive (signal timeout) b) connection lost c) etc..

ohh.. with smalltalk it is so easy to operate with those things (the fact the VM is green-threaded doesn't changes much).

As I said the layers are separate. So one idea was to keep it in multiple processes and use semaphores for orchestration. I need to think about that because I don’t have an immediate idea how this should be done. I mean the dispatcher/scheduler does the orchestration of the processes but I would need to make use of this inside a test. Need to think about that. Norbert

On 28 October 2013 23:52, Norbert Hartl <norbert@hartl.name> wrote:

Am 28.10.2013 um 23:50 schrieb Norbert Hartl <norbert@hartl.name>:

...

Are there any approaches to simulate coroutines in a single thread environment or approaches to deal with multiple processes within one process?

That should have been

Are there any approaches to simulate coroutines in a single thread environment or approaches to deal with multiple processes within one test?

Norbert

-- Best regards, Igor Stasenko.

If you're going to even think of using callbacks like they do in Javascript, strongly reconsider. Especially so since the Javascript community are abandoning callbacks in favour of promises, especially of the A+ sort. (Squeak has such Promises in the base image, and if anyone's interested I can break them out into a Pharo-consumable library.) frank On 29 October 2013 00:11, Sebastian Sastre <sebastian@flowingconcept.com> wrote:

you can do stuff like that, just not sync (which when it's about networks is a good idea)

sounds like you should design things javascript style.

I mean, like if "everything" is a callback reacting to something..

yourAPIClient sendCommand: aCommand do: aBlockHandlingResponse

my 2 cents

On Oct 28, 2013, at 8:50 PM, Norbert Hartl <norbert@hartl.name> wrote:

...

I’m working on project that deals with server to server communication. For the test suite it is necessary to simulate the two ends of the communication within one test. The problem is that in a single threaded environment you make a send call SC from server A to B but the response of B will happen before SC returned. That makes every state change the happens in A after SC bogus from a call flow perspective.

There are two approaches that have their own problems.

1) Until now I’m reordering parts of the call stack to simulate the return of a call before the actual call is sent. But now I have nested sends that make my simple approach not working anymore. I could elaborate my approach by using something like delimited continuations to make a less brittle approach to stack call reordering in order to cope with the problem. That would need some work and the stack in the debugger will look a little bit odd/confusing.

2.) The product will later have a dispatcher process that dispatches each send operation in its own process. That would solve one problem: the timing of when actions happen. But I’m not sure if it will be easy to orchestrate actions in a way that I would call controlled in order to have reliable tests.

Are there any approaches to simulate coroutines in a single thread environment or approaches to deal with multiple processes within one process?

It is hard to explain and I hope my description of the problem is understandable

thanks in advance,

Norbert