March 2013 - Pharo-dev - lists.pharo.org

Re: [Pharo-project] Some issues with UDP sockets in Pharo 1.4/Pharo 2.0

by Sven Van Caekenberghe

On 27 Mar 2013, at 10:55, Norbert Hartl <norbert(a)hartl.name> wrote: > > Am 27.03.2013 um 00:25 schrieb Sven Van Caekenberghe <sven(a)stfx.eu>: > >> >> On 26 Mar 2013, at 23:46, Norbert Hartl <norbert(a)hartl.name> wrote: >> >>> Am 26.03.2013 um 21:08 schrieb Sven Van Caekenberghe <sven(a)stfx.eu>: >>> >>>> Hi Holger, >>>> >>>> On 26 Mar 2013, at 12:12, Holger Hans Peter Freyther <holger(a)freyther.de> wrote: >>>> >>>>> Hi, >>>>> >>>>> I am porting a tooling class that I wrote on GST for my SIP and MGCP code >>>>> to Pharo. This class will create a socket and then fork a RX and TX process. >>>>> I have some issues with 'stopping' the socket. I have created an example >>>>> that shows the issue in Pharo 2.0 and Pharo 1.4 with the pharovm (and to >>>>> have more debug output with a re-compiled one). >>>>> >>>>> >>>>> | s rx | >>>>> s := Socket newUDP. >>>>> rx := [ >>>>> [s waitForData. s halt] >>>>> on: ConnectionClosed do: [ s halt] >>>>> ] fork. >>>>> "Wait for the process to hit the readSemaphore" >>>>> (Delay forSeconds: 3) wait. >>>>> s close. >>>>> >>>>> >>>>> My expectation is that I will get the ConnectionClosed signal or at least >>>>> that >>#waitForData will return once the socket is closed. This is based >>>>> on reading the comment of >>#waitForData. >>>>> >>>>> >>>>> The Socket>>#waitForData documentation reads like this: >>>>> "Wait for data to arrive. This method will block until >>>>> data is available or the socket is closed. If the socket is closed >>>>> a ConnectionClosed exception will be signaled." >>>>> >>>>> >>>>> I had a look at the socket plugin and the following is happening. First >>>>> aio is disabled, the state set to ThisEndClosed, then ::close is called >>>>> and in the above case the result is '0' and the following code will be >>>>> executed: >>>>> >>>>> else if (0 == result) >>>>> { >>>>> /* close completed synchronously */ >>>>> SOCKETSTATE(s)= Unconnected; >>>>> FPRINTF((stderr, "closeConnection: disconnected\n")); >>>>> SOCKET(s)= -1; >>>>> } >>>>> >>>>> >>>>> this means that my 'rx' process will happily wait on the readSemaphore >>>>> of the Socket and will never be woken up. >>>>> >>>>> >>>>> My question are probably: >>>>> >>>>> 1.) Is this the intended behavior for Socket>>#close and Socket>>#waitForData? >>>>> 2.) How should I terminate my RX process? >>>>> >>>>> kind regards >>>>> holger >>>>> >>>>> PS: The issue is probably socket specific and not related to UDP. >>>> >>>> Maybe I am telling you things you already know, but TCP and UDP are fundamentally different. UDP is really one-shot, try-and-hope, completely asynchronous. In networking you generally only get a ConnectionClosed when you actually read or write (and even then ;-). >>>> >>> That's not really true. It is implementation specific. IMHO you can open a UDP socket in send and receive mode where the communication is synchronous. So your socket blocks until a datagram socket is received. This has to advantage that the underlying OS registers local and remote port and automatically and accepts only datagrams for the right ports match. As it is in a lot of use cases not preferrable to have blocking sockets while using UDP you use two sockets instead, one for sending and one for receiving. For this you need to register the listening socket for listening to everything. >>> Disclaimer: I took this from the back of my head and this might by outdated/inproper information. I just wanted to say that you shouldn't mix asynchronous with unidirectional :) >>> >>> Norbert >> >> I politely disagree ;-) >> >> But I am willing to learn: so please provide some references of such UDP usage. >> >> http://en.wikipedia.org/wiki/Datagram >> > If I find the time I dig out some things from the past. I just can't follow your reasoning that the way UDP is defined means unidirectional. TCP defines a connection, UDP does not. Connections are not required to have end-to-end communication. A remote port is a service identifier and local port is a client identifier. A port pair defines a communication channel. TCP just adds roles to both ends and defines stream sequences and transmission windows. > Maybe I misunderstood you then I take everything back :) It is probably about terms and how we each interpret them. I was not trying to be very formal, maybe that was confusing. Some of the terms you are using are not my standard vocabulary. But I don't think I said 'unidirectional' in any sentence. Here is my non-formal high-level description: A datagram (UDP packet) is a (small) collection of bytes being sent from A to B. Either A or B can be client/server, or sender/receiver. To receive or to be server, you have to be bound to a port (or you listen on the default port assigned to you when sending). Datagrams know their length and sender/destination host:port - but that is mostly implementation. Semantically, datagrams can get lost or arrive out of order. A or B will never know as there are no guarantees. To add reliability, you need acknowledgements, retries, numbering, and so on. If you do all that with a stream interface, you are building your own TCP layer. As a (Smalltalk) user, that means that #sendUDPData:toHost:port: will not tell you if anything is wrong, for example Socket newUDP sendUDPData: #[ 1 2 3 ] toHost: #[ 217 11 221 200 ] port: 12345 sending 3 bytes to pharo.st:12345 will give no error at all #receiveUDPDataInto: will either return a datagram or not, but does not signal any error it will also return immediately, so you have to loop BTW, these are not necessarily disadvantages, but advantages to build specific, efficient, low-level protocols. Sven > Norbert > >>>> The Socket API has many strange methods, most of which you should not touch. You should use SocketStream. Looking in the VM code is also misleading because the 3 platforms (Mac, Win, *nix) are different. In Socket[Stream] there is lots of historic baggage. >>>> >>>> I would suggest you look at Smalltalk code that actually works (for TCP, Zinc Client and Server code would do, for UDP, ZTimestampSNTPClient for example). >>>> >>>> For your concrete question (2): just start a reading process, wrap it in a handler and clean up. For UDP this will never happen: you will keep on reading nothing. For TCP reading will eventually fail if the other end disappears. >>>> >>>> HTH, >>>> >>>> Sven >>>> >>>> -- >>>> Sven Van Caekenberghe >>>> http://stfx.eu >>>> Smalltalk is the Red Pill >>>> >>>> >>> >>> >> >> > >

March 27, 2013

[Pharo-project] [FugBugz] Simplififactions Issue tracker

by Marcus Denker

Hi, I simplified things a bit -> just one Milestone for 3.0 -> "Active" is retired, the default Active state is "Work Needed" In addtion, the filters are simplified. -> dedicted "Integration" and "Review" Filteres. They cross cut all Milestones, as we should quickly integrate 1.4 and 2.0 fixes. the "3.0 Work Needed" shows all issues tagged 3.0 that are not yet resolved. This means -> you want to review, take the review filter -> you want to code, take a "Work needed" filter.

March 27, 2013

Re: [Pharo-project] Some issues with UDP sockets in Pharo 1.4/Pharo 2.0

by Norbert Hartl

Am 27.03.2013 um 00:25 schrieb Sven Van Caekenberghe <sven(a)stfx.eu>: > > On 26 Mar 2013, at 23:46, Norbert Hartl <norbert(a)hartl.name> wrote: > >> Am 26.03.2013 um 21:08 schrieb Sven Van Caekenberghe <sven(a)stfx.eu>: >> >>> Hi Holger, >>> >>> On 26 Mar 2013, at 12:12, Holger Hans Peter Freyther <holger(a)freyther.de> wrote: >>> >>>> Hi, >>>> >>>> I am porting a tooling class that I wrote on GST for my SIP and MGCP code >>>> to Pharo. This class will create a socket and then fork a RX and TX process. >>>> I have some issues with 'stopping' the socket. I have created an example >>>> that shows the issue in Pharo 2.0 and Pharo 1.4 with the pharovm (and to >>>> have more debug output with a re-compiled one). >>>> >>>> >>>> | s rx | >>>> s := Socket newUDP. >>>> rx := [ >>>> [s waitForData. s halt] >>>> on: ConnectionClosed do: [ s halt] >>>> ] fork. >>>> "Wait for the process to hit the readSemaphore" >>>> (Delay forSeconds: 3) wait. >>>> s close. >>>> >>>> >>>> My expectation is that I will get the ConnectionClosed signal or at least >>>> that >>#waitForData will return once the socket is closed. This is based >>>> on reading the comment of >>#waitForData. >>>> >>>> >>>> The Socket>>#waitForData documentation reads like this: >>>> "Wait for data to arrive. This method will block until >>>> data is available or the socket is closed. If the socket is closed >>>> a ConnectionClosed exception will be signaled." >>>> >>>> >>>> I had a look at the socket plugin and the following is happening. First >>>> aio is disabled, the state set to ThisEndClosed, then ::close is called >>>> and in the above case the result is '0' and the following code will be >>>> executed: >>>> >>>> else if (0 == result) >>>> { >>>> /* close completed synchronously */ >>>> SOCKETSTATE(s)= Unconnected; >>>> FPRINTF((stderr, "closeConnection: disconnected\n")); >>>> SOCKET(s)= -1; >>>> } >>>> >>>> >>>> this means that my 'rx' process will happily wait on the readSemaphore >>>> of the Socket and will never be woken up. >>>> >>>> >>>> My question are probably: >>>> >>>> 1.) Is this the intended behavior for Socket>>#close and Socket>>#waitForData? >>>> 2.) How should I terminate my RX process? >>>> >>>> kind regards >>>> holger >>>> >>>> PS: The issue is probably socket specific and not related to UDP. >>> >>> Maybe I am telling you things you already know, but TCP and UDP are fundamentally different. UDP is really one-shot, try-and-hope, completely asynchronous. In networking you generally only get a ConnectionClosed when you actually read or write (and even then ;-). >>> >> That's not really true. It is implementation specific. IMHO you can open a UDP socket in send and receive mode where the communication is synchronous. So your socket blocks until a datagram socket is received. This has to advantage that the underlying OS registers local and remote port and automatically and accepts only datagrams for the right ports match. As it is in a lot of use cases not preferrable to have blocking sockets while using UDP you use two sockets instead, one for sending and one for receiving. For this you need to register the listening socket for listening to everything. >> Disclaimer: I took this from the back of my head and this might by outdated/inproper information. I just wanted to say that you shouldn't mix asynchronous with unidirectional :) >> >> Norbert > > I politely disagree ;-) > > But I am willing to learn: so please provide some references of such UDP usage. > > http://en.wikipedia.org/wiki/Datagram > If I find the time I dig out some things from the past. I just can't follow your reasoning that the way UDP is defined means unidirectional. TCP defines a connection, UDP does not. Connections are not required to have end-to-end communication. A remote port is a service identifier and local port is a client identifier. A port pair defines a communication channel. TCP just adds roles to both ends and defines stream sequences and transmission windows. Maybe I misunderstood you then I take everything back :) Norbert >>> The Socket API has many strange methods, most of which you should not touch. You should use SocketStream. Looking in the VM code is also misleading because the 3 platforms (Mac, Win, *nix) are different. In Socket[Stream] there is lots of historic baggage. >>> >>> I would suggest you look at Smalltalk code that actually works (for TCP, Zinc Client and Server code would do, for UDP, ZTimestampSNTPClient for example). >>> >>> For your concrete question (2): just start a reading process, wrap it in a handler and clean up. For UDP this will never happen: you will keep on reading nothing. For TCP reading will eventually fail if the other end disappears. >>> >>> HTH, >>> >>> Sven >>> >>> -- >>> Sven Van Caekenberghe >>> http://stfx.eu >>> Smalltalk is the Red Pill >>> >>> >> >> > >

March 27, 2013

Re: [Pharo-project] [Pharo-users] More GSoC ideas wanted, 3 days left ....

by ClÃ©ment Bera

Title: A Pharo Image running on top of VirtualBox with the help of a Linux micro kernel Level: intermediate Possible mentor: Clement Bera Possible second mentor: Marcus Denker, Camillo Bruni Description Archiving Pharo / Squeak Image is not easy : it depends on a VM, that depends on a processor and an OS. Sometimes, we find old Pharo / Squeak Image that we cannot run because we have lost the appropriate VM. This is even more relevant when the project rely on a specifically built VM. Moreover, sometimes we have also lost the Pharo / Squeak Image sources from that image and we cannot read easily the code. I propose then to save the working image with its sources, changes file and VM into a virtual box image in order to have it working on top of virtual box easily forever. This way the archived image will only depends on the computer processor, and not on VM, OS, computer processor, source file, change file Technical Details The student would have to identify a minimum linux kernel in order to run Pharo. Then, he would have to implement some Pharo code so that by entering 'Smalltalk saveAsVirtualBoxImage' in workspace it would create a virtual box image in the same folder, with the current VM, source file, change file, image and the minimum linux kernel. This command will probably go get some part of the virtual box image (for instance the linux micro kernel) or some external tool as Vagrant (http://www.vagrantup.com) somewhere on the internet. Running the virtual box image would directly run the Pharo image on the Pharo VM through a start-up process. Lastly, the student would have to set up a Jenkins job on the Pharo-contribution continuous integration server that create a Pharo virtual box image. If some time is remaining, the student would have to find a solution to manage the Pharo image crashes in virtual box. Benefits to the Student The student will learn some smalltalk (networking, commandLineHandler), how to set up a build in a Jenkins continuous integration server and some linux command line. He will also have a nice impressive open source tool to show. Benefits to the Community The community will benefit from the archiving tool. The Pharo virtual box image could also be interesting for some experiments. 2013/3/26 stephane ducasse <stephane.ducasse(a)free.fr> > > On Mar 26, 2013, at 2:36 PM, Janko MivÅ¡ek <janko.mivsek(a)eranova.si> wrote: > > > Hi Stef, > > > > Dne 26. 03. 2013 13:35, piÅ¡e stephane ducasse: > >> Janko > >> > >> Can you put andrei chis and nicolas passerini for Debugger improvment > projects. > > > > Put where? As mentors? > > Yes they are more into it. > > > > >> I know that alexandre should propose a couple of topics around roassal. > >> > >> I could come up with a couple of other topics. > >> Here is one: > > > > Added > > > > Best regards > > Janko > >> > >> > >>> Ideas so far: http://gsoc2013.esug.org/ideas > >>> > >>> To propose an idea just respond to this post by fulfilling this idea > >>> template: > >>> > >>> Title: > >> Better rewriting rule tool > >> > >> > >>> Level: (beginner, intermediate, advanced) > >> Intermediate > >> > >>> Possible mentor: (if already known) > >> Stephane Ducasse > >> > >>> Possible second mentor: (if already known) > >> Andre Hora > >>> > >>> Description > >> The refactoring browser is based on a powerful rewrite engine > which transforms AST. > >> However it is quite cumbersome to write rewrite rules. There is a > need to build a tool to help > >> programmers defining their own transformation. > >> > >>> Technical Details > >>> > >>> Benefits to the Student > >> learning AST > >> learning AST transofrmation > >> learning AST pattern matching > >> User interface definition > >> > >>> Benefits to the Community > >> A tool that is cruelly missing. > >> > >> > >>> > >>> Best regards > >>> Serge and Janko, > >>> your this year GSoC admins > >>> > >>> > >>> -- > >>> Janko MivÅ¡ek > >>> Aida/Web > >>> Smalltalk Web Application Server > >>> http://www.aidaweb.si > >>> > >> > >> > >> > > > > -- > > Janko MivÅ¡ek > > Aida/Web > > Smalltalk Web Application Server > > http://www.aidaweb.si > > > > >

March 27, 2013

Re: [Pharo-project] [Help to review/test] Crash when downloading FFI

by Luc Fabresse

Hi all, 2013/3/26 Guillermo Polito <guillermopolito(a)gmail.com> > Hi! > > I've given a shot trying to get a fix for this case. Thanks to Eliot, I > think I reached a solution, but since I had some random results at first, > I'd like if someone else can take 5 minutes and validate. > > > https://pharo.fogbugz.com/f/cases/10134/SpecialObjectsArray-should-not-chan… > I still cannot create an account there. > > To test, the following snippet does not have to crash: > > Gofer it > smalltalkhubUser: 'DBXTalk' project: 'DBXTalkDriver'; > configurationOf: 'OpenDBXDriver'; > loadStable. > it failed 3 times. two times woth a VM crash and one time with an unresponsive VM. I used: VM : 'NBCoInterpreter NativeBoost-CogPlugin-EstebanLorenzano.18 uuid: a53445f9-c0c0-4015-97a3-be7db8d9ed6b Mar 13 2013 NBCogit NativeBoost-CogPlugin-EstebanLorenzano.18 uuid: a53445f9-c0c0-4015-97a3-be7db8d9ed6b Mar 13 2013 git://gitorious.org/cogvm/blessed.git Commit: 412abef33cbed05cf1d75329e451d71c0c6aa5a7 Date: 2013-03-13 17:48:50 +0100 By: Esteban Lorenzano <estebanlm(a)gmail.com> Jenkins build #14535 ' image : #20593 Thanks Guillermo to take a look at that. Cheers, Luc > Thanks very much! > Guille >

March 27, 2013

Re: [Pharo-project] [update 3.0] #30014

by Camillo Bruni

On 2013-03-27, at 09:17, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > On 27 Mar 2013, at 08:25, Marcus Denker <marcus.denker(a)inria.fr> wrote: > >> 7465 version printing command line needs modifiers >> https://pharo.fogbugz.com/f/cases/7465 > > Cool. I was already thinking about this. > > Could we not further extend #printVersion with > - an option to print the VM version (through the image) > - an option to print the loaded versions of all installed packages why not? :) there is already the --version switch which is a bit a strange combo of everything ;) > Also, the we could not extend the #config option with a --trace to print what is interactively printed on the Transcript, and then also the final result of the load operation ? I guess we should do that :), I almost implemented that once since I hate not knowing what's going on.

March 27, 2013

Re: [Pharo-project] [update 3.0] #30014

by Sven Van Caekenberghe

On 27 Mar 2013, at 08:25, Marcus Denker <marcus.denker(a)inria.fr> wrote: > 7465 version printing command line needs modifiers > https://pharo.fogbugz.com/f/cases/7465 Cool. I was already thinking about this. Could we not further extend #printVersion with - an option to print the VM version (through the image) - an option to print the loaded versions of all installed packages ? Also, the we could not extend the #config option with a --trace to print what is interactively printed on the Transcript, and then also the final result of the load operation ? Sven -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill

March 27, 2013

Re: [Pharo-project] Some issues with UDP sockets in Pharo 1.4/Pharo 2.0

by Sven Van Caekenberghe

On 27 Mar 2013, at 08:11, Holger Hans Peter Freyther <holger(a)freyther.de> wrote: > On Tue, Mar 26, 2013 at 09:08:09PM +0100, Sven Van Caekenberghe wrote: > >> Maybe I am telling you things you already know, but TCP and UDP are fundamentally different. UDP is really one-shot, try-and-hope, completely asynchronous. In networking you generally only get a ConnectionClosed when you actually read or write (and even then ;-). > > Yes, but at least for the unix platform on close the aio/select handler is > removing the socket from the registration. So there is no way to read the > zero from the dead socket. :) > >> For your concrete question (2): just start a reading process, wrap it in a handler and clean up. For UDP this will never happen: you will keep on reading nothing. For TCP reading will eventually fail if the other end disappears. > > Do you have an example for "handler and clean up"? I resorted to code > like this for now in my 'blocking' read. > > [self dataAvailable] > whileFalse: [[self waitForDataFor: 10] on: ConnectionTimedOut do: []]. > > But I have not ported my client code so I don't know if the above is > working or not. This way I will get the ConnectionClosed exception at > least within 10 seconds. > > thank you > > holger It is still not clear to me if this is for UDP or TCP, I would be surprised of #dataAvailable or #waitForData: actually work for UDP. I gave you some pointers to examples that do work. Your code snippet looks reasonable to me, but I would set timeouts on the socket(streams) and use regular read/write operations and then loop. Sven

March 27, 2013

Re: [Pharo-project] Some issues with UDP sockets in Pharo 1.4/Pharo 2.0

by Sven Van Caekenberghe

On 27 Mar 2013, at 08:03, Holger Hans Peter Freyther <holger(a)freyther.de> wrote: > On Wed, Mar 27, 2013 at 12:25:20AM +0100, Sven Van Caekenberghe wrote: > >> I politely disagree ;-) >> >> But I am willing to learn: so please provide some references of such UDP usage. >> >> http://en.wikipedia.org/wiki/Datagram > > I am not sure which part you are disagreeing to. But you can definately use > connect on a SOCK_DGRAM. The benefit is that you can then use read/write on > filedescriptor instead of sendto/recv. An example is here[1]. > > holger > > [1] http://cgit.osmocom.org/openbsc/tree/openbsc/src/osmo-bsc_mgcp/mgcp_main.c#… OK, but I would doubt that such API usage would change the basic semantics of UDP. And I never said that you couldn't do a connect on a UDP socket. Sven

March 27, 2013

[Pharo-project] [update 3.0] #30014

by Marcus Denker

30014 ----- 10138 Config browser does not work in Pharo 3.0, requires new repo https://pharo.fogbugz.com/f/cases/10138/ 10111 DictionaryTest does not properly test subclass functionality https://pharo.fogbugz.com/f/cases/10111 7465 version printing command line needs modifiers https://pharo.fogbugz.com/f/cases/7465 Diff information: http://smalltalkhub.com/#!/~Pharo/Pharo30/diff/Tools-MarcusDenker.1069 http://smalltalkhub.com/#!/~Pharo/Pharo30/diff/System-CommandLine-MarcusDen… http://smalltalkhub.com/#!/~Pharo/Pharo30/diff/KernelTests-MarcusDenker.479 http://smalltalkhub.com/#!/~Pharo/Pharo30/diff/CollectionsTests-MarcusDenke… http://smalltalkhub.com/#!/~Pharo/Pharo30/diff/Collections-Unordered-Marcus…

March 27, 2013