- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] MQTT for Pharo

by Sven Van Caekenberghe

> On 21 Dec 2016, at 15:47, Ben Coman <btc(a)openinworld.com> wrote: > > You might like to collaborate with Tim and Craig. > http://forum.world.st/Graphing-weather-data-td4927701.html > cheers -ben Hmm, that code seems to be quite incomplete (from what I can see publicly): not all packet types and their IO is fully implemented, there are no unit tests, there is no client, and none of the more complex quality of service levels 0, 1 & 2 logic/flow, and there are no functional unit tests against public sandbox/test servers. > On Wed, Dec 21, 2016 at 10:17 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> >>> On 21 Dec 2016, at 15:11, phil(a)highoctane.be wrote: >>> >>> FWIW get yourself an account at http://enco.io >> >> OK, I'll put it on my list of things to look at. >> >>> The devportal has mqtt endpoints so that we can integrate with that and possibly have some market place solutions offered and running with Pharo. >>> >>> I have a small EnCoClient class in Pharo that works. Now making the API support more extensive. >>> >>> Docs: http://docs.enco.io/docs/device-sensor-apis >>> >>> Phil >>> >>> On Wed, Dec 21, 2016 at 2:59 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>> >>>> On 21 Dec 2016, at 12:50, phil(a)highoctane.be wrote: >>>> >>>> Sven, >>>> >>>> I am currently working with IoT and MQTT is necessary. At this point I use mosquitto as the broker and your Pharo client is nice to have and I will test it and give you feedback. >>> >>> That is good to here, Phil. >>> >>> Yes, such real world testing is much needed. This is actually quite fun stuff to experiment with. Let me know how it goes. >>> >>> Sven >>> >>>> I've got a working microPython codebase so can compare what is in there (this for the LoPy device): >>>> >>>> Thx for the module! >>>> >>>> Phil >>>> >>>> On Wed, Dec 21, 2016 at 12:15 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>>> Hi, >>>> >>>> MQTT is a light-weight publish/subscribe messaging protocol, originally created around 1998. It is now an official open industry ISO standard. It is perfect for large-scale Internet of Things applications and high performance mobile messaging. >>>> >>>> The publish/subscribe messaging pattern requires a message broker. The broker is responsible for distributing messages to interested clients based on the topic of a message. Parties communicating with each other over MQTT would all be clients in different roles, like producers and consumers, using the broker as middleware. >>>> >>>> Many client libraries for different programming languages and multiple brokers/servers are available. Facebook Messenger and Amazon AWS IOT are two users, among many others. >>>> >>>> >>>> A good client library for Pharo was not yet available. I started a new MQTT project and I am looking for collaborators to help me finish it. The official specification is quite readable and there is a lot of information available (see the References/Links section at the end). >>>> >>>> >>>> Right now, a first version of the following features is available: >>>> >>>> - reading & writing of all 14 binary packet types >>>> >>>> - an experimental client with support for connection open/close, ping, subscribe/unsubscribe, QoS levels 0 (at most once), 1 (at least once) and 2 (exactly once) for application (publish) messages in both directions, message/package IDs and keep alive (heartbeat) >>>> >>>> - unit tests, for packet reading/writing and for clients against 3 publicly available sandbox/test brokers >>>> >>>> Basically, the code works but needs polishing and maturing. Also, it might be useful to experiment with alternative client API design. Not all features are yet implemented. It would also be nice to implement an actual server/broker, not to replace production quality servers, but as a proof of concept and tools during development. >>>> >>>> >>>> Code >>>> >>>> http://smalltalkhub.com/#!/~SvenVanCaekenberghe/MQTT/ >>>> >>>> Right now, documentation is limited, but there are class comments and the most important public API methods are commented too. There is no Metacello configuration yet, just load the 3 packages. >>>> >>>> >>>> References/Links >>>> >>>> - http://mqtt.org >>>> - https://en.wikipedia.org/wiki/MQTT >>>> - http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html >>>> - https://github.com/mqtt/mqtt.github.io/wiki/software >>>> - http://mosquitto.org >>>> - https://github.com/emqtt/emqttd >>>> - http://kamilfb.github.io/mqtt-spy/ >>>> - https://github.com/eclipse/paho.mqtt-spy/wiki >>>> - https://eclipse.org/paho/ >>>> - https://eclipse.org/paho/clients/c/embedded/ >>>> - http://www.rabbitmq.com/mqtt.html >>>> - https://vernemq.com >>>> - https://www.ibm.com/developerworks/community/blogs/c565c720-fe84-4f63-873f-… >>>> >>>> >>>> Sandbox/test Servers/Brokers >>>> >>>> iot.eclipse.org:1883 >>>> test.mosquitto.org:1883 >>>> broker.mqtt-dashboard.com:1883 >>>> >>>> >>>> Sven >>>> >>>> -- >>>> Sven Van Caekenberghe >>>> http://stfx.eu >>>> Smalltalk is the Red Pill >>>> >>>> >>>> >>> >>> >>> >> >> >

Dec. 21, 2016

Re: [Pharo-dev] MQTT for Pharo

by Sven Van Caekenberghe

> On 21 Dec 2016, at 16:04, Henrik Johansen <henrik.s.johansen(a)veloxit.no> wrote: > >> >> On 21 Dec 2016, at 14:57 , Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> >> Hendrik, >> >> Thank you for this detailed feedback. >> >>> On 21 Dec 2016, at 12:44, Henrik Johansen <henrik.s.johansen(a)veloxit.no> wrote: >>> >>> Hi Sven! >>> One thing I noticed when testing the RabbitMQ client with keepalive > 0, was connection being closed and all subscriptions lost when receiving large payloads, due to timeout before the keepalive packet could be sent before waiting to receive next object. >> >> Indeed, I used my Stamp (STOMP) Rabbit MQ client as a model for the MQTT one - there is a lot of similarity, especially in concept. >> >> Keep alive processing is not that easy. I tried to do it by using read timeouts as a source of regular opportunities to check for the need to process keep alive logic. But of course, if you have no outstanding read (in a loop), that won't work. >> >> The fact that receiving a large payload would trigger an actual keep alive time out is not something that I have seen myself. It seems weird that the reading/transferring of incoming data would not count as activity against keep alive, no ? > > I never had a chance to investigate fully, but I distinctly remember having the same reaction! > It's quite awhile ago now, so my memory might be hazy, take the following with an appropriate amount of grains of salt. > > The first times I encountered it, it seemed quite random, occuring after extended periods of client inactivity after receiving only small payloads... > Setting a much shorter keepalive timeout than the default was/is very useful in reproducing/verifying if it is an issue. > The timeouts then occurred relatively shortly after I'd received a single payload with no other activity, and disappeared once I removed the resetting of lastActivity timestamp on reads, indicating that for at least rabbitmq (3.5 was the version at the time, I believe), receiving data was *not* being counted as keep-alive activity. > > The issue of receiving large payloads blocking writing in time was still unresolved, consistently cut off in the middle of (its own!) multi-MB payload transfers due to keepalive packet not being sent. > I couldn't see a solution other than abandoning the elegant single-threaded approach and do keepalive as a separate high-priority process, but the architectural choice for the app changed at this point to not include a MQ in first delivery, so the more involved rewrite for doing so before deploying in production, kinda got stranded :/ Instinctively it feels like messaging and multi-MB payloads would not be a good fit, at least I would suspect that they are an edge case. But maybe I am wrong. I think that the code in StampMedium>>#readBodyBytes: and StampMedium>>#readBodyString: could be refactored to use a loop with chunk buffers and at the same time check the elapsed time to fire back keep alive pings if necessary. A ping too much would not hurt, better safe than sorry. But is is hard to catch any/all network slowdowns, any IO operation could hang/timeout. But all this would assume that the problematic situation can be recreated. > Cheers, > Henry

Dec. 21, 2016

Re: [Pharo-dev] [Vm-dev] Problem with OSSubprocess / signals / heartbeat ?

by Guille Polito

Ok, so following a bit on this. I'll summarize some of our findings, some of them maybe obvious for some people in this list. I saw that there are actually two different kind of VMs for *nix [1] : - threaded heartbeat - itimer + signal heartbeat Specially, I'd like to cite the following paragraph for the lazy: A distinction on linux is between VMs with an itimer hearbeat or a threaded heartbeat. VMs with an itimer hearbeat use setitimer to deliver a SIGALRM signal at regular intervals to interrupt the VM to check for events. These signals can be troublesome, interrupting foreign code that cannot cope with such signals. VMs with a threaded heartbeat use a high-priority thread that loops, blocking on nanosleep and then interrupting the VM, performing the same function as the itimer heartbeat but without using signals. These VMs are to be preferred but suport for multiple thread priorities in user-level processes has only been available on linux in kernels later than 2.6.12. So, I downloaded the heartbeat squeak VM from bintray [2]. This VM requires so deploy some configuration files in /etc/security [2]. Under this configuration OSSubprocess worked like a charm (or I did not found the issue again so far). Now, this heartbeat threaded VM is the recommended in the README file, and we see that OSSubprocess generates the exact issue stated. The main problem remains for the moment since Pharo's default download includes not this VM but the itimer one. I talked with Esteban about it and he was aware of these two VM flavours, and the reason why we are using the itimer one is the need to deploy those permission files in /etc, which makes installation a bit less automatic. Cheers, Guille [1] https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/e17db79411cfec767e04… [2] https://bintray.com/opensmalltalk/vm/download_file?file_path=cog_linux32x86… [3] https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/tag/r3732#linux -------- Original Message -------- > > >> I asked Santa an OSSubprocess that won't hang :) >> > > > I really appreciate to see such not so small polar elfves helping santa :) > > > > Stef > > > > > > > >> Thanks, >> >> Guille >> >> >> >> >> >> -------- Original Message -------- >> >>> >>> >>>> On 19 Dec 2016, at 14:41, Mariano Martinez Peck >>>> <marianopeck(a)gmail.com> wrote: >>>> >>>> >>>> >>>> Hi guys, >>>> >>>> >>>> >>>> Guille Polito kept one of these images if someone can give us a >>>> hand. He also proposed the great idea of using `strace` to see what >>>> was going on. He (together with Pable Tesone) suspected that the >>>> heartbeat could be interrupting the `clone()` function which is (I >>>> think) called internally by the 'posix_spawn()' which is the one >>>> used by OSSubprocess. >>>> >>>> >>>> >>>> When these images are "hung" they found at a infinitive loop like this: >>>> >>> Okay, how many child processes do you have at that point? How many >>> processes does the system have? >>> >>> >>> >>> >>> >>>> [pid 17477] --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} --- >>>> >>>> [pid 17477] gettimeofday({1482152630, 593498}, NULL) = 0 >>>> >>>> [pid 17477] sigreturn() (mask []) = 120 >>>> >>>> [pid 17477] clone(child_stack=0, >>>> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, >>>> child_tidptr=0xf7578768) = ? ERESTARTNOINTR (To be restarted) >>>> >>>> [pid 17477] --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} --- >>>> >>>> [pid 17477] gettimeofday({1482152630, 600126}, NULL) = 0 >>>> >>>> [pid 17477] sigreturn() (mask []) = 120 >>>> >>>> [pid 17477] clone(child_stack=0, >>>> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, >>>> child_tidptr=0xf7578768) = ? ERESTARTNOINTR (To be restarted) >>>> >>> so above.. 7ms between the two gettimeofday calls? Nothing else? Set >>> a breakpoint on clone/fork in gdb and look at the c-stack at this >>> point? Could you strace with timestamps to see how much time is >>> spent? Is the process suspicious in other ways? >>> >>> >>> >>> So yes.. sounds like clone doesn't complete.. the question is why? >>> Is it out of resources? Is something in the VM blocking longer than >>> the heartbeat, is the heartbeat more frequent than expected? >>> >>> >>> >>> >>> >>>> As you can see, there is a SIGALARM involved. It also looks like >>>> the `gettimeofday` is used by the heartbeat ? Could it be that >>>> someone the heartbeat is interrupting the `clone()` ? >>>> >>>> >>>> >>>> Guille also showed me the `strace` output with a regular / working >>>> image: >>>> >>>> >>>> >>>> >>>> >>>> [pid 18647] --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} --- >>>> >>>> [pid 18647] gettimeofday({1482152829, 481014}, NULL) = 0 >>>> >>>> [pid 18647] sigreturn() (mask []) = -1 EINTR (Interrupted >>>> system call) >>>> >>>> [pid 18647] getitimer(ITIMER_REAL, {it_interval={0, 2000}, >>>> it_value={0, 1917}}) = 0 >>>> >>>> [pid 18647] recvmsg(3, 0xff7b0734, 0) = -1 EAGAIN (Resource >>>> temporarily unavailable) >>>> >>>> [pid 18647] select(4, [3], [], [3], {0, 1000}) = 0 (Timeout) >>>> >>>> [pid 18647] getitimer(ITIMER_REAL, {it_interval={0, 2000}, >>>> it_value={0, 797}}) = 0 >>>> >>>> [pid 18647] recvmsg(3, 0xff7b0734, 0) = -1 EAGAIN (Resource >>>> temporarily unavailable) >>>> >>>> [pid 18647] select(4, [3], [], [3], {0, 1000}) = ? ERESTARTNOHAND >>>> (To be restarted if no handler) >>>> >>>> [pid 18647] --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} --- >>>> >>>> >>>> >>>> >>>> >>>> Does anyone have any hint here? >>>> >>> Get timestamps in there. How long does it take to fail/end in this >>> situation? >>> >>> >>> >>> holger >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> >> >> >> >> >> > > > > > >

Dec. 21, 2016

Re: [Pharo-dev] MQTT for Pharo

by Henrik Johansen

> On 21 Dec 2016, at 14:57 , Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > Hendrik, > > Thank you for this detailed feedback. > >> On 21 Dec 2016, at 12:44, Henrik Johansen <henrik.s.johansen(a)veloxit.no> wrote: >> >> Hi Sven! >> One thing I noticed when testing the RabbitMQ client with keepalive > 0, was connection being closed and all subscriptions lost when receiving large payloads, due to timeout before the keepalive packet could be sent before waiting to receive next object. > > Indeed, I used my Stamp (STOMP) Rabbit MQ client as a model for the MQTT one - there is a lot of similarity, especially in concept. > > Keep alive processing is not that easy. I tried to do it by using read timeouts as a source of regular opportunities to check for the need to process keep alive logic. But of course, if you have no outstanding read (in a loop), that won't work. > > The fact that receiving a large payload would trigger an actual keep alive time out is not something that I have seen myself. It seems weird that the reading/transferring of incoming data would not count as activity against keep alive, no ? I never had a chance to investigate fully, but I distinctly remember having the same reaction! It's quite awhile ago now, so my memory might be hazy, take the following with an appropriate amount of grains of salt. The first times I encountered it, it seemed quite random, occuring after extended periods of client inactivity after receiving only small payloads... Setting a much shorter keepalive timeout than the default was/is very useful in reproducing/verifying if it is an issue. The timeouts then occurred relatively shortly after I'd received a single payload with no other activity, and disappeared once I removed the resetting of lastActivity timestamp on reads, indicating that for at least rabbitmq (3.5 was the version at the time, I believe), receiving data was *not* being counted as keep-alive activity. The issue of receiving large payloads blocking writing in time was still unresolved, consistently cut off in the middle of (its own!) multi-MB payload transfers due to keepalive packet not being sent. I couldn't see a solution other than abandoning the elegant single-threaded approach and do keepalive as a separate high-priority process, but the architectural choice for the app changed at this point to not include a MQ in first delivery, so the more involved rewrite for doing so before deploying in production, kinda got stranded :/ Cheers, Henry

Dec. 21, 2016

Re: [Pharo-dev] MQTT for Pharo

by Ben Coman

You might like to collaborate with Tim and Craig. http://forum.world.st/Graphing-weather-data-td4927701.html cheers -ben On Wed, Dec 21, 2016 at 10:17 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > >> On 21 Dec 2016, at 15:11, phil(a)highoctane.be wrote: >> >> FWIW get yourself an account at http://enco.io > > OK, I'll put it on my list of things to look at. > >> The devportal has mqtt endpoints so that we can integrate with that and possibly have some market place solutions offered and running with Pharo. >> >> I have a small EnCoClient class in Pharo that works. Now making the API support more extensive. >> >> Docs: http://docs.enco.io/docs/device-sensor-apis >> >> Phil >> >> On Wed, Dec 21, 2016 at 2:59 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> >> > On 21 Dec 2016, at 12:50, phil(a)highoctane.be wrote: >> > >> > Sven, >> > >> > I am currently working with IoT and MQTT is necessary. At this point I use mosquitto as the broker and your Pharo client is nice to have and I will test it and give you feedback. >> >> That is good to here, Phil. >> >> Yes, such real world testing is much needed. This is actually quite fun stuff to experiment with. Let me know how it goes. >> >> Sven >> >> > I've got a working microPython codebase so can compare what is in there (this for the LoPy device): >> > >> > Thx for the module! >> > >> > Phil >> > >> > On Wed, Dec 21, 2016 at 12:15 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> > Hi, >> > >> > MQTT is a light-weight publish/subscribe messaging protocol, originally created around 1998. It is now an official open industry ISO standard. It is perfect for large-scale Internet of Things applications and high performance mobile messaging. >> > >> > The publish/subscribe messaging pattern requires a message broker. The broker is responsible for distributing messages to interested clients based on the topic of a message. Parties communicating with each other over MQTT would all be clients in different roles, like producers and consumers, using the broker as middleware. >> > >> > Many client libraries for different programming languages and multiple brokers/servers are available. Facebook Messenger and Amazon AWS IOT are two users, among many others. >> > >> > >> > A good client library for Pharo was not yet available. I started a new MQTT project and I am looking for collaborators to help me finish it. The official specification is quite readable and there is a lot of information available (see the References/Links section at the end). >> > >> > >> > Right now, a first version of the following features is available: >> > >> > - reading & writing of all 14 binary packet types >> > >> > - an experimental client with support for connection open/close, ping, subscribe/unsubscribe, QoS levels 0 (at most once), 1 (at least once) and 2 (exactly once) for application (publish) messages in both directions, message/package IDs and keep alive (heartbeat) >> > >> > - unit tests, for packet reading/writing and for clients against 3 publicly available sandbox/test brokers >> > >> > Basically, the code works but needs polishing and maturing. Also, it might be useful to experiment with alternative client API design. Not all features are yet implemented. It would also be nice to implement an actual server/broker, not to replace production quality servers, but as a proof of concept and tools during development. >> > >> > >> > Code >> > >> > http://smalltalkhub.com/#!/~SvenVanCaekenberghe/MQTT/ >> > >> > Right now, documentation is limited, but there are class comments and the most important public API methods are commented too. There is no Metacello configuration yet, just load the 3 packages. >> > >> > >> > References/Links >> > >> > - http://mqtt.org >> > - https://en.wikipedia.org/wiki/MQTT >> > - http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html >> > - https://github.com/mqtt/mqtt.github.io/wiki/software >> > - http://mosquitto.org >> > - https://github.com/emqtt/emqttd >> > - http://kamilfb.github.io/mqtt-spy/ >> > - https://github.com/eclipse/paho.mqtt-spy/wiki >> > - https://eclipse.org/paho/ >> > - https://eclipse.org/paho/clients/c/embedded/ >> > - http://www.rabbitmq.com/mqtt.html >> > - https://vernemq.com >> > - https://www.ibm.com/developerworks/community/blogs/c565c720-fe84-4f63-873f-… >> > >> > >> > Sandbox/test Servers/Brokers >> > >> > iot.eclipse.org:1883 >> > test.mosquitto.org:1883 >> > broker.mqtt-dashboard.com:1883 >> > >> > >> > Sven >> > >> > -- >> > Sven Van Caekenberghe >> > http://stfx.eu >> > Smalltalk is the Red Pill >> > >> > >> > >> >> >> > >

Dec. 21, 2016

Re: [Pharo-dev] MQTT for Pharo

by Sven Van Caekenberghe

> On 21 Dec 2016, at 15:11, phil(a)highoctane.be wrote: > > FWIW get yourself an account at http://enco.io OK, I'll put it on my list of things to look at. > The devportal has mqtt endpoints so that we can integrate with that and possibly have some market place solutions offered and running with Pharo. > > I have a small EnCoClient class in Pharo that works. Now making the API support more extensive. > > Docs: http://docs.enco.io/docs/device-sensor-apis > > Phil > > On Wed, Dec 21, 2016 at 2:59 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > On 21 Dec 2016, at 12:50, phil(a)highoctane.be wrote: > > > > Sven, > > > > I am currently working with IoT and MQTT is necessary. At this point I use mosquitto as the broker and your Pharo client is nice to have and I will test it and give you feedback. > > That is good to here, Phil. > > Yes, such real world testing is much needed. This is actually quite fun stuff to experiment with. Let me know how it goes. > > Sven > > > I've got a working microPython codebase so can compare what is in there (this for the LoPy device): > > > > Thx for the module! > > > > Phil > > > > On Wed, Dec 21, 2016 at 12:15 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > Hi, > > > > MQTT is a light-weight publish/subscribe messaging protocol, originally created around 1998. It is now an official open industry ISO standard. It is perfect for large-scale Internet of Things applications and high performance mobile messaging. > > > > The publish/subscribe messaging pattern requires a message broker. The broker is responsible for distributing messages to interested clients based on the topic of a message. Parties communicating with each other over MQTT would all be clients in different roles, like producers and consumers, using the broker as middleware. > > > > Many client libraries for different programming languages and multiple brokers/servers are available. Facebook Messenger and Amazon AWS IOT are two users, among many others. > > > > > > A good client library for Pharo was not yet available. I started a new MQTT project and I am looking for collaborators to help me finish it. The official specification is quite readable and there is a lot of information available (see the References/Links section at the end). > > > > > > Right now, a first version of the following features is available: > > > > - reading & writing of all 14 binary packet types > > > > - an experimental client with support for connection open/close, ping, subscribe/unsubscribe, QoS levels 0 (at most once), 1 (at least once) and 2 (exactly once) for application (publish) messages in both directions, message/package IDs and keep alive (heartbeat) > > > > - unit tests, for packet reading/writing and for clients against 3 publicly available sandbox/test brokers > > > > Basically, the code works but needs polishing and maturing. Also, it might be useful to experiment with alternative client API design. Not all features are yet implemented. It would also be nice to implement an actual server/broker, not to replace production quality servers, but as a proof of concept and tools during development. > > > > > > Code > > > > http://smalltalkhub.com/#!/~SvenVanCaekenberghe/MQTT/ > > > > Right now, documentation is limited, but there are class comments and the most important public API methods are commented too. There is no Metacello configuration yet, just load the 3 packages. > > > > > > References/Links > > > > - http://mqtt.org > > - https://en.wikipedia.org/wiki/MQTT > > - http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html > > - https://github.com/mqtt/mqtt.github.io/wiki/software > > - http://mosquitto.org > > - https://github.com/emqtt/emqttd > > - http://kamilfb.github.io/mqtt-spy/ > > - https://github.com/eclipse/paho.mqtt-spy/wiki > > - https://eclipse.org/paho/ > > - https://eclipse.org/paho/clients/c/embedded/ > > - http://www.rabbitmq.com/mqtt.html > > - https://vernemq.com > > - https://www.ibm.com/developerworks/community/blogs/c565c720-fe84-4f63-873f-… > > > > > > Sandbox/test Servers/Brokers > > > > iot.eclipse.org:1883 > > test.mosquitto.org:1883 > > broker.mqtt-dashboard.com:1883 > > > > > > Sven > > > > -- > > Sven Van Caekenberghe > > http://stfx.eu > > Smalltalk is the Red Pill > > > > > > > > >

Dec. 21, 2016

Re: [Pharo-dev] MQTT for Pharo

by phil＠highoctane.be

FWIW get yourself an account at http://enco.io The devportal has mqtt endpoints so that we can integrate with that and possibly have some market place solutions offered and running with Pharo. I have a small EnCoClient class in Pharo that works. Now making the API support more extensive. Docs: http://docs.enco.io/docs/device-sensor-apis Phil On Wed, Dec 21, 2016 at 2:59 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > On 21 Dec 2016, at 12:50, phil(a)highoctane.be wrote: > > > > Sven, > > > > I am currently working with IoT and MQTT is necessary. At this point I > use mosquitto as the broker and your Pharo client is nice to have and I > will test it and give you feedback. > > That is good to here, Phil. > > Yes, such real world testing is much needed. This is actually quite fun > stuff to experiment with. Let me know how it goes. > > Sven > > > I've got a working microPython codebase so can compare what is in there > (this for the LoPy device): > > > > Thx for the module! > > > > Phil > > > > On Wed, Dec 21, 2016 at 12:15 PM, Sven Van Caekenberghe <sven(a)stfx.eu> > wrote: > > Hi, > > > > MQTT is a light-weight publish/subscribe messaging protocol, originally > created around 1998. It is now an official open industry ISO standard. It > is perfect for large-scale Internet of Things applications and high > performance mobile messaging. > > > > The publish/subscribe messaging pattern requires a message broker. The > broker is responsible for distributing messages to interested clients based > on the topic of a message. Parties communicating with each other over MQTT > would all be clients in different roles, like producers and consumers, > using the broker as middleware. > > > > Many client libraries for different programming languages and multiple > brokers/servers are available. Facebook Messenger and Amazon AWS IOT are > two users, among many others. > > > > > > A good client library for Pharo was not yet available. I started a new > MQTT project and I am looking for collaborators to help me finish it. The > official specification is quite readable and there is a lot of information > available (see the References/Links section at the end). > > > > > > Right now, a first version of the following features is available: > > > > - reading & writing of all 14 binary packet types > > > > - an experimental client with support for connection open/close, ping, > subscribe/unsubscribe, QoS levels 0 (at most once), 1 (at least once) and 2 > (exactly once) for application (publish) messages in both directions, > message/package IDs and keep alive (heartbeat) > > > > - unit tests, for packet reading/writing and for clients against 3 > publicly available sandbox/test brokers > > > > Basically, the code works but needs polishing and maturing. Also, it > might be useful to experiment with alternative client API design. Not all > features are yet implemented. It would also be nice to implement an actual > server/broker, not to replace production quality servers, but as a proof of > concept and tools during development. > > > > > > Code > > > > http://smalltalkhub.com/#!/~SvenVanCaekenberghe/MQTT/ > > > > Right now, documentation is limited, but there are class comments and > the most important public API methods are commented too. There is no > Metacello configuration yet, just load the 3 packages. > > > > > > References/Links > > > > - http://mqtt.org > > - https://en.wikipedia.org/wiki/MQTT > > - http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html > > - https://github.com/mqtt/mqtt.github.io/wiki/software > > - http://mosquitto.org > > - https://github.com/emqtt/emqttd > > - http://kamilfb.github.io/mqtt-spy/ > > - https://github.com/eclipse/paho.mqtt-spy/wiki > > - https://eclipse.org/paho/ > > - https://eclipse.org/paho/clients/c/embedded/ > > - http://www.rabbitmq.com/mqtt.html > > - https://vernemq.com > > - https://www.ibm.com/developerworks/community/ > blogs/c565c720-fe84-4f63-873f-607d87787327/entry/mqtt_security > > > > > > Sandbox/test Servers/Brokers > > > > iot.eclipse.org:1883 > > test.mosquitto.org:1883 > > broker.mqtt-dashboard.com:1883 > > > > > > Sven > > > > -- > > Sven Van Caekenberghe > > http://stfx.eu > > Smalltalk is the Red Pill > > > > > > > > >

Dec. 21, 2016

Re: [Pharo-dev] MQTT for Pharo

by Sven Van Caekenberghe

> On 21 Dec 2016, at 12:50, phil(a)highoctane.be wrote: > > Sven, > > I am currently working with IoT and MQTT is necessary. At this point I use mosquitto as the broker and your Pharo client is nice to have and I will test it and give you feedback. That is good to here, Phil. Yes, such real world testing is much needed. This is actually quite fun stuff to experiment with. Let me know how it goes. Sven > I've got a working microPython codebase so can compare what is in there (this for the LoPy device): > > Thx for the module! > > Phil > > On Wed, Dec 21, 2016 at 12:15 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > Hi, > > MQTT is a light-weight publish/subscribe messaging protocol, originally created around 1998. It is now an official open industry ISO standard. It is perfect for large-scale Internet of Things applications and high performance mobile messaging. > > The publish/subscribe messaging pattern requires a message broker. The broker is responsible for distributing messages to interested clients based on the topic of a message. Parties communicating with each other over MQTT would all be clients in different roles, like producers and consumers, using the broker as middleware. > > Many client libraries for different programming languages and multiple brokers/servers are available. Facebook Messenger and Amazon AWS IOT are two users, among many others. > > > A good client library for Pharo was not yet available. I started a new MQTT project and I am looking for collaborators to help me finish it. The official specification is quite readable and there is a lot of information available (see the References/Links section at the end). > > > Right now, a first version of the following features is available: > > - reading & writing of all 14 binary packet types > > - an experimental client with support for connection open/close, ping, subscribe/unsubscribe, QoS levels 0 (at most once), 1 (at least once) and 2 (exactly once) for application (publish) messages in both directions, message/package IDs and keep alive (heartbeat) > > - unit tests, for packet reading/writing and for clients against 3 publicly available sandbox/test brokers > > Basically, the code works but needs polishing and maturing. Also, it might be useful to experiment with alternative client API design. Not all features are yet implemented. It would also be nice to implement an actual server/broker, not to replace production quality servers, but as a proof of concept and tools during development. > > > Code > > http://smalltalkhub.com/#!/~SvenVanCaekenberghe/MQTT/ > > Right now, documentation is limited, but there are class comments and the most important public API methods are commented too. There is no Metacello configuration yet, just load the 3 packages. > > > References/Links > > - http://mqtt.org > - https://en.wikipedia.org/wiki/MQTT > - http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html > - https://github.com/mqtt/mqtt.github.io/wiki/software > - http://mosquitto.org > - https://github.com/emqtt/emqttd > - http://kamilfb.github.io/mqtt-spy/ > - https://github.com/eclipse/paho.mqtt-spy/wiki > - https://eclipse.org/paho/ > - https://eclipse.org/paho/clients/c/embedded/ > - http://www.rabbitmq.com/mqtt.html > - https://vernemq.com > - https://www.ibm.com/developerworks/community/blogs/c565c720-fe84-4f63-873f-… > > > Sandbox/test Servers/Brokers > > iot.eclipse.org:1883 > test.mosquitto.org:1883 > broker.mqtt-dashboard.com:1883 > > > Sven > > -- > Sven Van Caekenberghe > http://stfx.eu > Smalltalk is the Red Pill > > >

Dec. 21, 2016

Re: [Pharo-dev] MQTT for Pharo

by Sven Van Caekenberghe

Hendrik, Thank you for this detailed feedback. > On 21 Dec 2016, at 12:44, Henrik Johansen <henrik.s.johansen(a)veloxit.no> wrote: > > Hi Sven! > One thing I noticed when testing the RabbitMQ client with keepalive > 0, was connection being closed and all subscriptions lost when receiving large payloads, due to timeout before the keepalive packet could be sent before waiting to receive next object. Indeed, I used my Stamp (STOMP) Rabbit MQ client as a model for the MQTT one - there is a lot of similarity, especially in concept. Keep alive processing is not that easy. I tried to do it by using read timeouts as a source of regular opportunities to check for the need to process keep alive logic. But of course, if you have no outstanding read (in a loop), that won't work. The fact that receiving a large payload would trigger an actual keep alive time out is not something that I have seen myself. It seems weird that the reading/transferring of incoming data would not count as activity against keep alive, no ? > I see the default in ExperimentalClient is to disregard ConnectionTimedOut when that happens, and resend a ping. I use a similar technique, yes, > Are MQTT servers generally more lenient wrt dropping subscriptions once the timeout occurs, or will that lead to no new data being received from the broker? I don't know. I think this needs to be sorted out during actual usage. Note that MQTT has to the concept of resuming an existing session that keeps subscriptions and outstanding undelivered data. I have not explored that area. > (Also, doesn't there need to be a guarantee that timeout < keepalive when keepalive is > 0, for this logic to work as intended?) Yes, that is something that should be enforced. BTW, the MQTT spec says: << If the Keep Alive value is non-zero and the Server does not receive a Control Packet from the Client within one and a half times the Keep Alive time period, it MUST disconnect the Network Connection to the Client as if the network had failed [MQTT-3.1.2-24]. >> So I guess there is some leniency. Sven > Cheers, > Henry > >> On 21 Dec 2016, at 12:15 , Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> >> Hi, >> >> MQTT is a light-weight publish/subscribe messaging protocol, originally created around 1998. It is now an official open industry ISO standard. It is perfect for large-scale Internet of Things applications and high performance mobile messaging. >> >> The publish/subscribe messaging pattern requires a message broker. The broker is responsible for distributing messages to interested clients based on the topic of a message. Parties communicating with each other over MQTT would all be clients in different roles, like producers and consumers, using the broker as middleware. >> >> Many client libraries for different programming languages and multiple brokers/servers are available. Facebook Messenger and Amazon AWS IOT are two users, among many others. >> >> >> A good client library for Pharo was not yet available. I started a new MQTT project and I am looking for collaborators to help me finish it. The official specification is quite readable and there is a lot of information available (see the References/Links section at the end). >> >> >> Right now, a first version of the following features is available: >> >> - reading & writing of all 14 binary packet types >> >> - an experimental client with support for connection open/close, ping, subscribe/unsubscribe, QoS levels 0 (at most once), 1 (at least once) and 2 (exactly once) for application (publish) messages in both directions, message/package IDs and keep alive (heartbeat) >> >> - unit tests, for packet reading/writing and for clients against 3 publicly available sandbox/test brokers >> >> Basically, the code works but needs polishing and maturing. Also, it might be useful to experiment with alternative client API design. Not all features are yet implemented. It would also be nice to implement an actual server/broker, not to replace production quality servers, but as a proof of concept and tools during development. >> >> >> Code >> >> http://smalltalkhub.com/#!/~SvenVanCaekenberghe/MQTT/ >> >> Right now, documentation is limited, but there are class comments and the most important public API methods are commented too. There is no Metacello configuration yet, just load the 3 packages. >> >> >> References/Links >> >> - http://mqtt.org >> - https://en.wikipedia.org/wiki/MQTT >> - http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html >> - https://github.com/mqtt/mqtt.github.io/wiki/software >> - http://mosquitto.org >> - https://github.com/emqtt/emqttd >> - http://kamilfb.github.io/mqtt-spy/ >> - https://github.com/eclipse/paho.mqtt-spy/wiki >> - https://eclipse.org/paho/ >> - https://eclipse.org/paho/clients/c/embedded/ >> - http://www.rabbitmq.com/mqtt.html >> - https://vernemq.com >> - https://www.ibm.com/developerworks/community/blogs/c565c720-fe84-4f63-873f-… >> >> >> Sandbox/test Servers/Brokers >> >> iot.eclipse.org:1883 >> test.mosquitto.org:1883 >> broker.mqtt-dashboard.com:1883 >> >> >> Sven >> >> -- >> Sven Van Caekenberghe >> http://stfx.eu >> Smalltalk is the Red Pill >> >> > >

Dec. 21, 2016

Re: [Pharo-dev] Epicea user experience report

by Ben Coman

On Wed, Dec 21, 2016 at 3:03 PM, Tudor Girba <tudor(a)tudorgirba.com> wrote: > Hi, > > I think I would prefer to not have to think about this choice at all when in the world menu level. The user interface from the Epicea window already allows me to switch between current session and all sessions, and usually the decision of what to look at is based on whether I find what I am looking for or not. So, rather then asking the user to think about it upfront, I would prefer to have one command that opens the Epicea window on the current session and allow the user to dive deeper. This would also imply that we would have only one command in the menu item. What do you think? Good point. I had a similar thought earlier. The individual Session Changes window is almost completely embedded in the All Sessions window, so the former seems superfluous. The <Monitor> and <+> buttons would need to be added to the All Sessions window, probably above the "sesssion" pane. Two additional things I think are required to facilitate this... 1. The left-pane of the All Sessions windows needs to update when a new session is created - per <+> button. 2. It would be useful if a new session was created "when the image opened" rather than when a change is made, so that it can be preselected in the left-side pane when a fresh image is opened. But the file shouldn't be created until the first change, for the case like a web service image being invoked multiple times a second. Also one other request, That the current-session be tagged "Current" rather than just "Today". I can log issues if they all sound reasonable. cheers -ben > >> On Dec 20, 2016, at 10:16 PM, stepharong <stepharong(a)free.fr> wrote: >> >> I like the second because it associates the name with the function. >> >> Stef >> >> >> Hi all, >> >> One point of Ben's feedback is how Epicea code changes tool is presented in the World Menu. Below you can see the current state + 3 options to discuss it. >> >> >> 1. "Epicea" main entry > "Session Changes" + "All Changes" children [current state] >> >> <Mail Attachment.png> >> >> >> 2. Attach purpose to the name: "Epicea Code Changes" >> >> <Mail Attachment.png> >> >> >> 3. Just "Code Changes" + rephrase children: >> >> <Mail Attachment.png> >> >> >> 4. Flatten >> >> <Mail Attachment.png> >> >> >> >> Reminder: In World Menu > Help > Help Browser > Epicea you can find a description of the tool and it's model. >> >> Cheers. >> Martin >> >> On Mon, Oct 31, 2016 at 1:02 AM, Martin Dias <tinchodias(a)gmail.com> wrote: >> Hello Ben, >> >> About discussion points in 2 (a, b and c): I agree with your arguments... somebody that just moved from Pharo 5 to 6 and crashed an image will look for a "Recover lost changes" in the menu and can have a problem to discover it the replacement in a World->Tools->Epicea->... entry. >> >> Then, as a first step we could flatten the 2 menu entries and then at least anybody will easily find an entry related to changes in World->Tools. >> >> Second, we could try to merge both Epicea GUIs into one (suggestions are welcome). >> >> I still have to read more in detail the remaining of your report to answer. Anyway, thanks a lot for it. >> >> Cheers, >> Martin >> >> >> On Sat, Oct 29, 2016 at 5:22 AM, Ben Coman <btc(a)openinworld.com> wrote: >> 1. Created fresh Pharo image (build 60269) >> >> >> 2. Opened World > Tools > Epicea > All changes >> >> Points for discussion... >> >> a. How many submenu items are expected for Epicea? Can we push the >> current ones up so the Tools menu remains a flat menu. >> >> b. Do we need the two current menu items? "Current session" is >> encompassed by "All changes"? What expectations do people have of how >> often they'll use the former rather than the latter? >> >> c. When people move from Pharo 5 to Pharo 6 and in a panic want to >> "recover changes" for a crashed image, they'll be looking for that >> familiar feature name, not a new app name. Could the app name be left >> out or placed in brackets "Changes (Epicea)". >> >> btw, the interface looks really slick! nice work. >> >> >> 3. Opened World > System Browser. >> >> 4. Added package AAA >> All Changes window - no dynamic change. >> On <refresh>, still no change, i.e. no sessions >> #New All Changes window - not visible, no sessions. >> >> 5. Added class AA. >> All Changes window - no dynamic change. >> On <refresh>, shows new session with AAA & AA. >> >> 5. Added method... >> AA>a >> ^'something' >> Prompted for author, entered 'BenComan' >> All Changes window with session selected - dynamic update showing AA>>a. >> >> 6. Added package BBB. >> All Changes window - no dynamic update. >> On <refresh>, BBB still not visible in session. >> >> 7. Added class A to package AAA. >> All Changes window - dynamic update showing A. >> On <refresh>, BBB still not visible in session. >> >> 8. Added class BB to package BBB. >> All Changes window - dynamic update showing BBB & BB. >> >> a. Package creation event seems not handled properly, being only >> pushed through when a class is created in it. >> >> b. Since there is a dynamic update for class and method >> modifications, could the session creation also dynamically update it >> UI. >> >> ----------- >> 9. Killed the vm from command line >> $ ps -ef | grep pharo >> $ kill 29349 >> Restarted Pharo image >> >> 10. World > Tools > Epica > All changes. >> Authorship is inconsistent: >> * AAA and AA have blank author >> * AA>>a, A, BBB, B have author 'BenComan'. >> >> a. I understand this follows on from Author not being requested until >> the first method was defined. Did the old changes track the author of >> packages and classes at all? >> >> b. Since Epicea can track package and class authors, can we trigger >> the author prompt earlier for them? >> >> 11. Selected all previous changes AAA, AA, AA>>a, A, BBB, BB >> and did <Apply Changes>. >> Prompted for author. Entered 'DrWho' >> Existing All Changes window - no change >> New All Changes window - shows new session with all six changes. >> Authorship is a little inconsistent: >> * AAA and AA have author 'Unknown'. >> * AA>>a, A, BBB, B have blank author. >> >> 12. Killed the vm from command line >> $ ps -ef | grep pharo >> $ kill 30696 >> Restarted Pharo image >> >> 13. World > Tools > Epica > All changes. >> Authorship is a little inconsistent: >> First session >> * AAA and AA have blank author >> * AA>>a, A, BBB, B have author 'BenComan'. >> Second session >> * AAA and AA have blank author >> * AA>>a, A, BBB, B have author 'DrWho'. >> >> a. Epicea changes are reapplied as the current author. This seems a >> semantic change from Pharo 5 where changes were reapplied as their >> original author. Is this accidental or by design? Can we have some >> community discussion on this point (I don't remember seeing any)? >> >> cheers -ben >> >> P.S. I'll wait to see what arises out of discussion before creating any issues. >> >> >> >> >> >> >> -- >> Using Opera's mail client: http://www.opera.com/mail/ > > -- > www.tudorgirba.com > www.feenk.com > > "In a world where everything is moving ever faster, > one might have better chances to win by moving slower." > > > > >

Dec. 21, 2016