Communication between different images
Hi, I'm rearchitecting a web app to perform updates only when necessary (instead of computing them all the time) on each request, I can have a global announcer and subscribers to know when to update within an image, but is there a way to have something like that but for inter-image coordination? I'd only need to communicate the id and the class name (or a similar identifier), so on other images they'll update accordingly, and if there is an update in one image, it will notify the other images. The common data is on the database, so this is just to avoid re-reading a lot of things. Is a message queue a good fit for this? Pub/Sub? What is available in Pharo that works without having to set up a lot of things? Thanks! Esteban A. Maringolo
Hi Esteban,
On 29 Jun 2021, at 04:55, Esteban Maringolo <emaringolo@gmail.com> wrote:
Hi,
I'm rearchitecting a web app to perform updates only when necessary (instead of computing them all the time) on each request, I can have a global announcer and subscribers to know when to update within an image, but is there a way to have something like that but for inter-image coordination?
I'd only need to communicate the id and the class name (or a similar identifier), so on other images they'll update accordingly, and if there is an update in one image, it will notify the other images. The common data is on the database, so this is just to avoid re-reading a lot of things.
Is a message queue a good fit for this? Pub/Sub? What is available in Pharo that works without having to set up a lot of things?
Thanks!
Esteban A. Maringolo
RabbitMQ (for which there is the STOMP client 'STAMP' https://github.com/svenvc/stamp) is one option, but it is a bit more complex. I guess Redis would work too (https://medium.com/concerning-pharo/quick-write-me-a-redis-client-5fbe4ddfb1...). More recently I have been using MQTT (with the client https://github.com/svenvc/mqtt) which is much simpler. You post a message on a topic and have one or more listeners see it. You can even arrange for the broker to keep the messages you miss (within a reasonable window) - which is very nice for all kinds of reasons (especially operational). I have a system in production that uses this mechanism to coordinate different images (and handle ingress of data) for more than a year now. Sven
Hi Sven, I thought about both RabittMQ and MQTT too. For RabittMQ I noticed you provide a docker config to get a container running quickly. What is the easy-go solution for MQTT? Regards! Esteban A. Maringolo On Tue, Jun 29, 2021 at 3:15 AM Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi Esteban,
On 29 Jun 2021, at 04:55, Esteban Maringolo <emaringolo@gmail.com> wrote:
Hi,
I'm rearchitecting a web app to perform updates only when necessary (instead of computing them all the time) on each request, I can have a global announcer and subscribers to know when to update within an image, but is there a way to have something like that but for inter-image coordination?
I'd only need to communicate the id and the class name (or a similar identifier), so on other images they'll update accordingly, and if there is an update in one image, it will notify the other images. The common data is on the database, so this is just to avoid re-reading a lot of things.
Is a message queue a good fit for this? Pub/Sub? What is available in Pharo that works without having to set up a lot of things?
Thanks!
Esteban A. Maringolo
RabbitMQ (for which there is the STOMP client 'STAMP' https://github.com/svenvc/stamp) is one option, but it is a bit more complex.
I guess Redis would work too (https://medium.com/concerning-pharo/quick-write-me-a-redis-client-5fbe4ddfb1...).
More recently I have been using MQTT (with the client https://github.com/svenvc/mqtt) which is much simpler.
You post a message on a topic and have one or more listeners see it. You can even arrange for the broker to keep the messages you miss (within a reasonable window) - which is very nice for all kinds of reasons (especially operational). I have a system in production that uses this mechanism to coordinate different images (and handle ingress of data) for more than a year now.
Sven
On 29 Jun 2021, at 15:50, Esteban Maringolo <emaringolo@gmail.com> wrote:
Hi Sven,
I thought about both RabittMQ and MQTT too.
For RabittMQ I noticed you provide a docker config to get a container running quickly.
That was done for GitHub action testing, by Santiago, you could indeed (re)use part of that.
What is the easy-go solution for MQTT?
Installing mosquitto is very simple, just apt install. See also the GitHub action. There is much to say about both, you can find lots of information on the internet on how to configure these services, getting started is quite easy.
Regards!
Esteban A. Maringolo
On Tue, Jun 29, 2021 at 3:15 AM Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi Esteban,
On 29 Jun 2021, at 04:55, Esteban Maringolo <emaringolo@gmail.com> wrote:
Hi,
I'm rearchitecting a web app to perform updates only when necessary (instead of computing them all the time) on each request, I can have a global announcer and subscribers to know when to update within an image, but is there a way to have something like that but for inter-image coordination?
I'd only need to communicate the id and the class name (or a similar identifier), so on other images they'll update accordingly, and if there is an update in one image, it will notify the other images. The common data is on the database, so this is just to avoid re-reading a lot of things.
Is a message queue a good fit for this? Pub/Sub? What is available in Pharo that works without having to set up a lot of things?
Thanks!
Esteban A. Maringolo
RabbitMQ (for which there is the STOMP client 'STAMP' https://github.com/svenvc/stamp) is one option, but it is a bit more complex.
I guess Redis would work too (https://medium.com/concerning-pharo/quick-write-me-a-redis-client-5fbe4ddfb1...).
More recently I have been using MQTT (with the client https://github.com/svenvc/mqtt) which is much simpler.
You post a message on a topic and have one or more listeners see it. You can even arrange for the broker to keep the messages you miss (within a reasonable window) - which is very nice for all kinds of reasons (especially operational). I have a system in production that uses this mechanism to coordinate different images (and handle ingress of data) for more than a year now.
Sven
Maybe ZeroMQ fits you, but unlucky is not documented and there are only a few examples. I use ir in my jupyter kernel. You can install my port to pharo64 uFFI doing: Metacello new baseline: 'JupyterTalk'; repository: 'github://jmari/JupyterTalk:master/repository'; load:'zmq' But it needs ZeroMQ previously installed on your system. It was ported from http://smalltalkhub.com/#!/~panuw/zeromq now it is at: https://github.com/dellani/zeroMQ but I'm not sure if the original repo works on newer Pharos....I had no time to contact the original author to join both repos... El mar, 29 jun 2021 a las 4:56, Esteban Maringolo (<emaringolo@gmail.com>) escribió:
Hi,
I'm rearchitecting a web app to perform updates only when necessary (instead of computing them all the time) on each request, I can have a global announcer and subscribers to know when to update within an image, but is there a way to have something like that but for inter-image coordination?
I'd only need to communicate the id and the class name (or a similar identifier), so on other images they'll update accordingly, and if there is an update in one image, it will notify the other images. The common data is on the database, so this is just to avoid re-reading a lot of things.
Is a message queue a good fit for this? Pub/Sub? What is available in Pharo that works without having to set up a lot of things?
Thanks!
Esteban A. Maringolo
As far as I understand, ZeroMQ does not require a broker to perform the communication between publishers and subscribers, I don't know how that is implemented (what do they connect to? how each client discovers each other?). I already got MQTT running, It's not completely clear to me how the QoS setting works (atLeastOnce, exactlyOnce, atMostOnce), I don't remember using that setting in Java (it probably had a sensible default). Regards! Esteban A. Maringolo On Tue, Jun 29, 2021 at 2:45 PM Jesus Mari Aguirre <jmariaguirre@gmail.com> wrote:
Maybe ZeroMQ fits you, but unlucky is not documented and there are only a few examples. I use ir in my jupyter kernel. You can install my port to pharo64 uFFI doing:
Metacello new baseline: 'JupyterTalk'; repository: 'github://jmari/JupyterTalk:master/repository'; load:'zmq'
But it needs ZeroMQ previously installed on your system. It was ported from http://smalltalkhub.com/#!/~panuw/zeromq now it is at: https://github.com/dellani/zeroMQ but I'm not sure if the original repo works on newer Pharos....I had no time to contact the original author to join both repos...
El mar, 29 jun 2021 a las 4:56, Esteban Maringolo (<emaringolo@gmail.com>) escribió:
Hi,
I'm rearchitecting a web app to perform updates only when necessary (instead of computing them all the time) on each request, I can have a global announcer and subscribers to know when to update within an image, but is there a way to have something like that but for inter-image coordination?
I'd only need to communicate the id and the class name (or a similar identifier), so on other images they'll update accordingly, and if there is an update in one image, it will notify the other images. The common data is on the database, so this is just to avoid re-reading a lot of things.
Is a message queue a good fit for this? Pub/Sub? What is available in Pharo that works without having to set up a lot of things?
Thanks!
Esteban A. Maringolo
If you want to use Rabbit you can also take a look at Ansible <https://github.com/ba-st/Ansible>for connecting using the protocol instead of STOMP. On Tue, Jun 29, 2021 at 3:11 PM Esteban Maringolo <emaringolo@gmail.com> wrote:
As far as I understand, ZeroMQ does not require a broker to perform the communication between publishers and subscribers, I don't know how that is implemented (what do they connect to? how each client discovers each other?).
I already got MQTT running, It's not completely clear to me how the QoS setting works (atLeastOnce, exactlyOnce, atMostOnce), I don't remember using that setting in Java (it probably had a sensible default).
Regards!
Esteban A. Maringolo
On Tue, Jun 29, 2021 at 2:45 PM Jesus Mari Aguirre <jmariaguirre@gmail.com> wrote:
Maybe ZeroMQ fits you, but unlucky is not documented and there are only
a few examples. I use ir in my jupyter kernel. You can install my port to pharo64 uFFI doing:
Metacello new baseline: 'JupyterTalk'; repository: 'github://jmari/JupyterTalk:master/repository'; load:'zmq'
But it needs ZeroMQ previously installed on your system. It was ported from http://smalltalkhub.com/#!/~panuw/zeromq now it is
at: https://github.com/dellani/zeroMQ but I'm not sure if the original repo works on newer Pharos....I had no time to contact the original author to join both repos...
El mar, 29 jun 2021 a las 4:56, Esteban Maringolo (<emaringolo@gmail.com>)
escribió:
Hi,
I'm rearchitecting a web app to perform updates only when necessary (instead of computing them all the time) on each request, I can have a global announcer and subscribers to know when to update within an image, but is there a way to have something like that but for inter-image coordination?
I'd only need to communicate the id and the class name (or a similar identifier), so on other images they'll update accordingly, and if there is an update in one image, it will notify the other images. The common data is on the database, so this is just to avoid re-reading a lot of things.
Is a message queue a good fit for this? Pub/Sub? What is available in Pharo that works without having to set up a lot of
things?
Thanks!
Esteban A. Maringolo
Hi Gabriel, I like the Ansible approach using RabbitMQ, in particular because it would also work to delegate tasks to workers (even if in the same image), and the task queue would be preserved, whilst in MQTT if there are no subscribers for a topic, then all messages sent to it are discarded. Is there a way to "start fresh" when subscribing to a queue?, it is, to receive only messages _after_ you subscribed to it? How do you ensure that the publisher/subscriber (and particularly this last) keeps running for the lifetime of an image? It is, what are the failover/recovery options? Regards! Esteban A. Maringolo On Tue, Jun 29, 2021 at 3:44 PM Gabriel Cotelli <g.cotelli@gmail.com> wrote:
If you want to use Rabbit you can also take a look at Ansible for connecting using the protocol instead of STOMP.
On Tue, Jun 29, 2021 at 3:11 PM Esteban Maringolo <emaringolo@gmail.com> wrote:
As far as I understand, ZeroMQ does not require a broker to perform the communication between publishers and subscribers, I don't know how that is implemented (what do they connect to? how each client discovers each other?).
I already got MQTT running, It's not completely clear to me how the QoS setting works (atLeastOnce, exactlyOnce, atMostOnce), I don't remember using that setting in Java (it probably had a sensible default).
Regards!
Esteban A. Maringolo
On Tue, Jun 29, 2021 at 2:45 PM Jesus Mari Aguirre <jmariaguirre@gmail.com> wrote:
Maybe ZeroMQ fits you, but unlucky is not documented and there are only a few examples. I use ir in my jupyter kernel. You can install my port to pharo64 uFFI doing:
Metacello new baseline: 'JupyterTalk'; repository: 'github://jmari/JupyterTalk:master/repository'; load:'zmq'
But it needs ZeroMQ previously installed on your system. It was ported from http://smalltalkhub.com/#!/~panuw/zeromq now it is at: https://github.com/dellani/zeroMQ but I'm not sure if the original repo works on newer Pharos....I had no time to contact the original author to join both repos...
El mar, 29 jun 2021 a las 4:56, Esteban Maringolo (<emaringolo@gmail.com>) escribió:
Hi,
I'm rearchitecting a web app to perform updates only when necessary (instead of computing them all the time) on each request, I can have a global announcer and subscribers to know when to update within an image, but is there a way to have something like that but for inter-image coordination?
I'd only need to communicate the id and the class name (or a similar identifier), so on other images they'll update accordingly, and if there is an update in one image, it will notify the other images. The common data is on the database, so this is just to avoid re-reading a lot of things.
Is a message queue a good fit for this? Pub/Sub? What is available in Pharo that works without having to set up a lot of things?
Thanks!
Esteban A. Maringolo
On 30 Jun 2021, at 01:28, Esteban Maringolo <emaringolo@gmail.com> wrote:
I like the Ansible approach using RabbitMQ, in particular because it would also work to delegate tasks to workers (even if in the same image), and the task queue would be preserved, whilst in MQTT if there are no subscribers for a topic, then all messages sent to it are discarded.
That is correct but less of a problem in practice than it seems to be at first sight. As a client, you normally have to be able to start from scratch and load everything that came before the moment you start up. When a client with a specific ID has connected once with clean session false, QoS 1 and a subscription, the persistence of the queue will be enabled for a as long as the server lives (and maybe even beyond restarts). There is absolutely no question that Rabbit MQ has much more functionality, I found Mosquitto MQTT very nice to work with, but as always, YMMV. Sven
Hi Sven, Thanks, now I understand better the use of QoS and session ID with a practical use case. How do you deal with the lifetime of each "listener" (subscriber) on each image? I mean, how stable is it? E.g. MQTTClient new open; subscribeToTopic: '/updates'; runWith: [ :message | Transcript cr; show: '[UPDATED]'; space; show: message contents asString; cr. ]; yourself. You just fork that and it will continue to receive messages until you close the connection? Is there a watchdog class/library to ensure things like this continue working? (I'm used to having my images living for months sometimes). Thanks in advance. Regards! Esteban A. Maringolo On Wed, Jun 30, 2021 at 5:00 AM Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 30 Jun 2021, at 01:28, Esteban Maringolo <emaringolo@gmail.com> wrote:
I like the Ansible approach using RabbitMQ, in particular because it would also work to delegate tasks to workers (even if in the same image), and the task queue would be preserved, whilst in MQTT if there are no subscribers for a topic, then all messages sent to it are discarded.
That is correct but less of a problem in practice than it seems to be at first sight.
As a client, you normally have to be able to start from scratch and load everything that came before the moment you start up.
When a client with a specific ID has connected once with clean session false, QoS 1 and a subscription, the persistence of the queue will be enabled for a as long as the server lives (and maybe even beyond restarts).
There is absolutely no question that Rabbit MQ has much more functionality, I found Mosquitto MQTT very nice to work with, but as always, YMMV.
Sven
On 1 Jul 2021, at 20:58, Esteban Maringolo <emaringolo@gmail.com> wrote:
Hi Sven,
Thanks, now I understand better the use of QoS and session ID with a practical use case.
How do you deal with the lifetime of each "listener" (subscriber) on each image? I mean, how stable is it?
E.g. MQTTClient new open; subscribeToTopic: '/updates'; runWith: [ :message | Transcript cr; show: '[UPDATED]'; space; show: message contents asString; cr. ]; yourself.
You just fork that and it will continue to receive messages until you close the connection?
Basically, yes.
Is there a watchdog class/library to ensure things like this continue working? (I'm used to having my images living for months sometimes).
I have an MQTTListener class that wraps the MQTT client and its process, the process's run method looks like this: run [ self log: 'Starting MQTT reading'. self ensureMQTTClient ifTrue: [ mqttClient runWith: [ :message | self handleMessage: message ] ]. self log: 'End of MQTT reading, restarting in 5s'. self closeMQTTClient. 5 seconds wait ] repeat It is a bit crude but it does help to restore the connection when the broker becomes temporarily unavailable.
Thanks in advance.
Regards!
Esteban A. Maringolo
On Wed, Jun 30, 2021 at 5:00 AM Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 30 Jun 2021, at 01:28, Esteban Maringolo <emaringolo@gmail.com> wrote:
I like the Ansible approach using RabbitMQ, in particular because it would also work to delegate tasks to workers (even if in the same image), and the task queue would be preserved, whilst in MQTT if there are no subscribers for a topic, then all messages sent to it are discarded.
That is correct but less of a problem in practice than it seems to be at first sight.
As a client, you normally have to be able to start from scratch and load everything that came before the moment you start up.
When a client with a specific ID has connected once with clean session false, QoS 1 and a subscription, the persistence of the queue will be enabled for a as long as the server lives (and maybe even beyond restarts).
There is absolutely no question that Rabbit MQ has much more functionality, I found Mosquitto MQTT very nice to work with, but as always, YMMV.
Sven
On 29 Jun 2021, at 20:10, Esteban Maringolo <emaringolo@gmail.com> wrote:
I already got MQTT running, It's not completely clear to me how the QoS setting works (atLeastOnce, exactlyOnce, atMostOnce), I don't remember using that setting in Java (it probably had a sensible default).
From the comments in MQTTPacket class side
atMostOnce "Quality of service level 0 - no acks, send and forget" ^ 0 atLeastOnce "Quality of service level 1 - single acks, possible duplicates" ^ 1 exactlyOnce "Quality of service level 2 - no loss, no duplicates" ^ 2 The lowest one is the fastest, but won't work with persistent queues. In most cases the middle level is OK and enough - duplicates can only occur when you do not receive the ack.
As far as I know, zmq doesn't need a broker but subscribers should know the address of the publisher, if the network increases its complexity with more publishers you need a broker, that is a proxy on zmq. If I understand well you need any of them should be able to publish a change to all of the other images? El mar., 29 jun. 2021 20:11, Esteban Maringolo <emaringolo@gmail.com> escribió:
As far as I understand, ZeroMQ does not require a broker to perform the communication between publishers and subscribers, I don't know how that is implemented (what do they connect to? how each client discovers each other?).
I already got MQTT running, It's not completely clear to me how the QoS setting works (atLeastOnce, exactlyOnce, atMostOnce), I don't remember using that setting in Java (it probably had a sensible default).
Regards!
Esteban A. Maringolo
On Tue, Jun 29, 2021 at 2:45 PM Jesus Mari Aguirre <jmariaguirre@gmail.com> wrote:
Maybe ZeroMQ fits you, but unlucky is not documented and there are only
a few examples. I use ir in my jupyter kernel. You can install my port to pharo64 uFFI doing:
Metacello new baseline: 'JupyterTalk'; repository: 'github://jmari/JupyterTalk:master/repository'; load:'zmq'
But it needs ZeroMQ previously installed on your system. It was ported from http://smalltalkhub.com/#!/~panuw/zeromq now it is
at: https://github.com/dellani/zeroMQ but I'm not sure if the original repo works on newer Pharos....I had no time to contact the original author to join both repos...
El mar, 29 jun 2021 a las 4:56, Esteban Maringolo (<emaringolo@gmail.com>)
escribió:
Hi,
I'm rearchitecting a web app to perform updates only when necessary (instead of computing them all the time) on each request, I can have a global announcer and subscribers to know when to update within an image, but is there a way to have something like that but for inter-image coordination?
I'd only need to communicate the id and the class name (or a similar identifier), so on other images they'll update accordingly, and if there is an update in one image, it will notify the other images. The common data is on the database, so this is just to avoid re-reading a lot of things.
Is a message queue a good fit for this? Pub/Sub? What is available in Pharo that works without having to set up a lot of
things?
Thanks!
Esteban A. Maringolo
On Thu, Jul 1, 2021 at 3:00 PM Jesus Mari Aguirre <jmariaguirre@gmail.com> wrote:
As far as I know, zmq doesn't need a broker but subscribers should know the address of the publisher, if the network increases its complexity with more publishers you need a broker, that is a proxy on zmq. If I understand well you need any of them should be able to publish a change to all of the other images?
I want to broadcast notifications of updated objects so they can be recomputed or reloaded from the database to reflect the latest changes. The change might happen in any of the worker images, either by user UI or API call. Regards! Esteban A. Maringolo
On 1 Jul 2021, at 21:01, Esteban Maringolo <emaringolo@gmail.com> wrote:
On Thu, Jul 1, 2021 at 3:00 PM Jesus Mari Aguirre <jmariaguirre@gmail.com> wrote:
As far as I know, zmq doesn't need a broker but subscribers should know the address of the publisher, if the network increases its complexity with more publishers you need a broker, that is a proxy on zmq. If I understand well you need any of them should be able to publish a change to all of the other images?
I want to broadcast notifications of updated objects so they can be recomputed or reloaded from the database to reflect the latest changes. The change might happen in any of the worker images, either by user UI or API call.
This reminds me that there is a notification mechanism in PostgreSQL. I have not used it though. The SQL commands are NOTIFY and (UN)LISTEN. I think these will arrive as P3Notifications.
Regards!
Esteban A. Maringolo
On ZeroMQ it could be something like that...first you need a broker (a proxy) using XSUB and XPUB. In one pharo image... (I think this part better be done in C because it will freezy the image forever but it works) | frontend backend | frontend := ZmqXSubscriberSocket new. frontend bind: 'tcp://*:5559'. backend := ZmqXPublisherSocket new. backend bind: 'tcp://*:5560'. ZmqProxy frontend: frontend backend: backend capture: nil In any publisher or subscriber... publisher := ZmqPublisherSocket new connect: 'tcp://localhost:5559'. pr1 := [ |subscriber1| subscriber1 := ZmqSubscriberSocket new connect: 'tcp://localhost:5560'. [true] whileTrue:[ "receive is non blocking so this is an infinite loop" subscriber1 receiveIfAvailable: [ :messageData| messageData isEmpty ifFalse:[Transcript show: ('Subscriber 1 got stuff ' , messageData);cr]] ]] forkNamed: 'subscriber1'. "Publish doing that" publisher send: ('Hello Subscribers') asByteArray. El jue, 1 jul 2021 a las 21:03, Esteban Maringolo (<emaringolo@gmail.com>) escribió:
On Thu, Jul 1, 2021 at 3:00 PM Jesus Mari Aguirre <jmariaguirre@gmail.com> wrote:
As far as I know, zmq doesn't need a broker but subscribers should know
the address of the publisher, if the network increases its complexity with more publishers you need a broker, that is a proxy on zmq.
If I understand well you need any of them should be able to publish a change to all of the other images?
I want to broadcast notifications of updated objects so they can be recomputed or reloaded from the database to reflect the latest changes. The change might happen in any of the worker images, either by user UI or API call.
Regards!
Esteban A. Maringolo
participants (4)
-
Esteban Maringolo -
Gabriel Cotelli -
Jesus Mari Aguirre -
Sven Van Caekenberghe