Communication between different images
As far as I know, zmq doesn't need a broker but subscribers should know the
address of the publisher, if the network increases its complexity with more
publishers you need a broker, that is a proxy on zmq.
If I understand well you need any of them should be able to publish a
change to all of the other images?
El mar., 29 jun. 2021 20:11, Esteban Maringolo emaringolo@gmail.com
escribió:
As far as I understand, ZeroMQ does not require a broker to perform
the communication between publishers and subscribers, I don't know how
that is implemented (what do they connect to? how each client
discovers each other?).
I already got MQTT running, It's not completely clear to me how the
QoS setting works (atLeastOnce, exactlyOnce, atMostOnce), I don't
remember using that setting in Java (it probably had a sensible
default).
Regards!
Esteban A. Maringolo
On Tue, Jun 29, 2021 at 2:45 PM Jesus Mari Aguirre
jmariaguirre@gmail.com wrote:
Maybe ZeroMQ fits you, but unlucky is not documented and there are only
a few examples. I use ir in my jupyter kernel. You can install my port to
pharo64 uFFI doing:
Metacello new
baseline: 'JupyterTalk';
repository: 'github://jmari/JupyterTalk:master/repository';
load:'zmq'
But it needs ZeroMQ previously installed on your system.
It was ported from http://smalltalkhub.com/#!/~panuw/zeromq now it is
at: https://github.com/dellani/zeroMQ but I'm not sure if the original
repo works on newer Pharos....I had no time to contact the original author
to join both repos...
El mar, 29 jun 2021 a las 4:56, Esteban Maringolo (emaringolo@gmail.com)
escribió:
Hi,
I'm rearchitecting a web app to perform updates only when necessary
(instead of computing them all the time) on each request, I can have a
global announcer and subscribers to know when to update within an
image, but is there a way to have something like that but for
inter-image coordination?
I'd only need to communicate the id and the class name (or a similar
identifier), so on other images they'll update accordingly, and if
there is an update in one image, it will notify the other images. The
common data is on the database, so this is just to avoid re-reading a
lot of things.
Is a message queue a good fit for this? Pub/Sub?
What is available in Pharo that works without having to set up a lot of
things?
Thanks!
Esteban A. Maringolo
Hi Sven,
Thanks, now I understand better the use of QoS and session ID with a
practical use case.
How do you deal with the lifetime of each "listener" (subscriber) on
each image? I mean, how stable is it?
E.g.
MQTTClient new
open;
subscribeToTopic: '/updates';
runWith: [ :message |
Transcript cr; show: '[UPDATED]'; space; show: message contents
asString; cr.
];
yourself.
You just fork that and it will continue to receive messages until you
close the connection? Is there a watchdog class/library to ensure
things like this continue working? (I'm used to having my images
living for months sometimes).
Thanks in advance.
Regards!
Esteban A. Maringolo
On Wed, Jun 30, 2021 at 5:00 AM Sven Van Caekenberghe sven@stfx.eu wrote:
On 30 Jun 2021, at 01:28, Esteban Maringolo emaringolo@gmail.com wrote:
I like the Ansible approach using RabbitMQ, in particular because it
would also work to delegate tasks to workers (even if in the same
image), and the task queue would be preserved, whilst in MQTT if there
are no subscribers for a topic, then all messages sent to it are
discarded.
That is correct but less of a problem in practice than it seems to be at first sight.
As a client, you normally have to be able to start from scratch and load everything that came before the moment you start up.
When a client with a specific ID has connected once with clean session false, QoS 1 and a subscription, the persistence of the queue will be enabled for a as long as the server lives (and maybe even beyond restarts).
There is absolutely no question that Rabbit MQ has much more functionality, I found Mosquitto MQTT very nice to work with, but as always, YMMV.
Sven
On Thu, Jul 1, 2021 at 3:00 PM Jesus Mari Aguirre
jmariaguirre@gmail.com wrote:
As far as I know, zmq doesn't need a broker but subscribers should know the address of the publisher, if the network increases its complexity with more publishers you need a broker, that is a proxy on zmq.
If I understand well you need any of them should be able to publish a change to all of the other images?
I want to broadcast notifications of updated objects so they can be
recomputed or reloaded from the database to reflect the latest
changes.
The change might happen in any of the worker images, either by user UI
or API call.
Regards!
Esteban A. Maringolo
On 1 Jul 2021, at 20:58, Esteban Maringolo emaringolo@gmail.com wrote:
Hi Sven,
Thanks, now I understand better the use of QoS and session ID with a
practical use case.
How do you deal with the lifetime of each "listener" (subscriber) on
each image? I mean, how stable is it?
E.g.
MQTTClient new
open;
subscribeToTopic: '/updates';
runWith: [ :message |
Transcript cr; show: '[UPDATED]'; space; show: message contents
asString; cr.
];
yourself.
You just fork that and it will continue to receive messages until you
close the connection?
Basically, yes.
Is there a watchdog class/library to ensure
things like this continue working? (I'm used to having my images
living for months sometimes).
I have an MQTTListener class that wraps the MQTT client and its process, the process's run method looks like this:
run
[
self log: 'Starting MQTT reading'.
self ensureMQTTClient ifTrue: [
mqttClient runWith: [ :message |
self handleMessage: message ] ].
self log: 'End of MQTT reading, restarting in 5s'.
self closeMQTTClient.
5 seconds wait ] repeat
It is a bit crude but it does help to restore the connection when the broker becomes temporarily unavailable.
Thanks in advance.
Regards!
Esteban A. Maringolo
On Wed, Jun 30, 2021 at 5:00 AM Sven Van Caekenberghe sven@stfx.eu wrote:
On 30 Jun 2021, at 01:28, Esteban Maringolo emaringolo@gmail.com wrote:
I like the Ansible approach using RabbitMQ, in particular because it
would also work to delegate tasks to workers (even if in the same
image), and the task queue would be preserved, whilst in MQTT if there
are no subscribers for a topic, then all messages sent to it are
discarded.
That is correct but less of a problem in practice than it seems to be at first sight.
As a client, you normally have to be able to start from scratch and load everything that came before the moment you start up.
When a client with a specific ID has connected once with clean session false, QoS 1 and a subscription, the persistence of the queue will be enabled for a as long as the server lives (and maybe even beyond restarts).
There is absolutely no question that Rabbit MQ has much more functionality, I found Mosquitto MQTT very nice to work with, but as always, YMMV.
Sven
On 1 Jul 2021, at 21:01, Esteban Maringolo emaringolo@gmail.com wrote:
On Thu, Jul 1, 2021 at 3:00 PM Jesus Mari Aguirre
jmariaguirre@gmail.com wrote:
As far as I know, zmq doesn't need a broker but subscribers should know the address of the publisher, if the network increases its complexity with more publishers you need a broker, that is a proxy on zmq.
If I understand well you need any of them should be able to publish a change to all of the other images?
I want to broadcast notifications of updated objects so they can be
recomputed or reloaded from the database to reflect the latest
changes.
The change might happen in any of the worker images, either by user UI
or API call.
This reminds me that there is a notification mechanism in PostgreSQL. I have not used it though. The SQL commands are NOTIFY and (UN)LISTEN. I think these will arrive as P3Notifications.
Regards!
Esteban A. Maringolo
On ZeroMQ it could be something like that...first you need a broker (a
proxy) using XSUB and XPUB.
In one pharo image... (I think this part better be done in C because it
will freezy the image forever but it works)
| frontend backend |
frontend := ZmqXSubscriberSocket new.
frontend bind: 'tcp://:5559'.
backend := ZmqXPublisherSocket new.
backend bind: 'tcp://:5560'.
ZmqProxy frontend: frontend backend: backend capture: nil
In any publisher or subscriber...
publisher := ZmqPublisherSocket new connect: 'tcp://localhost:5559'.
pr1 := [
|subscriber1|
subscriber1 := ZmqSubscriberSocket new connect: 'tcp://localhost:5560'.
[true] whileTrue:[
"receive is non blocking so this is an infinite loop"
subscriber1 receiveIfAvailable: [ :messageData|
messageData isEmpty ifFalse:[Transcript show: ('Subscriber 1 got stuff ' ,
messageData);cr]]
]] forkNamed: 'subscriber1'.
"Publish doing that"
publisher send: ('Hello Subscribers') asByteArray.
El jue, 1 jul 2021 a las 21:03, Esteban Maringolo (emaringolo@gmail.com)
escribió:
On Thu, Jul 1, 2021 at 3:00 PM Jesus Mari Aguirre
jmariaguirre@gmail.com wrote:
As far as I know, zmq doesn't need a broker but subscribers should know
the address of the publisher, if the network increases its complexity with
more publishers you need a broker, that is a proxy on zmq.
If I understand well you need any of them should be able to publish a
change to all of the other images?
I want to broadcast notifications of updated objects so they can be
recomputed or reloaded from the database to reflect the latest
changes.
The change might happen in any of the worker images, either by user UI
or API call.
Regards!
Esteban A. Maringolo