Re: [Pharo-users] CannotWriteData errors in P3 and Seaside
Hi, I'm replying to the list as well... because the last two mails got replied to our personal addresses. On Thu, Aug 20, 2020 at 11:55 AM Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 20 Aug 2020, at 15:31, Esteban Maringolo <emaringolo@gmail.com> wrote:
Hi Sven,
If a socketstream doesn't know the state of the connection, then what is the #socketIsConnected method for? In particular the #isOtherEndClosed test.
ZdcAbstractSocketStream>>#socketIsConnected ^socket isConnected and: [ socket isOtherEndClosed not ]
I don't know what is going on inside Socket, I just stated my opinion.
Maybe there is something to investigate here?
With logging enabled, I can do the following:
$ grep P3 server-2020-08-20.log | grep CONNECT | tail -n 20 <snip> 2020-08-20 14:43:06 [P3] 30513 DISCONNECTING psql://client-xyz:hiddenpassword@client-xyz-db:5432/client-xyz 2020-08-20 14:44:06 [P3] 30516 CONNECTED psql://client-xyz:hiddenpassword@client-xyz-db:5432/client-xyz 2020-08-20 14:44:06 [P3] 30516 DISCONNECTING psql://client-xyz:hiddenpassword@client-xyz-db:5432/client-xyz 2020-08-20 14:44:06 [P3] 30517 CONNECTED psql://client-xyz:hiddenpassword@client-xyz-db:5432/client-xyz 2020-08-20 14:44:06 [P3] 30517 DISCONNECTING psql://client-xyz:hiddenpassword@client-xyz-db:5432/client-xyz
The number after [P3] is the session identifier (backend process id) of that connection. You should see each one being opened and closed in pairs.
Yes, I noticed the pid, and compared it with what I had on the pg_stat_activity table. I don't get the CONNECTED log because there is no way to set the logging in the P3DatabaseDriver before it creates (and connects) the P3Client. Maybe there could be a setting on P3Client class to set verbosity globally? Or at the P3DatabaseDriver instead. Summarizing... I'm pretty confident that P3 works correctly and also the PG server. At this point I'm factoring out what might be causing this. It's an issue that only happens to me in production, and I don't have a better instrumentation in place to debug it. Again, thanks for the support. Regards.
On 20 Aug 2020, at 19:51, Esteban Maringolo <emaringolo@gmail.com> wrote:
Hi,
I'm replying to the list as well... because the last two mails got replied to our personal addresses.
Oh, I did not notice that, that was not my intention.
On Thu, Aug 20, 2020 at 11:55 AM Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 20 Aug 2020, at 15:31, Esteban Maringolo <emaringolo@gmail.com> wrote:
Hi Sven,
If a socketstream doesn't know the state of the connection, then what is the #socketIsConnected method for? In particular the #isOtherEndClosed test.
ZdcAbstractSocketStream>>#socketIsConnected ^socket isConnected and: [ socket isOtherEndClosed not ]
I don't know what is going on inside Socket, I just stated my opinion.
Maybe there is something to investigate here?
Is it not so that in Docker all network connections in/out/between instances are mediated by some management software ? I even thought it was nginx. Maybe I am totally wrong here.
With logging enabled, I can do the following:
$ grep P3 server-2020-08-20.log | grep CONNECT | tail -n 20 <snip> 2020-08-20 14:43:06 [P3] 30513 DISCONNECTING psql://client-xyz:hiddenpassword@client-xyz-db:5432/client-xyz 2020-08-20 14:44:06 [P3] 30516 CONNECTED psql://client-xyz:hiddenpassword@client-xyz-db:5432/client-xyz 2020-08-20 14:44:06 [P3] 30516 DISCONNECTING psql://client-xyz:hiddenpassword@client-xyz-db:5432/client-xyz 2020-08-20 14:44:06 [P3] 30517 CONNECTED psql://client-xyz:hiddenpassword@client-xyz-db:5432/client-xyz 2020-08-20 14:44:06 [P3] 30517 DISCONNECTING psql://client-xyz:hiddenpassword@client-xyz-db:5432/client-xyz
The number after [P3] is the session identifier (backend process id) of that connection. You should see each one being opened and closed in pairs.
Yes, I noticed the pid, and compared it with what I had on the pg_stat_activity table.
Right.
I don't get the CONNECTED log because there is no way to set the logging in the P3DatabaseDriver before it creates (and connects) the P3Client. Maybe there could be a setting on P3Client class to set verbosity globally? Or at the P3DatabaseDriver instead.
What if you change P3DatabaseDriver >> connect: aLogin connection := self connectionClass new. connection host: aLogin host; port: aLogin port asInteger; database: aLogin databaseName; user: aLogin username; password: aLogin password. connection connect by inserting verbose: true before the last statement. You could also make a subclass of P3DatabaseDriver.
Summarizing... I'm pretty confident that P3 works correctly and also the PG server. At this point I'm factoring out what might be causing this. It's an issue that only happens to me in production, and I don't have a better instrumentation in place to debug it.
Again, thanks for the support.
Regards.
HI guys let us know if we have to do something from our side because we want to support business. S.
On 20 Aug 2020, at 20:01, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 20 Aug 2020, at 19:51, Esteban Maringolo <emaringolo@gmail.com <mailto:emaringolo@gmail.com>> wrote:
Hi,
I'm replying to the list as well... because the last two mails got replied to our personal addresses.
Oh, I did not notice that, that was not my intention.
On Thu, Aug 20, 2020 at 11:55 AM Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 20 Aug 2020, at 15:31, Esteban Maringolo <emaringolo@gmail.com> wrote:
Hi Sven,
If a socketstream doesn't know the state of the connection, then what is the #socketIsConnected method for? In particular the #isOtherEndClosed test.
ZdcAbstractSocketStream>>#socketIsConnected ^socket isConnected and: [ socket isOtherEndClosed not ]
I don't know what is going on inside Socket, I just stated my opinion.
Maybe there is something to investigate here?
Is it not so that in Docker all network connections in/out/between instances are mediated by some management software ? I even thought it was nginx. Maybe I am totally wrong here.
With logging enabled, I can do the following:
$ grep P3 server-2020-08-20.log | grep CONNECT | tail -n 20 <snip> 2020-08-20 14:43:06 [P3] 30513 DISCONNECTING psql://client-xyz:hiddenpassword@client-xyz-db:5432/client-xyz 2020-08-20 14:44:06 [P3] 30516 CONNECTED psql://client-xyz:hiddenpassword@client-xyz-db:5432/client-xyz 2020-08-20 14:44:06 [P3] 30516 DISCONNECTING psql://client-xyz:hiddenpassword@client-xyz-db:5432/client-xyz 2020-08-20 14:44:06 [P3] 30517 CONNECTED psql://client-xyz:hiddenpassword@client-xyz-db:5432/client-xyz 2020-08-20 14:44:06 [P3] 30517 DISCONNECTING psql://client-xyz:hiddenpassword@client-xyz-db:5432/client-xyz
The number after [P3] is the session identifier (backend process id) of that connection. You should see each one being opened and closed in pairs.
Yes, I noticed the pid, and compared it with what I had on the pg_stat_activity table.
Right.
I don't get the CONNECTED log because there is no way to set the logging in the P3DatabaseDriver before it creates (and connects) the P3Client. Maybe there could be a setting on P3Client class to set verbosity globally? Or at the P3DatabaseDriver instead.
What if you change
P3DatabaseDriver >> connect: aLogin connection := self connectionClass new. connection host: aLogin host; port: aLogin port asInteger; database: aLogin databaseName; user: aLogin username; password: aLogin password. connection connect
by inserting
verbose: true
before the last statement.
You could also make a subclass of P3DatabaseDriver.
Summarizing... I'm pretty confident that P3 works correctly and also the PG server. At this point I'm factoring out what might be causing this. It's an issue that only happens to me in production, and I don't have a better instrumentation in place to debug it.
Again, thanks for the support.
Regards.
-------------------------------------------- Stéphane Ducasse http://stephane.ducasse.free.fr / http://www.pharo.org 03 59 35 87 52 Assistant: Aurore Dalle FAX 03 59 57 78 50 TEL 03 59 35 86 16 S. Ducasse - Inria 40, avenue Halley, Parc Scientifique de la Haute Borne, Bât.A, Park Plaza Villeneuve d'Ascq 59650 France
Hi all, On Thu, Aug 20, 2020 at 3:02 PM Sven Van Caekenberghe <sven@stfx.eu> wrote:
I don't know what is going on inside Socket, I just stated my opinion. Maybe there is something to investigate here?
Is it not so that in Docker all network connections in/out/between instances are mediated by some management software ? I even thought it was nginx. Maybe I am totally wrong here.
No, it is not that, but inside a Swarm all containers run in a "overlay network" (basically a VPN) that is independent of the host network, this way you can distribute containers among different hosts. All the packages are routed by Docker itself, and apparently there is an issue there, that if a connection is idle for a certain time, it silently stops routing packages, leaving both sides of the connection unaware of it. https://github.com/moby/moby/issues/31208
What if you change
P3DatabaseDriver >> connect: aLogin
by inserting
verbose: true
before the last statement.
You could also make a subclass of P3DatabaseDriver.
I could, but that would only log one extra CONNECT entry, not of much use.
At this point I'm factoring out what might be causing this. It's an issue that only happens to me in production, and I don't have a better instrumentation in place to debug it.
I think I found the culprit, now I need to know how to bypass or setup the network to avoid these situations. Luckily this will be solved soon. :-) Regards.
Hi all, Good news. :-) On Thu, Aug 20, 2020 at 6:36 PM Esteban Maringolo <emaringolo@gmail.com> wrote:
Is it not so that in Docker all network connections in/out/between instances are mediated by some management software ? I even thought it was nginx. Maybe I am totally wrong here.
No, it is not that, but inside a Swarm all containers run in a "overlay network" (basically a VPN) that is independent of the host network, this way you can distribute containers among different hosts.
All the packages are routed by Docker itself, and apparently there is an issue there, that if a connection is idle for a certain time, it silently stops routing packages, leaving both sides of the connection unaware of it.
I finally was able to pinpoint what was causing this issue that gave me a lot of headaches. It effectively was the "overlay network" [1] of my Docker Swarm [2] deployment. The "mesh" router was terminating idle connections after 10 minutes (I noticed this exact timing some days ago). So configuring the PostgreSQL server container to use the host network instead of the overlay made all my connections stable again. No more silent drops. No more GlorpDatabaseReadError. And now with some additional health checks in place that will help me when there is an actual issue with the connections. Maybe there is another way to make it work within the overlay, but that's not something I'm interested in doing right now. So P3 behavior was okay (again), PostgreSQL was not doing anything odd either, it was what was sitting in between. Best regards, [1] https://docs.docker.com/network/overlay/ [2] I have a Swarm with 1 nginx for static content, 1 PostgreSQL and 1 Traefik as an HTTP gateway for several replicas of a Seaside application.
Hi Esteban, That is good to hear ! Sven
On 20 Aug 2020, at 23:36, Esteban Maringolo <emaringolo@gmail.com> wrote:
Hi all,
On Thu, Aug 20, 2020 at 3:02 PM Sven Van Caekenberghe <sven@stfx.eu> wrote:
I don't know what is going on inside Socket, I just stated my opinion. Maybe there is something to investigate here?
Is it not so that in Docker all network connections in/out/between instances are mediated by some management software ? I even thought it was nginx. Maybe I am totally wrong here.
No, it is not that, but inside a Swarm all containers run in a "overlay network" (basically a VPN) that is independent of the host network, this way you can distribute containers among different hosts.
All the packages are routed by Docker itself, and apparently there is an issue there, that if a connection is idle for a certain time, it silently stops routing packages, leaving both sides of the connection unaware of it. https://github.com/moby/moby/issues/31208
What if you change
P3DatabaseDriver >> connect: aLogin
by inserting
verbose: true
before the last statement.
You could also make a subclass of P3DatabaseDriver.
I could, but that would only log one extra CONNECT entry, not of much use.
At this point I'm factoring out what might be causing this. It's an issue that only happens to me in production, and I don't have a better instrumentation in place to debug it.
I think I found the culprit, now I need to know how to bypass or setup the network to avoid these situations.
Luckily this will be solved soon. :-)
Regards.
+1 and I like when this is not Pharo the problem :) S.
On 21 Aug 2020, at 08:24, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi Esteban,
That is good to hear !
Sven
On 20 Aug 2020, at 23:36, Esteban Maringolo <emaringolo@gmail.com> wrote:
Hi all,
On Thu, Aug 20, 2020 at 3:02 PM Sven Van Caekenberghe <sven@stfx.eu> wrote:
I don't know what is going on inside Socket, I just stated my opinion. Maybe there is something to investigate here?
Is it not so that in Docker all network connections in/out/between instances are mediated by some management software ? I even thought it was nginx. Maybe I am totally wrong here.
No, it is not that, but inside a Swarm all containers run in a "overlay network" (basically a VPN) that is independent of the host network, this way you can distribute containers among different hosts.
All the packages are routed by Docker itself, and apparently there is an issue there, that if a connection is idle for a certain time, it silently stops routing packages, leaving both sides of the connection unaware of it. https://github.com/moby/moby/issues/31208
What if you change
P3DatabaseDriver >> connect: aLogin
by inserting
verbose: true
before the last statement.
You could also make a subclass of P3DatabaseDriver.
I could, but that would only log one extra CONNECT entry, not of much use.
At this point I'm factoring out what might be causing this. It's an issue that only happens to me in production, and I don't have a better instrumentation in place to debug it.
I think I found the culprit, now I need to know how to bypass or setup the network to avoid these situations.
Luckily this will be solved soon. :-)
Regards.
-------------------------------------------- Stéphane Ducasse http://stephane.ducasse.free.fr / http://www.pharo.org 03 59 35 87 52 Assistant: Aurore Dalle FAX 03 59 57 78 50 TEL 03 59 35 86 16 S. Ducasse - Inria 40, avenue Halley, Parc Scientifique de la Haute Borne, Bât.A, Park Plaza Villeneuve d'Ascq 59650 France
participants (3)
-
Esteban Maringolo -
Stéphane Ducasse -
Sven Van Caekenberghe