February 2020 - Pharo-users

Re: [Pharo-users] P3 and concurrenty

by Petter Egesund

Yes, I see - that sounds sensible. Could it be an idea to subclass the pool-class and to prepare statements in this method, if there is a kind of init-method in the class? What do you think? Petter On Mon, Feb 10, 2020 at 4:36 PM Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > > On 10 Feb 2020, at 16:30, Petter Egesund <petter.egesund(a)gmail.com> > wrote: > > > > Hi and thanks for an answer above my expectations to our problem. > > > > This is really awesome, I was just about to start coding a pool myself, > but getting it directly from the author of the base library is off course > way better :) > > > > I will try out and get back as soon as we have tested. > > > > One quick question - are the prepared statements which are connected to > the pool thread safe? > > Prepared statements are scoped to a single connection in PSQL, AFAIK. > > What you will probably want to do is use a #configurator: block to set up > your prepared statements once, like you do now. Then each thread/process > grabs its own connection from the pool, (which will be already configured > properly) and uses it, for itself, and finally returns it to the pool. > > Interacting with the pool should be fully thread safe. Once you are using > a connection, it not longer is, but is also does not have to, since one > connection equals one client thread. > > > Petter > > > > On Mon, Feb 10, 2020 at 12:13 PM Sven Van Caekenberghe <sven(a)stfx.eu> > wrote: > > Hi Petter, > > > > > https://github.com/svenvc/P3/commit/a6b409d0d92cb92bf9b44452908bb9033523b863 > adds a connection pool. > > > > Here is the class comment: > > > > ====== > > > > I am P3ConnectionPool. > > > > I offer a pool of shared PSQL connections (P3Client instances) as a > resource. > > > > After configuring me with at least a url, you use me by calling > #withConnection: > > > > pool := P3ConnectionPool url: 'psql://sven@localhost'. > > pool withConnection: [ :p3Client | > > p3Client query: 'SELECT table_name FROM > information_schema.tables' ]. > > pool close. > > > > When a connection is in use, it is not part of the pool. > > When a connection is returned/released to the pool, it becomes available > for reuse. > > > > The pool's size is the number of open connection ready to be reused. > > The pool's capacity is the maximum number of connection that will be > pooled. > > Excess connections will be closed when they are returned/released to the > pool. > > > > New connections are created as needed. > > You can set a #configurator to further initialize new connections. > > You can use #warmUp or #warmUp: to precreate a number of connections. > > > > When an error occurs, the connection should not be reused and be closed > by the caller. > > > > ====== > > > > There are some unit tests as well. > > > > Let me know if this works for you. > > > > Sven > > > > > On 9 Feb 2020, at 19:04, Petter Egesund <petter.egesund(a)gmail.com> > wrote: > > > > > > Yes, thanks for good feedback. > > > > > > I will try the pooled way, I think - not primarily because of speed, > but due to that our library is built around prepared connections. > > > > > > Petter > > > > > > On Sun, Feb 9, 2020 at 6:01 PM Sven Van Caekenberghe <sven(a)stfx.eu> > wrote: > > > Hi Petter, > > > > > > > On 9 Feb 2020, at 17:27, Petter Egesund <petter.egesund(a)gmail.com> > wrote: > > > > > > > > Hi Sven and thanks for answering! > > > > > > > > I use Teapot with one common sql-connecton, not one for each user > session. At startup I create several sql statements and these does not seem > to be usable from different Teapot request at the same time. > > > > > > It is useable, but not concurrently. > > > > > > > I could create one connection pr. session and then close the > database connection when the user leaves, but then I also would need to > create all the prepared sql-statements for each session, which does not > sound right to me? > > > > > > Yes and no, see further. > > > > > > > It seems I have will have to look into the other solution, and see > if I can use some mutex stuff to avoid several tasks acessessing the same > resources at the same time. > > > > > > Yes, follow the pointer that I gave you, it is not hard. > > > > > > > Creating a pool sounds like the right solution to me now - any > meaning about this? > > > > > > Yes, you could create a connection pool. But that is harder than it > sounds: what is the minimum size, the maximum size, what do you do when you > go over it, how do you make sure that a resource (connection) is clean when > returning it to the pool (given authentication, possible errors), ... > > > > > > I don't know what you are doing, but I think you focus too much on > performance issues. I would first try to get the code correct and worry > about performance later on. > > > > > > Running > > > > > > P3ClientTest new runBenchmark1Bench. > > > > > > on my machine gives me a BenchmarkResult(67 iterations in 5 seconds 57 > milliseconds. 13.249 per second). This is a query that returns 10.000 > records with 5 columns. It is reusing the same client/connection for all > iterations. > > > > > > If I modify this slightly to use a new client/connection each time, > like this > > > > > > [ > > > (P3Client url: 'psql://sven@localhost') > > > query: 'SELECT * FROM benchmark1'; > > > close > > > ] benchFor: 5 seconds > > > > > > I get a BenchmarkResult(65 iterations in 5 seconds 34 milliseconds. > 12.912 per second) which almost as fast. Of course, for smaller queries, > the connect/disconnect overhead will be more significant. > > > > > > And note that this is not using prepared statements. > > > > > > So I would start by opening/closing a connection each time you need it. > > > > > > HTH, > > > > > > Sven > > > > > > > Petter > > > > > > > > > > > > On Sun, Feb 9, 2020 at 4:52 PM Sven Van Caekenberghe <sven(a)stfx.eu> > wrote: > > > > Hi Petter, > > > > > > > > [ CC-ing the Pharo Users list ] > > > > > > > > P3Client is not built/designed to be used by multiple processes > concurrently. Each database connection is represented by an instance of > P3Client and holds some state both at the client as well as at the server > side. > > > > > > > > Typically, in a multi user server application, each connection > should have its own P3Client / psql connection. For example, in Seaside, a > custom WASession subclass gives each session/user its own p3 > connection/client. > > > > > > > > Is that what you are doing ? > > > > > > > > If not, you could wrap your db accessing code so that mutual > exclusion is provided. For example, you can have a look at AbstractCache > #beThreadSafe and #critical: > > > > > > > > That will then serialise requests and possibly block one onto the > other. > > > > > > > > HTH, > > > > > > > > Sven > > > > > > > > PS: another thing to take care of if closing your sql connections > when the session is no longer needed. > > > > > > > > PS: Zinc HTTP does also provide a session mechanism > (ZnServerSession[Manager]) but these work with cookies and typically won't > help with a REST access pattern. > > > > > > > > > On 9 Feb 2020, at 14:21, Petter Egesund <petter.egesund(a)gmail.com> > wrote: > > > > > > > > > > Hi Sven > > > > > > > > > > We are using Pharo as our backend in a project and we have run > into a problem with P3. > > > > > > > > > > The problem seems to be connected to compiled sql statements and > concurrency. > > > > > > > > > > We keep getting this error: Bindcomplete message expected > > > > > > > > > > Problem seems to be easy to reproduce: > > > > > > > > > > 1) Compile any sql statement > > > > > 2) Use this statement in a query twice (!) in a teapot endpoint > > > > > > > > > > The run some concurrent queries, like "curl > http://localhost:8080/endpoint & curl http://localhost:8080/endpoint.." > (add several curls after here). > > > > > > > > > > One could also use ex. siege ( > https://manpages.ubuntu.com/manpages/trusty/man1/siege.1.html) for easy > reproducing. > > > > > > > > > > If we chain the curls after each other, like "curl > http://localhost:8080/endpoint && > https://manpages.ubuntu.com/manpages/trusty/man1/siege.1.html && > https://manpages.ubuntu.com/manpages/trusty/man1/siege.1.html.." it seems > to work fine, so doing the request sequentially seem to work fine. > > > > > > > > > > My conclusion is that this must be connected to how teapot handles > concurrency in companion with the compiled statements? > > > > > > > > > > Any clues on this one? We are on Pharo 8.0 with latest version of > P3, PG 9.x) > > > > > > > > > > Best regards > > > > > > > > > > Petter Egesund (I wrote the heysql-package based on P3) > > > > > > > > > > >

Feb. 10, 2020

Re: [Pharo-users] P3 and concurrenty

by Sven Van Caekenberghe

> On 10 Feb 2020, at 16:30, Petter Egesund <petter.egesund(a)gmail.com> wrote: > > Hi and thanks for an answer above my expectations to our problem. > > This is really awesome, I was just about to start coding a pool myself, but getting it directly from the author of the base library is off course way better :) > > I will try out and get back as soon as we have tested. > > One quick question - are the prepared statements which are connected to the pool thread safe? Prepared statements are scoped to a single connection in PSQL, AFAIK. What you will probably want to do is use a #configurator: block to set up your prepared statements once, like you do now. Then each thread/process grabs its own connection from the pool, (which will be already configured properly) and uses it, for itself, and finally returns it to the pool. Interacting with the pool should be fully thread safe. Once you are using a connection, it not longer is, but is also does not have to, since one connection equals one client thread. > Petter > > On Mon, Feb 10, 2020 at 12:13 PM Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > Hi Petter, > > https://github.com/svenvc/P3/commit/a6b409d0d92cb92bf9b44452908bb9033523b863 adds a connection pool. > > Here is the class comment: > > ====== > > I am P3ConnectionPool. > > I offer a pool of shared PSQL connections (P3Client instances) as a resource. > > After configuring me with at least a url, you use me by calling #withConnection: > > pool := P3ConnectionPool url: 'psql://sven@localhost'. > pool withConnection: [ :p3Client | > p3Client query: 'SELECT table_name FROM information_schema.tables' ]. > pool close. > > When a connection is in use, it is not part of the pool. > When a connection is returned/released to the pool, it becomes available for reuse. > > The pool's size is the number of open connection ready to be reused. > The pool's capacity is the maximum number of connection that will be pooled. > Excess connections will be closed when they are returned/released to the pool. > > New connections are created as needed. > You can set a #configurator to further initialize new connections. > You can use #warmUp or #warmUp: to precreate a number of connections. > > When an error occurs, the connection should not be reused and be closed by the caller. > > ====== > > There are some unit tests as well. > > Let me know if this works for you. > > Sven > > > On 9 Feb 2020, at 19:04, Petter Egesund <petter.egesund(a)gmail.com> wrote: > > > > Yes, thanks for good feedback. > > > > I will try the pooled way, I think - not primarily because of speed, but due to that our library is built around prepared connections. > > > > Petter > > > > On Sun, Feb 9, 2020 at 6:01 PM Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > Hi Petter, > > > > > On 9 Feb 2020, at 17:27, Petter Egesund <petter.egesund(a)gmail.com> wrote: > > > > > > Hi Sven and thanks for answering! > > > > > > I use Teapot with one common sql-connecton, not one for each user session. At startup I create several sql statements and these does not seem to be usable from different Teapot request at the same time. > > > > It is useable, but not concurrently. > > > > > I could create one connection pr. session and then close the database connection when the user leaves, but then I also would need to create all the prepared sql-statements for each session, which does not sound right to me? > > > > Yes and no, see further. > > > > > It seems I have will have to look into the other solution, and see if I can use some mutex stuff to avoid several tasks acessessing the same resources at the same time. > > > > Yes, follow the pointer that I gave you, it is not hard. > > > > > Creating a pool sounds like the right solution to me now - any meaning about this? > > > > Yes, you could create a connection pool. But that is harder than it sounds: what is the minimum size, the maximum size, what do you do when you go over it, how do you make sure that a resource (connection) is clean when returning it to the pool (given authentication, possible errors), ... > > > > I don't know what you are doing, but I think you focus too much on performance issues. I would first try to get the code correct and worry about performance later on. > > > > Running > > > > P3ClientTest new runBenchmark1Bench. > > > > on my machine gives me a BenchmarkResult(67 iterations in 5 seconds 57 milliseconds. 13.249 per second). This is a query that returns 10.000 records with 5 columns. It is reusing the same client/connection for all iterations. > > > > If I modify this slightly to use a new client/connection each time, like this > > > > [ > > (P3Client url: 'psql://sven@localhost') > > query: 'SELECT * FROM benchmark1'; > > close > > ] benchFor: 5 seconds > > > > I get a BenchmarkResult(65 iterations in 5 seconds 34 milliseconds. 12.912 per second) which almost as fast. Of course, for smaller queries, the connect/disconnect overhead will be more significant. > > > > And note that this is not using prepared statements. > > > > So I would start by opening/closing a connection each time you need it. > > > > HTH, > > > > Sven > > > > > Petter > > > > > > > > > On Sun, Feb 9, 2020 at 4:52 PM Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > Hi Petter, > > > > > > [ CC-ing the Pharo Users list ] > > > > > > P3Client is not built/designed to be used by multiple processes concurrently. Each database connection is represented by an instance of P3Client and holds some state both at the client as well as at the server side. > > > > > > Typically, in a multi user server application, each connection should have its own P3Client / psql connection. For example, in Seaside, a custom WASession subclass gives each session/user its own p3 connection/client. > > > > > > Is that what you are doing ? > > > > > > If not, you could wrap your db accessing code so that mutual exclusion is provided. For example, you can have a look at AbstractCache #beThreadSafe and #critical: > > > > > > That will then serialise requests and possibly block one onto the other. > > > > > > HTH, > > > > > > Sven > > > > > > PS: another thing to take care of if closing your sql connections when the session is no longer needed. > > > > > > PS: Zinc HTTP does also provide a session mechanism (ZnServerSession[Manager]) but these work with cookies and typically won't help with a REST access pattern. > > > > > > > On 9 Feb 2020, at 14:21, Petter Egesund <petter.egesund(a)gmail.com> wrote: > > > > > > > > Hi Sven > > > > > > > > We are using Pharo as our backend in a project and we have run into a problem with P3. > > > > > > > > The problem seems to be connected to compiled sql statements and concurrency. > > > > > > > > We keep getting this error: Bindcomplete message expected > > > > > > > > Problem seems to be easy to reproduce: > > > > > > > > 1) Compile any sql statement > > > > 2) Use this statement in a query twice (!) in a teapot endpoint > > > > > > > > The run some concurrent queries, like "curl http://localhost:8080/endpoint & curl http://localhost:8080/endpoint.." (add several curls after here). > > > > > > > > One could also use ex. siege (https://manpages.ubuntu.com/manpages/trusty/man1/siege.1.html) for easy reproducing. > > > > > > > > If we chain the curls after each other, like "curl http://localhost:8080/endpoint && https://manpages.ubuntu.com/manpages/trusty/man1/siege.1.html && https://manpages.ubuntu.com/manpages/trusty/man1/siege.1.html.." it seems to work fine, so doing the request sequentially seem to work fine. > > > > > > > > My conclusion is that this must be connected to how teapot handles concurrency in companion with the compiled statements? > > > > > > > > Any clues on this one? We are on Pharo 8.0 with latest version of P3, PG 9.x) > > > > > > > > Best regards > > > > > > > > Petter Egesund (I wrote the heysql-package based on P3) > > > > > >

Feb. 10, 2020

Re: [Pharo-users] P3 and concurrenty

by Petter Egesund

Hi and thanks for an answer above my expectations to our problem. This is really awesome, I was just about to start coding a pool myself, but getting it directly from the author of the base library is off course way better :) I will try out and get back as soon as we have tested. One quick question - are the prepared statements which are connected to the pool thread safe? Petter On Mon, Feb 10, 2020 at 12:13 PM Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > Hi Petter, > > > https://github.com/svenvc/P3/commit/a6b409d0d92cb92bf9b44452908bb9033523b863 > adds a connection pool. > > Here is the class comment: > > ====== > > I am P3ConnectionPool. > > I offer a pool of shared PSQL connections (P3Client instances) as a > resource. > > After configuring me with at least a url, you use me by calling > #withConnection: > > pool := P3ConnectionPool url: 'psql://sven@localhost'. > pool withConnection: [ :p3Client | > p3Client query: 'SELECT table_name FROM > information_schema.tables' ]. > pool close. > > When a connection is in use, it is not part of the pool. > When a connection is returned/released to the pool, it becomes available > for reuse. > > The pool's size is the number of open connection ready to be reused. > The pool's capacity is the maximum number of connection that will be > pooled. > Excess connections will be closed when they are returned/released to the > pool. > > New connections are created as needed. > You can set a #configurator to further initialize new connections. > You can use #warmUp or #warmUp: to precreate a number of connections. > > When an error occurs, the connection should not be reused and be closed by > the caller. > > ====== > > There are some unit tests as well. > > Let me know if this works for you. > > Sven > > > On 9 Feb 2020, at 19:04, Petter Egesund <petter.egesund(a)gmail.com> > wrote: > > > > Yes, thanks for good feedback. > > > > I will try the pooled way, I think - not primarily because of speed, but > due to that our library is built around prepared connections. > > > > Petter > > > > On Sun, Feb 9, 2020 at 6:01 PM Sven Van Caekenberghe <sven(a)stfx.eu> > wrote: > > Hi Petter, > > > > > On 9 Feb 2020, at 17:27, Petter Egesund <petter.egesund(a)gmail.com> > wrote: > > > > > > Hi Sven and thanks for answering! > > > > > > I use Teapot with one common sql-connecton, not one for each user > session. At startup I create several sql statements and these does not seem > to be usable from different Teapot request at the same time. > > > > It is useable, but not concurrently. > > > > > I could create one connection pr. session and then close the database > connection when the user leaves, but then I also would need to create all > the prepared sql-statements for each session, which does not sound right to > me? > > > > Yes and no, see further. > > > > > It seems I have will have to look into the other solution, and see if > I can use some mutex stuff to avoid several tasks acessessing the same > resources at the same time. > > > > Yes, follow the pointer that I gave you, it is not hard. > > > > > Creating a pool sounds like the right solution to me now - any meaning > about this? > > > > Yes, you could create a connection pool. But that is harder than it > sounds: what is the minimum size, the maximum size, what do you do when you > go over it, how do you make sure that a resource (connection) is clean when > returning it to the pool (given authentication, possible errors), ... > > > > I don't know what you are doing, but I think you focus too much on > performance issues. I would first try to get the code correct and worry > about performance later on. > > > > Running > > > > P3ClientTest new runBenchmark1Bench. > > > > on my machine gives me a BenchmarkResult(67 iterations in 5 seconds 57 > milliseconds. 13.249 per second). This is a query that returns 10.000 > records with 5 columns. It is reusing the same client/connection for all > iterations. > > > > If I modify this slightly to use a new client/connection each time, like > this > > > > [ > > (P3Client url: 'psql://sven@localhost') > > query: 'SELECT * FROM benchmark1'; > > close > > ] benchFor: 5 seconds > > > > I get a BenchmarkResult(65 iterations in 5 seconds 34 milliseconds. > 12.912 per second) which almost as fast. Of course, for smaller queries, > the connect/disconnect overhead will be more significant. > > > > And note that this is not using prepared statements. > > > > So I would start by opening/closing a connection each time you need it. > > > > HTH, > > > > Sven > > > > > Petter > > > > > > > > > On Sun, Feb 9, 2020 at 4:52 PM Sven Van Caekenberghe <sven(a)stfx.eu> > wrote: > > > Hi Petter, > > > > > > [ CC-ing the Pharo Users list ] > > > > > > P3Client is not built/designed to be used by multiple processes > concurrently. Each database connection is represented by an instance of > P3Client and holds some state both at the client as well as at the server > side. > > > > > > Typically, in a multi user server application, each connection should > have its own P3Client / psql connection. For example, in Seaside, a custom > WASession subclass gives each session/user its own p3 connection/client. > > > > > > Is that what you are doing ? > > > > > > If not, you could wrap your db accessing code so that mutual exclusion > is provided. For example, you can have a look at AbstractCache > #beThreadSafe and #critical: > > > > > > That will then serialise requests and possibly block one onto the > other. > > > > > > HTH, > > > > > > Sven > > > > > > PS: another thing to take care of if closing your sql connections when > the session is no longer needed. > > > > > > PS: Zinc HTTP does also provide a session mechanism > (ZnServerSession[Manager]) but these work with cookies and typically won't > help with a REST access pattern. > > > > > > > On 9 Feb 2020, at 14:21, Petter Egesund <petter.egesund(a)gmail.com> > wrote: > > > > > > > > Hi Sven > > > > > > > > We are using Pharo as our backend in a project and we have run into > a problem with P3. > > > > > > > > The problem seems to be connected to compiled sql statements and > concurrency. > > > > > > > > We keep getting this error: Bindcomplete message expected > > > > > > > > Problem seems to be easy to reproduce: > > > > > > > > 1) Compile any sql statement > > > > 2) Use this statement in a query twice (!) in a teapot endpoint > > > > > > > > The run some concurrent queries, like "curl > http://localhost:8080/endpoint & curl http://localhost:8080/endpoint.." > (add several curls after here). > > > > > > > > One could also use ex. siege ( > https://manpages.ubuntu.com/manpages/trusty/man1/siege.1.html) for easy > reproducing. > > > > > > > > If we chain the curls after each other, like "curl > http://localhost:8080/endpoint && > https://manpages.ubuntu.com/manpages/trusty/man1/siege.1.html && > https://manpages.ubuntu.com/manpages/trusty/man1/siege.1.html.." it seems > to work fine, so doing the request sequentially seem to work fine. > > > > > > > > My conclusion is that this must be connected to how teapot handles > concurrency in companion with the compiled statements? > > > > > > > > Any clues on this one? We are on Pharo 8.0 with latest version of > P3, PG 9.x) > > > > > > > > Best regards > > > > > > > > Petter Egesund (I wrote the heysql-package based on P3) > > > > > > >

Feb. 10, 2020

[ANN] Pharo TechTalk Feb 20: The Colony

by Marcus Denker

A regular chat about Pharo. Happens on Discord. This month a bit a special TechTalk, not about Pharo but about a tool that might be interesting for us to use. Topic: The Colony https://colony.io Introduction to "The Colony", a new platform to Organize and Incentivize Collaborators, Contributors, and Communities. Short demo, followed by some Q&A, help to get people onboarded. https://association.pharo.org/event-3697011 20 Feb 2020 â¢ 1:30 PM - 2:30 PM (EST) NOTE: this is EST time, for Europe, this is 19:30 (the easiest is to download the calendar entry)

Feb. 10, 2020

Re: [Pharo-users] Voluntarily cancelling requests ("applying an expiration date")

by Sven Van Caekenberghe

Hi Holger, That is a complicated story ;-) But, you running out of external semaphores means that you are using too many sockets, are not closing/releasing them (in time) and/or your GC does not run enough to keep up (it is easy to deplete the external semaphore table without the GC kicking in). You must have a loop somewhere that goes too fast and maybe does not clean up properly while doing so. YMMV, but I do similar things -- implement/offer REST services that call other REST/network services, all with timeouts, in several variations, for years, and I do not have problems like you describe. I would suggest enabling logging so that you can see better where the allocations happen and if your cleanup code does its work. Sven PS: Zinc logging is easy, just do ZnLogEvent logToTranscript > On 9 Feb 2020, at 16:31, Holger Freyther <holger(a)freyther.de> wrote: > > tl;dr: I am searching for a pattern (later code) to apply expiration to operations. > > > > Introduction: > > One nice aspect of Mongodb is that it has built-in data distribution[1] and configurable retention[2]. The upstream project has a document called "Server Discovery and Monitoring (SDAM)", defining how a client should behave. Martin Dias is currently implementing SDAM in MongoTalk/Voyage and I took it on a test drive. > > > Behavior: > > My software stack is using Zinc, Zinc-REST, Voyage and Mongo. When a new REST requests arrives I am using Voyage (e.g. >>#selectOne:) which will use MongoTalk. The MongoTalk code needs to select the right server. It's currently done by waiting for a result. > > Next I started to simulate database outages. The rest clients retried when not receiving a result within two seconds (no back-off/jitter). What happened was roughly the following: > > > [ > 1.) ZnServer accepts a new connection > 2.) MongoTalk waits for a server longer than 2s > "nothing.. the above waits..." > ] repeat. > > > > > Problem: > > What happened next surprised me. I expected to have a bad time when the database recovers and all the stale (remember the REST clients already gave up and closed the socket) requests will be answered. Instead my image crashed early in my test as the ExternalSemaphoreTable was full. > > Let's focus on the timeout behavior and discuss the existence of the ExternalSemaphoreTable and the number of entries separately at a different time. > > > > > To me the two main problems I see are: > > > 1.) Lack of back-pressure for ZnManagingMultiThreadedServer > > 2.) Disconnect of time between the Application Layer handling REST is allowed to take and down the stack how long MongoTalk may sleep and wait for a server. > > > The first item is difficult. Even answering HTTP 500 when we are out of space in the ExternalSemaphore is difficult... Let's ignore this for now as well. > > > > > > > What I look for: > > > 1.) Voluntarily Timeout > > Inside my Application code I would like to tag an operation with a timeout. This means everything that is done should complete within X seconds. It can be used on a voluntarily basis. > > >>> #lookupPerson > > "We expect all database operations to complete within two seconds" > person := ComputeContext current withTimeout: 2 seconds during: [ > repository selectOne: Person where: [:each name | ...], > ]. > > > > MongoTalk>>stuff > "See if the outer context timeout has expired and signal. E.g. before writing > something into the socket to keep consistency." > ComputeContext current checkExpired. > > > MongoTalk>>other > "Sleep for up to the remaining time out > (someSemaphore waitTimeoutContext: ComputeContext current) ifFalse: [ > SomethingExpired signal. > ] > > > > 2.) Cancellation > > > More difficult to write in pseudo code (without TaskIt?). In my above case we are waiting for the database to be ready while the client already closed the file descriptor. Now we are not able to see this until much later. > > The idea is that in addition to the timeout we can pass a block that is called when an operation should be cancelled and the ComputeContext can be checked if something has been cancelled? > > > > > The above takes inspiration from Go's context package[3]. In Go the context should be passed as parameter but we could make it a Process variable? > > > > > > Question: > > How do you handle this in your systems? Is this something we can consider for Pharo9? > > > > thanks > holger > > > > > > > > > [1] It has the concept of "replicationSet" and works by having a primary, secondary and arbiters running. > [2] For every write one can configure if the write should succeed immediately (before it is even on disk) or when it has been written to multiple stores (e.g. majority, US and EMEA) > [3] https://golang.org/pkg/context/ > > >

Feb. 10, 2020

Re: [Pharo-users] The results are in!

by Guillermo Polito

Hi Ben, > El 9 feb 2020, a las 21:01, Ben Coman <btc(a)openinworld.com> escribiÃ³: > > > This list is for people to *ask* questions about Pharo. Not to diffuse any kind of propaganda (pharo related or not). > > This list is not just for asking questions. There are plenty of [ANN] posts advertising new libraries > and blogs articles. Following completion of the competition, the MIT licensed code repository will be announced here. > Hopefully it may provide an engaging exercise for Pharo teachers to their with their students. I know, and I donât mind announcements from time to time. But announcements are generally marked with [ANN], and are kind of self explanatory. The mail Richard sent is, and Iâm sorry to be harsh, spam. > > I have the feeling this list has been lately flooded with such off-topic messages. > > Unfortunately the original post was quite bare and failed to show how it was Pharo-topical. > So I can understand how it triggered you. I hope the additional background info provided here alleviates some of your concern about it. Maybe. Maybe Richard could work his emails a bit more? Iâll quote his original email: > > El 8 feb 2020, a las 19:12, Richard Kenneth Eng <horrido.hobbies(a)gmail.com <mailto:horrido.hobbies@gmail.com>> escribiÃ³: > > > > Round 1 â #1 Leading Team: https://youtu.be/QWHeN5WXfBQ <https://youtu.be/QWHeN5WXfBQ> > > > > I'm actually quite amazed by their effort. They surpassed my expectations. > > > > At the risk of sounding immodest, I think this is a terrific way to promote Smalltalk (Pharo). I think the video is an absolute blast. > > > > Richard I particularly, who hasnât followed the competition, am completely lost with this email. With the risk of sounding harsh, again, but trying to give some feedback Richard can work on: - The subject does not imply it is an announcement, nor what the results are related to or how they got there - For those that are not following this competition: what is this round 1? Are there many of them? What is this about? are there many teams? - There is no single link to the competition website, what this is about, whoever contributed to it... - Clicking on the link gives a video with a title that is as ambiguous as this email :( - I check the description of the video: what is there in the video? not clear. How will it appear in a youtube search in the future? At least it seems there are links to the website... Yet, Really? do I have to watch the video before I can know if this is interesting for me or not? :P Happily these points could be addressed in the future. > So with this additional perspective, may one post a week be allowed to share the progress of the competition? Can the posts have better quality? :/ Still, Iâd feel more confortable with an alternative, such as using discord. What about having a dedicated #jmrpc channel on discord where people could ask questions too? What about twitter? I havent seen this on twitter. Is it there? Now, I already spent too much on this :) Time to get back to something productive. Cheers, Guille

Feb. 10, 2020

Re: [Pharo-users] The results are in!

by Ben Coman

On Mon, 10 Feb 2020 at 08:01, horrido <horrido.hobbies(a)gmail.com> wrote: > > > Although the "smalltalk" title of the video is not ideal, that was the > > name of the competition when it was originally going to use VisualWorks, > > and so it carried over. > > Not only that, but if you look at jrmpc.ca, it uses the word "Smalltalk" > all > over the place. The sponsors include LabWare (who doesn't use Pharo), > Simberon (who doesn't use Pharo), and TSUG (Toronto *Smalltalk* User > Group). > It's "Smalltalk" everywhere! That is what it is. Can't change it. The sponsors are being very generous. And that's fine because Pharo is a Smalltalk. > But pushing that line here may generate further controversy around the competition that will not help anyones cause. There are other forums you can do that. Here lets focus on the "Pharo" aspects of the competition. :) cheers -ben

Feb. 10, 2020

Re: [Pharo-users] P3 and concurrenty

by Sven Van Caekenberghe

Hi Petter, https://github.com/svenvc/P3/commit/a6b409d0d92cb92bf9b44452908bb9033523b863 adds a connection pool. Here is the class comment: ====== I am P3ConnectionPool. I offer a pool of shared PSQL connections (P3Client instances) as a resource. After configuring me with at least a url, you use me by calling #withConnection: pool := P3ConnectionPool url: 'psql://sven@localhost'. pool withConnection: [ :p3Client | p3Client query: 'SELECT table_name FROM information_schema.tables' ]. pool close. When a connection is in use, it is not part of the pool. When a connection is returned/released to the pool, it becomes available for reuse. The pool's size is the number of open connection ready to be reused. The pool's capacity is the maximum number of connection that will be pooled. Excess connections will be closed when they are returned/released to the pool. New connections are created as needed. You can set a #configurator to further initialize new connections. You can use #warmUp or #warmUp: to precreate a number of connections. When an error occurs, the connection should not be reused and be closed by the caller. ====== There are some unit tests as well. Let me know if this works for you. Sven > On 9 Feb 2020, at 19:04, Petter Egesund <petter.egesund(a)gmail.com> wrote: > > Yes, thanks for good feedback. > > I will try the pooled way, I think - not primarily because of speed, but due to that our library is built around prepared connections. > > Petter > > On Sun, Feb 9, 2020 at 6:01 PM Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > Hi Petter, > > > On 9 Feb 2020, at 17:27, Petter Egesund <petter.egesund(a)gmail.com> wrote: > > > > Hi Sven and thanks for answering! > > > > I use Teapot with one common sql-connecton, not one for each user session. At startup I create several sql statements and these does not seem to be usable from different Teapot request at the same time. > > It is useable, but not concurrently. > > > I could create one connection pr. session and then close the database connection when the user leaves, but then I also would need to create all the prepared sql-statements for each session, which does not sound right to me? > > Yes and no, see further. > > > It seems I have will have to look into the other solution, and see if I can use some mutex stuff to avoid several tasks acessessing the same resources at the same time. > > Yes, follow the pointer that I gave you, it is not hard. > > > Creating a pool sounds like the right solution to me now - any meaning about this? > > Yes, you could create a connection pool. But that is harder than it sounds: what is the minimum size, the maximum size, what do you do when you go over it, how do you make sure that a resource (connection) is clean when returning it to the pool (given authentication, possible errors), ... > > I don't know what you are doing, but I think you focus too much on performance issues. I would first try to get the code correct and worry about performance later on. > > Running > > P3ClientTest new runBenchmark1Bench. > > on my machine gives me a BenchmarkResult(67 iterations in 5 seconds 57 milliseconds. 13.249 per second). This is a query that returns 10.000 records with 5 columns. It is reusing the same client/connection for all iterations. > > If I modify this slightly to use a new client/connection each time, like this > > [ > (P3Client url: 'psql://sven@localhost') > query: 'SELECT * FROM benchmark1'; > close > ] benchFor: 5 seconds > > I get a BenchmarkResult(65 iterations in 5 seconds 34 milliseconds. 12.912 per second) which almost as fast. Of course, for smaller queries, the connect/disconnect overhead will be more significant. > > And note that this is not using prepared statements. > > So I would start by opening/closing a connection each time you need it. > > HTH, > > Sven > > > Petter > > > > > > On Sun, Feb 9, 2020 at 4:52 PM Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > Hi Petter, > > > > [ CC-ing the Pharo Users list ] > > > > P3Client is not built/designed to be used by multiple processes concurrently. Each database connection is represented by an instance of P3Client and holds some state both at the client as well as at the server side. > > > > Typically, in a multi user server application, each connection should have its own P3Client / psql connection. For example, in Seaside, a custom WASession subclass gives each session/user its own p3 connection/client. > > > > Is that what you are doing ? > > > > If not, you could wrap your db accessing code so that mutual exclusion is provided. For example, you can have a look at AbstractCache #beThreadSafe and #critical: > > > > That will then serialise requests and possibly block one onto the other. > > > > HTH, > > > > Sven > > > > PS: another thing to take care of if closing your sql connections when the session is no longer needed. > > > > PS: Zinc HTTP does also provide a session mechanism (ZnServerSession[Manager]) but these work with cookies and typically won't help with a REST access pattern. > > > > > On 9 Feb 2020, at 14:21, Petter Egesund <petter.egesund(a)gmail.com> wrote: > > > > > > Hi Sven > > > > > > We are using Pharo as our backend in a project and we have run into a problem with P3. > > > > > > The problem seems to be connected to compiled sql statements and concurrency. > > > > > > We keep getting this error: Bindcomplete message expected > > > > > > Problem seems to be easy to reproduce: > > > > > > 1) Compile any sql statement > > > 2) Use this statement in a query twice (!) in a teapot endpoint > > > > > > The run some concurrent queries, like "curl http://localhost:8080/endpoint & curl http://localhost:8080/endpoint.." (add several curls after here). > > > > > > One could also use ex. siege (https://manpages.ubuntu.com/manpages/trusty/man1/siege.1.html) for easy reproducing. > > > > > > If we chain the curls after each other, like "curl http://localhost:8080/endpoint && https://manpages.ubuntu.com/manpages/trusty/man1/siege.1.html && https://manpages.ubuntu.com/manpages/trusty/man1/siege.1.html.." it seems to work fine, so doing the request sequentially seem to work fine. > > > > > > My conclusion is that this must be connected to how teapot handles concurrency in companion with the compiled statements? > > > > > > Any clues on this one? We are on Pharo 8.0 with latest version of P3, PG 9.x) > > > > > > Best regards > > > > > > Petter Egesund (I wrote the heysql-package based on P3) > > >

Feb. 10, 2020

Re: [Pharo-users] The results are in!

by horrido

Thanks, Ben. > Although the "smalltalk" title of the video is not ideal, that was the > name of the competition when it was originally going to use VisualWorks, > and so it carried over. Not only that, but if you look at jrmpc.ca, it uses the word "Smalltalk" all over the place. The sponsors include LabWare (who doesn't use Pharo), Simberon (who doesn't use Pharo), and TSUG (Toronto *Smalltalk* User Group). It's "Smalltalk" everywhere! And that's fine because Pharo is a Smalltalk. Ben Coman wrote > Hi Guille, > > Full disclosure, I've spent the past 6 months using my discretionary hobby > time > programming the infrastructure for this competition using Pharo, so my > viewpoint may be biased. > > On Sun, 9 Feb 2020 at 03:25, Guillermo Polito < > guillermopolito@ > > > wrote: >> >> Hi Richard, >> >> Iâd like to invite you to refrain yourself from posting such off-topic > messages in the future. >> >> This list is for people to *ask* questions about Pharo. Not to diffuse > any kind of propaganda (pharo related or not). > > This list is not just for asking questions. There are plenty of [ANN] > posts advertising new libraries > and blogs articles. Following completion of the competition, the MIT > licensed code repository will be announced here. > Hopefully it may provide an engaging exercise for Pharo teachers to their > with their students. > > >> I have the feeling this list has been lately flooded with such off-topic > messages. > > Unfortunately the original post was quite bare and failed to show how it > was Pharo-topical. > So I can understand how it triggered you. I hope the additional > background > info provided here alleviates some of your concern about it. > > >> I know you donât agree, but I invite you to read here the **purpose** of > this list, just for respect to people subscribed to it: >> https://lists.pharo.org/mailman/listinfo/pharo-users_lists.pharo.org > > The purpose I see there seems fairly generic... "This is the general > mailing list for users of Pharo." > > The competition is programmed in Pharo and Pharo is promoted later in the > video with screens like... > > [image: image.png] > [image: image.png] > > But that promotion is not the main thing. The main thing is simply that > this is-A competition programmed using Pharo, > and as such this list seems reasonable a reasonable place discuss it. > Although the "smalltalk" title of the video is not ideal, > that was the name of the competition when it was originally going to use > VisualWorks, and so it carried over. > > A final thing to consider is how welcomed the dozens of competing students > may feel if they are watching this list > and see their Pharo based competition spurned by the community. > > So with this additional perspective, may one post a week be allowed to > share the progress of the competition? > > cheers -ben > > >> El 8 feb 2020, a las 19:12, Richard Kenneth Eng < > horrido.hobbies@ > > > escribiÃ³: >> >> Round 1 â #1 Leading Team: https://youtu.be/QWHeN5WXfBQ >> >> I'm actually quite amazed by their effort. They surpassed my >> expectations. >> >> At the risk of sounding immodest, I think this is a terrific way to > promote Smalltalk (Pharo). I think the video is an absolute blast. >> >> Richard >> >> > > > image.png (28K) > <http://forum.world.st/attachment/5111441/0/image.png> > image.png (91K) > <http://forum.world.st/attachment/5111441/1/image.png> -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Feb. 10, 2020

Re: [Pharo-users] [Browser] object is deprecated

by Steve Quezadas

Thank you. I see that this opens Nautilus. Is Calypso the Pharo8 equivalent? How do I find the class of the Calypso browser? Do I "halo click" a Calypso browser, I assume? And then click the wrench thing? What area of the browser do I click on to get the name of the object it is from? On Sun, Feb 9, 2020 at 9:44 AM Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > Hi Steve > > Smalltalk tools browser open > > Ð²Ñ, 9 ÑÐµÐ²Ñ. 2020 Ð³. Ð² 17:37, Steve Quezadas <steveeq1(a)gmail.com>: > >> Hello, >> >> I am porting a package into Pharo 8. One of the objects is referring to >> object [Browser], but that does not seem to present in Pharo8 and >> deprecated in Pharo5. What is the modern replacement for [Browser]? >> >> Also, is there a "smaltalk" way to find out recommended replacements for >> deprecated objects? >> >> - Steve >> >> PS How do you open object [Browser] in Pharo5, so I can at least visually >> see it? "Browser new open" and "Browser new openInWorld" both don't work. >> >

Feb. 9, 2020