October 2021 - Pharo-dev - lists.pharo.org

Re: Something weird in VirtualMachine>>#maxExternalSemaphores:

by Henrik Sperre Johansen

Seens it used to be in the Â«oldÂ» PharoVM before the reconcilliation: https://forum.world.st/external-semaphores-again-tp4712934p4713396.html Maybe itâs still possible to find it in the git history of the current pharovm repository? Cheers, Henry On Wednesday, October 13, 2021, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > Hi Henry, > > Thanks for your reply. > > I created an issue: https://github.com/pharo-project/pharo/issues/10156 > > Can we still find the file 'sqExternalSemaphores.c' from the old emails ? > > Sven > > > On 12 Oct 2021, at 19:20, Henrik Sperre Johansen < > henrik.s.johansen(a)veloxit.no> wrote: > > > > I believe Igor Stasenko made changes so signals could no longer get > lost, but Iâm not sure it ever got integrated in Cogâ¦ > > > > https://forum.world.st/Issue-5965-in-pharo-Can-t-allocate- > semaphores-VirtualMachine-maxExternalSemaphores-aSize- > Not-enough--tp4631607p4632388.html > > > > Maybe worth revisiting/reviewing? > > If it works, the limitation would be gone, and the image warnings could > be removed with no ill side-effects. > > > > Cheers, > > Henry > > > >> On 12 Oct 2021, at 18:55, Henrik Sperre Johansen < > henrik.s.johansen(a)veloxit.no> wrote: > >> > >> ï»¿Hi Sven - youâre correct the main intent was to make it clear one > should not really be setting this from the image during runtime - and make > it as obvious as possible when it had. > >> > >> The way I remember it, the Error was only raised if the image is in > development mode - so a developer running into this, would have the limit > raised, but also get a debugger (which is hard to ignore) explaining the > issue, while in runtime mode, the SystemNotification would get handled by > logging so it would at least be possible to spot that it had happened. > >> > >> If thereâs a less confusing way to achieve that it would be a good > thing, but Iâm not sure removing all warnings would be the best way, as the > related i/o problems would be nearly impossible to determine the root cause > of. > >> > >> Why forked? No idea/canât rememberâ¦ > >> > >> Cheers, > >> Henry > >> > >>> On 12 Oct 2021, at 16:01, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > >>> > >>> ï»¿Hi, > >>> > >>> The ExternalSemaphoreTable is an important resource since it holds 3 > Semaphores per Socket (network connection). The size of > ExternalSemaphoreTable determines how many such resources can be used > concurrently. > >>> > >>> VM parameter 49 holds this limit, it is accessed by VirtualMachine>>#maxExternalSemaphores. > The default for Pharo 9 seems to be 256 currently. > >>> > >>> There is a method VirtualMachine>>#maxExternalSemaphoresSilently: > that can be used to set this limit higher. This seems to work. > >>> > >>> Now, there are some limitations here: the number must be a power of 2, > higher that the one set before and lower than 64K. BTW the number of file > descriptors for files and sockets is also limited by the OS for its > processes. > >>> > >>> Also, while growing the table, the Semaphores become unavailable to > the VM for a very short period of time, so they could miss signals, leading > to IO issues. > >>> > >>> For this reason, it seems that the design goal was to not allow this > to grow during normal use, only during startup or upfront configuration. > >>> > >>> Now comes the weird thing: the implementation/code of VirtualMachine>># > maxExternalSemaphores: > >>> > >>> This is the method called when the current table is too small and when > it wants to grow larger than the current VirtualMachine>># > maxExternalSemaphores. > >>> > >>> The implementation calls #maxExternalSemaphoresSilently: in a forked > thread, after signalling a SystemNotification with a full explanation. But > why should this be forked in the first place ? > >>> > >>> Then in the main thread, it signals an error ! Presumably because the > design goals mentioned before. > >>> > >>> These two actions are in conflict. > >>> > >>> All this is probably the result of well meant evolution and > refactoring, but it is quite confusing. > >>> > >>> My first idea would be: let the caller raise the error and don't try > raising the limit, remove the current implementation and replace it by what > is now #maxExternalSemaphoresSilently: > >>> > >>> What do you think ? > >>> > >>> Sven > >>> > >>> > >>> > >>> >

Oct. 13, 2021

Re: Something weird in VirtualMachine>>#maxExternalSemaphores:

by Sven Van Caekenberghe

Hi Henry, Thanks for your reply. I created an issue: https://github.com/pharo-project/pharo/issues/10156 Can we still find the file 'sqExternalSemaphores.c' from the old emails ? Sven > On 12 Oct 2021, at 19:20, Henrik Sperre Johansen <henrik.s.johansen(a)veloxit.no> wrote: > > I believe Igor Stasenko made changes so signals could no longer get lost, but Iâm not sure it ever got integrated in Cogâ¦ > > https://forum.world.st/Issue-5965-in-pharo-Can-t-allocate-semaphores-Virtua… > > Maybe worth revisiting/reviewing? > If it works, the limitation would be gone, and the image warnings could be removed with no ill side-effects. > > Cheers, > Henry > >> On 12 Oct 2021, at 18:55, Henrik Sperre Johansen <henrik.s.johansen(a)veloxit.no> wrote: >> >> ï»¿Hi Sven - youâre correct the main intent was to make it clear one should not really be setting this from the image during runtime - and make it as obvious as possible when it had. >> >> The way I remember it, the Error was only raised if the image is in development mode - so a developer running into this, would have the limit raised, but also get a debugger (which is hard to ignore) explaining the issue, while in runtime mode, the SystemNotification would get handled by logging so it would at least be possible to spot that it had happened. >> >> If thereâs a less confusing way to achieve that it would be a good thing, but Iâm not sure removing all warnings would be the best way, as the related i/o problems would be nearly impossible to determine the root cause of. >> >> Why forked? No idea/canât rememberâ¦ >> >> Cheers, >> Henry >> >>> On 12 Oct 2021, at 16:01, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>> >>> ï»¿Hi, >>> >>> The ExternalSemaphoreTable is an important resource since it holds 3 Semaphores per Socket (network connection). The size of ExternalSemaphoreTable determines how many such resources can be used concurrently. >>> >>> VM parameter 49 holds this limit, it is accessed by VirtualMachine>>#maxExternalSemaphores. The default for Pharo 9 seems to be 256 currently. >>> >>> There is a method VirtualMachine>>#maxExternalSemaphoresSilently: that can be used to set this limit higher. This seems to work. >>> >>> Now, there are some limitations here: the number must be a power of 2, higher that the one set before and lower than 64K. BTW the number of file descriptors for files and sockets is also limited by the OS for its processes. >>> >>> Also, while growing the table, the Semaphores become unavailable to the VM for a very short period of time, so they could miss signals, leading to IO issues. >>> >>> For this reason, it seems that the design goal was to not allow this to grow during normal use, only during startup or upfront configuration. >>> >>> Now comes the weird thing: the implementation/code of VirtualMachine>>#maxExternalSemaphores: >>> >>> This is the method called when the current table is too small and when it wants to grow larger than the current VirtualMachine>>#maxExternalSemaphores. >>> >>> The implementation calls #maxExternalSemaphoresSilently: in a forked thread, after signalling a SystemNotification with a full explanation. But why should this be forked in the first place ? >>> >>> Then in the main thread, it signals an error ! Presumably because the design goals mentioned before. >>> >>> These two actions are in conflict. >>> >>> All this is probably the result of well meant evolution and refactoring, but it is quite confusing. >>> >>> My first idea would be: let the caller raise the error and don't try raising the limit, remove the current implementation and replace it by what is now #maxExternalSemaphoresSilently: >>> >>> What do you think ? >>> >>> Sven >>> >>> >>> >>>

Oct. 13, 2021

Re: Something weird in VirtualMachine>>#maxExternalSemaphores:

by Henrik Sperre Johansen

I believe Igor Stasenko made changes so signals could no longer get lost, but Iâm not sure it ever got integrated in Cogâ¦ https://forum.world.st/Issue-5965-in-pharo-Can-t-allocate-semaphores-Virtua… Maybe worth revisiting/reviewing? If it works, the limitation would be gone, and the image warnings could be removed with no ill side-effects. Cheers, Henry > On 12 Oct 2021, at 18:55, Henrik Sperre Johansen <henrik.s.johansen(a)veloxit.no> wrote: > > ï»¿Hi Sven - youâre correct the main intent was to make it clear one should not really be setting this from the image during runtime - and make it as obvious as possible when it had. > > The way I remember it, the Error was only raised if the image is in development mode - so a developer running into this, would have the limit raised, but also get a debugger (which is hard to ignore) explaining the issue, while in runtime mode, the SystemNotification would get handled by logging so it would at least be possible to spot that it had happened. > > If thereâs a less confusing way to achieve that it would be a good thing, but Iâm not sure removing all warnings would be the best way, as the related i/o problems would be nearly impossible to determine the root cause of. > > Why forked? No idea/canât rememberâ¦ > > Cheers, > Henry > >> On 12 Oct 2021, at 16:01, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> >> ï»¿Hi, >> >> The ExternalSemaphoreTable is an important resource since it holds 3 Semaphores per Socket (network connection). The size of ExternalSemaphoreTable determines how many such resources can be used concurrently. >> >> VM parameter 49 holds this limit, it is accessed by VirtualMachine>>#maxExternalSemaphores. The default for Pharo 9 seems to be 256 currently. >> >> There is a method VirtualMachine>>#maxExternalSemaphoresSilently: that can be used to set this limit higher. This seems to work. >> >> Now, there are some limitations here: the number must be a power of 2, higher that the one set before and lower than 64K. BTW the number of file descriptors for files and sockets is also limited by the OS for its processes. >> >> Also, while growing the table, the Semaphores become unavailable to the VM for a very short period of time, so they could miss signals, leading to IO issues. >> >> For this reason, it seems that the design goal was to not allow this to grow during normal use, only during startup or upfront configuration. >> >> Now comes the weird thing: the implementation/code of VirtualMachine>>#maxExternalSemaphores: >> >> This is the method called when the current table is too small and when it wants to grow larger than the current VirtualMachine>>#maxExternalSemaphores. >> >> The implementation calls #maxExternalSemaphoresSilently: in a forked thread, after signalling a SystemNotification with a full explanation. But why should this be forked in the first place ? >> >> Then in the main thread, it signals an error ! Presumably because the design goals mentioned before. >> >> These two actions are in conflict. >> >> All this is probably the result of well meant evolution and refactoring, but it is quite confusing. >> >> My first idea would be: let the caller raise the error and don't try raising the limit, remove the current implementation and replace it by what is now #maxExternalSemaphoresSilently: >> >> What do you think ? >> >> Sven >> >> >> >>

Oct. 12, 2021

Re: Something weird in VirtualMachine>>#maxExternalSemaphores:

by Henrik Sperre Johansen

Hi Sven - youâre correct the main intent was to make it clear one should not really be setting this from the image during runtime - and make it as obvious as possible when it had. The way I remember it, the Error was only raised if the image is in development mode - so a developer running into this, would have the limit raised, but also get a debugger (which is hard to ignore) explaining the issue, while in runtime mode, the SystemNotification would get handled by logging so it would at least be possible to spot that it had happened. If thereâs a less confusing way to achieve that it would be a good thing, but Iâm not sure removing all warnings would be the best way, as the related i/o problems would be nearly impossible to determine the root cause of. Why forked? No idea/canât rememberâ¦ Cheers, Henry > On 12 Oct 2021, at 16:01, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > ï»¿Hi, > > The ExternalSemaphoreTable is an important resource since it holds 3 Semaphores per Socket (network connection). The size of ExternalSemaphoreTable determines how many such resources can be used concurrently. > > VM parameter 49 holds this limit, it is accessed by VirtualMachine>>#maxExternalSemaphores. The default for Pharo 9 seems to be 256 currently. > > There is a method VirtualMachine>>#maxExternalSemaphoresSilently: that can be used to set this limit higher. This seems to work. > > Now, there are some limitations here: the number must be a power of 2, higher that the one set before and lower than 64K. BTW the number of file descriptors for files and sockets is also limited by the OS for its processes. > > Also, while growing the table, the Semaphores become unavailable to the VM for a very short period of time, so they could miss signals, leading to IO issues. > > For this reason, it seems that the design goal was to not allow this to grow during normal use, only during startup or upfront configuration. > > Now comes the weird thing: the implementation/code of VirtualMachine>>#maxExternalSemaphores: > > This is the method called when the current table is too small and when it wants to grow larger than the current VirtualMachine>>#maxExternalSemaphores. > > The implementation calls #maxExternalSemaphoresSilently: in a forked thread, after signalling a SystemNotification with a full explanation. But why should this be forked in the first place ? > > Then in the main thread, it signals an error ! Presumably because the design goals mentioned before. > > These two actions are in conflict. > > All this is probably the result of well meant evolution and refactoring, but it is quite confusing. > > My first idea would be: let the caller raise the error and don't try raising the limit, remove the current implementation and replace it by what is now #maxExternalSemaphoresSilently: > > What do you think ? > > Sven > > > >

Oct. 12, 2021

Re: Bloc and Spec2

by Martin Dias

Hello, Both are libraries for coding user interfaces, but at different abstraction levels. In few words, Spec2 provides widgets such as button, checkbox and table, while Bloc provides more low-level visual elements such as rectangle, circle and text. However, there is another library named Brick, that provides widgets based on Bloc, which is more comparable. But Spec2 is still "more abstract" than Brick: By default, a Spec UI opens using Morphic's widgets (which come with Pharo), but you could load other backends such as GTK or Brick to open "the same UI" using other widgets. I've been working in a backend for Brick, still quite incomplete, in: https://github.com/pharo-graphics/Spec-Brick Regards, MartÃn On Tue, Oct 12, 2021 at 11:39 AM o lu via Pharo-dev < pharo-dev(a)lists.pharo.org> wrote: > What is the difference between Bloc and Spec2? > > Sent with ProtonMail <https://protonmail.com/> Secure Email. >

Oct. 12, 2021

Bloc and Spec2

by o lu

What is the difference between Bloc and Spec2? Sent with [ProtonMail](https://protonmail.com/) Secure Email.

Oct. 12, 2021

Something weird in VirtualMachine>>#maxExternalSemaphores:

by Sven Van Caekenberghe

Hi, The ExternalSemaphoreTable is an important resource since it holds 3 Semaphores per Socket (network connection). The size of ExternalSemaphoreTable determines how many such resources can be used concurrently. VM parameter 49 holds this limit, it is accessed by VirtualMachine>>#maxExternalSemaphores. The default for Pharo 9 seems to be 256 currently. There is a method VirtualMachine>>#maxExternalSemaphoresSilently: that can be used to set this limit higher. This seems to work. Now, there are some limitations here: the number must be a power of 2, higher that the one set before and lower than 64K. BTW the number of file descriptors for files and sockets is also limited by the OS for its processes. Also, while growing the table, the Semaphores become unavailable to the VM for a very short period of time, so they could miss signals, leading to IO issues. For this reason, it seems that the design goal was to not allow this to grow during normal use, only during startup or upfront configuration. Now comes the weird thing: the implementation/code of VirtualMachine>>#maxExternalSemaphores: This is the method called when the current table is too small and when it wants to grow larger than the current VirtualMachine>>#maxExternalSemaphores. The implementation calls #maxExternalSemaphoresSilently: in a forked thread, after signalling a SystemNotification with a full explanation. But why should this be forked in the first place ? Then in the main thread, it signals an error ! Presumably because the design goals mentioned before. These two actions are in conflict. All this is probably the result of well meant evolution and refactoring, but it is quite confusing. My first idea would be: let the caller raise the error and don't try raising the limit, remove the current implementation and replace it by what is now #maxExternalSemaphoresSilently: What do you think ? Sven

Oct. 12, 2021

[Pharo 8.0] Build #1146: Fix invalid (non-Boolean) return when CPU watcher is enabled

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #1146 was: FAILURE. The Pull Request #10134 was integrated: "Fix invalid (non-Boolean) return when CPU watcher is enabled" Pull request url: https://github.com/pharo-project/pharo/pull/10134 Issue Url: https://github.com/pharo-project/pharo/issues/Pharo8.0 Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

Oct. 12, 2021

Re: Hardening Zinc's Core HTTP Server

by Sven Van Caekenberghe

Here is an update on the status of both instances. A couple of people tried fiddling with invalid input which did result in exceptions, but everything was handled correctly so that the server kept on functioning. There were no spurious crashes. There was one attack by Levente Uzonyi that was successful though. He used the fact that Seaside sessions are long lived (cached for a certain time) combined with the fact that the Reddit.st app kept one GLORP database connection through P3 open to PostgreSQL and fired off a small denial of service (DOS) attack. Eventually either the VM ran out of space in the ExternalSemaphoreTable, making Socket creation, either for database connections or for handling new HTTP request impossible or the database's own connection limit was reached. At that point the HTTP server or the Pharo VM stopped functioning. Although Levente used a sequential number of requests, a concurrent number of request could cause similar problems (related, but differently). In looking at what he reported it also became clear that I accidentally left a debugging exception handler active, which is not good in a production image. Both instances are now redeployed with the following changes: - the Reddit.st code was changed to not keep the database connection open all the time, but instead connect/disconnect per request. this might be a bit slower, but it conserves resources much better and solves the original issue Levente reported https://github.com/svenvc/Reddit/commit/f2e0a0dc00b9cbb68cfa4fb007906365ae6… - a new feature was added to Zinc HTTP Components' ZnManagingMultiThreadedServer (the default) to enforce a maximum number of concurrent connections being allowed (the limit is 32 by default, but changeable if you know what you are doing). when the limit is reached, 503 Service Unavailable responses are sent to the excess clients and the connection is closed. this should help protect against concurrent connection DOS attacks https://github.com/svenvc/zinc/commit/ac0f06e74e7ab129610c466cb1d7ea9533d29… - the deploy script was changed to use the more primitive WAErrorHandler https://github.com/svenvc/Reddit/commit/874b631e6dc0c04c8c0b687ef770d00540d… Thanks again to Levente for taking the time to try an attack and for reporting it clearly. Sven > On 29 Sep 2021, at 17:10, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > Both instances have been up for 5 days now, looking for more testers. > >> On 23 Sep 2021, at 17:03, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> >> Hi, >> >> Zinc HTTP Components [https://github.com/svenvc/zinc] has been a part of Pharo since version 1.3 (2011). It is an open-source framework to deal with the HTTP networking protocol, modelling all aspects involved. It also offers both client and server functionality. >> >> The reliability of the code base has improved steadily over the years, thanks to virtually all Pharo developers using it, directly or indirectly. Over the summer a number of issues that popped up after Pharo 9 was released were resolved. >> >> The robustness of the core HTTP server is one important aspect. To put this quality further to the test, I deployed two servers with the same demo Seaside application, Reddit.st, open to the internet, without any further protections. >> >> - On Amazon AWS: http://34.245.183.130:1701 >> >> - On Microsoft Azure: http://51.137.72.94:8080 >> >> The application's source code can be found at [https://github.com/svenvc/Reddit] For the technically curious there are also deploy instructions at [https://github.com/svenvc/Reddit/blob/main/DEPLOY.md] The demo app itself is described in an older article [https://medium.com/@svenvc/reddit-st-in-10-cool-pharo-classes-1b5327ca0740] Note that, by definition, there is no HTTPS/TLS variant. >> >> If you manage to break this server with (a) malicious request(s) in such a way that you can explain what you did for others to confirm your approach, you not only help me/us improve the code, but earn eternal fame as well ;-) >> >> Sven >> >> PS: I hope I won't regret this, I am looking for constructive criticism. >> >> >> -- >> Sven Van Caekenberghe >> Proudly supporting Pharo >> http://pharo.org >> http://association.pharo.org >> http://consortium.pharo.org >> >

Oct. 11, 2021

This week (40/2021) on the Pharo Issue Tracker MD

by Marcus Denker

We merged 23 PRs and fixed 19 Issue Tracker entries. Pharo9 ====== - Backport #10111: isReferenced-is-not-working-for-PoolVariables #10128 https://github.com/pharo-project/pharo/pull/10128 - 10123-Failing-test-Pharo9-ReleaseTesttestWorldMenuHasHelpForAllEntries #10127 https://github.com/pharo-project/pharo/pull/10127 - 10121-backport-Pharo9-Fix-failing-tests-in-CI-ZnHTTPSTesttestGForceInria #10122 https://github.com/pharo-project/pharo/pull/10122 - Fixes #10037. Now it only register world menu entries if the class "marked with praâ¦ #10055 https://github.com/pharo-project/pharo/pull/10055 - 9927-Check-Pharo9-Defining-variables-in-the-debugger-DNU-isForScripting-9925 #10120 https://github.com/pharo-project/pharo/pull/10120 - 10116-Backport-Pharo9-fixing-fuel-platform-dependency-10109 #10119 https://github.com/pharo-project/pharo/pull/10119 Fixes ===== - 10110-isReferenced-is-not-working-for-PoolVariables #10111 https://github.com/pharo-project/pharo/pull/10111 - Added two missing methods: tfPointerAddress https://github.com/pharo-project/pharo/pull/10099 - fixing-fuel-platform-dependency #10109 https://github.com/pharo-project/pharo/pull/10109 - 10067-CDClassDefinitionParserparseSelectorPartwithArgument-misses-subclass-definition-symbols #10092 https://github.com/pharo-project/pharo/pull/10092 - Fix failing tests in CI: ZnHTTPSTest>>testGForceInria #10091 https://github.com/pharo-project/pharo/pull/10091 Cleanups ========= - Cleanup-Unused-gtInspector-Rules #10112 https://github.com/pharo-project/pharo/pull/10112 - 10078-To-check-CompiledMethodlocalReadsSelf-does-isFFIMethod-check #10086 https://github.com/pharo-project/pharo/pull/10086 - ClassParser: Small Cleanup and Test for CompiledMethod #10097 https://github.com/pharo-project/pharo/pull/10097 - Deprecation-Rewrites-Ring #10115 https://github.com/pharo-project/pharo/pull/10115 - 10025-ObjectLayout-classlayoutForType-should-be-moved-to-the-Monticello-package #10087 https://github.com/pharo-project/pharo/pull/10087 - Cleanup Changeset class #10103 https://github.com/pharo-project/pharo/pull/10103 - Small cleanups: ByteString #10101 https://github.com/pharo-project/pharo/pull/10101 - Cleanup Color class #10105 https://github.com/pharo-project/pharo/pull/10105 - Deprecate #beginsWithEmpty:caseSensitive. Users should use beginsWithâ¦ #10016 https://github.com/pharo-project/pharo/pull/10016 - Fix #9907: improes DoesNotUnderstand exception's messageText #9982 https://github.com/pharo-project/pharo/pull/9982 - Use Jenkins git plugin #7141 https://github.com/pharo-project/pharo/pull/7141 - Small cleanups - AbstractWorldRenderer hierarchy #10082 https://github.com/pharo-project/pharo/pull/10082

Oct. 11, 2021