As I need an image that runs longer than 24 hours I'm looking at some stuff and wonder. Can anybody explain me the rationale for a code like this maxExternalSemaphores: aSize "This method should never be called as result of normal program execution. If it is however, handle it differently: - In development, signal an error to promt user to set a bigger size at startup immediately. - In production, accept the cost of potentially unhandled interrupts, but log the action for later review. See comment in maxExternalObjectsSilently: why this behaviour is desirable, " "Can't find a place where development/production is decided. Suggest Smalltalk image inProduction, but use an overridable temp meanwhile. " | inProduction | self maxExternalSemaphores ifNil: [^ 0]. inProduction := false. ^ inProduction ifTrue: [self maxExternalSemaphoresSilently: aSize. self crTrace: 'WARNING: Had to increase size of semaphore signal handling table due to many external objects concurrently in use'; crTrace: 'You should increase this size at startup using #maxExternalObjectsSilently:'; crTrace: 'Current table size: ' , self maxExternalSemaphores printString] ifFalse: ["Smalltalk image" self error: 'Not enough space for external objects, set a larger size at startup!' "Smalltalk image"] I have reported this once but got no feedback so I like to have a few opinions. The report is here: https://pharo.fogbugz.com/f/cases/10839/ Norbert
On Oct 7, 2013, at 11:16 , Norbert Hartl <norbert@hartl.name> wrote:
As I need an image that runs longer than 24 hours I'm looking at some stuff and wonder. Can anybody explain me the rationale for a code like this
maxExternalSemaphores: aSize "This method should never be called as result of normal program execution. If it is however, handle it differently: - In development, signal an error to promt user to set a bigger size at startup immediately. - In production, accept the cost of potentially unhandled interrupts, but log the action for later review.
See comment in maxExternalObjectsSilently: why this behaviour is desirable, " "Can't find a place where development/production is decided. Suggest Smalltalk image inProduction, but use an overridable temp meanwhile. " | inProduction | self maxExternalSemaphores ifNil: [^ 0]. inProduction := false. ^ inProduction ifTrue: [self maxExternalSemaphoresSilently: aSize. self crTrace: 'WARNING: Had to increase size of semaphore signal handling table due to many external objects concurrently in use'; crTrace: 'You should increase this size at startup using #maxExternalObjectsSilently:'; crTrace: 'Current table size: ' , self maxExternalSemaphores printString] ifFalse: ["Smalltalk image" self error: 'Not enough space for external objects, set a larger size at startup!' "Smalltalk image"]
I have reported this once but got no feedback so I like to have a few opinions.
The report is here: https://pharo.fogbugz.com/f/cases/10839/
Norbert
The rationale is that inProduction would be some global setting, not yet in place when the code was written⦠Excessive simultaneous Semaphore usage is something that should be caught during development, in which case it's better to get an active notification, than having it logged somewhere. If I've understood correctly, it's moot on newer Pharo VM's, where there's no limit on the semtable size, but for legacy code a startup item setting size using maxExternalObjectsSilently: (as suggested in the Warning text), is still a more proper fix than setting inProduction to true and crossing your fingers hoping no signals will be lost during table growth. Cheers. Henrik
On Oct 7, 2013, at 11:28 AM, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
On Oct 7, 2013, at 11:16 , Norbert Hartl <norbert@hartl.name> wrote:
I have reported this once but got no feedback so I like to have a few opinions.
The report is here: https://pharo.fogbugz.com/f/cases/10839/
Norbert
The rationale is that inProduction would be some global setting, not yet in place when the code was written⦠Excessive simultaneous Semaphore usage is something that should be caught during development, in which case it's better to get an active notification, than having it logged somewhere. If I've understood correctly, it's moot on newer Pharo VM's, where there's no limit on the semtable size, but for legacy code a startup item setting size using maxExternalObjectsSilently: (as suggested in the Warning text), is still a more proper fix than setting inProduction to true and crossing your fingers hoping no signals will be lost during table growth.
The problem with this "set this secret flag in deployment" is that nobody knows it. *And* I am convinced that *everything* that is different in development to deployment will come back and hurt you *hart* when you least expect it. Marcus
Am 07.10.2013 um 11:28 schrieb Henrik Johansen <henrik.s.johansen@veloxit.no>:
On Oct 7, 2013, at 11:16 , Norbert Hartl <norbert@hartl.name> wrote:
As I need an image that runs longer than 24 hours I'm looking at some stuff and wonder. Can anybody explain me the rationale for a code like this
maxExternalSemaphores: aSize "This method should never be called as result of normal program execution. If it is however, handle it differently: - In development, signal an error to promt user to set a bigger size at startup immediately. - In production, accept the cost of potentially unhandled interrupts, but log the action for later review.
See comment in maxExternalObjectsSilently: why this behaviour is desirable, " "Can't find a place where development/production is decided. Suggest Smalltalk image inProduction, but use an overridable temp meanwhile. " | inProduction | self maxExternalSemaphores ifNil: [^ 0]. inProduction := false. ^ inProduction ifTrue: [self maxExternalSemaphoresSilently: aSize. self crTrace: 'WARNING: Had to increase size of semaphore signal handling table due to many external objects concurrently in use'; crTrace: 'You should increase this size at startup using #maxExternalObjectsSilently:'; crTrace: 'Current table size: ' , self maxExternalSemaphores printString] ifFalse: ["Smalltalk image" self error: 'Not enough space for external objects, set a larger size at startup!' "Smalltalk image"]
I have reported this once but got no feedback so I like to have a few opinions.
The report is here: https://pharo.fogbugz.com/f/cases/10839/
Norbert
The rationale is that inProduction would be some global setting, not yet in place when the code was written⦠Excessive simultaneous Semaphore usage is something that should be caught during development, in which case it's better to get an active notification, than having it logged somewhere.
Agreed. But didn't work in my case because it needed roughly 20 hours and an instable remote backend to trigger the problem. And somehow I forgot to install my logger as Transcript so there is no warning message. I saw only dead images in the morning. This not satisfactory but on the other hand this type of problems are hard to solve anyway. My feeling tells me there is more to discover. Sockets resources get unregistered at finalization time but this didn't work either. I would have said that the unlikely situation that no garbage collection ran could be the case. But it can't because in ExternalSemaphoreTable>>#freedSlotsIn:ratherThanIncreaseSizeTo: there is explicit garbage collection.
If I've understood correctly, it's moot on newer Pharo VM's, where there's no limit on the semtable size, but for legacy code a startup item setting size using maxExternalObjectsSilently: (as suggested in the Warning text), is still a more proper fix than setting inProduction to true and crossing your fingers hoping no signals will be lost during table growth.
Ah, I didn't know about the risk of loosing signals while resizing the table. Thanks for that. Don't get me wrong I wasn't proposing to set inProduction in effect. I don't think that automatically growing resource management is a proper way to design a system. There is always a range of resources you need for your use case. Not setting an upper bound for this just covers leaking behavior. Norbert
1 thing. can you tell me what given expression yields for your VM/image: Smalltalk vm maxExternalSemaphores (if it gives you number less than 10000000 then i think i know what is your problem :) i just found that after one merge, my changes get lost we're just plugged them back in, and it should be back again with newer VMs.. but the problem could be more than just semaphores.. if merge broken this, it may break many other things, so we need time to check On 7 October 2013 12:31, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 11:28 schrieb Henrik Johansen < henrik.s.johansen@veloxit.no>:
On Oct 7, 2013, at 11:16 , Norbert Hartl <norbert@hartl.name> wrote:
As I need an image that runs longer than 24 hours I'm looking at some stuff and wonder. Can anybody explain me the rationale for a code like this
maxExternalSemaphores: aSize "This method should never be called as result of normal program execution. If it is however, handle it differently: - In development, signal an error to promt user to set a bigger size at startup immediately. - In production, accept the cost of potentially unhandled interrupts, but log the action for later review. See comment in maxExternalObjectsSilently: why this behaviour is desirable, " "Can't find a place where development/production is decided. Suggest Smalltalk image inProduction, but use an overridable temp meanwhile. " | inProduction | self maxExternalSemaphores ifNil: [^ 0]. inProduction := false. ^ inProduction ifTrue: [self maxExternalSemaphoresSilently: aSize. self crTrace: 'WARNING: Had to increase size of semaphore signal handling table due to many external objects concurrently in use'; crTrace: 'You should increase this size at startup using #maxExternalObjectsSilently:'; crTrace: 'Current table size: ' , self maxExternalSemaphores printString] ifFalse: ["Smalltalk image" self error: 'Not enough space for external objects, set a larger size at startup!' "Smalltalk image"]
I have reported this once but got no feedback so I like to have a few opinions.
The report is here: https://pharo.fogbugz.com/f/cases/10839/
Norbert
The rationale is that inProduction would be some global setting, not yet in place when the code was written⦠Excessive simultaneous Semaphore usage is something that should be caught during development, in which case it's better to get an active notification, than having it logged somewhere.
Agreed. But didn't work in my case because it needed roughly 20 hours and an instable remote backend to trigger the problem. And somehow I forgot to install my logger as Transcript so there is no warning message. I saw only dead images in the morning. This not satisfactory but on the other hand this type of problems are hard to solve anyway. My feeling tells me there is more to discover. Sockets resources get unregistered at finalization time but this didn't work either. I would have said that the unlikely situation that no garbage collection ran could be the case. But it can't because in ExternalSemaphoreTable>>#freedSlotsIn:ratherThanIncreaseSizeTo: there is explicit garbage collection.
If I've understood correctly, it's moot on newer Pharo VM's, where there's no limit on the semtable size, but for legacy code a startup item setting size using maxExternalObjectsSilently: (as suggested in the Warning text), is still a more proper fix than setting inProduction to true and crossing your fingers hoping no signals will be lost during table growth.
Ah, I didn't know about the risk of loosing signals while resizing the table. Thanks for that. Don't get me wrong I wasn't proposing to set inProduction in effect. I don't think that automatically growing resource management is a proper way to design a system. There is always a range of resources you need for your use case. Not setting an upper bound for this just covers leaking behavior.
Norbert
-- Best regards, Igor Stasenko.
On Oct 7, 2013, at 4:36 , Igor Stasenko <siguctua@gmail.com> wrote:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :)
Uhm, I didn't read your implementation, but the proper thing to do for a VM which doesn't care about the external objects table (splObjs at: 39) growing, is returning nil from maxExternalSemaphores, like it will for pre-Cog VM's. Otherwise, you'll get a domain error when you eventually (yes,yes, it'll take awhile) run out, and maxExternalSemaphoresSilently: tries to set the new max to a value that won't fit in the 16bit header field where it expects to put it⦠Cheers, Henry
On 7 October 2013 17:25, Henrik Johansen <henrik.s.johansen@veloxit.no>wrote:
On Oct 7, 2013, at 4:36 , Igor Stasenko <siguctua@gmail.com> wrote:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :)
Uhm, I didn't read your implementation, but the proper thing to do for a VM which doesn't care about the external objects table (splObjs at: 39) growing, is returning nil from maxExternalSemaphores, like it will for pre-Cog VM's.
or.. release VM + image, where image aware of current VM capabilities, and refuses to run on older ones, and get rid of all this stuff altogether. i changed the low-level implementation (C level), but not primitive level. sure thing more cleanup & polishing can be done, but the changes i made never get reviewed and properly tested (though, there was no problems reported since year from when it was introduced, except from last unfortunate rollback to Eliot's original implementation).
Otherwise, you'll get a domain error when you eventually (yes,yes, it'll take awhile) run out, and maxExternalSemaphoresSilently: tries to set the new max to a value that won't fit in the 16bit header field where it expects to put itâ¦
yes.. theoretically.. but i have gut feeling that your image/setup will experience a lot of other different problems (memory consumption/amount of available OS handles etc), before you get to that point. And at that point, you will need to modify VM/OS/image anyways (or figure how to work around limits in other way)..
Cheers, Henry
-- Best regards, Igor Stasenko.
On Oct 7, 2013, at 5:40 , Igor Stasenko <siguctua@gmail.com> wrote:
On 7 October 2013 17:25, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
On Oct 7, 2013, at 4:36 , Igor Stasenko <siguctua@gmail.com> wrote:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :)
Uhm, I didn't read your implementation, but the proper thing to do for a VM which doesn't care about the external objects table (splObjs at: 39) growing, is returning nil from maxExternalSemaphores, like it will for pre-Cog VM's.
or.. release VM + image, where image aware of current VM capabilities, and refuses to run on older ones, and get rid of all this stuff altogether.
i changed the low-level implementation (C level), but not primitive level. sure thing more cleanup & polishing can be done, but the changes i made never get reviewed and properly tested (though, there was no problems reported since year from when it was introduced, except from last unfortunate rollback to Eliot's original implementation).
I know, which is why I thought you might like the feedback :P AFAICT, it would be nice to couple the changes you made with changing parameterAt: 49 back to returning nil, as well as returning an error for parameterAt: 49 put: that the section is not in use/obsolete, instead of returning bogus values. Though, sounds a bit strange how parameterAt: 49 can return a huge number without any changes to the related code (which would then still be reading from a 16bit field :P)
Otherwise, you'll get a domain error when you eventually (yes,yes, it'll take awhile) run out, and maxExternalSemaphoresSilently: tries to set the new max to a value that won't fit in the 16bit header field where it expects to put itâ¦
yes.. theoretically.. but i have gut feeling that your image/setup will experience a lot of other different problems (memory consumption/amount of available OS handles etc), before you get to that point. And at that point, you will need to modify VM/OS/image anyways (or figure how to work around limits in other way)..
Yap, I guess you could say changing to a nil return would be more a change to play by established conventions when reporting capabilities/needs of the VM than anything with practical impact. Cheers, Henry
On 7 October 2013 18:01, Henrik Johansen <henrik.s.johansen@veloxit.no>wrote:
On Oct 7, 2013, at 5:40 , Igor Stasenko <siguctua@gmail.com> wrote:
On 7 October 2013 17:25, Henrik Johansen <henrik.s.johansen@veloxit.no>wrote:
On Oct 7, 2013, at 4:36 , Igor Stasenko <siguctua@gmail.com> wrote:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :)
Uhm, I didn't read your implementation, but the proper thing to do for a VM which doesn't care about the external objects table (splObjs at: 39) growing, is returning nil from maxExternalSemaphores, like it will for pre-Cog VM's.
or.. release VM + image, where image aware of current VM capabilities, and refuses to run on older ones, and get rid of all this stuff altogether.
i changed the low-level implementation (C level), but not primitive level. sure thing more cleanup & polishing can be done, but the changes i made never get reviewed and properly tested (though, there was no problems reported since year from when it was introduced, except from last unfortunate rollback to Eliot's original implementation).
I know, which is why I thought you might like the feedback :P AFAICT, it would be nice to couple the changes you made with changing parameterAt: 49 back to returning nil, as well as returning an error for parameterAt: 49 put: that the section is not in use/obsolete, instead of returning bogus values.
Though, sounds a bit strange how parameterAt: 49 can return a huge number without any changes to the related code (which would then still be reading from a 16bit field :P)
Otherwise, you'll get a domain error when you eventually (yes,yes, it'll take awhile) run out, and maxExternalSemaphoresSilently: tries to set the new max to a value that won't fit in the 16bit header field where it expects to put itâ¦
yes.. theoretically.. but i have gut feeling that your image/setup will experience a lot of other different problems (memory consumption/amount of available OS handles etc), before you get to that point. And at that point, you will need to modify VM/OS/image anyways (or figure how to work around limits in other way)..
Yap, I guess you could say changing to a nil return would be more a change to play by established conventions when reporting capabilities/needs of the VM than anything with practical impact.
i agree. i will create a bug entry for it then..
Cheers, Henry
-- Best regards, Igor Stasenko.
Just a note: in a Pharo20, the number is 256, no matter if in OSX or Windows (8 here). In Pharo30 with vmLatest I get the Smalltalk vm maxExternalSemaphores 10000000 indeed. Phil On Tue, Oct 8, 2013 at 11:15 AM, Igor Stasenko <siguctua@gmail.com> wrote:
On 7 October 2013 18:01, Henrik Johansen <henrik.s.johansen@veloxit.no>wrote:
On Oct 7, 2013, at 5:40 , Igor Stasenko <siguctua@gmail.com> wrote:
On 7 October 2013 17:25, Henrik Johansen <henrik.s.johansen@veloxit.no>wrote:
On Oct 7, 2013, at 4:36 , Igor Stasenko <siguctua@gmail.com> wrote:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :)
Uhm, I didn't read your implementation, but the proper thing to do for a VM which doesn't care about the external objects table (splObjs at: 39) growing, is returning nil from maxExternalSemaphores, like it will for pre-Cog VM's.
or.. release VM + image, where image aware of current VM capabilities, and refuses to run on older ones, and get rid of all this stuff altogether.
i changed the low-level implementation (C level), but not primitive level. sure thing more cleanup & polishing can be done, but the changes i made never get reviewed and properly tested (though, there was no problems reported since year from when it was introduced, except from last unfortunate rollback to Eliot's original implementation).
I know, which is why I thought you might like the feedback :P AFAICT, it would be nice to couple the changes you made with changing parameterAt: 49 back to returning nil, as well as returning an error for parameterAt: 49 put: that the section is not in use/obsolete, instead of returning bogus values.
Though, sounds a bit strange how parameterAt: 49 can return a huge number without any changes to the related code (which would then still be reading from a 16bit field :P)
Otherwise, you'll get a domain error when you eventually (yes,yes, it'll take awhile) run out, and maxExternalSemaphoresSilently: tries to set the new max to a value that won't fit in the 16bit header field where it expects to put itâ¦
yes.. theoretically.. but i have gut feeling that your image/setup will experience a lot of other different problems (memory consumption/amount of available OS handles etc), before you get to that point. And at that point, you will need to modify VM/OS/image anyways (or figure how to work around limits in other way)..
Yap, I guess you could say changing to a nil return would be more a change to play by established conventions when reporting capabilities/needs of the VM than anything with practical impact.
i agree. i will create a bug entry for it then..
Cheers, Henry
-- Best regards, Igor Stasenko.
Am 07.10.2013 um 16:36 schrieb Igor Stasenko <siguctua@gmail.com>:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :)
It is 10000000 What would be the problem if it would be smaller?
i just found that after one merge, my changes get lost we're just plugged them back in, and it should be back again with newer VMs.. but the problem could be more than just semaphores.. if merge broken this, it may break many other things, so we need time to check
I try to look at it some more time. I'm using the pharo-vm from the launchpad build. Are the changes supposed to be in this one? Norbert
On 7 October 2013 12:31, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 11:28 schrieb Henrik Johansen <henrik.s.johansen@veloxit.no>:
On Oct 7, 2013, at 11:16 , Norbert Hartl <norbert@hartl.name> wrote:
As I need an image that runs longer than 24 hours I'm looking at some stuff and wonder. Can anybody explain me the rationale for a code like this
maxExternalSemaphores: aSize "This method should never be called as result of normal program execution. If it is however, handle it differently: - In development, signal an error to promt user to set a bigger size at startup immediately. - In production, accept the cost of potentially unhandled interrupts, but log the action for later review.
See comment in maxExternalObjectsSilently: why this behaviour is desirable, " "Can't find a place where development/production is decided. Suggest Smalltalk image inProduction, but use an overridable temp meanwhile. " | inProduction | self maxExternalSemaphores ifNil: [^ 0]. inProduction := false. ^ inProduction ifTrue: [self maxExternalSemaphoresSilently: aSize. self crTrace: 'WARNING: Had to increase size of semaphore signal handling table due to many external objects concurrently in use'; crTrace: 'You should increase this size at startup using #maxExternalObjectsSilently:'; crTrace: 'Current table size: ' , self maxExternalSemaphores printString] ifFalse: ["Smalltalk image" self error: 'Not enough space for external objects, set a larger size at startup!' "Smalltalk image"]
I have reported this once but got no feedback so I like to have a few opinions.
The report is here: https://pharo.fogbugz.com/f/cases/10839/
Norbert
The rationale is that inProduction would be some global setting, not yet in place when the code was written⦠Excessive simultaneous Semaphore usage is something that should be caught during development, in which case it's better to get an active notification, than having it logged somewhere.
Agreed. But didn't work in my case because it needed roughly 20 hours and an instable remote backend to trigger the problem. And somehow I forgot to install my logger as Transcript so there is no warning message. I saw only dead images in the morning. This not satisfactory but on the other hand this type of problems are hard to solve anyway. My feeling tells me there is more to discover. Sockets resources get unregistered at finalization time but this didn't work either. I would have said that the unlikely situation that no garbage collection ran could be the case. But it can't because in ExternalSemaphoreTable>>#freedSlotsIn:ratherThanIncreaseSizeTo: there is explicit garbage collection.
If I've understood correctly, it's moot on newer Pharo VM's, where there's no limit on the semtable size, but for legacy code a startup item setting size using maxExternalObjectsSilently: (as suggested in the Warning text), is still a more proper fix than setting inProduction to true and crossing your fingers hoping no signals will be lost during table growth.
Ah, I didn't know about the risk of loosing signals while resizing the table. Thanks for that. Don't get me wrong I wasn't proposing to set inProduction in effect. I don't think that automatically growing resource management is a proper way to design a system. There is always a range of resources you need for your use case. Not setting an upper bound for this just covers leaking behavior.
Norbert
-- Best regards, Igor Stasenko.
On 7 October 2013 18:36, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 16:36 schrieb Igor Stasenko <siguctua@gmail.com>:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :)
It is 10000000
What would be the problem if it would be smaller?
that just means your VM don't have external object size cap. I changed the implementation to not have hard limit (the arbitrary large number is there just to be "compatible" with previous implementation). This means, that you can actually change in your image the check and completely ignore limits and just keep growing if it necessary. Now, since you using VM which don't have a limit, but problem still persists, it seems like it somewhere else.. :/ i just found that after one merge, my changes get lost we're just plugged them back in, and it should be back again with newer VMs.. but the problem could be more than just semaphores.. if merge broken this, it may break many other things, so we need time to check I try to look at it some more time. I'm using the pharo-vm from the launchpad build. Are the changes supposed to be in this one? Norbert Launchpad? You mean ppa? I can't say i remember all the details how changes to VM source gets into ppa distro, and how fast they get there. @Damien, can you enlighten us? Well, the VM which i downloaded recently using zero-conf script, having limit back to 256. Just some merge mistake, which now is fixed.. means that couple builds will use limit-based implementation.. but then it will be back to my implementaiton. On 7 October 2013 12:31, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 11:28 schrieb Henrik Johansen < henrik.s.johansen@veloxit.no>:
On Oct 7, 2013, at 11:16 , Norbert Hartl <norbert@hartl.name> wrote:
As I need an image that runs longer than 24 hours I'm looking at some stuff and wonder. Can anybody explain me the rationale for a code like this
maxExternalSemaphores: aSize "This method should never be called as result of normal program execution. If it is however, handle it differently: - In development, signal an error to promt user to set a bigger size at startup immediately. - In production, accept the cost of potentially unhandled interrupts, but log the action for later review. See comment in maxExternalObjectsSilently: why this behaviour is desirable, " "Can't find a place where development/production is decided. Suggest Smalltalk image inProduction, but use an overridable temp meanwhile. " | inProduction | self maxExternalSemaphores ifNil: [^ 0]. inProduction := false. ^ inProduction ifTrue: [self maxExternalSemaphoresSilently: aSize. self crTrace: 'WARNING: Had to increase size of semaphore signal handling table due to many external objects concurrently in use'; crTrace: 'You should increase this size at startup using #maxExternalObjectsSilently:'; crTrace: 'Current table size: ' , self maxExternalSemaphores printString] ifFalse: ["Smalltalk image" self error: 'Not enough space for external objects, set a larger size at startup!' "Smalltalk image"]
I have reported this once but got no feedback so I like to have a few opinions.
The report is here: https://pharo.fogbugz.com/f/cases/10839/
Norbert
The rationale is that inProduction would be some global setting, not yet in place when the code was written⦠Excessive simultaneous Semaphore usage is something that should be caught during development, in which case it's better to get an active notification, than having it logged somewhere.
Agreed. But didn't work in my case because it needed roughly 20 hours and an instable remote backend to trigger the problem. And somehow I forgot to install my logger as Transcript so there is no warning message. I saw only dead images in the morning. This not satisfactory but on the other hand this type of problems are hard to solve anyway. My feeling tells me there is more to discover. Sockets resources get unregistered at finalization time but this didn't work either. I would have said that the unlikely situation that no garbage collection ran could be the case. But it can't because in ExternalSemaphoreTable>>#freedSlotsIn:ratherThanIncreaseSizeTo: there is explicit garbage collection.
If I've understood correctly, it's moot on newer Pharo VM's, where there's no limit on the semtable size, but for legacy code a startup item setting size using maxExternalObjectsSilently: (as suggested in the Warning text), is still a more proper fix than setting inProduction to true and crossing your fingers hoping no signals will be lost during table growth.
Ah, I didn't know about the risk of loosing signals while resizing the table. Thanks for that. Don't get me wrong I wasn't proposing to set inProduction in effect. I don't think that automatically growing resource management is a proper way to design a system. There is always a range of resources you need for your use case. Not setting an upper bound for this just covers leaking behavior.
Norbert
-- Best regards, Igor Stasenko. -- Best regards, Igor Stasenko.
I do have a version #20619 with a VM from get.pharo.org (built Oct 2, jenkins #14731) which gives me 256... Is it bad Mr Stasenko ? (maybe no wonder I had socket issues when scraping a lot of things) Phil On Tue, Oct 8, 2013 at 11:27 AM, Igor Stasenko <siguctua@gmail.com> wrote:
On 7 October 2013 18:36, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 16:36 schrieb Igor Stasenko <siguctua@gmail.com>:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :)
It is 10000000
What would be the problem if it would be smaller?
that just means your VM don't have external object size cap. I changed the implementation to not have hard limit (the arbitrary large number is there just to be "compatible" with previous implementation).
This means, that you can actually change in your image the check and completely ignore limits and just keep growing if it necessary.
Now, since you using VM which don't have a limit, but problem still persists, it seems like it somewhere else.. :/
i just found that after one merge, my changes get lost we're just plugged them back in, and it should be back again with newer VMs.. but the problem could be more than just semaphores.. if merge broken this, it may break many other things, so we need time to check
I try to look at it some more time. I'm using the pharo-vm from the launchpad build. Are the changes supposed to be in this one?
Norbert
Launchpad? You mean ppa? I can't say i remember all the details how changes to VM source gets into ppa distro, and how fast they get there. @Damien, can you enlighten us?
Well, the VM which i downloaded recently using zero-conf script, having limit back to 256. Just some merge mistake, which now is fixed.. means that couple builds will use limit-based implementation.. but then it will be back to my implementaiton.
On 7 October 2013 12:31, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 11:28 schrieb Henrik Johansen < henrik.s.johansen@veloxit.no>:
On Oct 7, 2013, at 11:16 , Norbert Hartl <norbert@hartl.name> wrote:
As I need an image that runs longer than 24 hours I'm looking at some stuff and wonder. Can anybody explain me the rationale for a code like this
maxExternalSemaphores: aSize "This method should never be called as result of normal program execution. If it is however, handle it differently: - In development, signal an error to promt user to set a bigger size at startup immediately. - In production, accept the cost of potentially unhandled interrupts, but log the action for later review. See comment in maxExternalObjectsSilently: why this behaviour is desirable, " "Can't find a place where development/production is decided. Suggest Smalltalk image inProduction, but use an overridable temp meanwhile. " | inProduction | self maxExternalSemaphores ifNil: [^ 0]. inProduction := false. ^ inProduction ifTrue: [self maxExternalSemaphoresSilently: aSize. self crTrace: 'WARNING: Had to increase size of semaphore signal handling table due to many external objects concurrently in use'; crTrace: 'You should increase this size at startup using #maxExternalObjectsSilently:'; crTrace: 'Current table size: ' , self maxExternalSemaphores printString] ifFalse: ["Smalltalk image" self error: 'Not enough space for external objects, set a larger size at startup!' "Smalltalk image"]
I have reported this once but got no feedback so I like to have a few opinions.
The report is here: https://pharo.fogbugz.com/f/cases/10839/
Norbert
The rationale is that inProduction would be some global setting, not yet in place when the code was written⦠Excessive simultaneous Semaphore usage is something that should be caught during development, in which case it's better to get an active notification, than having it logged somewhere.
Agreed. But didn't work in my case because it needed roughly 20 hours and an instable remote backend to trigger the problem. And somehow I forgot to install my logger as Transcript so there is no warning message. I saw only dead images in the morning. This not satisfactory but on the other hand this type of problems are hard to solve anyway. My feeling tells me there is more to discover. Sockets resources get unregistered at finalization time but this didn't work either. I would have said that the unlikely situation that no garbage collection ran could be the case. But it can't because in ExternalSemaphoreTable>>#freedSlotsIn:ratherThanIncreaseSizeTo: there is explicit garbage collection.
If I've understood correctly, it's moot on newer Pharo VM's, where there's no limit on the semtable size, but for legacy code a startup item setting size using maxExternalObjectsSilently: (as suggested in the Warning text), is still a more proper fix than setting inProduction to true and crossing your fingers hoping no signals will be lost during table growth.
Ah, I didn't know about the risk of loosing signals while resizing the table. Thanks for that. Don't get me wrong I wasn't proposing to set inProduction in effect. I don't think that automatically growing resource management is a proper way to design a system. There is always a range of resources you need for your use case. Not setting an upper bound for this just covers leaking behavior.
Norbert
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
On 8 October 2013 11:45, phil@highoctane.be <phil@highoctane.be> wrote:
I do have a version #20619 with a VM from get.pharo.org (built Oct 2, jenkins #14731) which gives me 256...
Is it bad Mr Stasenko ? (maybe no wonder I had socket issues when scraping a lot of things)
Well, it is not that bad unless you hit the limit because either to too many open connections, or too lazy/wasteful/errorneous management of them :) But please, search the year-old discussion about it, if you want to know more. Phil
On Tue, Oct 8, 2013 at 11:27 AM, Igor Stasenko <siguctua@gmail.com> wrote:
On 7 October 2013 18:36, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 16:36 schrieb Igor Stasenko <siguctua@gmail.com>:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :)
It is 10000000
What would be the problem if it would be smaller?
that just means your VM don't have external object size cap. I changed the implementation to not have hard limit (the arbitrary large number is there just to be "compatible" with previous implementation).
This means, that you can actually change in your image the check and completely ignore limits and just keep growing if it necessary.
Now, since you using VM which don't have a limit, but problem still persists, it seems like it somewhere else.. :/
i just found that after one merge, my changes get lost we're just plugged them back in, and it should be back again with newer VMs.. but the problem could be more than just semaphores.. if merge broken this, it may break many other things, so we need time to check
I try to look at it some more time. I'm using the pharo-vm from the launchpad build. Are the changes supposed to be in this one?
Norbert
Launchpad? You mean ppa? I can't say i remember all the details how changes to VM source gets into ppa distro, and how fast they get there. @Damien, can you enlighten us?
Well, the VM which i downloaded recently using zero-conf script, having limit back to 256. Just some merge mistake, which now is fixed.. means that couple builds will use limit-based implementation.. but then it will be back to my implementaiton.
On 7 October 2013 12:31, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 11:28 schrieb Henrik Johansen < henrik.s.johansen@veloxit.no>:
On Oct 7, 2013, at 11:16 , Norbert Hartl <norbert@hartl.name> wrote:
As I need an image that runs longer than 24 hours I'm looking at some stuff and wonder. Can anybody explain me the rationale for a code like this
maxExternalSemaphores: aSize "This method should never be called as result of normal program execution. If it is however, handle it differently: - In development, signal an error to promt user to set a bigger size at startup immediately. - In production, accept the cost of potentially unhandled interrupts, but log the action for later review. See comment in maxExternalObjectsSilently: why this behaviour is desirable, " "Can't find a place where development/production is decided. Suggest Smalltalk image inProduction, but use an overridable temp meanwhile. " | inProduction | self maxExternalSemaphores ifNil: [^ 0]. inProduction := false. ^ inProduction ifTrue: [self maxExternalSemaphoresSilently: aSize. self crTrace: 'WARNING: Had to increase size of semaphore signal handling table due to many external objects concurrently in use'; crTrace: 'You should increase this size at startup using #maxExternalObjectsSilently:'; crTrace: 'Current table size: ' , self maxExternalSemaphores printString] ifFalse: ["Smalltalk image" self error: 'Not enough space for external objects, set a larger size at startup!' "Smalltalk image"]
I have reported this once but got no feedback so I like to have a few opinions.
The report is here: https://pharo.fogbugz.com/f/cases/10839/
Norbert
The rationale is that inProduction would be some global setting, not yet in place when the code was written⦠Excessive simultaneous Semaphore usage is something that should be caught during development, in which case it's better to get an active notification, than having it logged somewhere.
Agreed. But didn't work in my case because it needed roughly 20 hours and an instable remote backend to trigger the problem. And somehow I forgot to install my logger as Transcript so there is no warning message. I saw only dead images in the morning. This not satisfactory but on the other hand this type of problems are hard to solve anyway. My feeling tells me there is more to discover. Sockets resources get unregistered at finalization time but this didn't work either. I would have said that the unlikely situation that no garbage collection ran could be the case. But it can't because in ExternalSemaphoreTable>>#freedSlotsIn:ratherThanIncreaseSizeTo: there is explicit garbage collection.
If I've understood correctly, it's moot on newer Pharo VM's, where there's no limit on the semtable size, but for legacy code a startup item setting size using maxExternalObjectsSilently: (as suggested in the Warning text), is still a more proper fix than setting inProduction to true and crossing your fingers hoping no signals will be lost during table growth.
Ah, I didn't know about the risk of loosing signals while resizing the table. Thanks for that. Don't get me wrong I wasn't proposing to set inProduction in effect. I don't think that automatically growing resource management is a proper way to design a system. There is always a range of resources you need for your use case. Not setting an upper bound for this just covers leaking behavior.
Norbert
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
On Oct 8, 2013, at 11:27 , Igor Stasenko <siguctua@gmail.com> wrote:
On 7 October 2013 18:36, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 16:36 schrieb Igor Stasenko <siguctua@gmail.com>:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :)
It is 10000000
What would be the problem if it would be smaller?
that just means your VM don't have external object size cap. I changed the implementation to not have hard limit (the arbitrary large number is there just to be "compatible" with previous implementation).
This means, that you can actually change in your image the check and completely ignore limits and just keep growing if it necessary.
⦠it also means you'll never trigger the GC which is part of table growth, which would finalize connections. (Which is how it's been since until recently anyways, right?) I'd reckon that would be taken care of by the retryWithGC: when creating Sockets if you run into plugin failures due to lacking OS resources triggering finalize instead, but I can't say I'm sure how the plugin behaves in that case⦠A stress test checking graceful/expected behavior of the socket plugin when whatever limits exists are reached would probably be nice :) Cheers, Henry
Hi Igor, On Tue, Oct 8, 2013 at 2:27 AM, Igor Stasenko <siguctua@gmail.com> wrote:
On 7 October 2013 18:36, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 16:36 schrieb Igor Stasenko <siguctua@gmail.com>:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :)
It is 10000000
What would be the problem if it would be smaller?
that just means your VM don't have external object size cap. I changed the implementation to not have hard limit (the arbitrary large number is there just to be "compatible" with previous implementation).
If you've really done this why haven't you pushed changes back to me? You think I like the limit ?!? ;-). But is your new implementation lock-free? I went to some lengths to make sure that the Cog implementation is thread-safe, by making signalling lock-free. But making it lock-free while allowing growing was too much work. If your new implementation isn't lock-free and/or isn't thread-safe then IMO the cure is worse than the disease, because signals can get lost and that's much harder to diagnose than deal with a limit that can only be set at startup.
This means, that you can actually change in your image the check and completely ignore limits and just keep growing if it necessary.
Now, since you using VM which don't have a limit, but problem still persists, it seems like it somewhere else.. :/
i just found that after one merge, my changes get lost we're just plugged them back in, and it should be back again with newer VMs.. but the problem could be more than just semaphores.. if merge broken this, it may break many other things, so we need time to check
I try to look at it some more time. I'm using the pharo-vm from the launchpad build. Are the changes supposed to be in this one?
Norbert
Launchpad? You mean ppa? I can't say i remember all the details how changes to VM source gets into ppa distro, and how fast they get there. @Damien, can you enlighten us?
Well, the VM which i downloaded recently using zero-conf script, having limit back to 256. Just some merge mistake, which now is fixed.. means that couple builds will use limit-based implementation.. but then it will be back to my implementaiton.
On 7 October 2013 12:31, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 11:28 schrieb Henrik Johansen < henrik.s.johansen@veloxit.no>:
On Oct 7, 2013, at 11:16 , Norbert Hartl <norbert@hartl.name> wrote:
As I need an image that runs longer than 24 hours I'm looking at some stuff and wonder. Can anybody explain me the rationale for a code like this
maxExternalSemaphores: aSize "This method should never be called as result of normal program execution. If it is however, handle it differently: - In development, signal an error to promt user to set a bigger size at startup immediately. - In production, accept the cost of potentially unhandled interrupts, but log the action for later review. See comment in maxExternalObjectsSilently: why this behaviour is desirable, " "Can't find a place where development/production is decided. Suggest Smalltalk image inProduction, but use an overridable temp meanwhile. " | inProduction | self maxExternalSemaphores ifNil: [^ 0]. inProduction := false. ^ inProduction ifTrue: [self maxExternalSemaphoresSilently: aSize. self crTrace: 'WARNING: Had to increase size of semaphore signal handling table due to many external objects concurrently in use'; crTrace: 'You should increase this size at startup using #maxExternalObjectsSilently:'; crTrace: 'Current table size: ' , self maxExternalSemaphores printString] ifFalse: ["Smalltalk image" self error: 'Not enough space for external objects, set a larger size at startup!' "Smalltalk image"]
I have reported this once but got no feedback so I like to have a few opinions.
The report is here: https://pharo.fogbugz.com/f/cases/10839/
Norbert
The rationale is that inProduction would be some global setting, not yet in place when the code was written⦠Excessive simultaneous Semaphore usage is something that should be caught during development, in which case it's better to get an active notification, than having it logged somewhere.
Agreed. But didn't work in my case because it needed roughly 20 hours and an instable remote backend to trigger the problem. And somehow I forgot to install my logger as Transcript so there is no warning message. I saw only dead images in the morning. This not satisfactory but on the other hand this type of problems are hard to solve anyway. My feeling tells me there is more to discover. Sockets resources get unregistered at finalization time but this didn't work either. I would have said that the unlikely situation that no garbage collection ran could be the case. But it can't because in ExternalSemaphoreTable>>#freedSlotsIn:ratherThanIncreaseSizeTo: there is explicit garbage collection.
If I've understood correctly, it's moot on newer Pharo VM's, where there's no limit on the semtable size, but for legacy code a startup item setting size using maxExternalObjectsSilently: (as suggested in the Warning text), is still a more proper fix than setting inProduction to true and crossing your fingers hoping no signals will be lost during table growth.
Ah, I didn't know about the risk of loosing signals while resizing the table. Thanks for that. Don't get me wrong I wasn't proposing to set inProduction in effect. I don't think that automatically growing resource management is a proper way to design a system. There is always a range of resources you need for your use case. Not setting an upper bound for this just covers leaking behavior.
Norbert
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
-- best, Eliot
Hi eliot
that just means your VM don't have external object size cap. I changed the implementation to not have hard limit (the arbitrary large number is there just to be "compatible" with previous implementation).
If you've really done this why haven't you pushed changes back to me?
He did. I remember that he asked you for review on the code multiple times. I know because I was asking him to know what we should do. And in addition the code was issue was documented on the Cog bug tracker.
You think I like the limit ?!? ;-). But is your new implementation lock-free? I went to some lengths to make sure that the Cog implementation is thread-safe, by making signalling lock-free. But making it lock-free while allowing growing was too much work. If your new implementation isn't lock-free and/or isn't thread-safe then IMO the cure is worse than the disease, because signals can get lost and that's much harder to diagnose than deal with a limit that can only be set at startup.
I will let igor reply for the technical part. But we did some tests with companies doing web dev for a while before pushing this fix in Pharo. Stef
On 8 October 2013 22:03, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Igor,
On Tue, Oct 8, 2013 at 2:27 AM, Igor Stasenko <siguctua@gmail.com> wrote:
On 7 October 2013 18:36, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 16:36 schrieb Igor Stasenko <siguctua@gmail.com>:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :)
It is 10000000
What would be the problem if it would be smaller?
that just means your VM don't have external object size cap. I changed the implementation to not have hard limit (the arbitrary large number is there just to be "compatible" with previous implementation).
If you've really done this why haven't you pushed changes back to me? You think I like the limit ?!? ;-). But is your new implementation lock-free? I went to some lengths to make sure that the Cog implementation is thread-safe, by making signalling lock-free. But making it lock-free while allowing growing was too much work. If your new implementation isn't lock-free and/or isn't thread-safe then IMO the cure is worse than the disease, because signals can get lost and that's much harder to diagnose than deal with a limit that can only be set at startup.
I understand your concerns. And inviting you to verify my implementation. It should be lock-free and thread-safe. All changes are in single file, so you can just take it and replace existing one.. https://github.com/pharo-project/pharo-vm/blob/master/platforms/Cross/vm/sqE... should be easy to pick up for you.
If you don't remember, in short, i changed the way to it works, by accumulating signals into static-sized buffer, which then flushed upon each interrupt. As long as time between two interrupts remain relatively small, it is highly improbable that this buffer will be overrun. -- Best regards, Igor Stasenko.
I can report that the behavior is different now. There were two new vm releases this week in ppa. The first one didn't work but the second changed something. My application was never running that long. It is more than a day now having an actual external objects table size of 623 which wasn't ever reached before. So I would say that there is chance that this particular problem is gone. I monitor this further and I think that this wasn't the only problem. But then it is another problem. Thanks to all of you who've helped solving this. If it comes to the VM being the source of problems it is always extra annoying because it is way harder to change something there. Norbert Am 08.10.2013 um 11:27 schrieb Igor Stasenko <siguctua@gmail.com>:
On 7 October 2013 18:36, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 16:36 schrieb Igor Stasenko <siguctua@gmail.com>:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :)
It is 10000000
What would be the problem if it would be smaller?
that just means your VM don't have external object size cap. I changed the implementation to not have hard limit (the arbitrary large number is there just to be "compatible" with previous implementation).
This means, that you can actually change in your image the check and completely ignore limits and just keep growing if it necessary.
Now, since you using VM which don't have a limit, but problem still persists, it seems like it somewhere else.. :/
i just found that after one merge, my changes get lost we're just plugged them back in, and it should be back again with newer VMs.. but the problem could be more than just semaphores.. if merge broken this, it may break many other things, so we need time to check
I try to look at it some more time. I'm using the pharo-vm from the launchpad build. Are the changes supposed to be in this one?
Norbert
Launchpad? You mean ppa? I can't say i remember all the details how changes to VM source gets into ppa distro, and how fast they get there. @Damien, can you enlighten us?
Well, the VM which i downloaded recently using zero-conf script, having limit back to 256. Just some merge mistake, which now is fixed.. means that couple builds will use limit-based implementation.. but then it will be back to my implementaiton.
On 7 October 2013 12:31, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 11:28 schrieb Henrik Johansen <henrik.s.johansen@veloxit.no>:
On Oct 7, 2013, at 11:16 , Norbert Hartl <norbert@hartl.name> wrote:
As I need an image that runs longer than 24 hours I'm looking at some stuff and wonder. Can anybody explain me the rationale for a code like this
maxExternalSemaphores: aSize "This method should never be called as result of normal program execution. If it is however, handle it differently: - In development, signal an error to promt user to set a bigger size at startup immediately. - In production, accept the cost of potentially unhandled interrupts, but log the action for later review.
See comment in maxExternalObjectsSilently: why this behaviour is desirable, " "Can't find a place where development/production is decided. Suggest Smalltalk image inProduction, but use an overridable temp meanwhile. " | inProduction | self maxExternalSemaphores ifNil: [^ 0]. inProduction := false. ^ inProduction ifTrue: [self maxExternalSemaphoresSilently: aSize. self crTrace: 'WARNING: Had to increase size of semaphore signal handling table due to many external objects concurrently in use'; crTrace: 'You should increase this size at startup using #maxExternalObjectsSilently:'; crTrace: 'Current table size: ' , self maxExternalSemaphores printString] ifFalse: ["Smalltalk image" self error: 'Not enough space for external objects, set a larger size at startup!' "Smalltalk image"]
I have reported this once but got no feedback so I like to have a few opinions.
The report is here: https://pharo.fogbugz.com/f/cases/10839/
Norbert
The rationale is that inProduction would be some global setting, not yet in place when the code was written⦠Excessive simultaneous Semaphore usage is something that should be caught during development, in which case it's better to get an active notification, than having it logged somewhere.
Agreed. But didn't work in my case because it needed roughly 20 hours and an instable remote backend to trigger the problem. And somehow I forgot to install my logger as Transcript so there is no warning message. I saw only dead images in the morning. This not satisfactory but on the other hand this type of problems are hard to solve anyway. My feeling tells me there is more to discover. Sockets resources get unregistered at finalization time but this didn't work either. I would have said that the unlikely situation that no garbage collection ran could be the case. But it can't because in ExternalSemaphoreTable>>#freedSlotsIn:ratherThanIncreaseSizeTo: there is explicit garbage collection.
If I've understood correctly, it's moot on newer Pharo VM's, where there's no limit on the semtable size, but for legacy code a startup item setting size using maxExternalObjectsSilently: (as suggested in the Warning text), is still a more proper fix than setting inProduction to true and crossing your fingers hoping no signals will be lost during table growth.
Ah, I didn't know about the risk of loosing signals while resizing the table. Thanks for that. Don't get me wrong I wasn't proposing to set inProduction in effect. I don't think that automatically growing resource management is a proper way to design a system. There is always a range of resources you need for your use case. Not setting an upper bound for this just covers leaking behavior.
Norbert
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
On 11 Oct 2013, at 10:24, Norbert Hartl <norbert@hartl.name> wrote:
I can report that the behavior is different now. There were two new vm releases this week in ppa. The first one didn't work but the second changed something. My application was never running that long. It is more than a day now having an actual external objects table size of 623 which wasn't ever reached before. So I would say that there is chance that this particular problem is gone. I monitor this further and I think that this wasn't the only problem. But then it is another problem.
Yeah, but not knowing your application load, 623, which would be about 200 sockets (3 semaphores per sockets), is still a lot to be active at the same time. Can you in some way invoke a full GC externally, like using ZnReadEvalPrintDelegate and see if it eventually drops due to finalization ? It should, at least that is what I see.
Thanks to all of you who've helped solving this. If it comes to the VM being the source of problems it is always extra annoying because it is way harder to change something there.
Norbert
Am 08.10.2013 um 11:27 schrieb Igor Stasenko <siguctua@gmail.com>:
On 7 October 2013 18:36, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 16:36 schrieb Igor Stasenko <siguctua@gmail.com>:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :)
It is 10000000
What would be the problem if it would be smaller?
that just means your VM don't have external object size cap. I changed the implementation to not have hard limit (the arbitrary large number is there just to be "compatible" with previous implementation).
This means, that you can actually change in your image the check and completely ignore limits and just keep growing if it necessary.
Now, since you using VM which don't have a limit, but problem still persists, it seems like it somewhere else.. :/
i just found that after one merge, my changes get lost we're just plugged them back in, and it should be back again with newer VMs.. but the problem could be more than just semaphores.. if merge broken this, it may break many other things, so we need time to check
I try to look at it some more time. I'm using the pharo-vm from the launchpad build. Are the changes supposed to be in this one?
Norbert
Launchpad? You mean ppa? I can't say i remember all the details how changes to VM source gets into ppa distro, and how fast they get there. @Damien, can you enlighten us?
Well, the VM which i downloaded recently using zero-conf script, having limit back to 256. Just some merge mistake, which now is fixed.. means that couple builds will use limit-based implementation.. but then it will be back to my implementaiton.
On 7 October 2013 12:31, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 11:28 schrieb Henrik Johansen <henrik.s.johansen@veloxit.no>:
On Oct 7, 2013, at 11:16 , Norbert Hartl <norbert@hartl.name> wrote:
As I need an image that runs longer than 24 hours I'm looking at some stuff and wonder. Can anybody explain me the rationale for a code like this
maxExternalSemaphores: aSize "This method should never be called as result of normal program execution. If it is however, handle it differently: - In development, signal an error to promt user to set a bigger size at startup immediately. - In production, accept the cost of potentially unhandled interrupts, but log the action for later review.
See comment in maxExternalObjectsSilently: why this behaviour is desirable, " "Can't find a place where development/production is decided. Suggest Smalltalk image inProduction, but use an overridable temp meanwhile. " | inProduction | self maxExternalSemaphores ifNil: [^ 0]. inProduction := false. ^ inProduction ifTrue: [self maxExternalSemaphoresSilently: aSize. self crTrace: 'WARNING: Had to increase size of semaphore signal handling table due to many external objects concurrently in use'; crTrace: 'You should increase this size at startup using #maxExternalObjectsSilently:'; crTrace: 'Current table size: ' , self maxExternalSemaphores printString] ifFalse: ["Smalltalk image" self error: 'Not enough space for external objects, set a larger size at startup!' "Smalltalk image"]
I have reported this once but got no feedback so I like to have a few opinions.
The report is here: https://pharo.fogbugz.com/f/cases/10839/
Norbert
The rationale is that inProduction would be some global setting, not yet in place when the code was written⦠Excessive simultaneous Semaphore usage is something that should be caught during development, in which case it's better to get an active notification, than having it logged somewhere.
Agreed. But didn't work in my case because it needed roughly 20 hours and an instable remote backend to trigger the problem. And somehow I forgot to install my logger as Transcript so there is no warning message. I saw only dead images in the morning. This not satisfactory but on the other hand this type of problems are hard to solve anyway. My feeling tells me there is more to discover. Sockets resources get unregistered at finalization time but this didn't work either. I would have said that the unlikely situation that no garbage collection ran could be the case. But it can't because in ExternalSemaphoreTable>>#freedSlotsIn:ratherThanIncreaseSizeTo: there is explicit garbage collection.
If I've understood correctly, it's moot on newer Pharo VM's, where there's no limit on the semtable size, but for legacy code a startup item setting size using maxExternalObjectsSilently: (as suggested in the Warning text), is still a more proper fix than setting inProduction to true and crossing your fingers hoping no signals will be lost during table growth.
Ah, I didn't know about the risk of loosing signals while resizing the table. Thanks for that. Don't get me wrong I wasn't proposing to set inProduction in effect. I don't think that automatically growing resource management is a proper way to design a system. There is always a range of resources you need for your use case. Not setting an upper bound for this just covers leaking behavior.
Norbert
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
On 11 October 2013 10:53, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 11 Oct 2013, at 10:24, Norbert Hartl <norbert@hartl.name> wrote:
I can report that the behavior is different now. There were two new vm releases this week in ppa. The first one didn't work but the second changed something. My application was never running that long. It is more than a day now having an actual external objects table size of 623 which wasn't ever reached before. So I would say that there is chance that this particular problem is gone. I monitor this further and I think that this wasn't the only problem. But then it is another problem.
Yeah, but not knowing your application load, 623, which would be about 200 sockets (3 semaphores per sockets), is still a lot to be active at the same time. Can you in some way invoke a full GC externally, like using ZnReadEvalPrintDelegate and see if it eventually drops due to finalization ? It should, at least that is what I see.
yes, 600 semaphores indicating there are a possibly leak somewhere.
Thanks to all of you who've helped solving this. If it comes to the VM being the source of problems it is always extra annoying because it is way harder to change something there.
Norbert
-- Best regards, Igor Stasenko.
Am 11.10.2013 um 10:53 schrieb Sven Van Caekenberghe <sven@stfx.eu>:
On 11 Oct 2013, at 10:24, Norbert Hartl <norbert@hartl.name> wrote:
I can report that the behavior is different now. There were two new vm releases this week in ppa. The first one didn't work but the second changed something. My application was never running that long. It is more than a day now having an actual external objects table size of 623 which wasn't ever reached before. So I would say that there is chance that this particular problem is gone. I monitor this further and I think that this wasn't the only problem. But then it is another problem.
Yeah, but not knowing your application load, 623, which would be about 200 sockets (3 semaphores per sockets), is still a lot to be active at the same time. Can you in some way invoke a full GC externally, like using ZnReadEvalPrintDelegate and see if it eventually drops due to finalization ? It should, at least that is what I see.
Yes, that's what I meant. There is always only one outgoing connection at a time. Every 15 seconds one request is issued. So you see why expect more to find. I'm travelling right now and will have a deeper look after being back Norbert
Thanks to all of you who've helped solving this. If it comes to the VM being the source of problems it is always extra annoying because it is way harder to change something there.
Norbert
Am 08.10.2013 um 11:27 schrieb Igor Stasenko <siguctua@gmail.com>:
On 7 October 2013 18:36, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 16:36 schrieb Igor Stasenko <siguctua@gmail.com>:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :) It is 10000000
What would be the problem if it would be smaller?
that just means your VM don't have external object size cap. I changed the implementation to not have hard limit (the arbitrary large number is there just to be "compatible" with previous implementation).
This means, that you can actually change in your image the check and completely ignore limits and just keep growing if it necessary.
Now, since you using VM which don't have a limit, but problem still persists, it seems like it somewhere else.. :/
i just found that after one merge, my changes get lost we're just plugged them back in, and it should be back again with newer VMs.. but the problem could be more than just semaphores.. if merge broken this, it may break many other things, so we need time to check I try to look at it some more time. I'm using the pharo-vm from the launchpad build. Are the changes supposed to be in this one?
Norbert
Launchpad? You mean ppa? I can't say i remember all the details how changes to VM source gets into ppa distro, and how fast they get there. @Damien, can you enlighten us?
Well, the VM which i downloaded recently using zero-conf script, having limit back to 256. Just some merge mistake, which now is fixed.. means that couple builds will use limit-based implementation.. but then it will be back to my implementaiton.
On 7 October 2013 12:31, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 11:28 schrieb Henrik Johansen <henrik.s.johansen@veloxit.no>:
On Oct 7, 2013, at 11:16 , Norbert Hartl <norbert@hartl.name> wrote:
As I need an image that runs longer than 24 hours I'm looking at some stuff and wonder. Can anybody explain me the rationale for a code like this
maxExternalSemaphores: aSize "This method should never be called as result of normal program execution. If it is however, handle it differently: - In development, signal an error to promt user to set a bigger size at startup immediately. - In production, accept the cost of potentially unhandled interrupts, but log the action for later review.
See comment in maxExternalObjectsSilently: why this behaviour is desirable, " "Can't find a place where development/production is decided. Suggest Smalltalk image inProduction, but use an overridable temp meanwhile. " | inProduction | self maxExternalSemaphores ifNil: [^ 0]. inProduction := false. ^ inProduction ifTrue: [self maxExternalSemaphoresSilently: aSize. self crTrace: 'WARNING: Had to increase size of semaphore signal handling table due to many external objects concurrently in use'; crTrace: 'You should increase this size at startup using #maxExternalObjectsSilently:'; crTrace: 'Current table size: ' , self maxExternalSemaphores printString] ifFalse: ["Smalltalk image" self error: 'Not enough space for external objects, set a larger size at startup!' "Smalltalk image"]
I have reported this once but got no feedback so I like to have a few opinions.
The report is here: https://pharo.fogbugz.com/f/cases/10839/
Norbert
The rationale is that inProduction would be some global setting, not yet in place when the code was written⦠Excessive simultaneous Semaphore usage is something that should be caught during development, in which case it's better to get an active notification, than having it logged somewhere.
Agreed. But didn't work in my case because it needed roughly 20 hours and an instable remote backend to trigger the problem. And somehow I forgot to install my logger as Transcript so there is no warning message. I saw only dead images in the morning. This not satisfactory but on the other hand this type of problems are hard to solve anyway. My feeling tells me there is more to discover. Sockets resources get unregistered at finalization time but this didn't work either. I would have said that the unlikely situation that no garbage collection ran could be the case. But it can't because in ExternalSemaphoreTable>>#freedSlotsIn:ratherThanIncreaseSizeTo: there is explicit garbage collection.
If I've understood correctly, it's moot on newer Pharo VM's, where there's no limit on the semtable size, but for legacy code a startup item setting size using maxExternalObjectsSilently: (as suggested in the Warning text), is still a more proper fix than setting inProduction to true and crossing your fingers hoping no signals will be lost during table growth.
Ah, I didn't know about the risk of loosing signals while resizing the table. Thanks for that. Don't get me wrong I wasn't proposing to set inProduction in effect. I don't think that automatically growing resource management is a proper way to design a system. There is always a range of resources you need for your use case. Not setting an upper bound for this just covers leaking behavior.
Norbert
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
So, finally it turned out that the culprit is in my own code. I was logging exception objects that have a signaler context pointing to the socket. This way every connection timeout I added the exception to a collection preventing unregistering of external resources. Norbert Am 11.10.2013 um 15:02 schrieb Norbert Hartl <norbert@hartl.name>:
Am 11.10.2013 um 10:53 schrieb Sven Van Caekenberghe <sven@stfx.eu>:
On 11 Oct 2013, at 10:24, Norbert Hartl <norbert@hartl.name> wrote:
I can report that the behavior is different now. There were two new vm releases this week in ppa. The first one didn't work but the second changed something. My application was never running that long. It is more than a day now having an actual external objects table size of 623 which wasn't ever reached before. So I would say that there is chance that this particular problem is gone. I monitor this further and I think that this wasn't the only problem. But then it is another problem.
Yeah, but not knowing your application load, 623, which would be about 200 sockets (3 semaphores per sockets), is still a lot to be active at the same time. Can you in some way invoke a full GC externally, like using ZnReadEvalPrintDelegate and see if it eventually drops due to finalization ? It should, at least that is what I see.
Yes, that's what I meant. There is always only one outgoing connection at a time. Every 15 seconds one request is issued. So you see why expect more to find. I'm travelling right now and will have a deeper look after being back
Norbert
Thanks to all of you who've helped solving this. If it comes to the VM being the source of problems it is always extra annoying because it is way harder to change something there.
Norbert
Am 08.10.2013 um 11:27 schrieb Igor Stasenko <siguctua@gmail.com>:
On 7 October 2013 18:36, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 16:36 schrieb Igor Stasenko <siguctua@gmail.com>:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what is your problem :) It is 10000000
What would be the problem if it would be smaller?
that just means your VM don't have external object size cap. I changed the implementation to not have hard limit (the arbitrary large number is there just to be "compatible" with previous implementation).
This means, that you can actually change in your image the check and completely ignore limits and just keep growing if it necessary.
Now, since you using VM which don't have a limit, but problem still persists, it seems like it somewhere else.. :/
i just found that after one merge, my changes get lost we're just plugged them back in, and it should be back again with newer VMs.. but the problem could be more than just semaphores.. if merge broken this, it may break many other things, so we need time to check I try to look at it some more time. I'm using the pharo-vm from the launchpad build. Are the changes supposed to be in this one?
Norbert
Launchpad? You mean ppa? I can't say i remember all the details how changes to VM source gets into ppa distro, and how fast they get there. @Damien, can you enlighten us?
Well, the VM which i downloaded recently using zero-conf script, having limit back to 256. Just some merge mistake, which now is fixed.. means that couple builds will use limit-based implementation.. but then it will be back to my implementaiton.
On 7 October 2013 12:31, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 11:28 schrieb Henrik Johansen <henrik.s.johansen@veloxit.no>:
On Oct 7, 2013, at 11:16 , Norbert Hartl <norbert@hartl.name> wrote:
As I need an image that runs longer than 24 hours I'm looking at some stuff and wonder. Can anybody explain me the rationale for a code like this
maxExternalSemaphores: aSize "This method should never be called as result of normal program execution. If it is however, handle it differently: - In development, signal an error to promt user to set a bigger size at startup immediately. - In production, accept the cost of potentially unhandled interrupts, but log the action for later review.
See comment in maxExternalObjectsSilently: why this behaviour is desirable, " "Can't find a place where development/production is decided. Suggest Smalltalk image inProduction, but use an overridable temp meanwhile. " | inProduction | self maxExternalSemaphores ifNil: [^ 0]. inProduction := false. ^ inProduction ifTrue: [self maxExternalSemaphoresSilently: aSize. self crTrace: 'WARNING: Had to increase size of semaphore signal handling table due to many external objects concurrently in use'; crTrace: 'You should increase this size at startup using #maxExternalObjectsSilently:'; crTrace: 'Current table size: ' , self maxExternalSemaphores printString] ifFalse: ["Smalltalk image" self error: 'Not enough space for external objects, set a larger size at startup!' "Smalltalk image"]
I have reported this once but got no feedback so I like to have a few opinions.
The report is here: https://pharo.fogbugz.com/f/cases/10839/
Norbert
The rationale is that inProduction would be some global setting, not yet in place when the code was written⦠Excessive simultaneous Semaphore usage is something that should be caught during development, in which case it's better to get an active notification, than having it logged somewhere.
Agreed. But didn't work in my case because it needed roughly 20 hours and an instable remote backend to trigger the problem. And somehow I forgot to install my logger as Transcript so there is no warning message. I saw only dead images in the morning. This not satisfactory but on the other hand this type of problems are hard to solve anyway. My feeling tells me there is more to discover. Sockets resources get unregistered at finalization time but this didn't work either. I would have said that the unlikely situation that no garbage collection ran could be the case. But it can't because in ExternalSemaphoreTable>>#freedSlotsIn:ratherThanIncreaseSizeTo: there is explicit garbage collection.
If I've understood correctly, it's moot on newer Pharo VM's, where there's no limit on the semtable size, but for legacy code a startup item setting size using maxExternalObjectsSilently: (as suggested in the Warning text), is still a more proper fix than setting inProduction to true and crossing your fingers hoping no signals will be lost during table growth.
Ah, I didn't know about the risk of loosing signals while resizing the table. Thanks for that. Don't get me wrong I wasn't proposing to set inProduction in effect. I don't think that automatically growing resource management is a proper way to design a system. There is always a range of resources you need for your use case. Not setting an upper bound for this just covers leaking behavior.
Norbert
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
On 12 October 2013 11:08, Norbert Hartl <norbert@hartl.name> wrote:
So, finally it turned out that the culprit is in my own code. I was logging exception objects that have a signaler context pointing to the socket. This way every connection timeout I added the exception to a collection preventing unregistering of external resources.
Congratulations finding the bug! :)
Norbert
Am 11.10.2013 um 15:02 schrieb Norbert Hartl <norbert@hartl.name>:
Am 11.10.2013 um 10:53 schrieb Sven Van Caekenberghe <sven@stfx.eu>:
On 11 Oct 2013, at 10:24, Norbert Hartl <norbert@hartl.name> wrote:
I can report that the behavior is different now. There were two new vm
releases this week in ppa. The first one didn't work but the second changed something. My application was never running that long. It is more than a day now having an actual external objects table size of 623 which wasn't ever reached before. So I would say that there is chance that this particular problem is gone. I monitor this further and I think that this wasn't the only problem. But then it is another problem.
Yeah, but not knowing your application load, 623, which would be about
200 sockets (3 semaphores per sockets), is still a lot to be active at the same time. Can you in some way invoke a full GC externally, like using ZnReadEvalPrintDelegate and see if it eventually drops due to finalization ? It should, at least that is what I see.
Yes, that's what I meant. There is always only one outgoing connection at a time. Every 15 seconds one request is issued. So you see why expect more to find. I'm travelling right now and will have a deeper look after being back
Norbert
Thanks to all of you who've helped solving this. If it comes to the VM being the source of problems it is always extra annoying because it is way harder to change something there.
Norbert
Am 08.10.2013 um 11:27 schrieb Igor Stasenko <siguctua@gmail.com>:
On 7 October 2013 18:36, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 16:36 schrieb Igor Stasenko <siguctua@gmail.com>:
1 thing.
can you tell me what given expression yields for your VM/image:
Smalltalk vm maxExternalSemaphores
(if it gives you number less than 10000000 then i think i know what
is your problem :)
It is 10000000
What would be the problem if it would be smaller?
that just means your VM don't have external object size cap. I changed the implementation to not have hard limit (the arbitrary large number is there just to be "compatible" with previous implementation).
This means, that you can actually change in your image the check and completely ignore limits and just keep growing if it necessary.
Now, since you using VM which don't have a limit, but problem still persists, it seems like it somewhere else.. :/
i just found that after one merge, my changes get lost we're just plugged them back in, and it should be back again with newer VMs.. but the problem could be more than just semaphores.. if merge broken this, it may break many other things, so we need time to check I try to look at it some more time. I'm using the pharo-vm from the launchpad build. Are the changes supposed to be in this one?
Norbert
Launchpad? You mean ppa? I can't say i remember all the details how changes to VM source gets into ppa distro, and how fast they get there. @Damien, can you enlighten us?
Well, the VM which i downloaded recently using zero-conf script, having limit back to 256. Just some merge mistake, which now is fixed.. means that couple builds will use limit-based implementation.. but then it will be back to my implementaiton.
On 7 October 2013 12:31, Norbert Hartl <norbert@hartl.name> wrote:
Am 07.10.2013 um 11:28 schrieb Henrik Johansen <
henrik.s.johansen@veloxit.no>:
On Oct 7, 2013, at 11:16 , Norbert Hartl <norbert@hartl.name>
wrote:
As I need an image that runs longer than 24 hours I'm looking at
some stuff and wonder. Can anybody explain me the rationale for a code like this
maxExternalSemaphores: aSize "This method should never be called as result of normal program execution. If it is however, handle it differently: - In development, signal an error to promt user to set a bigger
size
at startup immediately. - In production, accept the cost of potentially unhandled interrupts, but log the action for later review.
See comment in maxExternalObjectsSilently: why this behaviour is desirable, " "Can't find a place where development/production is decided. Suggest Smalltalk image inProduction, but use an overridable temp meanwhile. " | inProduction | self maxExternalSemaphores ifNil: [^ 0]. inProduction := false. ^ inProduction ifTrue: [self maxExternalSemaphoresSilently: aSize. self crTrace: 'WARNING: Had to increase size of semaphore signal handling table due to many external objects concurrently in use'; crTrace: 'You should increase this size at startup using #maxExternalObjectsSilently:'; crTrace: 'Current table size: ' , self maxExternalSemaphores printString] ifFalse: ["Smalltalk image" self error: 'Not enough space for external objects, set a larger size at startup!' "Smalltalk image"]
I have reported this once but got no feedback so I like to have a few opinions.
The report is here: https://pharo.fogbugz.com/f/cases/10839/
Norbert
The rationale is that inProduction would be some global setting, not yet in place when the code was written⦠Excessive simultaneous Semaphore usage is something that should be caught during development, in which case it's better to get an active notification, than having it logged somewhere.
Agreed. But didn't work in my case because it needed roughly 20 hours and an instable remote backend to trigger the problem. And somehow I forgot to install my logger as Transcript so there is no warning message. I saw only dead images in the morning. This not satisfactory but on the other hand this type of problems are hard to solve anyway. My feeling tells me there is more to discover. Sockets resources get unregistered at finalization time but this didn't work either. I would have said that the unlikely situation that no garbage collection ran could be the case. But it can't because in ExternalSemaphoreTable>>#freedSlotsIn:ratherThanIncreaseSizeTo: there is explicit garbage collection.
If I've understood correctly, it's moot on newer Pharo VM's, where there's no limit on the semtable size, but for legacy code a startup item setting size using maxExternalObjectsSilently: (as suggested in the Warning text), is still a more proper fix than setting inProduction to true and crossing your fingers hoping no signals will be lost during table growth.
Ah, I didn't know about the risk of loosing signals while resizing the table. Thanks for that. Don't get me wrong I wasn't proposing to set inProduction in effect. I don't think that automatically growing resource management is a proper way to design a system. There is always a range of resources you need for your use case. Not setting an upper bound for this just covers leaking behavior.
Norbert
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
participants (8)
-
Eliot Miranda -
Henrik Johansen -
Igor Stasenko -
Marcus Denker -
Norbert Hartl -
phil＠highoctane.be -
Stéphane Ducasse -
Sven Van Caekenberghe