[Pharo-project] Protecting against stack overflow ?
Hi, Would it be possible to build some kind of protection against stack overflow ? This is actually the most common situation for loop/cycle that either results in an unusable image or a crash. Interrupting with CMD-. is not always possible. A possible solution could be to just limit the stack size and throw an exception. For example, in LispWorks, it goes like this: CL-USER 5 > (defun foo () (cons (random 100) (foo))) FOO CL-USER 6 > (foo) Stack overflow (stack size 48128). 1 (continue) Extend stack by 50%. 2 (abort) Return to level 0. 3 Return to top loop level 0. Type :b for backtrace or :c <option number> to proceed. Type :bug-form "<subject>" for a bug report template or :? for other options. CL-USER 7 : 1 > :bq ERROR <- RUNTIME:BAD-ARGS-OR-STACK <- LENGTH <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- EVAL <- CAPI::CAPI-TOP-LEVEL-FUNCTION <- CAPI::INTERACTIVE-PANE-TOP-LOOP <- MP::PROCESS-SG-FUNCTION Obviously, this is possible, the question is, can it be done easily and without paying an unbearable performance price ? Sven -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
Sven, this is one of the nice features of VA Smalltalk, so yes it can be done and no the performance hit is not unbearable. I'm not sure it captures every loop, but it's certainly saved me a few times. ...Stan -- View this message in context: http://forum.world.st/Protecting-against-stack-overflow-tp4658633p4658713.ht... Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
2012/12/9 Sven Van Caekenberghe <sven@stfx.eu>
Hi,
Would it be possible to build some kind of protection against stack overflow ?
note that there already is some mechanism. The low space mechanism is supposed to protect against stack overflow but with today's memories and the difficulty of testing it, this mechanism often a) takes way too long to kick in, and b) either leaves the system in a very sluggish state or simply fails to kick in and the system crashes with an out-of-memory error. Part of the problem is in designing a mechanism which is in keeping with the reflective nature of the system. What I mean is that we now have two different VMs, the Interpreter and the STack/Cog VMs. First one wants a check which is cheap. That means a check which isn't run on every send. In the interpreter a natural time to check for recursion would be on e.g. fullGC, checking for repetition in the current process's context chain. The the Stack/Cog VMs the natural time is on evacuating a stack page when the stack zone is full. But these are two different mechanisms, both of which are deep in the VM, neither of which can be conveniently intercepted from Smalltalk, unlike e.g. doesNotUnderstand:. Personally I like the idea of reviving the low space mechanism and maintaining it properly. For example, if the low space mechanism causes the low space semaphore to fire often, because the limit was set only a little way above the current memory size, and not, as it is now, a little below the absolute limit, then when it fires, the low space process could inspect the current process's stack (e.g. via accelerated via a primitive for performance) and see how deep it is, and possibly whether there is recursion (e.g. a bag of context methods would soon reveal high-frequency methods, and then those contexts could be inspected for repetition, but KISS says just have a per-process limit on the maximum depth of stack).
This is actually the most common situation for loop/cycle that either results in an unusable image or a crash. Interrupting with CMD-. is not always possible. A possible solution could be to just limit the stack size and throw an exception.
For example, in LispWorks, it goes like this:
CL-USER 5 > (defun foo () (cons (random 100) (foo))) FOO
CL-USER 6 > (foo)
Stack overflow (stack size 48128). 1 (continue) Extend stack by 50%. 2 (abort) Return to level 0. 3 Return to top loop level 0.
Type :b for backtrace or :c <option number> to proceed. Type :bug-form "<subject>" for a bug report template or :? for other options.
CL-USER 7 : 1 > :bq
ERROR <- RUNTIME:BAD-ARGS-OR-STACK <- LENGTH <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- EVAL <- CAPI::CAPI-TOP-LEVEL-FUNCTION <- CAPI::INTERACTIVE-PANE-TOP-LOOP <- MP::PROCESS-SG-FUNCTION
Obviously, this is possible, the question is, can it be done easily and without paying an unbearable performance price ?
Sven
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
-- best, Eliot
Hi Eliot, On 11 Dec 2012, at 02:33, Eliot Miranda <eliot.miranda@gmail.com> wrote:
2012/12/9 Sven Van Caekenberghe <sven@stfx.eu> Hi,
Would it be possible to build some kind of protection against stack overflow ?
note that there already is some mechanism. The low space mechanism is supposed to protect against stack overflow but with today's memories and the difficulty of testing it, this mechanism often a) takes way too long to kick in, and b) either leaves the system in a very sluggish state or simply fails to kick in and the system crashes with an out-of-memory error.
Part of the problem is in designing a mechanism which is in keeping with the reflective nature of the system. What I mean is that we now have two different VMs, the Interpreter and the STack/Cog VMs. First one wants a check which is cheap. That means a check which isn't run on every send. In the interpreter a natural time to check for recursion would be on e.g. fullGC, checking for repetition in the current process's context chain. The the Stack/Cog VMs the natural time is on evacuating a stack page when the stack zone is full. But these are two different mechanisms, both of which are deep in the VM, neither of which can be conveniently intercepted from Smalltalk, unlike e.g. doesNotUnderstand:.
Personally I like the idea of reviving the low space mechanism and maintaining it properly. For example, if the low space mechanism causes the low space semaphore to fire often, because the limit was set only a little way above the current memory size, and not, as it is now, a little below the absolute limit, then when it fires, the low space process could inspect the current process's stack (e.g. via accelerated via a primitive for performance) and see how deep it is, and possibly whether there is recursion (e.g. a bag of context methods would soon reveal high-frequency methods, and then those contexts could be inspected for repetition, but KISS says just have a per-process limit on the maximum depth of stack).
Thanks for the reply. Yes, getting a low space warning sooner and then deal with that intelligently at the image level sounds like a good solution. It is probably more portable, requires miminal VM collaboration, allows for many behaviours as a reaction and would catch other problems than just stack overflow. Is this already possible today ? Preferrably, dynamic control over the limit would be nice: in a headless server you want other behavior than during interactive development. Sven
This is actually the most common situation for loop/cycle that either results in an unusable image or a crash. Interrupting with CMD-. is not always possible. A possible solution could be to just limit the stack size and throw an exception.
For example, in LispWorks, it goes like this:
CL-USER 5 > (defun foo () (cons (random 100) (foo))) FOO
CL-USER 6 > (foo)
Stack overflow (stack size 48128). 1 (continue) Extend stack by 50%. 2 (abort) Return to level 0. 3 Return to top loop level 0.
Type :b for backtrace or :c <option number> to proceed. Type :bug-form "<subject>" for a bug report template or :? for other options.
CL-USER 7 : 1 > :bq
ERROR <- RUNTIME:BAD-ARGS-OR-STACK <- LENGTH <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- EVAL <- CAPI::CAPI-TOP-LEVEL-FUNCTION <- CAPI::INTERACTIVE-PANE-TOP-LOOP <- MP::PROCESS-SG-FUNCTION
Obviously, this is possible, the question is, can it be done easily and without paying an unbearable performance price ?
Sven
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
-- best, Eliot
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
i would be happy to have something, which can be expressed as following 'unit test': Smalltalk vm setStackDepthLimit: 100. self should: [ self recurseTimes: 101 ] throw: Exception. Smalltalk vm setStackDepthLimit: 0. "no limit" self shouldNot: [ self recurseTimes: 101 ] throw: Exception. -- Best regards, Igor Stasenko.
I don't understand how setting a fixed-size stack could ever be useful. No one would risk blowing up a production app only because they guessed wrong about the stackDepthLimit, so for production it would always be set to 0. So that leaves the development use-case. During development the system is already under heavy scrutiny, the real requirement is just the ability to _get back control_ when the system doesn't seem to be doing anything so that infinite recursions can be debugged. So if simply Command+. interrupt could be made to just work then there's no need to fiddle with the "guess the stack-limit" game. I like Eliots idea of paying attention to how long the LowSpaceSemaphore fires, except not trying to analyze the stack. Instead, check how much space was _freed_ by the handlers of low-space (#freeSomeSpace I think) and, if it is Zero or very low, THEN open a debugger. On Tue, Dec 11, 2012 at 8:00 AM, Igor Stasenko <siguctua@gmail.com> wrote:
i would be happy to have something, which can be expressed as following 'unit test':
Smalltalk vm setStackDepthLimit: 100.
self should: [ self recurseTimes: 101 ] throw: Exception.
Smalltalk vm setStackDepthLimit: 0. "no limit"
self shouldNot: [ self recurseTimes: 101 ] throw: Exception.
-- Best regards, Igor Stasenko.
On 11 Dec 2012, at 16:54, Chris Muller <asqueaker@gmail.com> wrote:
I don't understand how setting a fixed-size stack could ever be useful. No one would risk blowing up a production app only because they guessed wrong about the stackDepthLimit, so for production it would always be set to 0.
I kind of disagree: any system has limits, and most server software is set up to enforce limits earlier (maximum connections, file descriptors per process, â¦) and deal as gracefully as possible with them, to protect itself from external abuse or internal errors. Yes this might involve some guessing, so what. I would rather be able to deal with an infinite recursion and return an internal server error, than have the whole server crash.
So that leaves the development use-case. During development the system is already under heavy scrutiny, the real requirement is just the ability to _get back control_ when the system doesn't seem to be doing anything so that infinite recursions can be debugged. So if simply Command+. interrupt could be made to just work then there's no need to fiddle with the "guess the stack-limit" game.
With a reasonable stack or low memory limit, you won't even have to press cmd-. when something goes wrong ;-) But of course, cmd-. should just work.
I like Eliots idea of paying attention to how long the LowSpaceSemaphore fires, except not trying to analyze the stack. Instead, check how much space was _freed_ by the handlers of low-space (#freeSomeSpace I think) and, if it is Zero or very low, THEN open a debugger.
On Tue, Dec 11, 2012 at 8:00 AM, Igor Stasenko <siguctua@gmail.com> wrote:
i would be happy to have something, which can be expressed as following 'unit test':
Smalltalk vm setStackDepthLimit: 100.
self should: [ self recurseTimes: 101 ] throw: Exception.
Smalltalk vm setStackDepthLimit: 0. "no limit"
self shouldNot: [ self recurseTimes: 101 ] throw: Exception.
-- Best regards, Igor Stasenko.
I don't understand how setting a fixed-size stack could ever be useful. No one would risk blowing up a production app only because they guessed wrong about the stackDepthLimit, so for production it would always be set to 0.
I kind of disagree: any system has limits, and most server software is set up to enforce limits earlier (maximum connections, file descriptors per process, â¦) and deal as gracefully as possible with them, to protect itself from external abuse or internal errors. Yes this might involve some guessing, so what.
You can't accurately guess an arbitrary stack limit and still have it be effective for what you want it to do (application stability). The limit will be exceeded at 3am and grumpy on-call support will increase it so that server will run out of memory first anyway. Server software needs to enforce its memory limit anyway, so that may as well be also the solution for a runaway stack. One solution for two problems.
I would rather be able to deal with an infinite recursion and return an internal server error, than have the whole server crash.
If the app registers with LowSpaceSemaphore and implements #freeSomeSpace then the whole server wouldn't crash.
On 11 December 2012 17:41, Chris Muller <ma.chris.m@gmail.com> wrote:
I don't understand how setting a fixed-size stack could ever be useful. No one would risk blowing up a production app only because they guessed wrong about the stackDepthLimit, so for production it would always be set to 0.
I kind of disagree: any system has limits, and most server software is set up to enforce limits earlier (maximum connections, file descriptors per process, â¦) and deal as gracefully as possible with them, to protect itself from external abuse or internal errors. Yes this might involve some guessing, so what.
You can't accurately guess an arbitrary stack limit and still have it be effective for what you want it to do (application stability). The limit will be exceeded at 3am and grumpy on-call support will increase it so that server will run out of memory first anyway.
You don't need a grumpy guy to do that. you can always do: [ i don't care what happens inside, recurse: 10000000 times ] on: StackOverflow do: [: ex pass ]. or, if you care, you can do: [ inside, recurse: 10000000 times ] on: StackOverflow do: [: ex | self showErrorAndTerminateBugger ]. all you need is a levers to control that. The idea is that you set the stack limit per process, and when it hits the limit, you get a signal, with exact information about who ran onto it. Right now, that "lowspace" is just a notification: you're going to explode...in 5..4..3..2.. now in a few lapses left, you should guess right and cut right wire (among 10000) , and if you cut wrong one... booom.. the point is that you don't know right wire, because you're not imposed any limits(rules) on your model, so you cannot find the violator.
Server software needs to enforce its memory limit anyway, so that may as well be also the solution for a runaway stack. One solution for two problems.
I would rather be able to deal with an infinite recursion and return an internal server error, than have the whole server crash.
If the app registers with LowSpaceSemaphore and implements #freeSomeSpace then the whole server wouldn't crash.
-- Best regards, Igor Stasenko.
hehe.. it looks like we can do it: - when memory hits the waterline mark, VM should do the following: - for every currently scheduled process , push an artificial stack frame on top , which is StackOverflow signal. Then the first thing what this guy should do, when he will get control, is to analyze how deep is the stack of given process and if it surpassed the allowed threshold, and if there's no handler, then decide whether to kill the process or not. Also, since it is exception, a user code can intercept this exception and either ignore it (effectively forcing system to keep running), or otherwise terminate process, but in an educated manner, since he might have a better knowledge, what can eat so much resources in his code.. as well, as do a gracious failure handling, e.g., correctly clean associated system resources etc. -- Best regards, Igor Stasenko.
Cool, could you maybe do a proof of concept someday ;-) On 11 Dec 2012, at 19:26, Igor Stasenko <siguctua@gmail.com> wrote:
hehe.. it looks like we can do it:
- when memory hits the waterline mark, VM should do the following: - for every currently scheduled process , push an artificial stack frame on top , which is StackOverflow signal.
Then the first thing what this guy should do, when he will get control, is to analyze how deep is the stack of given process and if it surpassed the allowed threshold, and if there's no handler, then decide whether to kill the process or not.
Also, since it is exception, a user code can intercept this exception and either ignore it (effectively forcing system to keep running), or otherwise terminate process, but in an educated manner, since he might have a better knowledge, what can eat so much resources in his code.. as well, as do a gracious failure handling, e.g., correctly clean associated system resources etc.
-- Best regards, Igor Stasenko.
On Tue, Dec 11, 2012 at 10:26 AM, Igor Stasenko <siguctua@gmail.com> wrote:
hehe.. it looks like we can do it:
- when memory hits the waterline mark, VM should do the following: - for every currently scheduled process , push an artificial stack frame on top , which is StackOverflow signal.
If the VM supports the LowSpaceSignal mechanism as it used to be then this can be done in the image with a high-priority process. And tehrefore it can e.g. be avoided for processes such as the Delay process, where IMO it is inappropriate.
Then the first thing what this guy should do, when he will get control, is to analyze how deep is the stack of given process and if it surpassed the allowed threshold, and if there's no handler, then decide whether to kill the process or not.
Also, since it is exception, a user code can intercept this exception and either ignore it (effectively forcing system to keep running), or otherwise terminate process, but in an educated manner, since he might have a better knowledge, what can eat so much resources in his code.. as well, as do a gracious failure handling, e.g., correctly clean associated system resources etc.
-- Best regards, Igor Stasenko.
-- best, Eliot
On Tue, Dec 11, 2012 at 3:35 PM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
On Tue, Dec 11, 2012 at 10:26 AM, Igor Stasenko <siguctua@gmail.com>wrote:
hehe.. it looks like we can do it:
- when memory hits the waterline mark, VM should do the following: - for every currently scheduled process , push an artificial stack frame on top , which is StackOverflow signal.
If the VM supports the LowSpaceSignal mechanism as it used to be then this can be done in the image with a high-priority process. And tehrefore it can e.g. be avoided for processes such as the Delay process, where IMO it is inappropriate.
And if it is done in the image it is easy to e.g. avoid pushing two such frames on the same process because the process has made no progress since the last occurrence, etc.
Then the first thing what this guy should do, when he will get control, is to analyze how deep is the stack of given process and if it surpassed the allowed threshold, and if there's no handler, then decide whether to kill the process or not.
Also, since it is exception, a user code can intercept this exception and either ignore it (effectively forcing system to keep running), or otherwise terminate process, but in an educated manner, since he might have a better knowledge, what can eat so much resources in his code.. as well, as do a gracious failure handling, e.g., correctly clean associated system resources etc.
-- Best regards, Igor Stasenko.
-- best, Eliot
-- best, Eliot
On 12 December 2012 00:43, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Tue, Dec 11, 2012 at 3:35 PM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Tue, Dec 11, 2012 at 10:26 AM, Igor Stasenko <siguctua@gmail.com> wrote:
hehe.. it looks like we can do it:
- when memory hits the waterline mark, VM should do the following: - for every currently scheduled process , push an artificial stack frame on top , which is StackOverflow signal.
If the VM supports the LowSpaceSignal mechanism as it used to be then this can be done in the image with a high-priority process. And tehrefore it can e.g. be avoided for processes such as the Delay process, where IMO it is inappropriate.
And if it is done in the image it is easy to e.g. avoid pushing two such frames on the same process because the process has made no progress since the last occurrence, etc.
indeed.. i did not realized that this can be done inside an image :) i was thinking more about stack limit, not space limit. because scanning the stack will trigger putting all contexts on heap (which in own turn will put even more pressure on memory).. while in VM, to determine stack depth, it can walk the stack frames without marrying them with heap contexts to count stack depth. or it is not the case? is accessing the context state triggers "marrying" it with heap object, i.e. [ count := count + 1. context := context sender. ] whileNotNil ? -- Best regards, Igor Stasenko.
On Tue, Dec 11, 2012 at 9:26 PM, Igor Stasenko <siguctua@gmail.com> wrote:
On 12 December 2012 00:43, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Tue, Dec 11, 2012 at 3:35 PM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Tue, Dec 11, 2012 at 10:26 AM, Igor Stasenko <siguctua@gmail.com> wrote:
hehe.. it looks like we can do it:
- when memory hits the waterline mark, VM should do the following: - for every currently scheduled process , push an artificial stack frame on top , which is StackOverflow signal.
If the VM supports the LowSpaceSignal mechanism as it used to be then
this
can be done in the image with a high-priority process. And tehrefore it can e.g. be avoided for processes such as the Delay process, where IMO it is inappropriate.
And if it is done in the image it is easy to e.g. avoid pushing two such frames on the same process because the process has made no progress since the last occurrence, etc.
indeed.. i did not realized that this can be done inside an image :)
i was thinking more about stack limit, not space limit. because scanning the stack will trigger putting all contexts on heap (which in own turn will put even more pressure on memory).. while in VM, to determine stack depth, it can walk the stack frames without marrying them with heap contexts to count stack depth. or it is not the case? is accessing the context state triggers "marrying" it with heap object, i.e.
[ count := count + 1. context := context sender. ] whileNotNil
?
exactly. The VM can provide a primitive to walk the stack without creating objects (which worsens the space situation). Even better the VM could maintain stack depth in Process, but that's for the future, because it needs a) a slot in Process and b) the support writing in both VMs. --
Best regards, Igor Stasenko.
-- best, Eliot
On Tue, Dec 11, 2012 at 2:44 AM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi Eliot,
On 11 Dec 2012, at 02:33, Eliot Miranda <eliot.miranda@gmail.com> wrote:
2012/12/9 Sven Van Caekenberghe <sven@stfx.eu> Hi,
Would it be possible to build some kind of protection against stack overflow ?
note that there already is some mechanism. The low space mechanism is supposed to protect against stack overflow but with today's memories and the difficulty of testing it, this mechanism often a) takes way too long to kick in, and b) either leaves the system in a very sluggish state or simply fails to kick in and the system crashes with an out-of-memory error.
Part of the problem is in designing a mechanism which is in keeping with the reflective nature of the system. What I mean is that we now have two different VMs, the Interpreter and the STack/Cog VMs. First one wants a check which is cheap. That means a check which isn't run on every send. In the interpreter a natural time to check for recursion would be on e.g. fullGC, checking for repetition in the current process's context chain. The the Stack/Cog VMs the natural time is on evacuating a stack page when the stack zone is full. But these are two different mechanisms, both of which are deep in the VM, neither of which can be conveniently intercepted from Smalltalk, unlike e.g. doesNotUnderstand:.
Personally I like the idea of reviving the low space mechanism and maintaining it properly. For example, if the low space mechanism causes the low space semaphore to fire often, because the limit was set only a little way above the current memory size, and not, as it is now, a little below the absolute limit, then when it fires, the low space process could inspect the current process's stack (e.g. via accelerated via a primitive for performance) and see how deep it is, and possibly whether there is recursion (e.g. a bag of context methods would soon reveal high-frequency methods, and then those contexts could be inspected for repetition, but KISS says just have a per-process limit on the maximum depth of stack).
Thanks for the reply.
Yes, getting a low space warning sooner and then deal with that intelligently at the image level sounds like a good solution. It is probably more portable, requires miminal VM collaboration, allows for many behaviours as a reaction and would catch other problems than just stack overflow.
Is this already possible today ?
Wow, I'm amazed. The low space limit in the VM has been ripped out So no, it is not possible, but it would be easy to put back. There is a variable in ObjectMemory (the VM's heap manager/garbage collector class) that defines the low-space limit, but there is no setter. Back in the day (and still in VisualWorks) there was a primitive SystemDictionary methods for memory space signal: aSemaphore atOopsLeft: numOops wordsLeft: numWords "Tell the object memory to signal the Semaphore when either the number of object pointers remaining drops below numOops, or the number of words in the object space remaining drops below numWords. Fail if the frist argument is neither a Semaphore nor nil. Fail if numOops is not a 16-bit Integer, or if numWords is not a 32-bit LargePositiveInteger. Essential. See Object documentation whatIsAPrimitive." <primitive: 116> self primitiveFailed But this and its support is gone. Hmm. Hmmph :)
Preferrably, dynamic control over the limit would be nice: in a headless server you want other behavior than during interactive development.
Right. The low space process would readjust the value every time low space fired. But one could (can) call the primitive at any time to install a new limit.
Sven
This is actually the most common situation for loop/cycle that either results in an unusable image or a crash. Interrupting with CMD-. is not always possible. A possible solution could be to just limit the stack size and throw an exception.
For example, in LispWorks, it goes like this:
CL-USER 5 > (defun foo () (cons (random 100) (foo))) FOO
CL-USER 6 > (foo)
Stack overflow (stack size 48128). 1 (continue) Extend stack by 50%. 2 (abort) Return to level 0. 3 Return to top loop level 0.
Type :b for backtrace or :c <option number> to proceed. Type :bug-form "<subject>" for a bug report template or :? for other options.
CL-USER 7 : 1 > :bq
ERROR <- RUNTIME:BAD-ARGS-OR-STACK <- LENGTH <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- EVAL <- CAPI::CAPI-TOP-LEVEL-FUNCTION <- CAPI::INTERACTIVE-PANE-TOP-LOOP <- MP::PROCESS-SG-FUNCTION
Obviously, this is possible, the question is, can it be done easily and without paying an unbearable performance price ?
Sven
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
-- best, Eliot
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
-- best, Eliot
On Tue, Dec 11, 2012 at 3:33 PM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
On Tue, Dec 11, 2012 at 2:44 AM, Sven Van Caekenberghe <sven@stfx.eu>wrote:
Hi Eliot,
On 11 Dec 2012, at 02:33, Eliot Miranda <eliot.miranda@gmail.com> wrote:
2012/12/9 Sven Van Caekenberghe <sven@stfx.eu> Hi,
Would it be possible to build some kind of protection against stack overflow ?
note that there already is some mechanism. The low space mechanism is supposed to protect against stack overflow but with today's memories and the difficulty of testing it, this mechanism often a) takes way too long to kick in, and b) either leaves the system in a very sluggish state or simply fails to kick in and the system crashes with an out-of-memory error.
Part of the problem is in designing a mechanism which is in keeping with the reflective nature of the system. What I mean is that we now have two different VMs, the Interpreter and the STack/Cog VMs. First one wants a check which is cheap. That means a check which isn't run on every send. In the interpreter a natural time to check for recursion would be on e.g. fullGC, checking for repetition in the current process's context chain. The the Stack/Cog VMs the natural time is on evacuating a stack page when the stack zone is full. But these are two different mechanisms, both of which are deep in the VM, neither of which can be conveniently intercepted from Smalltalk, unlike e.g. doesNotUnderstand:.
Personally I like the idea of reviving the low space mechanism and maintaining it properly. For example, if the low space mechanism causes the low space semaphore to fire often, because the limit was set only a little way above the current memory size, and not, as it is now, a little below the absolute limit, then when it fires, the low space process could inspect the current process's stack (e.g. via accelerated via a primitive for performance) and see how deep it is, and possibly whether there is recursion (e.g. a bag of context methods would soon reveal high-frequency methods, and then those contexts could be inspected for repetition, but KISS says just have a per-process limit on the maximum depth of stack).
Thanks for the reply.
Yes, getting a low space warning sooner and then deal with that intelligently at the image level sounds like a good solution. It is probably more portable, requires miminal VM collaboration, allows for many behaviours as a reaction and would catch other problems than just stack overflow.
Is this already possible today ?
Um, ignore this. I was smoking something. (actually the message selector browser completely hid the inst var access browser so I didn't see the inst var accesses). Hang on and I'll try again...
Wow, I'm amazed. The low space limit in the VM has been ripped out So no, it is not possible, but it would be easy to put back. There is a variable in ObjectMemory (the VM's heap manager/garbage collector class) that defines the low-space limit, but there is no setter. Back in the day (and still in VisualWorks) there was a primitive
SystemDictionary methods for memory space signal: aSemaphore atOopsLeft: numOops wordsLeft: numWords "Tell the object memory to signal the Semaphore when either the number of object pointers remaining drops below numOops, or the number of words in the object space remaining drops below numWords. Fail if the frist argument is neither a Semaphore nor nil. Fail if numOops is not a 16-bit Integer, or if numWords is not a 32-bit LargePositiveInteger. Essential. See Object documentation whatIsAPrimitive."
<primitive: 116> self primitiveFailed
But this and its support is gone. Hmm. Hmmph :)
Preferrably, dynamic control over the limit would be nice: in a headless server you want other behavior than during interactive development.
Right. The low space process would readjust the value every time low space fired. But one could (can) call the primitive at any time to install a new limit.
Sven
This is actually the most common situation for loop/cycle that either results in an unusable image or a crash. Interrupting with CMD-. is not always possible. A possible solution could be to just limit the stack size and throw an exception.
For example, in LispWorks, it goes like this:
CL-USER 5 > (defun foo () (cons (random 100) (foo))) FOO
CL-USER 6 > (foo)
Stack overflow (stack size 48128). 1 (continue) Extend stack by 50%. 2 (abort) Return to level 0. 3 Return to top loop level 0.
Type :b for backtrace or :c <option number> to proceed. Type :bug-form "<subject>" for a bug report template or :? for other options.
CL-USER 7 : 1 > :bq
ERROR <- RUNTIME:BAD-ARGS-OR-STACK <- LENGTH <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- EVAL <- CAPI::CAPI-TOP-LEVEL-FUNCTION <- CAPI::INTERACTIVE-PANE-TOP-LOOP <- MP::PROCESS-SG-FUNCTION
Obviously, this is possible, the question is, can it be done easily and without paying an unbearable performance price ?
Sven
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
-- best, Eliot
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
-- best, Eliot
-- best, Eliot
On Tue, Dec 11, 2012 at 2:44 AM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi Eliot,
On 11 Dec 2012, at 02:33, Eliot Miranda <eliot.miranda@gmail.com> wrote:
2012/12/9 Sven Van Caekenberghe <sven@stfx.eu> Hi,
Would it be possible to build some kind of protection against stack overflow ?
note that there already is some mechanism. The low space mechanism is supposed to protect against stack overflow but with today's memories and the difficulty of testing it, this mechanism often a) takes way too long to kick in, and b) either leaves the system in a very sluggish state or simply fails to kick in and the system crashes with an out-of-memory error.
Part of the problem is in designing a mechanism which is in keeping with the reflective nature of the system. What I mean is that we now have two different VMs, the Interpreter and the STack/Cog VMs. First one wants a check which is cheap. That means a check which isn't run on every send. In the interpreter a natural time to check for recursion would be on e.g. fullGC, checking for repetition in the current process's context chain. The the Stack/Cog VMs the natural time is on evacuating a stack page when the stack zone is full. But these are two different mechanisms, both of which are deep in the VM, neither of which can be conveniently intercepted from Smalltalk, unlike e.g. doesNotUnderstand:.
Personally I like the idea of reviving the low space mechanism and maintaining it properly. For example, if the low space mechanism causes the low space semaphore to fire often, because the limit was set only a little way above the current memory size, and not, as it is now, a little below the absolute limit, then when it fires, the low space process could inspect the current process's stack (e.g. via accelerated via a primitive for performance) and see how deep it is, and possibly whether there is recursion (e.g. a bag of context methods would soon reveal high-frequency methods, and then those contexts could be inspected for repetition, but KISS says just have a per-process limit on the maximum depth of stack).
Thanks for the reply.
Yes, getting a low space warning sooner and then deal with that intelligently at the image level sounds like a good solution. It is probably more portable, requires miminal VM collaboration, allows for many behaviours as a reaction and would catch other problems than just stack overflow.
Is this already possible today ?
Yes. The primitive is primitive 125. SmalltalkImage methods for memory space primSignalAtBytesLeft: numBytes "Tell the interpreter the low-space threshold in bytes. When the free space falls below this threshold, the interpreter will signal the low-space semaphore, if one has been registered. Disable low-space interrupts if the argument is zero. Fail if numBytes is not an Integer." <primitive: 125> self primitiveFailed There should be an installLowSpaceWatcher to access this, and a lowSpaceWatcher process that implements the low space policy.
Preferrably, dynamic control over the limit would be nice: in a headless server you want other behavior than during interactive development.
Right. One can call the primitive at any time to modify the limit.
Sven
This is actually the most common situation for loop/cycle that either results in an unusable image or a crash. Interrupting with CMD-. is not always possible. A possible solution could be to just limit the stack size and throw an exception.
For example, in LispWorks, it goes like this:
CL-USER 5 > (defun foo () (cons (random 100) (foo))) FOO
CL-USER 6 > (foo)
Stack overflow (stack size 48128). 1 (continue) Extend stack by 50%. 2 (abort) Return to level 0. 3 Return to top loop level 0.
Type :b for backtrace or :c <option number> to proceed. Type :bug-form "<subject>" for a bug report template or :? for other options.
CL-USER 7 : 1 > :bq
ERROR <- RUNTIME:BAD-ARGS-OR-STACK <- LENGTH <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- EVAL <- CAPI::CAPI-TOP-LEVEL-FUNCTION <- CAPI::INTERACTIVE-PANE-TOP-LOOP <- MP::PROCESS-SG-FUNCTION
Obviously, this is possible, the question is, can it be done easily and without paying an unbearable performance price ?
Sven
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
-- best, Eliot
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
-- best, Eliot
participants (6)
-
Chris Muller -
Chris Muller -
Eliot Miranda -
Igor Stasenko -
Stan Shepherd -
Sven Van Caekenberghe