[Pharo-project] Exceptions with Continuations
It always bugged me that SUnit has to rerun the tests again when you want to debug them. So I made a little use-case where exceptions have an optional continuation, so you can resume/debug them later on. Here's a small example: |e| [ 0 / 0 ] on: Error do: [ :error| e := error freeze ]. e debug freeze simply creates a continuation (at the moment a full one, but that could be of course reduced to a partial one by the TestRunner). Now when you would click on the failing test result you would simply have to activate the corresponding error by sending #debug to it. This would have a major advantage for non-deterministic tests (which of course they should not be). Because in this case you're lost with the current SUnit since the debug stack will differ from the actual test-run which produced the error. Needed Changes: - Continuations (the one from Seaside will just do fine) - Exception >> #freeze saving the continuation - TestCase has to store the actual exception
On 26 August 2011 14:54, Camillo Bruni <camillo.bruni@inria.fr> wrote:
It always bugged me that SUnit has to rerun the tests again when you want to debug them. So I made a little use-case where exceptions have an optional continuation, so you can resume/debug them later on.
Here's a small example:
|e| [ 0 / 0 ]     on: Error     do: [ :error| e := error freeze ]. e debug
freeze simply creates a continuation (at the moment a full one, but that could be of course reduced to a partial one by the TestRunner). Now when you would click on the failing test result you would simply have to activate the corresponding error by sending #debug to it.
This would have a major advantage for non-deterministic tests (which of course they should not be). Because in this case you're lost with the current SUnit since the debug stack will differ from the actual test-run which produced the error.
Needed Changes: - Continuations (the one from Seaside will just do fine) - Exception >> #freeze saving the continuation - TestCase has to store the actual exception
Hi Camillo, Indeed, partial continuations would make an excellent addition to SUnit. Levente & I discussed doing do a short while back in squeak-dev. Have you had a look at http://www.squeaksource.com/Control.html by any chance? It implements the shift/reset control operator, which is almost what Seaside uses. frank
On 2011-08-26, at 16:02, Frank Shearar wrote:
On 26 August 2011 14:54, Camillo Bruni <camillo.bruni@inria.fr> wrote:
It always bugged me that SUnit has to rerun the tests again when you want to debug them. So I made a little use-case where exceptions have an optional continuation, so you can resume/debug them later on.
Here's a small example:
|e| [ 0 / 0 ] on: Error do: [ :error| e := error freeze ]. e debug
freeze simply creates a continuation (at the moment a full one, but that could be of course reduced to a partial one by the TestRunner). Now when you would click on the failing test result you would simply have to activate the corresponding error by sending #debug to it.
This would have a major advantage for non-deterministic tests (which of course they should not be). Because in this case you're lost with the current SUnit since the debug stack will differ from the actual test-run which produced the error.
Needed Changes: - Continuations (the one from Seaside will just do fine) - Exception >> #freeze saving the continuation - TestCase has to store the actual exception
Hi Camillo,
Indeed, partial continuations would make an excellent addition to SUnit. Levente & I discussed doing do a short while back in squeak-dev. Have you had a look at http://www.squeaksource.com/Control.html by any chance? It implements the shift/reset control operator, which is almost what Seaside uses.
frank
No I was unaware of that, having a look at it right now. thanks
Storing a continuation per failure/error potentially consumes a lot of memory. Memory leaks are also the reason why SUnit doesn't even remeber the exception (which only refer to one or two unwound stack frames). Furthermore continuations interfeer with the guarantees of tearDown methods in test-cases and test-resources. This opens a box full of worms: when do you run them, how often do you run them, how do you ensure a consistent state, ...? Lukas On Friday, 26 August 2011, Camillo Bruni <camillo.bruni@inria.fr> wrote:
On 2011-08-26, at 16:02, Frank Shearar wrote:
On 26 August 2011 14:54, Camillo Bruni <camillo.bruni@inria.fr> wrote:
It always bugged me that SUnit has to rerun the tests again when you
want to debug them. So I made a little use-case where exceptions have an optional continuation, so you can resume/debug them later on.
Here's a small example:
|e| [ 0 / 0 ] on: Error do: [ :error| e := error freeze ]. e debug
freeze simply creates a continuation (at the moment a full one, but that
could be of course reduced to a partial one by the TestRunner). Now when you would click on the failing test result you would simply have to activate the corresponding error by sending #debug to it.
This would have a major advantage for non-deterministic tests (which of
course they should not be). Because in this case you're lost with the current SUnit since the debug stack will differ from the actual test-run which produced the error.
Needed Changes: - Continuations (the one from Seaside will just do fine) - Exception >> #freeze saving the continuation - TestCase has to store the actual exception
Hi Camillo,
Indeed, partial continuations would make an excellent addition to SUnit. Levente & I discussed doing do a short while back in squeak-dev. Have you had a look at http://www.squeaksource.com/Control.html by any chance? It implements the shift/reset control operator, which is almost what Seaside uses.
frank
No I was unaware of that, having a look at it right now.
thanks
-- Lukas Renggli www.lukas-renggli.ch
On 2011-08-26, at 16:28, Lukas Renggli wrote:
Storing a continuation per failure/error potentially consumes a lot of memory. Memory leaks are also the reason why SUnit doesn't even remeber the exception (which only refer to one or two unwound stack frames).
point taken.
Furthermore continuations interfeer with the guarantees of tearDown methods in test-cases and test-resources. This opens a box full of worms: when do you run them, how often do you run them, how do you ensure a consistent state, �
right, but they might also help debugging annoying nondeterministic edge cases. so I will implement it with an optional setting in the TestRunner. When you find an annoying test that won't run consistently you can rerun it with the continuation being recorded.
Lukas
On Friday, 26 August 2011, Camillo Bruni <camillo.bruni@inria.fr> wrote:
On 2011-08-26, at 16:02, Frank Shearar wrote:
On 26 August 2011 14:54, Camillo Bruni <camillo.bruni@inria.fr> wrote:
It always bugged me that SUnit has to rerun the tests again when you want to debug them. So I made a little use-case where exceptions have an optional continuation, so you can resume/debug them later on.
Here's a small example:
|e| [ 0 / 0 ] on: Error do: [ :error| e := error freeze ]. e debug
freeze simply creates a continuation (at the moment a full one, but that could be of course reduced to a partial one by the TestRunner). Now when you would click on the failing test result you would simply have to activate the corresponding error by sending #debug to it.
This would have a major advantage for non-deterministic tests (which of course they should not be). Because in this case you're lost with the current SUnit since the debug stack will differ from the actual test-run which produced the error.
Needed Changes: - Continuations (the one from Seaside will just do fine) - Exception >> #freeze saving the continuation - TestCase has to store the actual exception
Hi Camillo,
Indeed, partial continuations would make an excellent addition to SUnit. Levente & I discussed doing do a short while back in squeak-dev. Have you had a look at http://www.squeaksource.com/Control.html by any chance? It implements the shift/reset control operator, which is almost what Seaside uses.
frank
No I was unaware of that, having a look at it right now.
thanks
-- Lukas Renggli www.lukas-renggli.ch
Why don't you just debug the test then? If you have to enable and rerun the test anyway, you can as well tell the test to open a debugger in case of a problem. Isn't this what "MyTest debug: #testMe" does? Lukas On Friday, 26 August 2011, Camillo Bruni <camillo.bruni@inria.fr> wrote:
On 2011-08-26, at 16:28, Lukas Renggli wrote:
Storing a continuation per failure/error potentially consumes a lot of
memory. Memory leaks are also the reason why SUnit doesn't even remeber the exception (which only refer to one or two unwound stack frames).
point taken.
Furthermore continuations interfeer with the guarantees of tearDown
methods in test-cases and test-resources. This opens a box full of worms: when do you run them, how often do you run them, how do you ensure a consistent state, �
right, but they might also help debugging annoying nondeterministic edge
cases. so I will implement it with an optional setting in the TestRunner. When you find an annoying test that won't run consistently you can rerun it with the continuation being recorded.
Lukas
On Friday, 26 August 2011, Camillo Bruni <camillo.bruni@inria.fr> wrote:
On 2011-08-26, at 16:02, Frank Shearar wrote:
On 26 August 2011 14:54, Camillo Bruni <camillo.bruni@inria.fr> wrote:
It always bugged me that SUnit has to rerun the tests again when you
want to debug them. So I made a little use-case where exceptions have an optional continuation, so you can resume/debug them later on.
Here's a small example:
|e| [ 0 / 0 ] on: Error do: [ :error| e := error freeze ]. e debug
freeze simply creates a continuation (at the moment a full one, but
that could be of course reduced to a partial one by the TestRunner). Now when you would click on the failing test result you would simply have to activate the corresponding error by sending #debug to it.
This would have a major advantage for non-deterministic tests (which of
course they should not be). Because in this case you're lost with the current SUnit since the debug stack will differ from the actual test-run which produced the error.
Needed Changes: - Continuations (the one from Seaside will just do fine) - Exception >> #freeze saving the continuation - TestCase has to store the actual exception
Hi Camillo,
Indeed, partial continuations would make an excellent addition to SUnit. Levente & I discussed doing do a short while back in squeak-dev. Have you had a look at http://www.squeaksource.com/Control.html by any chance? It implements the shift/reset control operator, which is almost what Seaside uses.
frank
No I was unaware of that, having a look at it right now.
thanks
-- Lukas Renggli www.lukas-renggli.ch <http://www.lukas-renggli.ch/>
-- Lukas Renggli www.lukas-renggli.ch
well the problem with a non deterministic test is that the UI might not correspond to the opened debugger :) another application I see here is for instance when running tests in CI and you see only a couple of tests failing, rerun them again, save the image. when you open the tests locally you can directly see where it failed on the server (e.g. very annoying with platform dependent stuff) And I personally would prefer this as default behavior, even though it might not fully work on certain tests (so I will add it as an option). Ah, and long running tests which fail, are debuggable instantaneously ;) cami On 2011-08-26, at 16:54, Lukas Renggli wrote:
Why don't you just debug the test then? If you have to enable and rerun the test anyway, you can as well tell the test to open a debugger in case of a problem. Isn't this what "MyTest debug: #testMe" does?
Lukas
On Friday, 26 August 2011, Camillo Bruni <camillo.bruni@inria.fr> wrote:
On 2011-08-26, at 16:28, Lukas Renggli wrote:
Storing a continuation per failure/error potentially consumes a lot of memory. Memory leaks are also the reason why SUnit doesn't even remeber the exception (which only refer to one or two unwound stack frames).
point taken.
Furthermore continuations interfeer with the guarantees of tearDown methods in test-cases and test-resources. This opens a box full of worms: when do you run them, how often do you run them, how do you ensure a consistent state, �
right, but they might also help debugging annoying nondeterministic edge cases. so I will implement it with an optional setting in the TestRunner. When you find an annoying test that won't run consistently you can rerun it with the continuation being recorded.
Lukas
On Friday, 26 August 2011, Camillo Bruni <camillo.bruni@inria.fr> wrote:
On 2011-08-26, at 16:02, Frank Shearar wrote:
On 26 August 2011 14:54, Camillo Bruni <camillo.bruni@inria.fr> wrote:
It always bugged me that SUnit has to rerun the tests again when you want to debug them. So I made a little use-case where exceptions have an optional continuation, so you can resume/debug them later on.
Here's a small example:
|e| [ 0 / 0 ] on: Error do: [ :error| e := error freeze ]. e debug
freeze simply creates a continuation (at the moment a full one, but that could be of course reduced to a partial one by the TestRunner). Now when you would click on the failing test result you would simply have to activate the corresponding error by sending #debug to it.
This would have a major advantage for non-deterministic tests (which of course they should not be). Because in this case you're lost with the current SUnit since the debug stack will differ from the actual test-run which produced the error.
Needed Changes: - Continuations (the one from Seaside will just do fine) - Exception >> #freeze saving the continuation - TestCase has to store the actual exception
Hi Camillo,
Indeed, partial continuations would make an excellent addition to SUnit. Levente & I discussed doing do a short while back in squeak-dev. Have you had a look at http://www.squeaksource.com/Control.html by any chance? It implements the shift/reset control operator, which is almost what Seaside uses.
frank
No I was unaware of that, having a look at it right now.
thanks
-- Lukas Renggli www.lukas-renggli.ch <http://www.lukas-renggli.ch/>
-- Lukas Renggli www.lukas-renggli.ch
On 26 August 2011 17:19, Camillo Bruni <camillo.bruni@inria.fr> wrote:
well the problem with a non deterministic test is that the UI might not correspond to the opened debugger :)
I don't understand. #debug: simply runs the test without wrapping them into exception handlers, so it will just open a debugger on the spot where the problem happens. So if you test fails once in a while you should be able to reproduce it with 1000 timesRepeat: [ MyTest debug: #testMe ] and be exactly in the execution where the problem happend.
another application I see here is for instance when running tests in CI and you see only a couple of tests failing, rerun them again, save the image. when you open the tests locally you can directly see where it failed on the server (e.g. very annoying with platform dependent stuff) And I personally would prefer this as default behavior, even though it might not fully work on certain tests (so I will add it as an option). Ah, and long running tests which fail, are debuggable instantaneously ;)
Yeah maybe, but then again you can just run the tests with #debug: and leave the debuggers open in the image, no? Lukas -- Lukas Renggli www.lukas-renggli.ch
participants (3)
-
Camillo Bruni -
Frank Shearar -
Lukas Renggli