Different behavior in running tests
I use Sunit + Phexample in my project. The model uses a dynamic variable. In order to ease the writing of tests I implemented in my TestCase IAMModelTests>>#runCase self createContext do: [ super runCase ] with IAMContext>>#do: aBlock ^ IAMCurrentContext value: self during: aBlock It works basically. The method #runCase is only execute once. The #given: calls inside a test method do not trigger further runCase executions. The strange thing is that all tests succeed when I run them alone e.g. in Nautilus. Whenever I run the tests via the testrunner 2 tests fail. Those 2 tests are not any more special than the other tests. And they fail reliably. Does anyone has a hint what is different when being run in testrunner. I cannot debug the problem easily because then I would run the test alone and it will succeed. thanks, Norbert
The problem seems to be Phexample. There is an implementation of Given>>#consumeReturnValueAt: testMethod ^(cache includesKey: testMethod) ifTrue: [ cache removeKey: testMethod ] ifFalse: [ |ret| testMethod setUp. ret := testMethod perform: testMethod selector. testMethod tearDown. ret ] that caches test executions. I don't know why it does this and I don't know why an existing key is removed. But in my case I have tests with side effects while storing objects in MongoDB. The DB is properly reseted at the beginning of a test and if a cached test is not executed the necessary state is not produced. I'm not sure what exactly triggers the problem. It is probably the way how I use Phexample. I do a lot of MyTestCase new given: #anotherStateProducingTest because I cannot align the inheritance chain in parallel to the state needs to be there (and Given is used as a singleton). This is a Phexample problem IMHO. I really like Phexample because state preparation is often something that is annoying while writing tests. On the other hand your tests become a database of test execution state. The management of that state is not properly solved and only works in the simple cases as expected. Norbert Am 23.06.2014 um 14:32 schrieb Norbert Hartl <norbert@hartl.name>:
I use Sunit + Phexample in my project. The model uses a dynamic variable. In order to ease the writing of tests I implemented in my TestCase
IAMModelTests>>#runCase
self createContext do: [ super runCase ]
with
IAMContext>>#do: aBlock ^ IAMCurrentContext value: self during: aBlock
It works basically. The method #runCase is only execute once. The #given: calls inside a test method do not trigger further runCase executions. The strange thing is that all tests succeed when I run them alone e.g. in Nautilus. Whenever I run the tests via the testrunner 2 tests fail. Those 2 tests are not any more special than the other tests. And they fail reliably.
Does anyone has a hint what is different when being run in testrunner. I cannot debug the problem easily because then I would run the test alone and it will succeed.
thanks,
Norbert
Norbert Hartl wrote:
I use Sunit + Phexample in my project. The model uses a dynamic variable. In order to ease the writing of tests I implemented in my TestCase
IAMModelTests>>#runCase
self createContext do: [ super runCase ]
with
IAMContext>>#do: aBlock ^ IAMCurrentContext value: self during: aBlock
It works basically. The method #runCase is only execute once. The #given: calls inside a test method do not trigger further runCase executions. The strange thing is that all tests succeed when I run them alone e.g. in Nautilus. Whenever I run the tests via the testrunner 2 tests fail. Those 2 tests are not any more special than the other tests. And they fail reliably.
Does anyone has a hint what is different when being run in testrunner. I cannot debug the problem easily because then I would run the test alone and it will succeed.
thanks,
Norbert
I don't know what might be different, but maybe a way to attack it is to put a halt in a test method. Then look down the debugger stack and remove "on: self class failure , self class skip, Warning, self class error" from TestCase>>runCase: Now you'll get a debugger at the moment the failure occurs, rather than filing it to be later clicked on to run it by itself. cheers -ben
participants (2)
-
Ben Coman -
Norbert Hartl