Marcus Denker wrote:

On 02 Feb 2014, at 16:27, btc@openinworld.com wrote:

Thanks for your feedback Marcus.� A couple of thought inline...

Marcus Denker wrote: 
On 02 Feb 2014, at 14:31, btc@openinworld.com wrote:

  
For diligence and curiosity leading up to the Pharo3 release, I downloaded build image 30733 (with PharoLauncher)
    
Another wonderful thing is that tests are fine when run individually, but fail when running all tests.

It seems that when clicking on the failed test, the test is re-run and that result is debugged - but sometimes the error doesn't occur.� It would be great the context of the error in the original execution was stored so that actual error could be inspected.� Indeed, (to dream) presuming you're working off fresh CI build, it would be cool if using Fuel in one step you send the context of the original error (with CI build meta-info attached) to someone else's PharoLauncher, which automatically downloads the required CI build, and launches with the Fuel file loaded and ready to debug.

On the ci server this is already done: failing tests are serialise with Fuel for debugging. In addition, the image is save and can be downloaded (in the state after the test run).

Cool.� Does the CI use TestRunner? Where should I dig to try to get similar behaviour for failed tests from user initiated Test Runner - in Pharo 4 :)?
For now I am resorting to modifying TestResult>>runCase: to remove the exception catching� to allow a debugger to open on failures.�

btw, watching the execution of method "TestRunner>>runTestSuites: testSuites"
I notice that parameter 'testSuites' holds a Set, such that the testSuites are run in random order.� That might be great for coverage but is poor for reproducibility, and one reason that multiple <Run Selected> "all tests" can produce different results each time. It would useful if that was an OrderedList with several different randomization choices (fully manual, each Image startup, each TestRunner session, each <Run Selected>) and that list could be exported to be shared with others.


We need to think about automatic UI testing� 

Loading an external package changes *everything*. We do not have a a release criterium that all tests are green after loading external packages.

No external packages loaded here.�
Launcher? It�s external in the sense that this is not code that is in the image when we run all tests on the build server.

Sorry I wasn't clear.� I was not testing PharoLauncher images. I was using PharoLauncher to create fresh Pharo images for testing.

cheers -ben