[Pharo-dev] GUI TestRunner failures not found by CI (was Re: [pharo-project/pharo-core] 1953a9: 30829)
Sven Van Caekenberghe
sven at stfx.eu
Wed Apr 23 14:35:35 EDT 2014
On 23 Apr 2014, at 19:55, Ben Coman <btc at openInWorld.com> wrote:
> Sven Van Caekenberghe wrote:
>> And we're all GREEN, great work !
>> Let's keep it that way until release.
>> Everything that breaks a test should now be reverted.
> Thats great news. However while the CI monkey is great, there are still a few tests (at least in Windows) that fail in a fresh image since:
> * they are bypassed by the CI
> * they are not run on a Windows CI slave
> I fixed a few similar failures but ran out of time to finish this. It might be worthwhile some others doing a pass the old fashioned way via the TestRunner GUI. Thats what users will end up seeing. It would be "real nice" if TestRunner showed users ALL GREEN for the Released image. I've just started a run with the latest image while I head off to bed. I'll report more on the issue tracker tomorrow.
Agreed, but we know from experience that running the tests in a very controlled environment is the only way to have a common frame of reference. Yes, there are always a couple of guys that will download the release, run all tests and complain about a couple of failures. It would be super if we could avoid that, but I am pretty sure that will not happen (i.e. they will not succeed 100% for everyone).
More eyes (especially on, the many different variants of, Windows) will always help, but the community is what it is.
More information about the Pharo-dev