On Sun, Feb 2, 2014 at 2:43 PM, Marcus Denker <marcus.denker@inria.fr> wrote:

On 02 Feb 2014, at 14:31, btc@openinworld.com wrote:

> For diligence and curiosity leading up to the Pharo3 release, I downloaded build image 30733 (with PharoLauncher)

Getting an image green is a lot of little, trivial, hard work… and green always means “Green in this context”. Just headless vs. non-headless
changes a lot, for example.

Another wonderful thing is that tests are fine when run individually, but fail when running all tests.

We need to think about automatic UI testing…

It will be awesome to have automatic UI tests.

Currently, I can go as far as invoking the UI but then to complete the fields in another process/thread, I haven't really spent enough time. I think that's already discussed on the mailing list but some time I should have a look. Automated tests before a release/roll-out will be a dream coming true ;-).

Usman

Loading an external package changes *everything*. We do not have a a release criterium that all tests are green after loading external packages.

> and immediately ran all tests, which reported 7 failures and 1 error (see attached snapshot). Now presumably these did not occur in the CI build since otherwise it would not have been available to PharoLauncher, so this raises a few questions/ideas:
>
> 1. Why the discrepancy / how did these slip through ? Perhaps some difference in local environment / difference in VM? Can anyone else reproduce this for 30733? I've logged an new case [1] for this where I plan to either link to existing cases or add a subcase for each failure. Please add your results there, particularly if you see a known issue.
>
> Here are my failures...
> BlockClosureTest>>#testTrace

Not failing for me

> JobTest>>#testCurrent
Not failing for me

> MetacelloRepositorySqueakCommonTestCase>>#testAsRepositorySpecFor
> MetacelloRepositorySqueakCommonTestCase>>#testDirectoryRepository
> MetacelloRepositorySqueakCommonTestCase>>#testFileTreeRepository

Not failing fo rme

> ReleaseTest>>#testObsoleteClasses

Not failing for me
> SimmulateMouseSpecification>>#testSimulateRightClick
yes, that is failing

> and errors...
> OpenToolTest>>#testOpenBrowseOnInstalledTraitMethod
>
yes.

The last two are examples of UI tests that are not run on the Build server in a way that they could fail.

> 3. Assuming Pharo3 will go through a Release Candidate phase,

Normally the idea is that we will just release. In the past we used an elaborate process, but the reality is that nobody looks at release candidates.
You can tell 5 times that the release candidate will be released unchanged and people should check: they will not. They will download
the release, though, and then complain that the “obvious” thing X is not fixed that that the people who managed the release process did
everything wrong. Because “release” means bug free. By magic.

Marcus