On 02 Feb 2014, at 16:27, btc@openinworld.com wrote:

Thanks for your feedback Marcus. A couple of thought inline...

Marcus Denker wrote:

...

On 02 Feb 2014, at 14:31, btc@openinworld.com wrote:

...

For diligence and curiosity leading up to the Pharo3 release, I downloaded build image 30733 (with PharoLauncher)

Another wonderful thing is that tests are fine when run individually, but fail when running all tests.

It seems that when clicking on the failed test, the test is re-run and that result is debugged - but sometimes the error doesn't occur. It would be great the context of the error in the original execution was stored so that actual error could be inspected. Indeed, (to dream) presuming you're working off fresh CI build, it would be cool if using Fuel in one step you send the context of the original error (with CI build meta-info attached) to someone else's PharoLauncher, which automatically downloads the required CI build, and launches with the Fuel file loaded and ready to debug.

On the ci server this is already done: failing tests are serialise with Fuel for debugging. In addition, the image is save and can be downloaded (in the state after the test run).

...

We need to think about automatic UI testing…

Loading an external package changes *everything*. We do not have a a release criterium that all tests are green after loading external packages.

No external packages loaded here. Launcher? It’s external in the sense that this is not code that is in the image when we run all tests on the build server.

btw, What was your OS environment?

Mac

...

...

3. Assuming Pharo3 will go through a Release Candidate phase,

Normally the idea is that we will just release. In the past we used an elaborate process, but the reality is that nobody looks at release candidates. You can tell 5 times that the release candidate will be released unchanged and people should check: they will not. They will download the release, though, and then complain that the “obvious” thing X is not fixed that that the people who managed the release process did everything wrong. Because “release” means bug free. By magic.

I've seen that happening previously. In trying to understand the reality of this, perhaps... * "check if this is okay" is an open ended question that doesn't have a deliverable to drive people to action. How do you know when its time to report success/failure? * people do download and try it but its only shallow testing * people only report by exception.

Perhaps a more defined task such as "Run All Tests" is easier for people to do (single button press) and to report success/failure. PharoLaucher also makes it easier and quicker to do this. You don't need to call it a Release Candidate. Just have a specified version that multiple people run in multiple environments. Of course, that could also open up a can of worms of being hard from your end to dealing every test failure due to other people's environments. There would need a pragmatic approach about which ones to deal with to stay on schedule with the release date.

yes… its important that we don burn ourselves while doing the release, that is why the idea was to move as much things upfront as possible (“the build is the release unchanged”) Marcus

Take a look at the Fuel job. Just put the file into the working directory (same directory as the image is in) and add a rule to archive Fuel files: **/*.fuel On 25.02.2014, at 17:35, btc@openInWorld.com wrote:

btc@openinworld.com wrote:

...

btc@openinworld.com wrote:

...

Marcus Denker wrote:

...

On the ci server this is already done: failing tests are serialise with Fuel for debugging. In addition, the image is save and can be downloaded (in the state after the test run).

How do I find the fuel files and saved image for this validation report? https://ci.inria.fr/pharo/job/Pharo-3.0-Issue-Validator/11185//artifact/vali...

Hi Camillo,

I am interested by your comment on Case 12957 that "I think we can/should add this code to the standard CI project" regarding my ImageWorker hack to try getting hold of the test failure fuel file. What was it you liked?

Its not suitable for integration in its current form since its a very specific hack hardcoding the target folder, that would leave rubbish on the CI server if I didn't pay attention to manually clean up afterward. Also getting the fuel file back to my local machine is awkward using scp to bounce a copy through the inria ssh gateway. Ideally the fuel file would just end up as an artifact of the Jenkins job. What needs doing? I'd be interested in helping out - though I wouldn't want to risk breaking anything prior to Pharo 3 release.

cheers -ben

On 26.02.2014, at 17:47, btc@openInWorld.com wrote:

Thanks for that info Max. I can see "Files to archive" in the PharoLauncher configuration that I have access to. I don't have access to the configuration of Pharo-3.0-Issue-Validator to check, but I guess it should have an entry... **/*/*.fuel to gather fuel files from the subdirectories of the workspace.

That’s right. Althoug I think that one “/*” is enough (the double asterisk should take care of subdirectories. see http://ant.apache.org/manual/Types/fileset.html)

I have observed from running Pharo-3.0-Issue-Tracker-Image locally that Fuel files are created, that on the build server would located as follows... /builds/workspace/Pharo-3.0-Issue-Validator/3570607022163494/Pharo-3.-2.0.fuel

Cool that they are being generated, not so cool that nobody knew :)

cheers -ben

Max Leske wrote:

...

Take a look at the Fuel job. Just put the file into the working directory (same directory as the image is in) and add a rule to archive Fuel files: **/*.fuel

On 25.02.2014, at 17:35, btc@openInWorld.com wrote:

...

btc@openinworld.com wrote:

...

btc@openinworld.com wrote:

...

Marcus Denker wrote:

...

On the ci server this is already done: failing tests are serialise with Fuel for debugging. In addition, the image is save and can be downloaded (in the state after the test run).

How do I find the fuel files and saved image for this validation report? https://ci.inria.fr/pharo/job/Pharo-3.0-Issue-Validator/11185//artifact/vali...

Hi Camillo,

I am interested by your comment on Case 12957 that "I think we can/should add this code to the standard CI project" regarding my ImageWorker hack to try getting hold of the test failure fuel file. What was it you liked?

Its not suitable for integration in its current form since its a very specific hack hardcoding the target folder, that would leave rubbish on the CI server if I didn't pay attention to manually clean up afterward. Also getting the fuel file back to my local machine is awkward using scp to bounce a copy through the inria ssh gateway. Ideally the fuel file would just end up as an artifact of the Jenkins job. What needs doing? I'd be interested in helping out - though I wouldn't want to risk breaking anything prior to Pharo 3 release.

cheers -ben

On Sun, Feb 2, 2014 at 2:43 PM, Marcus Denker <marcus.denker@inria.fr>wrote:

On 02 Feb 2014, at 14:31, btc@openinworld.com wrote:

...

For diligence and curiosity leading up to the Pharo3 release, I downloaded build image 30733 (with PharoLauncher)

Getting an image green is a lot of little, trivial, hard work... and green always means "Green in this context". Just headless vs. non-headless changes a lot, for example.

Another wonderful thing is that tests are fine when run individually, but fail when running all tests.

We need to think about automatic UI testing...

It will be awesome to have automatic UI tests. Currently, I can go as far as invoking the UI but then to complete the fields in another process/thread, I haven't really spent enough time. I think that's already discussed on the mailing list but some time I should have a look. Automated tests before a release/roll-out will be a dream coming true ;-). Usman

Loading an external package changes *everything*. We do not have a a release criterium that all tests are green after loading external packages.

...

and immediately ran all tests, which reported 7 failures and 1 error (see attached snapshot). Now presumably these did not occur in the CI build since otherwise it would not have been available to PharoLauncher, so this raises a few questions/ideas:

1. Why the discrepancy / how did these slip through ? Perhaps some difference in local environment / difference in VM? Can anyone else reproduce this for 30733? I've logged an new case [1] for this where I plan to either link to existing cases or add a subcase for each failure. Please add your results there, particularly if you see a known issue.

Here are my failures... BlockClosureTest>>#testTrace

Not failing for me

...

JobTest>>#testCurrent Not failing for me

...

MetacelloRepositorySqueakCommonTestCase>>#testAsRepositorySpecFor MetacelloRepositorySqueakCommonTestCase>>#testDirectoryRepository MetacelloRepositorySqueakCommonTestCase>>#testFileTreeRepository Not failing fo rme

...

ReleaseTest>>#testObsoleteClasses

Not failing for me

...

SimmulateMouseSpecification>>#testSimulateRightClick yes, that is failing

...

and errors... OpenToolTest>>#testOpenBrowseOnInstalledTraitMethod

yes.

The last two are examples of UI tests that are not run on the Build server in a way that they could fail.

...

3. Assuming Pharo3 will go through a Release Candidate phase,

Normally the idea is that we will just release. In the past we used an elaborate process, but the reality is that nobody looks at release candidates. You can tell 5 times that the release candidate will be released unchanged and people should check: they will not. They will download the release, though, and then complain that the "obvious" thing X is not fixed that that the people who managed the release process did everything wrong. Because "release" means bug free. By magic.

Marcus

On Tue, Feb 4, 2014 at 9:08 AM, Goubier Thierry <thierry.goubier@cea.fr>wrote:

Le 03/02/2014 21:41, Usman Bhatti a écrit :

...

On Sun, Feb 2, 2014 at 2:43 PM, Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>> wrote:

On 02 Feb 2014, at 14:31, btc@openinworld.com <mailto:btc@openinworld.com> wrote:

> For diligence and curiosity leading up to the Pharo3 release, I downloaded build image 30733 (with PharoLauncher)

Getting an image green is a lot of little, trivial, hard work... and green always means "Green in this context". Just headless vs. non-headless changes a lot, for example.

Another wonderful thing is that tests are fine when run individually, but fail when running all tests.

We need to think about automatic UI testing...

It will be awesome to have automatic UI tests.

Well, it can be done as it is now.

It's fairly easy in Pharo to write tests that open a GUI, test a few things in it, and close it. There, you can already test automatically that your gui updates correctly when the model change.

Yes, I use that already with Glamour and Roassal, and other morph widgets that I have. The harder part for me so far is responding to dialog boxes e.g. answer the user questions automatically and once I have that, I'd be able to completely test my current set of UIs automatically.

It may be a bit harder to trigger events on specific positions, but not that much difficult. A way to write a test method out of Sean event recorder would be nice there.

Currently, I can go as far as invoking the UI but then to complete the

...

fields in another process/thread, I haven't really spent enough time. I think that's already discussed on the mailing list but some time I should have a look. Automated tests before a release/roll-out will be a dream coming true ;-).

Yes. Especially on all the gui parts: drag and drop for example.

But, technically, we're not very far. It's just that nobody writes tests that way.

Thierry -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95

On Tue, Feb 4, 2014 at 9:29 AM, Goubier Thierry <thierry.goubier@cea.fr>wrote:

Le 04/02/2014 09:23, Usman Bhatti a écrit :

...

Yes, I use that already with Glamour and Roassal, and other morph widgets that I have. The harder part for me so far is responding to dialog boxes e.g. answer the user questions automatically and once I have that, I'd be able to completely test my current set of UIs automatically.

What about writing a Shim over the dialog creator with a catalog of predefined answers that could be picked by each dialog creation? When calling a dialog, we're usually going through a factory interface usually.

I do not know what is a Shim. Sometimes, we would like to provide our own answers to the dialogue to see if the tests pass in different configurations (e.g. when we have two or more checkboxes, to see if we get different results selecting different boxes). Or may be we can start with a simpler solution and then evolve towards a completely customizable one.

Thierry -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95

On Tue, Feb 4, 2014 at 10:01 AM, Goubier Thierry <thierry.goubier@cea.fr>wrote:

Le 04/02/2014 09:48, Usman Bhatti a écrit :

...

On Tue, Feb 4, 2014 at 9:29 AM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:

Le 04/02/2014 09:23, Usman Bhatti a écrit :

Yes, I use that already with Glamour and Roassal, and other morph widgets that I have. The harder part for me so far is responding to dialog boxes e.g. answer the user questions automatically and once I have that, I'd be able to completely test my current set of UIs automatically.

What about writing a Shim over the dialog creator with a catalog of predefined answers that could be picked by each dialog creation? When calling a dialog, we're usually going through a factory interface usually.

I do not know what is a Shim.

From Wikipedia:

In computer programming, a shim (from shim) or shiv is a small library that transparently intercepts an API and changes the parameters passed, handles the operation itself, or redirects the operation elsewhere.

Tx for this info. I was not aware of it and it is surprising for me because half of my doctoral thesis was about AOP and related concepts, and Shim seems to be a very similar concept, in its definition, to AOP.

Sometimes, we would like to provide our own answers to the dialogue to

...

see if the tests pass in different configurations (e.g. when we have two or more checkboxes, to see if we get different results selecting different boxes).

Or may be we can start with a simpler solution and then evolve towards a completely customizable one.

My simple approach would be :

in normal code, instead of calling UIManager directly as in:

UIManager default chooseFrom: aCollection title: aString withCRs.

use a shim (self class myUIManager ....). In the test case, setup so that:

`self class myUIManager` returns a shim over UIManager capturing the chooseFrom:title: and containing a dictionary with 'title' -> value to choose. You preload the shim with the right answers and the keys for the shim to find to which dialog this refers (it could be ordered: answer for first dialog, answer for second dialog, etc...). The shim, instead of creating the dialog, returns the value it already has (or an error if it hasn't, which will help you fill the needed values !).

For stuff not related to dialogs you are using the shim for, you can just do a passthrough (i.e. UIManager default perform: ) which can even be handled automagically via doesNotUnderstand:.

As you can see, there isn't much to it.

I see and I've been achieving it in another way. aClass >> export fileRef := UITheme builder fileSave: 'Choose a file' extensions: #('txt') path: nil. fileRef isNil ifFalse: [ self exportOn: fileRef writeStream ] Instead of testing the method export, I test exportOn: with a stream as a parameter. What I am thinking is that, with this approach, we are not testing all the execution paths in the method export. But then, with a Shim, we will not be testing that path either. Usman

Thierry -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95